Diary of making getLink vip Fshare system in 48 hours

Tram Ho

Diary of the month …
On a nice day, I found the websites that provide getlink VIP fshare service simultaneously reported an error, could not get VIP link download maxbandwith . And I also see a lot of new posts created in one group (specializing in IT sharing), about sharing Fcode (fshare code) that they just bought, and not used up. The fcode is limited in number, and use time.
That sparked me thinking about making my own getLink VIP website. And Fshare is the service I will test.

// (If possible, please pay fee for fshare to be able to use the service better)

1. Original design ideas

Fshare is a famous file storage service provider. And to download files with max bandwidth speed, you need a VIP account. And to have a VIP account, you will have to pay. And according to fshare’s policy , a VIP account will not be limited to downloads, and download bandwidth.
=> I will make a website that the client will fill in the form containing the URL to the file to download. The backend server will then use its VIP account, request fshare to get the download link, and then return it to the client.

  • Obstacle 1 …

This cannot be done, because fshare has the mechanism that is only the IP address that requests the download link, only that IP address can be downloaded. In the download url that fshare has returned, which has been signed (the download link has passed additional parameters to validate), I “guess” one of the parameters to be signed, which has the client’s IP address .
=> If the request link does not occur in the backend service, then I will do it in the frontend.

  • Obstacle 2 …

Having the client (frontend) automatically request the link directly with fshare, makes me worry about the security of my VIP account. I don’t want clients to get any sensitive information about VIP accounts. Moreover, I have also tested, quite complex to CORS to fshare (using ajax call cross to another domain is running domain). I debugged cors fshare using f12 of web browser. And I see fshare still accepts cors , but it’s too complicated and hard to control.
=> For the 2 reasons above, I decided to remove this way. And back to the first way. Get the link at the backend server. However, after the download link is created, pay it back to the client. I will provide always a proxy route, which is responsible for forwarding traffic downloaded from the client to my backend server, then to fshare.

  • Obstacle 3 …

This method is also not perfect, because forward traffic will be very easy to cause bottleneck (bottlenecks) at your server. However, the server that I have, according to the advertisement, is bandwith up to 10Gbps. I’m not sure about the performance when the number of clients use up, but it will be a problem in another field. So I still decided to develop in this direction.

2. Select tech stack round 1

  • Java vs Spring Framework (Springboot, Spring webservice …): java is the language I use most, and Spring is a famous framework. So I used it as a backend.
  • Frontend: html + css + jquery + ajax. I will use ajax to call api to the backend server.

3. Find a way to connect with fshare

I did google, but I don’t see fshare that there is a document about public API for developers.
I started to think again about debugging f12 web browser fshare.vn to find the format for httpClient.
Fortunately, after I searched on github , I saw a repo about getLink fshare. That’s this repo: https://github.com/tudoanh/get_fshare (use python)
To make sure the API in the repo is correct, I used PostMan to test it. And the API results are still good.
There are basically 2 APIs:

  1. Use user_email + password on fshare, to get session_id and token
    • Request example:

    • Response example:

  2. Use token to request download link
    • Request example:

    • Response example:

// The examples above I export from Postman. I used Postman and it was ok. But do not understand why when using curl errors. Pretty confusing. Moreover, we only see the use of token , not where they use session_id . Later when I converted to Java code, the second request always reported an “not logged in” error. Then I found out, in the 2nd request, I have to pass more header containing Cookie , in which the Cookie value has session_id to pass.

4. Select tech stack round 2

4.1 Selection

The construction of the spring boot project has a backend and a frontend in a repo rather quickly.

  • Combo Spring framework
    • To call the fshare API, a httpClient is required. Initially I intended to use FeignClient , its code is quite short. However, I had a hard time debugging errors with fshare. So I decided to go back to using Spring’s RestTemplate .
    • As soon as the application code is run, I need to get the value of the session_id , the token and store it in the static variable. I use CommandLineRunner .
    • I discovered that session_id + token only be used for a certain period of time, maybe this is the policy of fshare. So I used Spring Scheduled to add timer API rotation/ refresh to the token value + new session_id.
    • Client request form request get download link, and server request fshare, should be run asynchronously. And because of asynchronous, I use Spring Websocket (spring-boot-starter-websocket) to make a channel to send results to the client, after the Server run task get the download link. This asynchrony also contributes to make your website more friendly.
    • I use Spring’s ThreadPoolTaskExecutor/TaskExecutor , to create task for each request get link. (running multi thread)
  • On the client side, I use sockjs-client + stomp-websocket to subscribe to the socket channel. (subcribe so that when the server runs the getlink task, it will show the results to the frontend, without having to refresh the page). For each request client session, I will generate a new socket topic. Purpose to ID. I am quite worried about this, it will lead to much reduced system performance.

4.2 Quick code

  • Spring RestTemplate

  • CommandLineRunner

  • Scheduled Spring

  • Spring Websocket

  • ThreadPoolTaskExecutor

  • sockjs-client + stomp-websocket

5. Obstacle 4

The application code has basically completed quite a bit. I started looking for a solution for the proxy traffic that I mentioned above. Traffic download will be from the client then forwarded to your server, then to fshare. I have google a lot about this way, but quite confused with the current skill.
While contemplating, I suddenly remembered NGINX , a guy I used to be able to do, something I forgot.
=> Ideas:
Similarly, fdare linkdownload returns the following format

In which download022.fshare.vn is the fshare server. Now I will turn it into download022.tungexplorer.me – is my server address. And the download link for the client is:

  • The replacement of this String in java only 1 note.
  • Nginx configuration reference:

6. Obstacle 5

I discovered the endpoint that fshare provides, not only download022.fshare.vn , but it loadbalancer through many other endpoints as well. For example, 023, 024, 011 … I can’t passively configure each endpoint like that.
=> Config with regex to dynamically route the proxy. This took me a lot of time, because I’m not too proficient in this. It took me 3-4 hours to test regex, and google debug.
Finally the format worked well:

7. Select tech stack round 3

Basically the system worked almost as it wanted. However, I want to do 2 more things:

  1. Control limit bandwith for each download. Although my server has a transmission line of up to 10Gbps, however I want each download, only a speed limit of XXX Mbps . => NGINX Plus offers this feature, but Nginx Plus is quite expensive. $ 25000 / year. => Until now I still leave this problem open.
  2. I need my webserver monitor system, I want to keep track of how many downloads I have, the total traffic on the network ports, how much cpu resources, ram is full load or not. After pondering, I decided to use the combo: Prometheus + Grafana + Prometheus Exporter
    • Prometheus server: get metrics from client/ device to track.
    • Prometheus Exporter: Prometheus client, collecting metrics on the running “client”, to send to the prometheus server
    • Grafana: connecting with prometheus, then graphing the metric display to the admin, via webUI.

Image of 1 Dashboard monitor traffic của Grafana Grafana

Note: the use of Prometheus + Grafana + Prometheus Exporter combos at first, it seems redundant, finding it “complicated” the problem. However, this system will further develop itself. So I still decided to choose to use them.

8. Full SourceCode

Share the news now

Source : Viblo