Diary of the month …
On a nice day, I found the websites that provide getlink VIP fshare
service simultaneously reported an error, could not get VIP link download maxbandwith
. And I also see a lot of new posts created in one group (specializing in IT sharing), about sharing Fcode (fshare code) that they just bought, and not used up. The fcode
is limited in number, and use time.
That sparked me thinking about making my own getLink VIP website. And Fshare
is the service I will test.
// (If possible, please pay fee for fshare to be able to use the service better)
1. Original design ideas
Fshare is a famous file storage service provider. And to download files with max bandwidth speed, you need a VIP account. And to have a VIP account, you will have to pay. And according to fshare’s policy
, a VIP account will not be limited to downloads, and download bandwidth.
=> I will make a website that the client will fill in the form containing the URL to the file to download. The backend server will then use its VIP account, request fshare to get the download link, and then return it to the client.
- Obstacle 1 …
This cannot be done, because fshare has the mechanism that is only the IP address that requests the download link, only that IP address can be downloaded. In the download url that fshare has returned, which has been signed
(the download link has passed additional parameters to validate), I “guess” one of the parameters to be signed, which has the client’s IP address
.
=> If the request link does not occur in the backend service, then I will do it in the frontend.
- Obstacle 2 …
Having the client (frontend) automatically request the link directly with fshare, makes me worry about the security of my VIP account. I don’t want clients to get any sensitive information about VIP accounts. Moreover, I have also tested, quite complex to CORS
to fshare (using ajax
call cross to another domain is running domain). I debugged cors
fshare using f12 of web browser. And I see fshare still accepts cors
, but it’s too complicated and hard to control.
=> For the 2 reasons above, I decided to remove this way. And back to the first way. Get the link at the backend server. However, after the download link is created, pay it back to the client. I will provide always a proxy route, which is responsible for forwarding traffic downloaded from the client to my backend server, then to fshare.
- Obstacle 3 …
This method is also not perfect, because forward traffic will be very easy to cause bottleneck
(bottlenecks) at your server. However, the server that I have, according to the advertisement, is bandwith up to 10Gbps. I’m not sure about the performance when the number of clients use up, but it will be a problem in another field. So I still decided to develop in this direction.
2. Select tech stack round 1
- Java vs Spring Framework (Springboot, Spring webservice …): java is the language I use most, and Spring is a famous framework. So I used it as a backend.
- Frontend: html + css + jquery + ajax. I will use ajax to call api to the backend server.
3. Find a way to connect with fshare
I did google, but I don’t see fshare that there is a document about public
API for developers.
I started to think again about debugging f12 web browser fshare.vn
to find the format for httpClient.
Fortunately, after I searched on github
, I saw a repo about getLink fshare. That’s this repo: https://github.com/tudoanh/get_fshare (use python)
To make sure the API in the repo is correct, I used PostMan
to test it. And the API results are still good.
There are basically 2 APIs:
- Use
user_email
+password
on fshare, to getsession_id
andtoken
- Request example:
123456789<span class="token function">curl</span> --location --request POST <span class="token string">'https://api.fshare.vn/api/user/login'</span>--header <span class="token string">'Content-Type: application/json'</span>--data-raw <span class="token string">'{"user_email": " <a class="__cf_email__" href="/cdn-cgi/l/email-protection">[email protected]</a> ","password": "passWord","app_key": "L2S7R6ZMagggC5wWkQhX2+aDi467PPuftWUMRFSn"}'</span>- Response example:
1234567<span class="token punctuation">{</span><span class="token string">"code"</span> <span class="token keyword">:</span> 200,<span class="token string">"msg"</span> <span class="token keyword">:</span> <span class="token string">"Login successfully!"</span> ,<span class="token string">"token"</span> <span class="token keyword">:</span> <span class="token string">"483f5bb8ab3812070f95b2f52d0ff5645a4f4"</span> ,<span class="token string">"session_id"</span> <span class="token keyword">:</span> <span class="token string">"f35jj7d0q5v91dt6b4r3vns"</span><span class="token punctuation">}</span> - Use
token
to request download link- Request example:
1234567<span class="token function">curl</span> --location --request POST <span class="token string">'https://api.fshare.vn/api/session/download'</span>--header <span class="token string">'Content-Type: application/json'</span>--data-raw <span class="token string">'{"token": "483f5bb8ab3812070f95b2f52d0ff5645a4f4","url": "https://www.fshare.vn/file/VG8998AQNPB"}'</span>- Response example:
1234<span class="token punctuation">{</span><span class="token string">"location"</span> <span class="token keyword">:</span> <span class="token string">"http://download022.fshare.vn/dl/g1igLGM7IScOwdkwo3iwyPmJPDJk1roRViiVDbNszLZwk4k2xb-6TogmIK9Rjxq4y+Ggl2V0Z/ipz0g78hhb.wmv"</span><span class="token punctuation">}</span>
// The examples above I export from Postman. I used Postman
and it was ok. But do not understand why when using curl
errors. Pretty confusing. Moreover, we only see the use of token
, not where they use session_id
. Later when I converted to Java code, the second request always reported an “not logged in” error. Then I found out, in the 2nd request, I have to pass more header
containing Cookie
, in which the Cookie
value has session_id
to pass.
4. Select tech stack round 2
4.1 Selection
The construction of the spring boot
project has a backend and a frontend in a repo rather quickly.
- Combo Spring framework
- To call the fshare API, a httpClient is required. Initially I intended to use
FeignClient
, its code is quite short. However, I had a hard time debugging errors with fshare. So I decided to go back to using Spring’sRestTemplate
. - As soon as the
application
code is run, I need to get the value of thesession_id
, thetoken
and store it in thestatic
variable. I useCommandLineRunner
. - I discovered that
session_id
+token
only be used for a certain period of time, maybe this is the policy of fshare. So I usedSpring Scheduled
to add timer APIrotation/ refresh
to the token value + new session_id. - Client request form request get download link, and server request fshare, should be run asynchronously. And because of asynchronous, I use
Spring Websocket
(spring-boot-starter-websocket) to make a channel to send results to the client, after the Server run task get the download link. This asynchrony also contributes to make your website more friendly. - I use Spring’s
ThreadPoolTaskExecutor/TaskExecutor
, to create task for each request get link. (running multi thread)
- To call the fshare API, a httpClient is required. Initially I intended to use
- On the client side, I use
sockjs-client
+stomp-websocket
to subscribe to the socket channel. (subcribe so that when the server runs the getlink task, it will show the results to the frontend, without having to refresh the page). For each request client session, I will generate a new socket topic. Purpose to ID. I am quite worried about this, it will lead to much reduced system performance.
4.2 Quick code
- Spring RestTemplate
1 2 3 4 5 6 7 8 9 10 11 12 | <span class="token keyword">public</span> <span class="token keyword">void</span> <span class="token function">setToken</span> <span class="token punctuation">(</span> <span class="token punctuation">)</span> <span class="token punctuation">{</span> LoginRequest loginRequest <span class="token operator">=</span> <span class="token keyword">new</span> <span class="token class-name">LoginRequest</span> <span class="token punctuation">(</span> <span class="token punctuation">)</span> <span class="token punctuation">;</span> loginRequest <span class="token punctuation">.</span> <span class="token function">setUser_email</span> <span class="token punctuation">(</span> userMail <span class="token punctuation">)</span> <span class="token punctuation">;</span> loginRequest <span class="token punctuation">.</span> <span class="token function">setPassword</span> <span class="token punctuation">(</span> password <span class="token punctuation">)</span> <span class="token punctuation">;</span> loginRequest <span class="token punctuation">.</span> <span class="token function">setApp_key</span> <span class="token punctuation">(</span> appKey <span class="token punctuation">)</span> <span class="token punctuation">;</span> RestTemplate restTemplate <span class="token operator">=</span> <span class="token keyword">new</span> <span class="token class-name">RestTemplate</span> <span class="token punctuation">(</span> <span class="token punctuation">)</span> <span class="token punctuation">;</span> LoginResponse loginResponse <span class="token operator">=</span> restTemplate <span class="token punctuation">.</span> <span class="token function">postForObject</span> <span class="token punctuation">(</span> Const <span class="token punctuation">.</span> FSHARE_ENDPOINT_LOGIN <span class="token punctuation">,</span> loginRequest <span class="token punctuation">,</span> LoginResponse <span class="token punctuation">.</span> <span class="token keyword">class</span> <span class="token punctuation">)</span> <span class="token punctuation">;</span> <span class="token keyword">assert</span> loginResponse <span class="token operator">!=</span> null <span class="token punctuation">;</span> Const <span class="token punctuation">.</span> FSHARE_SESSION_ID <span class="token operator">=</span> loginResponse <span class="token punctuation">.</span> <span class="token function">getSession_id</span> <span class="token punctuation">(</span> <span class="token punctuation">)</span> <span class="token punctuation">;</span> Const <span class="token punctuation">.</span> FSHARE_TOKEN <span class="token operator">=</span> loginResponse <span class="token punctuation">.</span> <span class="token function">getToken</span> <span class="token punctuation">(</span> <span class="token punctuation">)</span> <span class="token punctuation">;</span> <span class="token punctuation">}</span> |
- CommandLineRunner
1 2 3 4 5 6 7 8 9 10 11 | <span class="token annotation punctuation">@Component</span> <span class="token keyword">public</span> <span class="token keyword">class</span> <span class="token class-name">GetToken</span> <span class="token keyword">implements</span> <span class="token class-name">CommandLineRunner</span> <span class="token punctuation">{</span> <span class="token annotation punctuation">@Autowired</span> <span class="token keyword">private</span> FshareService fshareService <span class="token punctuation">;</span> <span class="token annotation punctuation">@Override</span> <span class="token keyword">public</span> <span class="token keyword">void</span> <span class="token function">run</span> <span class="token punctuation">(</span> String <span class="token punctuation">.</span> <span class="token punctuation">.</span> <span class="token punctuation">.</span> strings <span class="token punctuation">)</span> <span class="token punctuation">{</span> fshareService <span class="token punctuation">.</span> <span class="token function">setToken</span> <span class="token punctuation">(</span> <span class="token punctuation">)</span> <span class="token punctuation">;</span> <span class="token punctuation">}</span> <span class="token punctuation">}</span> |
- Scheduled Spring
1 2 3 4 5 6 7 8 9 10 11 | <span class="token annotation punctuation">@Component</span> <span class="token keyword">public</span> <span class="token keyword">class</span> <span class="token class-name">RefreshToken</span> <span class="token punctuation">{</span> <span class="token annotation punctuation">@Autowired</span> <span class="token keyword">private</span> FshareService fshareService <span class="token punctuation">;</span> <span class="token annotation punctuation">@Scheduled</span> <span class="token punctuation">(</span> fixedRate <span class="token operator">=</span> <span class="token number">1000</span> <span class="token operator">*</span> <span class="token number">60</span> <span class="token operator">*</span> <span class="token number">60</span> <span class="token operator">*</span> <span class="token number">2</span> <span class="token punctuation">)</span> <span class="token keyword">public</span> <span class="token keyword">void</span> <span class="token function">run</span> <span class="token punctuation">(</span> <span class="token punctuation">)</span> <span class="token punctuation">{</span> fshareService <span class="token punctuation">.</span> <span class="token function">setToken</span> <span class="token punctuation">(</span> <span class="token punctuation">)</span> <span class="token punctuation">;</span> <span class="token punctuation">}</span> <span class="token punctuation">}</span> |
- Spring Websocket
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 | <span class="token annotation punctuation">@Configuration</span> <span class="token annotation punctuation">@EnableWebSocketMessageBroker</span> <span class="token keyword">public</span> <span class="token keyword">class</span> <span class="token class-name">WebSocketConfig</span> <span class="token keyword">extends</span> <span class="token class-name">AbstractWebSocketMessageBrokerConfigurer</span> <span class="token punctuation">{</span> <span class="token annotation punctuation">@Override</span> <span class="token keyword">public</span> <span class="token keyword">void</span> <span class="token function">registerStompEndpoints</span> <span class="token punctuation">(</span> StompEndpointRegistry stompEndpointRegistry <span class="token punctuation">)</span> <span class="token punctuation">{</span> stompEndpointRegistry <span class="token punctuation">.</span> <span class="token function">addEndpoint</span> <span class="token punctuation">(</span> <span class="token string">"/websocket-receive-link"</span> <span class="token punctuation">)</span> <span class="token punctuation">.</span> <span class="token function">withSockJS</span> <span class="token punctuation">(</span> <span class="token punctuation">)</span> <span class="token punctuation">;</span> <span class="token punctuation">}</span> <span class="token annotation punctuation">@Override</span> <span class="token keyword">public</span> <span class="token keyword">void</span> <span class="token function">configureMessageBroker</span> <span class="token punctuation">(</span> MessageBrokerRegistry registry <span class="token punctuation">)</span> <span class="token punctuation">{</span> registry <span class="token punctuation">.</span> <span class="token function">enableSimpleBroker</span> <span class="token punctuation">(</span> <span class="token string">"/topic"</span> <span class="token punctuation">)</span> <span class="token punctuation">;</span> registry <span class="token punctuation">.</span> <span class="token function">setApplicationDestinationPrefixes</span> <span class="token punctuation">(</span> <span class="token string">"/app"</span> <span class="token punctuation">)</span> <span class="token punctuation">;</span> <span class="token punctuation">}</span> <span class="token punctuation">}</span> |
- ThreadPoolTaskExecutor
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 | <span class="token annotation punctuation">@Bean</span> <span class="token annotation punctuation">@Primary</span> TaskExecutor <span class="token function">taskExecutor</span> <span class="token punctuation">(</span> <span class="token punctuation">)</span> <span class="token punctuation">{</span> ThreadPoolTaskExecutor t <span class="token operator">=</span> <span class="token keyword">new</span> <span class="token class-name">ThreadPoolTaskExecutor</span> <span class="token punctuation">(</span> <span class="token punctuation">)</span> <span class="token punctuation">;</span> t <span class="token punctuation">.</span> <span class="token function">setCorePoolSize</span> <span class="token punctuation">(</span> <span class="token number">10</span> <span class="token punctuation">)</span> <span class="token punctuation">;</span> t <span class="token punctuation">.</span> <span class="token function">setMaxPoolSize</span> <span class="token punctuation">(</span> <span class="token number">100</span> <span class="token punctuation">)</span> <span class="token punctuation">;</span> t <span class="token punctuation">.</span> <span class="token function">setQueueCapacity</span> <span class="token punctuation">(</span> <span class="token number">500</span> <span class="token punctuation">)</span> <span class="token punctuation">;</span> <span class="token keyword">return</span> t <span class="token punctuation">;</span> <span class="token punctuation">}</span> <span class="token comment">//</span> <span class="token annotation punctuation">@Autowired</span> <span class="token keyword">private</span> TaskExecutor task <span class="token punctuation">;</span> <span class="token comment">//</span> task <span class="token punctuation">.</span> <span class="token function">execute</span> <span class="token punctuation">(</span> <span class="token keyword">new</span> <span class="token class-name">TaskGetLink</span> <span class="token punctuation">(</span> requestLink <span class="token punctuation">,</span> template <span class="token punctuation">,</span> fshareService <span class="token punctuation">)</span> <span class="token punctuation">)</span> <span class="token punctuation">;</span> |
- sockjs-client + stomp-websocket
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 | <span class="token keyword">function</span> <span class="token function">initSocket</span> <span class="token punctuation">(</span> <span class="token punctuation">)</span> <span class="token punctuation">{</span> <span class="token keyword">var</span> socket <span class="token operator">=</span> <span class="token keyword">new</span> <span class="token class-name">SockJS</span> <span class="token punctuation">(</span> <span class="token string">'/websocket-receive-link'</span> <span class="token punctuation">)</span> <span class="token punctuation">;</span> stompClient <span class="token operator">=</span> Stomp <span class="token punctuation">.</span> <span class="token function">over</span> <span class="token punctuation">(</span> socket <span class="token punctuation">)</span> <span class="token punctuation">;</span> stompClient <span class="token punctuation">.</span> <span class="token function">connect</span> <span class="token punctuation">(</span> <span class="token punctuation">{</span> <span class="token string">"X-Token"</span> <span class="token punctuation">:</span> <span class="token string">"tokenvalue"</span> <span class="token punctuation">}</span> <span class="token punctuation">,</span> onConnected <span class="token punctuation">,</span> onError <span class="token punctuation">)</span> <span class="token punctuation">;</span> <span class="token keyword">function</span> <span class="token function">onConnected</span> <span class="token punctuation">(</span> <span class="token punctuation">)</span> <span class="token punctuation">{</span> stompClient <span class="token punctuation">.</span> <span class="token function">subscribe</span> <span class="token punctuation">(</span> <span class="token string">"/topic/"</span> <span class="token operator">+</span> requestId <span class="token punctuation">,</span> onMessageReceived <span class="token punctuation">)</span> <span class="token punctuation">;</span> <span class="token punctuation">}</span> <span class="token keyword">function</span> <span class="token function">onMessageReceived</span> <span class="token punctuation">(</span> payload <span class="token punctuation">)</span> <span class="token punctuation">{</span> <span class="token comment">// todo some thing</span> <span class="token punctuation">}</span> <span class="token keyword">function</span> <span class="token function">onError</span> <span class="token punctuation">(</span> error <span class="token punctuation">)</span> <span class="token punctuation">{</span> <span class="token comment">// todo some thing</span> <span class="token punctuation">}</span> <span class="token punctuation">}</span> |
5. Obstacle 4
The application code has basically completed quite a bit. I started looking for a solution for the proxy traffic
that I mentioned above. Traffic download will be from the client then forwarded to your server, then to fshare. I have google a lot about this way, but quite confused with the current skill.
While contemplating, I suddenly remembered NGINX
, a guy I used to be able to do, something I forgot.
=> Ideas:
Similarly, fdare linkdownload returns the following format
1 2 | http://download022.fshare.vn/dl/g1igLGM7IScOwdkwo3iwyPmJPDJk1roRViimGIL8t7VDbNxb-6TogmIK9Rjxq4y+Ggl2V0Z/ipz0g78hhb.wmv |
In which download022.fshare.vn
is the fshare server. Now I will turn it into download022.tungexplorer.me
– is my server address. And the download link for the client is:
1 2 | http://download022.tungexplorer.me/dl/g1igLGM7IScOwdkwo3iwyPmJPDJk1roRViimGIL8t7VDbNxb-6TogmIK9Rjxq4y+Ggl2V0Z/ipz0g78hhb.wmv |
- The replacement of this String in java only 1 note.
Nginx
configuration reference:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 | server <span class="token punctuation">{</span> listen 80 <span class="token punctuation">;</span> server_name download03333.tungexplorer.me <span class="token punctuation">;</span> access_log off <span class="token punctuation">;</span> location / <span class="token punctuation">{</span> proxy_pass http://download022.fshare.vn <span class="token punctuation">;</span> proxy_read_timeout 300 <span class="token punctuation">;</span> proxy_connect_timeout 300 <span class="token punctuation">;</span> proxy_redirect off <span class="token punctuation">;</span> proxy_set_header X-Forwarded-Proto <span class="token variable">$scheme</span> <span class="token punctuation">;</span> proxy_set_header Host <span class="token variable">$http_host</span> <span class="token punctuation">;</span> proxy_set_header X-Real-IP <span class="token variable">$remote_addr</span> <span class="token punctuation">;</span> <span class="token punctuation">}</span> <span class="token punctuation">}</span> |
6. Obstacle 5
I discovered the endpoint
that fshare provides, not only download022.fshare.vn
, but it loadbalancer through many other endpoints as well. For example, 023, 024, 011 … I can’t passively configure each endpoint like that.
=> Config with regex
to dynamically route the proxy. This took me a lot of time, because I’m not too proficient in this. It took me 3-4 hours to test regex, and google debug.
Finally the format worked well:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 | server <span class="token punctuation">{</span> listen 80 <span class="token punctuation">;</span> server_name ~^ <span class="token punctuation">(</span> www. <span class="token punctuation">)</span> ? <span class="token punctuation">[</span> ^. <span class="token punctuation">]</span> +.tungexplorer.me$ <span class="token punctuation">;</span> access_log off <span class="token punctuation">;</span> <span class="token keyword">if</span> <span class="token punctuation">(</span> <span class="token variable">$host</span> ~* ^ <span class="token punctuation">(</span> www. <span class="token punctuation">)</span> ? <span class="token punctuation">(</span> <span class="token punctuation">[</span> ^. <span class="token punctuation">]</span> + <span class="token punctuation">)</span> .tungexplorer.me$ <span class="token punctuation">)</span> <span class="token punctuation">{</span> <span class="token keyword">set</span> <span class="token variable">$subdomain</span> <span class="token variable">$2</span> <span class="token punctuation">;</span> <span class="token punctuation">}</span> resolver 8.8.8.8 valid <span class="token operator">=</span> 10s <span class="token punctuation">;</span> location / <span class="token punctuation">{</span> proxy_pass http:// <span class="token variable">$subdomain</span> .fshare.vn <span class="token punctuation">;</span> proxy_read_timeout 300 <span class="token punctuation">;</span> proxy_connect_timeout 300 <span class="token punctuation">;</span> proxy_redirect off <span class="token punctuation">;</span> proxy_set_header X-Forwarded-Proto <span class="token variable">$scheme</span> <span class="token punctuation">;</span> proxy_set_header Host <span class="token variable">$http_host</span> <span class="token punctuation">;</span> proxy_set_header X-Real-IP <span class="token variable">$remote_addr</span> <span class="token punctuation">;</span> <span class="token punctuation">}</span> <span class="token punctuation">}</span> |
7. Select tech stack round 3
Basically the system worked almost as it wanted. However, I want to do 2 more things:
- Control limit bandwith for each download. Although my server has a transmission line of up to 10Gbps, however I want each download, only a speed limit of XXX
Mbps
. => NGINX Plus offers this feature, but Nginx Plus is quite expensive. $ 25000 / year. => Until now I still leave this problem open. - I need my webserver monitor system, I want to keep track of how many downloads I have, the total traffic on the network ports, how much cpu resources, ram is full load or not. After pondering, I decided to use the combo:
Prometheus + Grafana + Prometheus Exporter
- Prometheus server: get metrics from
client/ device
to track. - Prometheus Exporter: Prometheus client, collecting metrics on the running “client”, to send to the
prometheus server
- Grafana: connecting with prometheus, then graphing the metric display to the admin, via webUI.
- Prometheus server: get metrics from
Image of 1 Dashboard monitor traffic của Grafana
Note: the use of Prometheus + Grafana + Prometheus Exporter
combos at first, it seems redundant, finding it “complicated” the problem. However, this system will further develop itself. So I still decided to choose to use them.
8. Full SourceCode
- I public source application here: https://github.com/tungtv202/getlink_fshare