Construction reason
- After looking at your web shop topics, I was bored and decided to do something new.
- Meanwhile, my club held a contest, the system at that time was an open source, but we wanted to customize a few places but we couldn’t because we weren’t qualified.
That’s the main reason I decided to build my own system.
The first step of the system
Original resources:
- Backend: Nodejs.
- OS: Ubuntu 20.4
Problem 1: Run code in what kind of programming languages?
Usually when we want to run code programs we often press the run button in IDEs. But here we want to run the code by code =)
Remember when we want to code a certain language we usually have to install the environment (compiler). With C / C ++ is gcc it supports compiling and running C / C ++ code files through the command line and other languages are similar.
Example of compiling a C++ file with gcc
1 2 | g++ main.cpp -o main |
and run then type
1 2 | ./main |
Problem 2: How to run command line with Nodejs?
After a long google search, I found it, we will use Spawned Child Processes .
Simple implementation as follows:
1 2 3 4 5 6 7 8 9 10 11 12 13 | <span class="token keyword">const</span> <span class="token punctuation">{</span> spawn <span class="token punctuation">}</span> <span class="token operator">=</span> <span class="token function">require</span> <span class="token punctuation">(</span> <span class="token string">'child_process'</span> <span class="token punctuation">)</span> <span class="token punctuation">;</span> <span class="token keyword">const</span> child <span class="token operator">=</span> <span class="token function">spawn</span> <span class="token punctuation">(</span> <span class="token string">'g++'</span> <span class="token punctuation">,</span> <span class="token punctuation">[</span> <span class="token string">'main.cpp -o main && ./main'</span> <span class="token punctuation">]</span> <span class="token punctuation">,</span> <span class="token punctuation">{</span> shell <span class="token operator">:</span> <span class="token boolean">true</span> <span class="token punctuation">}</span> <span class="token punctuation">)</span> <span class="token punctuation">;</span> child <span class="token punctuation">.</span> stdout <span class="token punctuation">.</span> <span class="token function">on</span> <span class="token punctuation">(</span> <span class="token string">'data'</span> <span class="token punctuation">,</span> <span class="token punctuation">(</span> <span class="token parameter">data</span> <span class="token punctuation">)</span> <span class="token operator">=></span> <span class="token punctuation">{</span> console <span class="token punctuation">.</span> <span class="token function">log</span> <span class="token punctuation">(</span> <span class="token template-string"><span class="token template-punctuation string">`</span> <span class="token string">output: </span><span class="token interpolation"><span class="token interpolation-punctuation punctuation">${</span> data <span class="token interpolation-punctuation punctuation">}</span></span> <span class="token template-punctuation string">`</span></span> <span class="token punctuation">)</span> <span class="token punctuation">;</span> <span class="token punctuation">}</span> <span class="token punctuation">)</span> <span class="token punctuation">;</span> child <span class="token punctuation">.</span> stderr <span class="token punctuation">.</span> <span class="token function">on</span> <span class="token punctuation">(</span> <span class="token string">"data"</span> <span class="token punctuation">,</span> <span class="token punctuation">(</span> <span class="token parameter">err</span> <span class="token punctuation">)</span> <span class="token operator">=></span> <span class="token punctuation">{</span> console <span class="token punctuation">.</span> <span class="token function">log</span> <span class="token punctuation">(</span> <span class="token template-string"><span class="token template-punctuation string">`</span> <span class="token string">error, </span><span class="token interpolation"><span class="token interpolation-punctuation punctuation">${</span> err <span class="token punctuation">.</span> <span class="token function">toString</span> <span class="token punctuation">(</span> <span class="token punctuation">)</span> <span class="token interpolation-punctuation punctuation">}</span></span> <span class="token template-punctuation string">`</span></span> <span class="token punctuation">)</span> <span class="token punctuation">;</span> <span class="token punctuation">}</span> <span class="token punctuation">)</span> <span class="token punctuation">;</span> |
Writing a file that defines command line commands of other languages is ok.
Problem 3: What type of input is input into the program?
After a while of searching google sml, I found a way. We will write the input to a file in.txt . Later, if one article has many test cases, there will be many txt files
Example content is as follows:
1 2 | 5 |
We will take that txt file as input as follows:
1 2 | ./main <span class="token operator"><</span> in.txt |
Problem 4: What kind of time limit?
First I thought of the setTimeout function, so I decided to use it to limit the time of the program.
It seemed fine at first, but when I touched the java guy, it kept getting time limit even though the program ran for less than 1s (1 second was the limit I set in that post)
Then I noticed this radish guy compiles so long I don’t know (maybe my vps is leper). As shown below, the total time is 1.5s , exceeding 1s is correct.
So when calculating the time, only the running time is calculated.
Issue 5: What is the memory limit?
After learning about ostrich types for a while, I temporarily put this problem on hold 🥲🥲, maybe the nodejs guy can’t handle this problem.
Problem 6: Where is the answer stored?
Initially I saved the answer in mysql as json. Then I see this answer, I get it regularly for a period of time and I am also afraid that the mysql query is slow (as if the database has 1 million records not equal to )
Then I fixed it up a bit. Still save to mysql, but will cache with memcached.
- First check if the data is in memcached?
- If yes then get the data in memcached.
- Otherwise, the mysql query is also saved to memcached.
Issue 7: Sanbox
Do not exclude cases where bosses write programs to run commands against the server miscellaneous. One fine day when I entered the server with an error to see if there was no belt, it was fun.
Then I sat and searched for sml. I decided to use docker quickly, for the bosses to play with. Every time someone submits a post, I will create an environment container corresponding to the language people submit. Then read the output of the container, then let it fly.
Last Activity Stream
- Request comes with id, code file, language
- Initialize the container corresponding to the language submitted by the user and get the results of the program.
- Find the answer in memcached and compare it with the output of the program.
- If there is no answer in memcached, query mysql and save it in memcached.
Evaluate
- The problem of time limit, bad memory needs another solution.
- Container initialization and deletion takes a long time. Total time to run a cpp file that only prints “hello world” takes 6s.
- An article not only has 1 test case but will have many test cases. Handling this guy I will say later, but in this initial system, I handled it quite stupid =)).
- Another problem is when many people submit at once, the system may be overloaded 🥲. I have looked through websites like leetcode or codesignal, people will kick the queue to solve this problem. (As for your crappy system kid, there’s no need to worry because there must be so many people already )
Conclusion: I’m still not very satisfied with this V1 version, there are still many problems to be solved. Then I decided to find out how opensource people handle the above problems and rewrote the V2 version.
See you in part 2