Background job and queue for farmer

Tram Ho

Written by Minh Monmen

Since learning programming, we have all heard of concepts such as queue , stack , linked list, and a bunch of things that she did in basic programming courses, data types and algorithms. But understanding it is one thing, knowing it is one thing, and using it in reality is another.

But the story below is in another world, the world does not have a queue or stack , the story is about a farmer who raises vegetables, raises pigs, but the pigs are too sick so they move to bananas and write blog about my golden era of bananas. That’s it.

First things first

So who is that farmer? Of course, people are typing these words for the readers. And of course, he only slashed the wind about his golden era. But because he didn’t have programming training, he continued to write, so it would be easier to make a story closer to him.

So what will this story of this banana trade do for you?

  • Queue is not as high and far away as you think
  • Queue can absolutely be applied to your project, right … tomorrow

The dog tomorrow is tomorrow, tomorrow is not sure

The story before the queue

On the first day of selling bananas, I sat at home and had an order, I excitedly closed the application with the guests, put the order in the book, charged and wrote the bill. Done to save costs, I also ship bananas to the customers’ houses. It took half a morning, but I was very happy because of a good start.

The second day has 3 orders, 1 single morning, 1 single afternoon, 1 single evening, still the application process, book entry, cash register, invoice, ship goods. I made one application after another, and everything was still smooth.

The third day has 5 orders, a little overloaded but because the menu is spread evenly throughout the day, I can still handle it.

The problem happened on the 7th day. It was the 1st day, early in the morning I received an application quite far away. Because I previously saved 3G, I turned off the network. I finished all the steps, excitedly entered the house to check the message, then realized that I just missed … 10 customer conversations. I contacted people again, but because I waited for him for so long, people ordered another place.

And that is the price to pay for the lesson about waiting type handling.

Reach the queue and background job

After that day, I immediately changed the way I work so that I can always close the application, not process the order to wait for the other application to be processed, but then handle it according to the queue, when I close the deal with the guest Once the order is finished, I will give you an appointment and will only record that order on a queue list, hire another worker to take care of the book entry, charge, bill ( job ) , hire some other people ( workers ) to work as banana shipper ( job ) according to the other order list.

And so I was able to handle several dozen orders a day.

This is a simple example of applying queues and background jobs for the purpose of speeding up processing. Instead of just having to close the application, register the book, charge, write the bill, ship the goods and then accept the new application, I will only close the application, schedule the ship and handle all the rest by hiring others more professional. My customers will get feedback sooner, and I can serve more customers at the same time.

But do I do programming, where do I sell bananas?

Yes, because you don’t sell bananas but do programming, try to imagine the situation:

  • Your system handles a task that takes 0.5 seconds, nothing to discuss.

alt text

  • Your system handles a job that takes 10 seconds. And users will have to sit and watch the browser spin in 10 seconds to know what happens next.
  • The system can only open 100 connections at the same time . So the 101st connection will have to wait 10 seconds, then connecting behind will be waiting, waiting forever, waiting forever, … then timeout.

alt text

And that’s when you have to think about queues and background jobs . You only need 0.5 seconds to record the client’s request to the queue , responding to them that you will handle, then disconnect them and create the background jobs that handle this request on the workers .

alt text

Queue is a queue, obviously, comes first before processing (FIFO)

Background jobs are jobs that are done in the background, ie do not interrupt user interaction (non-blocking user activities).

Workers are separate components (usually separate processes, separate services, etc.) that handle certain specific tasks.

Maybe you are interested

15 “gold” certificates are worthwhile in the programming industry

Great way to ensure immutability in JavaScript

When should I use this form?

Although this background job approach makes the application’s processing capability greatly increased, not all systems need it, and not every system is suitable for it. Here are some signs that you should use queue and background job:

  • Large-scale recording systems (such as log, tracking, …). Pushing into queue processing and background jobs will reduce the risk of overloading the database.
  • Specialized reading systems but with reporting and reporting properties, the number of requests is small but very time consuming to synthesize.
  • The systems have a long response time because of the nature of work, objective limitations, etc. The feedback to users immediately and then run in the background will create a better user experience. The system is also capable of serving more users.
  • Jobs arising from the main business, working with many external services but not critical . For example, collect system history, email, update information from sources, …
  • The work is independent and less affected by chains or order . Make sure to scale the system by adding multiple workers at the same time.

When should not be used?

The following are examples that do not encourage the use of queues and background jobs unless there is an appropriate architecture:

  • Specialized reading systems operate as systems for reading articles and products, etc. These systems will be optimized by other ways.
  • The work is important , decisive but there is not too long response time . For example, requests related to payment, scramble for unique key system (such as booking, buying products)

A few practical examples

Here are some of the systems I actually used to queue and background jobs. You can consult to have a closer look:

  • Tracking system: client calls the tracking system to record user information such as browser information, date and time of access, ip, etc. The API system will then push that information into the queue on the previous redis, then the response OK for the client. The new worker then gets the information from the queue and writes it to the database. Because the number of requests is very large and the amount of data recorded on the db is very large, using a queue will avoid overloading the database
  • Logging system: After some user activity with the API system, such as login, logout, change profile, update post, view post, … the system will generate events, but these events will not be processed immediately. put in a queue to avoid user activity block .
  • Notification system: This system is in charge of sending notifications, sms, email to users. This is a business -critical, non-critical work and uses many external services, so it will be pushed into the queue and sent in turn.
  • The analytic system: this is an internal reporting system, although the number of reports is not much, but each report is very time consuming to respond to data aggregation. So pushing the queue to create a better user experience. The user will send a report request and receive a processing response. After being processed by the workers will notify the user again via the notification.

A few points to note and sum up

Through this article, you probably want to roll up the sleeve to follow him to sell bananas to make the queue, right? To put it simply, to say that you can apply it all the time , you need to pay attention to some problems when using queue and background job:

  • Job tracking: You must have a mechanism to monitor job activity when applying this method. Dealing with background jobs means handling jobs silently, but the mistake is always silently, not hitting your face like outside API.
  • Error handling: After monitoring the operation of the jobs, the next thing is how to handle the error that arises, how to retry or rollback, notify the responsible person, …
  • Retryable job: Your background job needs to be designed so that it can be retryed, that is, whether errors or errors can run again without duplicate data or excess data.

Of course, the story I talked about about youth selling bananas is just an absolutely fictitious story . You should not study but quit the job to sell bananas like me. Not so easy

Share the news now

Source : Viblo