Definitive guide for performance engineer.
Having problems with performance?
Does your API have too long response times?
Is your server often overloaded?
Or cloud invoices hit you in the face with horrible numbers that only use a handful of users?
If you are facing the above problems, or are you simply a developer looking to raise your salary but do not know why?
PLEASE READ, THIS WRITING IS FOR YOU .
First things first
Hello everyone , it’s me again, Minh Monmen in his spectacular comeback in early 2021 with the Performance Optimization Guideline series – Performance engineer bedside guide. This is a challenging personal project that I have cherished for a long time since entering Solution Architect with the desire to provide you, developers, devops, solution architects or more specifically, performance engineers. The most basic guide to solving chronic difficult problems is the optimize performance .
The Performance Optimization Guideline will be synthesized from my own knowledge of the few years I work, as well as the results of my research in the field of optimization performance, so it will not only have a summary of the theory. Or common sense approach, which will contain a lot of usecase I have encountered in the past. We hope that all the brothers and sisters, regardless of age, young, girl, boy, after reading this series, will also draw out experiences to apply for their own work, and will contribute more to themselves. for this series to improve and reach more people.
Since it’s a long-term project, I’m not sure how long I will get it done, or even afford it. But when I go there, I hope I can keep the 1-2 week release 1 chapter schedule to serve you. And also because this is a project that spans many articles, so I am not completely sure about the content of the future articles, so I leave here a dew index. I think there will be many of you like me who are often afraid to read books, so if you see long things, you will panic =))), but I hope you will feel secure to read the series, because you are who is even more pragmatic and lazy than you guys, won’t put anything out of the book if you don’t really find it necessary. In addition, I will try to incorporate a lot of practical stories to help you feel less sleepy.
Here is the preliminary content of this series:
- Chapter 1: Performance Optimization: what, why, who, when, where, how?
- Chap 2: Scalability and story about distributed delusion
- Chapter 3: The Art of Finding Bottleneck
- Chap 4: Surveying application with monitoring / profiling
- Chapter 5: Database optimization – Arrest a criminal
- Chapter 6: Caching optimization – The road was very tough
- Chapter 7: Buffering optimization – Step buffering is necessary
- Chapter 8: Connection optimization – Fatal deaths
- Chapter 9: Coding optimization – Is it worth it?
- Chapter 10: System optimization – When the heart rate is at 3GHz
- Chapter 11: Architect optimization – Back to the starting line
CALL TO ACTION : UPVOTE and CLIP this article for later reading and practice. (It also encourages me to write more). You don’t really WANT to learn something until you feel NEED it. Really need it! Saying that means that the things that I am about to say, or the things you have been forced to learn, will definitely help , just now you do not need it, so if you can remember, it is good not to remember it. It’s fine to save it.
Let do it! NOW !!!
Introduction (continued)
Continuing the program is 1 minute for advertising.
Author
Name: Minh
Stage Name: Monmen
Areas of work: Backend , Devops , Solution Architect
Performance Optimization 101: what, why, who, when, where, how?
This chapter will answer 6 questions that originated all of these problems: 5W + 1H, containing the basic definitions of performance optimization (a systematic and scientific way I have made up).
At first, my thought was: A bunch of theories about stride swimming isn’t as important as plunging your face into the sea and letting the ocean hit the ground with tsunamis. And it is true that after trying like that, having already been full of seawater, people will automatically know … the way without thinking so stupid. When I need it, I feel that the days of learning basic programming are boring with pascal, the nights of reading about basic concepts so that I can be useful for an interview the next day.
So this article is a bit of theory so that you have an overview and understand what we are going to do in the next articles. In addition, in order to prove that I do not cheat, these knowledge has long been summarized in books.
Okay, let’s get started now.
What is Performance Optimization?
Performance Optimization is the operation of optimizing the system’s performance . Speaking from the perspective of our economic people, it is the optimal output based on limited input resources . For example:
- You have a finite input period of 8 working hours
- You have 1 output that is the product number
- Your current performance is 100 items per 8 hours
So Performance Optimization is how to increase your fraction 100sp / 8h up, maybe 1 factor increase or decrease like 150sp / 8h or 100sp / 6h , or both factors like 150sp / 6h .
Coming here is still easy to understand, right? I come from an economic population, so you will find my approach will be filled with economic colors, and my vision will also be very practical. Get used to this.
So for the specific jobs in our industry such as Developer, System Admin or higher, Solution Architect, what does Performance Optimization mean? And how will we tie the above concepts to our industry?
TIPs : Most of the books or articles that I have read from past to present, this issue is often summed up into a rather vague definition, the type is always in the reader’s face, but does not explain why it is necessary to do it. so. So sometimes I will miss the thinking part and move on to the memorization part. Don’t do that. Try to systemize those things into something familiar and you won’t need to memorize it anymore.
The Performance Optimization that I am talking specifically here for the software will still be to increase the output based on limited input resources . What you need to do is find out what the outputs are and what your input resources are.
Some specific outputs:
- Number of requests that can be met
- Number of user / connection concurrent
- Waiting time for 1 request
- The rate of successful request
Some specific input resources:
- Time
- CPU
- RAM
- Disk IO
- Network bandwidth
Why do Performance Optimization be performed?
Question 2: Why do we have to do that?
Answering this question is important because it will determine whether and how much you do it. There have been many traumatic lessons that happen to me and those around me just because they cannot answer this question, or answer incorrectly, leading to failure or too wasteful of resources to perform.
Performance Optimization will forever be a programmer’s illusion until it becomes a real problem. The difference between the vision of an experienced profession and the dream of a fresh graduate is also evident when they answer this question.
Actually the answer is inherently very simple: The economic problem
- If you only make $ 100 but your server money is already $ 1000 ~> You need to optimize!
- If your number of customers is 1 million but your system can only meet 1000 ~> You need optimization!
- If your website loads too long causing users to leave ~> You need optimization!
- If google evaluates your web low because it is too heavy and doesn’t show users in the search results ~> You need optimization!
I have encountered this situation at work, although a running system consumes a lot of resources and I can fully optimize it to save server costs. But the stake holder (who has related interests) finds it unnecessary and does not want to do it. Simply because this system has no economic problems to solve for them, it is obvious that there will be no need to optimize and will save resources (yourself) for other tasks.
Thinking back, their economic problem was bigger than their own. And identifying the economic problem is the first step to having reason for the optimal story.
How about not? Okay, you can play like: “If you like it, I’ll do it”.
Who performs Performance Optimization?
Well, everyone’s work has an optimal part. Therefore, certainly no one person or role will always undertake this job. Why is that?
System performance is a broad category affected by many factors, both hardware and software. Therefore, to optimize it also needs a combination of many roles in the company:
- For dev, it is code optimization
- For system admin, it is the optimization of hardware or settings in os
- For database admin, it is the optimization of the query, the optimization of the db design
- For solution architect, it is the system design optimization
- For devops, the integration process is optimized
Currently in some companies there is also a role as Performance Engineer , ie engineers who specialize in optimal performance. Their job will be to analyze, evaluate, find problems, give directions to solve performance issues. The job is promising, just unfortunately it is not very popular in Vietnam. Mainly, optimizing performance will be integrated into the work of many different roles, not completely separated like that.
Many people can participate, but who is most likely to perform this job? Because performance optimization is the process of having knowledge of many roles, then knowledge of operation in particular as well as knowledge of the whole system in general. Therefore, I often see people playing the role of DevOps , Sys Admin , Solution Architect who have access to more performance problems. And the fact that in the companies I used to work with, the DevOps team, with the right condition that the team knows the operational metrics, as well as knowledge of both resources and code will be the team that drives the optimization. power.
Even so, a tree should not be young. One person will also have certain limitations in knowledge, so optimizing performance in general will need the coordination of many parties.
When do I need to perform Performance Optimization?
Although we said that only when performance becomes an economic problem, there are sufficient reasons to do it, but at all times we need the performance mindset in mind. That is, right from the application design step, the db design should also be calculated a little carefully, not just indiscriminate design and then slow processing.
But there is another school of criticism: usually optimizing the performance will make the code more complicated, difficult to read and maintain. So performance optimization should only be done at the end of the development process, when everything is on track. This is also true. Therefore, deciding “when” is the right time to optimize performance will also be a difficult question to answer. After drawing from my personal experience, I have some suggestions as follows:
- Always have a performance mindset when designing and implementing a software solution.
- Performance Optimization refers to the whole process from design, implementation to operation, feedback, … Therefore, understanding the entire flow of the system is very important.
- Having a performance mindset is different from performing performance optimization right from the start. Don’t be confused. Having a mindset means knowing this place can be optimized, this can be faster, designing this place can be optimized later, … but not always and immediately, but only until needed. new implementation.
Example of implementation of my caching:
- Having a mindset on caching means knowing where the data can be cached , and using it can reduce what resources .
- Implementing caching right from the start of the code is not always good, because caching raises many problems related to business logic, data misalignment , … but in the initial stage (which is the logic test phase application) should not be stumbled.
So in a lot of apps that I have done, colleagues often ask: “Why don’t you cache this API, save the db reading things” . I simply answered simply: “It’s not time yet!” .
Where is Performance Optimization done?
The answer that pops up in my mind is probably: everywhere .
Well, actually this answer is not wrong, all the stages, the components of the system will be able to be optimized. Example drafting a web page optimization plan according to guise:
- Reduce the number of HTTP requests
- Use HTTP version 2
- Drop REST to GRPC
- Minify css, js
- Cache the response
- Denormalize Database to reduce JOIN query …
Lots of things, right? The question is, can you do anything?
Of course. Your possibilities are limitless .
But, time is limited . So choosing where to perform optimal will be the process of finding the problem , ranking the problem and solving it by priority . And sometimes it’s time consuming to find the bottleneck of your system. The system is complicated but the problem sometimes lies in only a few small ones, so normally we will deviate =))). Yeah, don’t be afraid to admit this harsh truth. Sometimes we search the wrong place, leading to no optimization results or results that are not as expected.
An example: After accessing the micro services system with the interlocking request network, 1 of you asked yourself: Why doesn’t the system use GRPC? Then list out about a dozen advantages of GRPC over RESTful API, typically request to another service will reduce connection time, reduce transmission payload, blablo … Thus, the service will cross call. each other faster
I answered: Everyone sees that, and yes, it is an optimal thing, but just do a simple comparison:
- 1 Request is taking 300ms to respond
- Using GRPC saves 10ms
- Optimizing the query saves 200ms
So where will our present optimal choose? I will leave it here and say nothing more.
How can we do it?
After answering the above questions, you must also start to see a larger picture of Performance Optimization, right? A lot of pitfall, many mistakes have been, and will be made in this optimization process. In general it will be like this:
- Lack of overview of Performance Optimization. Optimizing system performance isn’t just about increasing the req / s.
- There is no exact reason to do it. Remember the reason that will decide the problem, if there is a new problem, how to solve it.
- Lack of combination of components in optimal performance.
- Right problem but at the wrong time.
- Wrong problem always.
So in order not to get stuck in the pitfalls above, I should have a scientific approach to performance optimization, not just saying, “I feel like doing this will be quick.” Also because the system is so complex that we lack a valid perspective, so we often approach Performance Optimization by blaming a certain element of the system (this sentence quoted in the book). and started to prove this assumption true or false. This approach is very dangerous, because it will take you a lot of time and effort but often have to start over. I think it will suit the elders with many years of experience, aiming hard to make the hypothesis of a system bottleneck rather than the dreamer like us.
However, you can practice making your own judgment. My imagination is an important advantage in the art of system bottleneck analysis and has helped me in countless net saves. But DO NOT APPLY or CLICK on your hypothesis when it’s not strong enough. Always keep an eye on other unusual things during the analysis.
Implementation process
Let’s go back to the method. In the book System Performance – Enterprise and the Cloud by Brendan Gregg, there are 9 steps to evaluate and optimize performance. However, I would like to summarize with some of the following steps:
- Select the performance scale and set the target of performance : From the requirements of the business such as how many users access you must choose a performance measurement method (can be CCU, throughput, latency, …). Next is the resource planning (Capacity Planning). This is the most important step, laying the foundation for all other activities such as choosing what to optimize, how to optimize, …
- Static performance analysis : Evaluate system performance based on solution review, architecture review, code review, … This step takes place in the dev process, requires reviewers (techlead, solution architect, …) to knowledgeable about business, knowledgeable about the system and able to evaluate performance based on imagination =)). Do not laugh at yourself, this is an extremely important skill and it takes a lot of practice and a lot of experience to be able to do it. In the process of doing, we must constantly learn and improve to adapt to newer and bigger problems. Good reviewers will have a very important basis to hypothesize system bottlenecks without the system running.
- Benchmarking : Usually people think of performance optimization only or think about this step, ie benchmarking to see if your app can withstand a load or not. But in fact this is only part of the optimization process only. Benchmarking will reproduce a number of specific conditions to test whether the application is achieving the desired performance at first.
- Monitoring / Profiling : Good performance engineer is someone who understands monitoring as well as how to monitor. This is also an important step in the whole process from dev to operation. We often make false hypotheses because of lack of monitoring information. I also find it strange that in a lot of big companies, although all developers have Senior titles, few people care about knowledge about monitoring and system operation. Not knowing your application monitor or not understanding the indicators is a big disadvantage and a huge barrier of performance optimization. Monitoring also acts as an early warning for performance problems that may occur.
- Dynamic performance analysis when there is a problem : The real problem is always far from what we think in the lab and when it happens we are limited to processing time too, so this is usually the step the most difficult and the most demanding. You will have to combine a lot of understanding, imagination and improvisation to solve the performance problems that arise. Except for the preparation of advanced debugging skills and advanced understanding of the system, I think I can only build with experience. Every few times the system is overloaded and you will be enlightened.
Approach
There are two approaches when it comes to performance analysis and will influence many of our decisions later:
- Analyze workload : Set the target of the system’s workload and see if the application is responsive. An example is to target 10,000 CCU and run the test to track the following metrics:
- The number of requests to the system
- Latency
- Error rate This approach is often taken from the direction of developers to have optimal code.
- Resource Analysis : Monitor the usage of resources (CPU, RAM, disk, network, …) in different cases. For example, with a 2 core 4 GB server, it will run the test to track the following metrics:
- Throughput (req / s) max
- Utilization: resource usage level
- Saturation: Point overload This approach is usually done from the system admin direction to have resource planning, scaling, …
In fact, I often combine both approaches to get the best overview.
For example, I have 1 service A providing 1 API and the request is to meet 10,000 req / s
I will go from Resource Analysis to find the performance limit for a certain amount of resources, assuming service A is capable of responding to 1000 req / s with 1CPU 2GB of ram. So here I just need to run 10 servers to have 10CPU 20GB ram to be able to meet 10,000 req / s (let’s say service A scale is good). But running 10 servers for service A is too costly for a very small service, so I will have to fix its resources. For example, 5 servers go and start analyzing the workload to see if with their requested workloads the system is able to respond, what the error rate and latency is and begin to optimize.
summary
It is said that a hundred words are not equal to a seeing, a hundred words are not equal to 1 image. I will summarize all knowledge in this article with the following mind map:
So I have to answer 6 questions 5W + 1H with you about Performance Optimization. The above is a rough understanding of performance optimization that I have gained in the past several years. In the following articles, I will go deeper into the bottleneck searching methods of the system as well as the handling methods that I used to perform. Specifically, in the following article, I will immediately mention the Scalability story and the story of distributed illusion . Please look forward to it.
P / s: Do not forget to upvote + clip this article to cheer me up and continue to follow my next posts.