The materialistic dialectical methodology of the hottest profession in the current IT recruitment market: DevOps
Hello everyone, it’s been a while. Well, last time I started working as a trainer in some classes about Kubernetes so I was a little busy. Yesterday was just or last session with my presentation on Zero downtime application . The presentation is not only about technology knowledge, configuration, … but also many other things related to process integration, application design, devops, … have the opportunity to systematize the achievements in recent years when approaching DevOps and Kubernetes.
In this article, I hope to give you a practical perspective on so-called DevOps at companies, in which the perspectives are subjective, there are perspectives from others, not from others. Right or wrong, I hope you can freely give your own opinion after the article to comment for yourself.
First things first
The first is the self-introduction, still me – Minh Monmen of every day, but instead of being the one leading you through the technological experience, today he acts as the philosopher Dép-Le Oop to Tells about a working philosophy that I have pursued since I started working until now. The article not only mentions the concept that is trending today, but also recounts the difficulties in recruiting and why it is so difficult to find a genuine DevOps. We hope you will support with many bricks.
!!! Note: !!! After articles with large technical content, this article is almost NON-TECH , you should not wait for super high-tech here, but expect it to give you a little experience for the you build team.
Story 2 front lines
The story starts from a training session for businesses, I asked you guys below:
Các anh em nghĩ DevOps là cái gì và khoá học này để làm gì?
One developer said (I can’t quote correctly, but just remember):
Thì giờ DevOps rồi container docker đang là xu hướng rồi, đi học để hợp với xu thế và không lạc hậu, Devops là vừa dev vừa triển khai trên server các thứ.
1 brother sysadmin said:
Theo mình thì DevOps là các công cụ tự động thay thế việc của con người, thay vì mình phải deploy code bằng tay thì sẽ có tool làm tự động
These are two opinions from the two front lines . Why do I call it two front lines, is because since I went to work with the first company, I have seen an invisible wall separating Dev team and Sys team of a company. Stemming from the opposite demand, the opposite habits, the opposite knowledge that leads to working together on the same product which is quite troublesome but ineffective. For example, when the system has an error, Mr. dev is not authorized to enter the system, he requires log, debug, Mr. sys has a log, but does not understand what the content is, … leading to very handling. long and sloppy.
At that time, I had to ask the leader permission to go night with team Sys, which meant I made an exception so I could access the system and debug to handle errors. Of course this is not something that should be done in large systems but it is the way we can handle errors as quickly as possible. At that time, everyone implicitly formed the so-called fast response team specializing in error handling and had the right to surpass the dev teams at that time when they both had in-depth application knowledge and access to server resources. This is what we call later DevOps .
Going back to the story of the other business, you can clearly see that the two-sided perspective comes from the experiences and characteristics of each side’s work. The Dev side thinks DevOps is Dev and adds server settings, while the Ops side thinks DevOps is Sys, adds automation. Because of this point, in many Vietnamese companies, I find it a bit abusive the DevOps concept to give their existing teams a new name. That’s their perspective, how about my perspective?
Who needs change?
After about 4 years of approaching docker, k8s, then DevOps, what have we gained? Unlike many other brothers, after the working process, what are things such as understanding of container technology, cloud, gitops, helm chart, configuration, … then I get something else and understand at Why talking about DevOps is not about tools but about a working culture. That is to find the common voice of the two teams Dev and Ops (system, operation) through sharing information , feedback and establishing a workflow . This is the process that both teams have to change to meet, not just install a few CI / CD and can be called DevOps. Let’s take a look at the DevOps concept for a bit:
This is a picture that says it all, but few people fully understand it. DevOps must start right from the plan stage, not until release , like most current teamwork. I take an example of a procedure at my old company:
- Dev and Dev lead come up with a code plan
- Dev code logic, implement it
- Send email asking Sys for server provision / DB installation
- Sys, after installing the infrastructure, will build code and deploy to the environments.
This is a very simple process, Sys Admin only participates in the last stage, once Dev has finished coding. This raises infrastructure related problems that the Dev side may not anticipate such as:
- Sys team has no experience operating a new type of DB
- The app architecture is not suitable for the deployment environment such as logging to multiple files, env fix in code, …
- Sys resource level is insufficient when the application is under load or is redundant
- and so on and clouds …
This process is still in use today, as many companies already have a DevOps team. And that’s what creates a wall between the two teams, when one team doesn’t know (or is not aware) what the other team is doing. This is something that must change, and must change from the manager’s mindset.
The walls are difficult to pass
Okay, so when approaching DevOps, when two teams have to understand each other’s work, what difficulty will it be?
For team Dev:
- Setup the environment , so that the dev environment is the same as the production environment
- Ops mindset, ie mindset about operations, product tracking, resources, …
- System-related problem , like application life cycle, kill signal, …
- Docker , build and optimize the docker image
For team Sys:
- Understanding of language , knowledge related to package management, dependencies, …
- Application logic , knowledge related to application logic, the flow of things, …
- Application architecture , API components architecture, worker, cronjob, … and how to interact.
These difficulties cause team dev to create products that are not suitable for the system and vice versa, the system created by team sys is not suitable for the application of team dev. These are not only difficulties in terms of knowledge (which can be improved through learning) but also there are difficulties in terms of organization, corporate regulation, confidentiality, class division. .. which causes the parties to be underinformed or overlooked. To overcome these very high walls, there is no other way to change.
Change or … live the hard way
Of course, when referring to the following changes, I do not mean that every business, every environment must apply. We can completely keep the traditional team organization, the traditional operation, it’s just that it will be a little more difficult for us on the above issues.
If you want to change, and are determined to change, then first of all, you need to find a common voice from both sides. You can make a dog or a tiger, but you have to cry together. And we did that by the DevOps team will come up with some rules such as:
- 12factor.net : This is the easiest thing to see because it has been widely recommended in the community. Here are the rules for easy application development to deploy on infrastructures, especially containers, clouds, …:
- Config load from environment variables
- Run the app as a stateless process
- Log out stdout, stderr
- App has the ability to run multiple instances
- App must handle the life cycle (graceful shutdown) …
- Migration rule : In addition to the app design, migration will be when the two teams have to rub each other the most, so setting rules for this part is also very important.
- Fast migration
- Backward compatible …
- DevOps review : In addition to letting dev and dev lead make the product themselves, DevOps’ opinion is very important and must be involved from the beginning of the product, not when everything is done. In which we will review about:
- Is the app architecture infrastructure appropriate?
- Are there any problems with app resources using …
To accomplish these, it requires both Dev to cultivate knowledge of the system and DevOps must also improve understanding of the application to provide quality advice. Well, not everyone wants to learn things that they are not strong enough, or are not familiar with, and the bosses do not appreciate DevOps’s role so they do not allow participation in the first place, … As we had to be very difficult and struggled to be able to properly put the DevOps team role and force was followed by the development team according to what we offer. But if they can follow, your life will naturally be a lot easier.
One rule process adds all
Overall, the DevOps team will have a process to talk to Dev, through which we will be asked to participate in the different stages of development as well as allow the dev team to participate in the operation of the product. . Specifically, the process will be as follows:
Step Planning : Planning and designing products, we will participate in the review of 3 things:
- Application Architecture : Architecture of the application, what components of the app are, how are they interconnected, their role …
- Dependencies : Conditions necessary to run the application, such as what DB to use, what cache, what message queue, … Whether the system has those components yet, is easy to maintain, …
- Sizing : Dev will make an estimate of resource usage based on calculating the amount of data that may arise. This section will be used for us to arrange resources properly.
In addition, in the planning step, we also provide guideline about Git , how to divide branches, divide the environment, …
Step Coding : This part of the code will mainly give developers the guideline for developing applications that meet the underlying infrastructure (specifically, Kubernetes), in which there will be parts of this 12factor:
- Config : Where the Config is located, what should be put in the config, …
- Logging : Log is fired according to stdout and stderr output levels
- Healthcheck : To deploy, the service must implement healthcheck.
- Concurrency : The application supports horizontal scaling
- Disposability : The application must handle system signals like
SIGTERM,SIGINT, … to graceful shutdown - Tracing : There is also a need to integrate the tool tracing of the system that is the simplest error tracing via sentry for example.
Deploying step : This part is a frequently repeated step in the team dev update application process, we will give a flow to deploy the application with the following requirements:
- Deploy request : Create a deploy request for the DevOps team with details related to new app information, when to release what environment, …
- Release note : Each of these deployments will need a release note to see if anything has changed. This will help the team be able to focus on those things when the deployment is done.
- Affected service : Give a list of other services affected and the level of influence, how we know and have a monitoring plan.
- Migration steps : This is a detailed section of the steps to perform migration, usually used for database or breaking change. We will also review whether this migration plan is feasible and has any impact on the operation of the system.
Step Operating : This is the end, we will give developers back the tools to monitor the system, allowing Dev to further intervene in the operation of the product. Include:
- Centralize logs : Centralized log system
- Resource monitoring : System monitor server resources, application metrics, …
- CI / CD dashboard : Dev will know where the CI / CD process is being performed, on the server app deployed.
- Error tracing : The trace error or trace activity tool
- Notification : Setup notifies system events so that Dev can know such as deployment progress, downtime, …
One final thought
Although thinking of DevOps will think of automated tools to replace people, but in order to achieve that, the human role must come first. It sounds very ridiculous, but very convincing, =))) Because DevOps not only has a bunch of tools and set it up, but more importantly, what do you use those tools for, arrange it. how and especially the information sharing as well as the role of the two Dev – Ops teams.
The most important thing that we have done is asking to participate in product development from the first stage, then the developer will have our comments on the infrastructure as well as we have enough information. information for planning testing, scaling, … later. Certainly, this is not easy for both sides because of the constraints in terms of organization as well as knowledge. But be confident and make changes with the company now, because it is necessary for rapid and sustainable development.
It’s over, thank you for listening.