Testing on a production environment is essential if you want to test the software as closely as possible.
Why so? Although early in-pipe testing (that is, shift-left testing) is necessary and strongly recommended, it simply isn’t really enough to ensure software quality is perfect. Companies implement agile testing methods and build a disposable infrastructure that is ready to perform testing in production environments, sometimes called shift-right testing .
By testing in a production environment, you create another level of confidence in releases after performing various tests in a live production environment. Testing in a production environment allows a company to see how an application responds when new code is pushed into the real user world. It will become an important component of your future application quality strategy in the future.
Below, I will explain why testing in a production environment is important, and then give tips for developing a shift-right test strategy .
Ask yourself: What are the advantages of testing on a production environment?
A major difference here is that the specifications change by extending the continuous test feedback loop from live production data and actual user traffic while performing tests on production environments. You will find a set of errors that you did not find when testing in other test environments (dev, staging, or pre-prod). The bugs collected in the production environment will help the development team isolate the bugs to improve application quality, thereby providing a better customer experience.
It encourages and empowers developers and SDETs (software developers in test) to test early on production environments. The goal is to improve the quality by building various quality builds around the application.
Here are the advantages of testing in a production environment:
- Beta programs where customers provide early feedback on new features and user experience.
- Prevent disasters with test recovery and better resilience. The application can recover from unexpected (chaotic) or unexpected events without loss of functionality and data.
- Designing and constructing a disaster recovery process will help free up chaos in the pre-production environment before performing it in a live production environment.
- You are testing with production data. (It is difficult to simulate the flow and data on the production environment, making it difficult to detect any probable situation for testing).
- It eliminates the risk of frequent development on a production environment when done daily, while you monitor application performance in real time with tools like New Relic. (For example, if there is regular testing on the product environment, the error detection before the customer reports the error is higher)
What are the risks of testing on a production environment?
Although testing on production environment has many advantages, it also has risks. The final risk is the design of the infrastructure. Has it been scrutinized? Is it repeatable and disposable? Otherwise any of the following results may occur with or without testing in the production environment:
- There is no backup plan in case the application is at risk of data loss
- There are no rollback plans when releasing a new version.
- Revealing potential holes
- Unable to recover from unexpected chaos
- Testing time caused bad experience for users
Take some time to understand the following Devops term: Pet with wildlife. In order to minimize the risk that can occur when testing on the production environment, the infrastructure of your application needs to be a wildlife, not a pet. It is important to have a repeatable and disposable infrastructure (using Chef, Ansible, Puppet or Docker) to handle any of the situations that may occur above.
Potential tactics for testing on a production environment
The purpose of testing is to prevent errors from appearing on production. Finding an issue after the application has been deployed to users is too late. We should continue to test shift left at every stage of the pipeline to allow for faster feedback than code testing and integration. It allows teams to find problems as soon as possible.
Testing on production environments is just a quality safeguard around your application. The new strategy of testing in the production environment is an important part of our testing strategy to provide quality applications to our customers.
I will divide it into three parts: deployment strategies, test methods in production environments, and monitoring:
Implementation strategies:
- Blue-Green Deployment ( https://www.sumologic.com/glossary/blue-green-deployment/ )
- Canary test
- A / B testing
- Automatic rollback strategy
Test methods on production environment:
- Testing New Relic Synthetic users ( https://techblog.vn/gioi-thieu-ve-cac-san-pham-application-performance-monitoring-cua-new-relic )
- Lightweight acceptance test
- Infrastructure integration test
- Visual testing with Applitools
- Test disaster recovery
Supervision
- Application performance in real time with New Relic
- Alerts Policies ( https://docs.microsoft.com/en-us/microsoft-365/compliance/alert-policies?view=o365-worldwide )
5 tips for testing in a Production environment
- Divide testing on production into layers:
While we talk about the idea of ’testing in a production environment’, it involves testing applications that run on the production platform alone, running tests directly with 100% deployed code, and testing the whole thing. test server list in the production datacenter.
Therefore, testing on the production environment should be divided into tiers to test different aspects of the production environment in different ways.
- Plan testing at a time when users are less likely to use:
Performance testing can have a big impact on the entire user base. It can make the server environment inoperable, which is something no one wants. We should research the analysis and determine when is the best time to schedule testing on a production environment.
- Collect original traffic data for testing:
We should collect and use actual traffic data in the production environment (such as user workflows, resources, and user behavior) to drive the load generation of test cases. When you perform tests in a production environment, you have confidence that the simulation behavior is real.
- Centralized supervision:
While running an experiment on production, always keep an eye on user performance metrics to know whether testing has unacceptable effects on user experience. Be prepared to stop testing if that happens.
- Create an “Opt-in” experience:
A great way to test how the application will behave with actual users is to have some “opt-ins” in the new feature releases. This will allow users to track and collect data from users in real time and make adjustments consistent with the testing strategy without worrying about affecting their experience.
Conclude
Application testing is always a high priority for any software development organization. Most organizations prioritize testing as much as possible before deploying to a production environment to ensure that the post-deployment transition is as smooth as possible.
But testing on production environment can provide some unique benefits for a DevOps organization that should not be overlooked. From team preparation to dealing with disastrous situations in the production environment to providing a better user experience for customers, over time testing on the environment has become an increasingly essential part. of application testing.
Testing on production environments should be part of the test routine, which is highly scalable and highly resilient. We should also continue to test early and often, and consider whether testing in production environments (or shift-right testing) is part of our testing strategy.
And although testing on the environment can be a lot of risks so if you are required to test on production environments, be very careful.
Good luck!
The reference article from the link: