Faster test: How to halve test time in a life cycle

Tram Ho


In brief:

In just one year, a team test has been reduced by more than 50% in a test life cycle. They analyzed, planned and worked – they first looked at their time usage, and then they questioned whether they could reduce time in any of them. is not. Once they know where to be more effective, they can begin to solve bottlenecks. This is also how you can do it.

“No way!”

This is the response from a team when asked if you can speed up system testing within a week?

In a project, we usually have a nine-month delivery cycle, including 3 months for system testing, after milestone completion, and full effort every day for those three months. We used to work all night, all weekend, in the last weeks. At meetings where I explore with the team.

So we made another tactic. We took the time to ask where time went. We have started listing all the things we need to do as a checklist within 12 weeks and we ended up with a time model dummy equation:

System test time = RC * (TC / TV + B * BHT) / PC RC = Re-test cycle or number of retest due to application changes TC = Number of test cases performed in each TV cycle = test speed, or average, how many test cases have been performed in a unit of time B = Number of bugs found and fixed in the test period BHT = Time error handling or amount of effort handling each error (diagnose, document, verify errors …) PC = Human ability, or number of testers and their productivity

Now that we have this model as a way to represent time in a simpler way, we can ask more tangible questions:

  • What can we do to reduce the number of cycles to be retested?
  • Do all cases need to be tested?
  • How can we speed up testing?
  • How can we reduce the number of errors we have to handle during the test phase?
  • Is it effective in handling errors?
  • Is it possible to find additional helpers in testing? And how to work more effectively?

Taking time to analyze the big problem, the problem that feels hard to turn into easier questions to help us see that there’s really a difference.

We restarted the meetings by asking just one of these questions, then exploring ways to improve on just that one variable. Finally, after a year of improvements, we were able to reduce our system test time to five weeks. When we started talking about it, we didn’t think we could take a day in the twelve-week test cycle; We ended the cut time more than half.

Reduce errors in system testing

We started to handle the number of errors we found during the system test, because we found and fixed many bugs, so there was a good chance to improve. But more importantly, error reduction is important for the main goal of providing high quality software.

So far, we have not recorded the root cause of errors found in the system test, so we have to use some judgment and cooperate with the development team. We read through a sample of errors found in previous testing cycles and looked for any patterns. We have found some simple coding errors and some memory leaks (it’s a C ++ application).

We have invested a little time to make sure we make the most of our code reviews to help reduce the number of error codes. We conducted a training review code, tracked the review code activity, and reported the code review results to the team. We also started using some tools designed to detect memory leaks. These two improvements have begun to eliminate the effort needed to handle errors during system testing.

We finally started recording the root cause of the error and analyzing it regularly to find other opportunities for improvement.

Improve error handling time

When looking back at the list of bugs, the dev team often feels very confused to see that half of the bugs posted by QA have been closed without fixing. Two of the biggest contributors are developers who cannot reproduce the bug, and that bug is a copy that already exists on the system.

This requires a simple change: We trained a test team to find the bug tracking system before creating a new one. If they find a similar bug, then they will consider modifying the original bug report with updated data or consult the develop team assigned to the bug.

For the bugs that could not be duplicated, we tried an experiment. Instead of a tester writing a bug and continuing, the tester will keep a “bug huddle” to reproduce the bug for the development team. This error usually occurs at the end of the day. The tester will then rewrite the error report after discussing it with the tester. This makes the fix much faster, as the Development team often says: “Ah, I saw what happened.” This reduces any ambiguities during the error reproduction process.

After these improvements, we found that more than 80% of the reported bugs were actually closed without re-occurring or being buggy in other areas.

Speed ​​up the test

Increasing the scope of test automation seems to be a clear direction to speed up testing and automation is really helpful. But there are also other things we find will improve the speed.

We built tools to help automatically fill test results after installation, so the tester installed the build with the necessary test data available, when they came to work in the morning.

We have also created a list of bugs that are fixed according to the user’s wishes and raise the priority of these errors. The most desirable errors are those that have blocked the testing process that has been completed, so we have set the priority of the log bug to develop fix in order to help the tester’s test run smoothly when there are The bug causes pending to re-issue the number of test cases in a given function. This reduces the waiting time for the test team.

Reduce the number of testcase

From the idea of ​​reducing the number of test cases it was controversial for us to start thinking about it. But if we thought it was a risk-based test, where we put the most effort into testing it in the areas most at risk, then we started to make progress.

We evaluated each testcase set in two dimensions: the probability that those test cases will find bugs or faults and the severity of the consequences for customers if there are errors in that area of ​​the product. Priority is ranked from high to medium or low, and we used the chart to determine our method:

We have reviewed and analyzed this issue with the development team, asking them for the areas they consider most risky. They can name certain areas immediately, and they have also done some research in the change log for code lines with regular bug fixes.

We reviewed this data with the product management team to assess customer impact on severity. Likewise, they have some immediate concerns and have also performed further analysis based on usage analysis.

Test cases with P1 priority are the highest priority in our tests. We make sure to check these P1 mounting priorities early and often early in the cycle and then again, ensuring there is no regression in the cycle.

The test case sets have a lower priority than we installed as P2, and for these sets, we lost a bit more lag in the regression test until the end of the cycle. For the P3 testcase, we tested them in detail and truncated the test cases, used sampling and only performed them once in the system test.

Find more people

I saved this improvement for the last time because that was the first thing the team requested: a bigger team. I was concerned about asking for a larger budget at first; But, after making some of the improvements above and showing their progress, our leaders asked what else we could do.

Making drastic improvements requires a timely investment in your team. We were able to ask more people and show the specific improvements we were expecting with the new investment.

We ended up finding a foreign partner to help a lot of regression testing. This gave the current team more time to improve.

Before engaging with foreign partners, we received a little help by enticing others in the organization to help during the test. Product managers, customer support and technical documentation teams have really invested in helping create a better product, and they volunteered their time to help improve the testing process.

We also did a number of online tests, in which we gathered the whole group for a day to test different areas of the product. The develop and management team joined and everyone tested it for a day. We have awarded the person who found the most interesting and most serious error. The test congestion was fun and a great team building activity.


Finally, we reduced the test time from twelve weeks to five weeks. It is not easy or simple, but once we create that equation to represent time and begin to have conflicts in each variable, we are finally able to successfully solve our challenges. .

Because of this project, I have used this technique many times to speed up the test cycle. Teams really like the process of breaking down time into specific variables and finding opportunities for immediate improvement. And you, how can your team check faster?

Share the news now

Source : Viblo