Often, when organizations plan and implement an effective test automation strategy, they believe that they are implementing continuous testing. They are wrong.
Test automation is certainly one of the most critical components of continuous testing. But implementing continuous testing begins with a multi-layered test strategy that includes all the types and levels of testing required—including unit, integration, functional, exploratory, and automated. Continuous testing must also have a strategy for incorporating testing throughout the continuous integration pipeline.
Organizations turn to DevOps to increase value and improve the customer experience by maximizing the speed of delivery without sacrificing quality. DevOps delivers by implementing a continuous delivery pipeline, which requires continuous testing.
But just what is continuous testing? The practice goes beyond executing automated tests throughout the software delivery pipeline and providing immediate feedback on the risks associated with a potential release.
Continuous testing requires continuous business and technical risk analysis, as well as process improvement and automation throughout the continuous integration process. Equally critical is developing a culture in which quality is a shared responsibility of every team member. It is based on an approach to managing risk that eliminates testing bottlenecks by improving the effectiveness and increasing the efficiency of your test processes.
Here are four key steps for an effective continuous test strategy, plus what you need to know about each.
1. Streamline the test process
This has three components: focusing on business risk, identifying and mitigating bottlenecks, and optimizing your tests.
The ultimate goal of DevOps and, therefore, the goal of continuous testing, is to mitigate business risk, which is composed of both customer risk and organizational risk.
Customer risk involves understanding which application workflows are most important to the customer and planning your risk-based test coverage accordingly.
Organizational risk involves understanding the complexities of the business climate as well as the product itself. For example, is it critical to be first to market, or is the product health or safety more important? Once the overall business risk is accurately assessed, you should map requirements, application components, and tests to those risks.
It is critical to identify and mitigate bottlenecks; they impede both quality and velocity. Constraints to the test process can be found throughout, from requirements through post-production checkout. Some examples include testers not being included in backlog refinement so they don't have the opportunity to help build the acceptance criteria, defects not being addressed on a timely basis, automated test suites running too long, and post-production checkouts being done manually.
Optimize your tests
Test optimization, the foundation for an effective automation strategy for continuous testing, is the practice of selecting the right tests that provide the test coverage you need with the fewest number of test cases.
This practice has many applications, from risk-based selection of tests for a full regression suite to choosing tests for targeted testing and regression of new features. It is a dynamic, ongoing process, especially when applied as part of a continuous testing framework. Test optimization should take place prior to automation and must continue throughout the continuous testing process.
The first step here is optimizing the test scope by understanding all integrations involved in critical user workflows—including the technologies employed in those applications (web, mobile, message/API layer, etc.).
Once you have a clear understanding of the test scope, your next step is to optimize the test cases. This includes not only analyzing test cases for their quality and level of detail, but also selecting tests that provide the greatest level of test coverage. Your test suites should be designed to provide the maximum amount of coverage with the fewest number of test cases to increase both quality and velocity.
Test optimization is a dynamic process. Evaluating effectiveness is a critical part of maintaining an optimized test suite. The suite should be evaluated continually to deprecate tests that are no longer needed, address flaky tests, and add new tests based on new features.
2. Automate testing throughout the CI pipeline
Continuous testing requires test automation throughout the delivery pipeline. Test automation increases the speed of deployment and reduces the risk inherent in continuous delivery.
But automation within the continuous testing framework is about more than just developing and maintaining an automated regression suite. In fact, automated regression suites, especially those that run overnight, create bottlenecks in the continuous deployment process. Continuous testing requires a test automation strategy that enhances, rather than impedes, the continuous delivery process.
It is not enough to implement an optimized regression test suite. The automation test strategy must include automated checkpoints at each step in the build process. This begins with unit tests that verify individual pieces of code and component tests to verify key features. Risk-based regression test suites should be customized based on the features you're currently implementing.
Automated testing continues into production, with post-deployment health checks to ensure the applications are up and running. As a final step, production monitoring should find functional and performance issues before the customer finds them.
Within your continuous testing strategy, test automation must be designed to run efficiently while providing reliable, consistent, reproducible results. Your automation strategy must be well developed and include the test maintenance plan and a schedule of which automated tests should run at specific points along the delivery pipeline.
3. Shift left
To implement a successful continuous testing strategy, it is critical that the team embrace the shift-left tenet of quality engineering, which advocates testing as early as possible in the development process. Quality engineering is the practice of building quality in throughout the process; it starts with development and focuses on preventing defects, rather than finding them. Shifting testing left enables quality engineering and continuous testing.
Test-driven development (TDD), behavior-driven development (BDD), and acceptance test-driven development (ATDD) are all techniques used to "shift left."
In TDD, developers and testers work together to design the test first, and then develop the code to make the test pass. BDD is TDD with a design focus on customer behavior. ATDD focuses on customer-centric design where the "three amigos"—developers, testers, and product owners—share responsibility for and work together in design sessions.
It’s important to take a shift-left approach to nonfunctional testing as well. Since the root cause of many performance and security defects can be found in design, it's critical to find defects early, before the design is finalized. When you don't find those defects until late in the process, they become much more complicated and expensive to fix.
4. Take responsibility for quality
This step is the underpinning of the continuous testing strategy. Unless all team members accept their roles in quality assurance, it will be impossible to successfully implement the other elements.
Taking responsibility for quality requires a shift in mindset that must be led from the top. Education is critical, since each member of your team must understand the nature of his or her role in quality engineering.
Want to know more? Come to my Agile + DevOps Virtual conference session, "Test Automation and Beyond: Implementing Continuous Testing Effectively," when I'll offer more tips on creating a highly effective continuous testing strategy. The conference runs November 9-12, 2020.
Keep learning
Take a deep dive into the state of quality with TechBeacon's Guide. Plus: Download the free World Quality Report 2022-23.
Put performance engineering into practice with these top 10 performance engineering techniques that work.
Find to tools you need with TechBeacon's Buyer's Guide for Selecting Software Test Automation Tools.
Discover best practices for reducing software defects with TechBeacon's Guide.
- Take your testing career to the next level. TechBeacon's Careers Topic Center provides expert advice to prepare you for your next move.
