Hello everyone!
Firstly, a small announcement: I haven't been posting regularly lately. It doesn't mean the blog is over — far from it! Increased workload and personal matters have reduced my time for writing. Now, onto the topic.
Lately, I've been asked several times about "AGILE test automation." Applying agile testing methods to automation depends on various factors like the project, team, and business. Some examples off the top of my head might include a business mentally stuck in a waterfall approach, attempting to rush through tasks while claiming to be agile. There is no universal solution, but I'll provide an outline of how I would handle it.
The Case
Tests are plugged into the pipeline. Automated suites run before each merge to the master.
Problems emerge
The first problem we'll likely encounter is tests running on a different machine than our local one, leading to potential timeouts. The solution might involve increasing timeouts and adding 'probing' — repeated attempts. However, this could extend the length of the test run.
The second problem caused by the first solution is that the test run can block development. Imagine having a test suite of 100 tests, all falling into a regression pack run before each merge. Devs working on different branches merge changes rapidly, leading to queued test runs.
Each test run lasts around 40-60+ minutes, with a range of failed tests almost every time (false negatives caused by timeouts due to weak Application Under Test (AUT) performance on the remote, while locally all pass!).
In such a case, the entire team will be irritated, and QA will spend more time fixing emerging issues. In the worst-case scenario, devs might skip testing to avoid blocking development.
Business pressure might push for parallel testing, but this introduces other problems. First, consider if our tests CAN run in parallel. Second, consider if our AUT will be able to be tested in parallel. Third, is test framework capable of running in parallel?
Stuck in a waterfall approach, attempting to rush through tasks while claiming to be agile => WAGILE
Ultimate solution:
While not a 'golden path' or 'universal,' this solution seems sensible to me.
Check for Test Suite Duplication: Look for duplication of test cases in your test suite. Many times in automation, we duplicate scenarios, forgetting equivalence partitioning. Each test checking the same "input group" is considered wasted resources and time that you can save.
Reduce Regression Suite: Reduce the regression suite to eliminate overlapping scenarios, saving time.
Use Risk-Based Testing (RBT): Use RBT and consider what is eligible to be tested and what does not have to be tested. Using tags, mark tests that have to be either ignored or run as regression. Decide what to test using RBT and a Risk Matrix.
Risk Assessment: Categorize risk into High, Medium, Low, and No-Risk items. Assess risk at the start of the sprint, involving the dev team and other testers. Continuously update the risk assessment as the sprint progresses.
This risk assessment can help reduce the scope of testing even further. For example, from 80 tests, determine only 50 that are in the range of critical-regular. Run these tests. For the remaining 30, run a fully automated regression on your own if you have time and resources.
Risk-Based Testing (RBT) Categories:
High Risk:
Code Modifications
Bug Fixes
Critical Functionality
Medium Risk:
Regularly Used Functionality
Functional Enhancements
Low Risk:
Stable and Untouched Code
Non-Code Related Changes
No Risk:
Thoroughly Tested Areas
External Elements (e.g., third-party services)
risk assessment can help reduce the scope
Summary:
To reduce testing time and unblock agile development, consider mixing techniques from manual testing like equivalence partitioning with RBT methods. It will help reduce time and the risk of errors. More tests increase the risk of failure.
Happy testing!
Comments