Why automating tests in the first place?
To contextualize our reflection, we must step back regarding our objectives. The Test activity aims to support Business to reach a Flow state with fast, reliable and efficient software delivery. Thus, the test activity must also be fast, reliable and efficient. Consequently, this is where automated tests done right help to validate existing and new features in short release cycles . Various reports and market surveys, like the World Quality Report*, do share the main pains linked with test automation. Usually mentioned, the tooling, integration and ability to adapt to a constantly changing product. In summary, test automation can bring a significant competitive advantage leveraging software delivery, but still hard to achieve.
Without technical and organisational constraints, the Traditional Test Pyramid focus the testing effort on the main business and customer facing value, from successful customer experience to a flexible architecture.
The definition and requirements of a Test Strategy
A properly defined Test Strategy is a key element of successful execution of the activity in its given context. It is usually bypass or partially done in organisations, resulting in missed opportunities for improvements in the delivery process.
The requirements we defined are the following ones:
- Qualitative: The main value resides in the capacity to validate a successful user experience, a flexible and testable software architecture.
- Accessible: Understandable, shared and accessible for all stakeholders, from business to operations.
- Reliable: Systematically executed during the delivery process and in production, with a flaky test ratio under 0.25%.
- Scalable: The organisation and execution can be scaled across the organisation, test management and execution. The target being to execute all tests at maximum parallelisation capacity with functional constraints.
- Maintainable: Flexible to adapt to the change of our product with minimal rework. Changing one behaviour must translate to one change on the test configuration, not more.
- Optimized: Optimize the investment in time on automation, execution to find most bugs at the lower cost. The number of defects found and not found by the process is a metric that can support this objective.
The Agile Testing Pyramid, in contradiction with our Strategy Objectives?
The Agile Test Automation Pyramid is referenced for a proper Testing Strategy implementation, in traditional and agile environments. The core motivations are linked to Agile objectives: reducing cycle-time, meeting frequent deadlines, rapid feedback, improved and co-located collaboration.
Aligned on those imperatives, the pyramid recommends focusing mostly on unit tests done by developers and testers, to ensure rapid feedback in the cycle. Therefore, it recommends limiting end-to-end and functional tests, being costly to develop and maintain, slow to execute, complex and unstable.
Looking at the pyramid versus our objectives or validating business behaviour as early as possible, we can question this model mainly focused on opportunities provided by Agile models, and not on addressing business testing priorities.
We may prefer Unit and Integration tests for bad reasons
The points referred in the Agile Pyramid but focusing mainly on unit tests could be done for bad reasons. Three different typologies will be covered: Organisational, Process and Tooling.
A first reason can be related to lack of maturity or budget for test activity and test engineer allocation. It usually results with developers validating their own development, increasing the technical debt with to much unit tests.
Another reason usually mention that tests are done too late in the process, thus having low value and slowing down the whole delivery. In fact, most of time the inclusion of test from the design – also known as shift-left pattern – is not done. It is a typical factor of having tests started too late in the process.
The last one, related to tooling, miss an automated testing solution fulfilling the automation requirements: flexible configuration, fast execution and feedback loop, CI/CD ready, accessible to the whole team.
Functional and end-to-end tests usually fail due to lack of Skills, Automation & Tooling. We can reverse the point of view of the previous point, from a functional test automation perspective. The difficulties raised in the ecosystem reside in the capacity to deliver a successful test automation strategy.
In a lot of cases, the tests are partially automated, resulting in costly tests, harder to maintain over time. Other experiences usually lack trained testers and software testing competencies. The outcomes resulting in fragile tests, with manual data setting, lacking reuse, technical and design quality. This type of experience also results in a poor functional test automation experience.
Organisations also implement models that were not proven to scale or focused on the bad priority, thus increasing the tests complexity, execution time, maintenance. It usually resides in lack of design and analysis, and method selection like equivalent partitioning, boundary value analysis, decision-table testing, state transition, use-cases, … appropriated to each context. We get back to lack of proper investment and skill in the test activity.
The Traditional Test Pyramid assumptions can be challenged
The analysis of the different points of view and pyramid can make us challenge and assume the value of each type of automated testing layer.
This view raises the following open questions:
- How to implement end-to-end and functional tests meeting our test requirements?
- Do we need integration test with good end-to-end and functional tests?
- How to balance unit test effort and value? Focus them on architectural validation mainly?
Designing the Reversed Test Pyramid for a Test Automation Strategy
Our approach resides in applying the Traditional Pyramid, with an aggressive automation focus on end-to-end and functional testing, while limiting integration tests as possible, and keeping unit tests for architecture validation.
You can find practical information about our experience using the pyramid in this article using our open source testing solution, Cerberus.
Each test has its set of objectives, limiting duplicated effort on the test implementation and maintenance. An inherent assumption of this model is to invest in the testing area, meaning budget, skilled profiles, testing and evolution plan, valorising test within your organisation and careers. It can be translated in the metrics you follow in the cross-functional tests, adding the test strategy ones, number of defects, technical debt, adding testers early in projects as developers.
Tests need to be considered a part of the digital products you are delivering, so add their build and execution as part of the CI/CD chain, design them to be executed in production on a regular basis.
We also highly recommend shared dashboards for the various tests to be transparent and aligned with the stakeholders. Investing in an end-to-end and functional solution meeting the mentioned requirements, accessible to each member of the team, will also make a huge difference over-time.
Testing as a building block in the accelerated digital and more complex world
The evolving technological ecosystem is growing in complexity: distributed event-driven microservices, serverless functions, APIs, screen-less interactions, low-code and developer citizens. In this environment, we highly recommend investing in your Test strategy and implementation!
References
The Agile Testing Pyramid, The Agile Coach Journal, https://www.agilecoachjournal.com/2014-01-28/the-agile-testing-pyramid
The Forgotten Layer of the Test Automation Pyramid, Mike Cohn, https://www.mountaingoatsoftware.com/blog/the-forgotten-layer-of-the-test-automation-pyramid
The Practical Test Pyramid, Martin Flower, https://martinfowler.com/articles/practical-test-pyramid.html
The World Quality Report, 2019-2020, https://www.capgemini.com/research/world-quality-report-2019/