The need to accelerate software delivery cycles is driving companies to set up automated deployment pipelines to streamline various activities through automation and repeatability.
Nevertheless, 47% of companies consider non-regression testing to be one of the major unresolved issues, with underlying problems of test environments (55%) and insufficient test coverage (39%).
As a result, many of these pipelines struggle to meet their objectives, becoming more fragile and rapidly obsolete in the face of the various stimuli an organization undergoes, such as product evolution or employee turnover.
This article explores how the systemic approach supported by the MAMOS model can not only solve these problems, but also promote long-term sustainability of the software production system.
Non-regression testing, an art of balance
Non-regression testing is unattractive for many teams for multiple reasons: automating repetitive scenarios that either don’t exist or are carried out manually, solving background problems such as test data or environments, and maintaining them.
Typical arguments against non-regression testing are that it doesn’t justify the investment. Accused of slowing down delivery cycles and being costly to maintain, ultimately bringing little improvement for the whole team, from product to operations.
The solution therefore often lies in local approaches of this type:
- Product team defines requirements and use cases
- Developers run unit and integration tests with their tools
- A tester profile, if present, to set up test campaigns
- The product team explores and uses analytics in production
- Operations implement the monitoring they deem appropriate.
In such a scenario, inefficiencies abound:
- Requirements are tested several times by several people
- Reference systems end up out of sync without global measurement
- Each team optimizes its local vision without considering the global picture.
These organizations have locally optimized non-regression tests, with performance focused on indicators such as the number of tests implemented, without offsetting negative effects such as increased execution time, duplication of tests, or a high false positive rate.
The result? With each software change, the cost of initial analysis becomes higher due to the multiple diluted impacts without consolidation, and the cost of execution increases drastically, with each team having a heavy manual adaptation burden.
A sustainable solution requires an integrated approach
Software delivery where non-regression testing is value-adding, robust and efficient requires a more integrated approach to find the right balance between investment justifying results, and true team satisfaction.
When trying to go too fast, the following symptoms materialize:
- The team will not be aligned on the subject, continuing in part as before
- Management won’t support the process nor the necessary resources
- Tests will be difficult to maintain, often ending up slow and unstable
- Test results will only be used by part of the team
- With time or delivery stress, the tests will be ignored.
An integrated approach means considering the entire software production system, so that the non-regression testing approach enables overall performance improvement, beyond technical and organizational silos.
This integrated approach is described in the MAMOS model:
MAMOS describes an organization’s software production system through 5 domains (Methods, Architecture, Management, Organization, Skills) and 50 software production capabilities at the heart of which the players may or may not perform.
These capabilities need to be developed and aligned to solve systemic problems with systemic solutions. Let’s see how to use it to implement deployment pipelines with perennial non-regression tests.
Systemic deployment pipeline architecture
Valuable, sustainable and optimized deployment pipelines are the consequence of a production system with (i) aligned system capabilities, (ii) systematic processes, and (iii) positive and negative reinforcement forces.
Here, we use the AAA (Assess, Architecture, Accelerate) methodology, focusing on the two Architect and Accelerate steps, as these are the ones that can be applied while abstracting from the context.
The Architect stage consists of:
- Aligning the system structure
- Ensuring minimum capabilities
- Define a systematic process.
Aligning the system structure
This first step involves aligning the structuring elements of the software production system to speed up the implementation of pipelines, and also reduce the forces that might otherwise slow down or inhibit the dynamics.
In our case, these 3 capabilities need to be aligned:
- Vision to ensure executive sponsorship and cross-functional alignment
- Structure of organizational design clarifying responsibilities
- Focus to define the areas to be addressed first.
Executive support is to be sought at a level with the ability to act and influence end-to-end in order to optimize the global view and solve structural problems—beyond the product, engineering or operations teams of a given scope.
This overview will make it possible to define, in relation to business challenges, which areas need perennial acceleration of their digital products to maintain energy over time, sharing value creation on a regular basis.
Ensuring minimum capabilities
Going too fast without foundations is a guarantee of having to solve foreseeable problems that will be a waste of resources for the organization, and a source of demotivation for a team just starting out.
It is therefore necessary to identify and secure the minimum production capability required, depending on the objective being pursued. For our deployment pipelines, we need to analyze and carry out focused, robust tests, maintained by profiles within the team.
These 3 MAMOS capabilities come into play:
- Modularity to assess the functional cohesion of components
- Integration to ensure environments, data sets and availability
- Skills mapping to guarantee the skills required for each activity.
The application of this grid can lead you firstly to develop the capabilities of an application perimeter suffering from too strong coupling between its components, with unstable environments with no refreshed data sets, no skills to define tests, nor to design and maintain them correctly.
Define a systematic process
With the system aligned, it’s time to architect a systematic process for (i) integrating deployment pipelines into the existing production process, and (ii) measuring the achievement of non-regression deliverables.
The integration points in the production process are as follows:
- Specify to identify the requirements to be covered by non-regression tests
- Deploy to run non-regression tests on each deployment
- Operate forcing the creation of a non-regression test for each incident.
These 3 steps allow us to systematize the activities required for the non-regression test lifecycle within deployment pipelines. The challenge is to make this a reality by changing team habits through practice and coaching.
Accelerating the deployment of systemic pipelines
The Accelerate phase follows on from the Architect phase, which defined the system, its prerequisites and the process to be implemented. The priority here is to implement the process efficiently, solidly and sustainably.
These 3 actions are necessary and complementary:
- Test and adapt quickly
- Developing support capabilities
- Expand while containing complexity.
Test and adapt quickly
Concrete process practice enables us to confront system alignment and process definition with real system dynamics. The aim is not to force the process, but to adapt our measures or the system through action.
Each stage is therefore measured by implementation metrics, which are not performance metrics in any way; they are designed to monitor the effective implementation of pipelines within the defined perimeters, and to clarify expectations for everyone involved.
The metrics to follow for rapid testing and iteration are:
- Covering requirements with non-regression tests
- Results of tests executed across pipelines
- Number of open defects by environment and criticality.
At this level, 3 production capabilities are to be used in the development cycle:
- Measurement to develop the fact-based measures defined
- Fail-fast to help teams quickly confront and overcome failure
- Development by leading solution-finding sessions.
In addition, systematic retrospectives are needed to ensure that any structural problems encountered are resolved. In this case, Learn, Improve, Plan are the capabilities to be activated.
Developing support capabilities
Once the iterations have been able to develop a functional model in a given software production system, we need to ensure that the foundations are sufficiently solid before deploying the new practices more widely.
The support capabilities of the software production system are those that make the process repeatable, while guaranteeing compliance with a framework thanks to observability, self-service and the replication of learning mechanisms.
MAMOS identifies three types of support capabilities:
- Observability to architecture standardize visualization
- Self-service to provide a turnkey portal for other teams
- Organizational learning to encourage inter-team sharing of practices.
Observability here means automating standard metrics defined upstream, directly collected during process execution, which must be accessible and activatable on demand via a portal for authorized profiles.
Expanding while containing complexity
Complex systems have an unfortunate tendency to grow in complexity, and to accelerate without any counter-forces in place. Their aim is to contain the accumulation of complexity and avoid local over-optimization with negative effects on the system as a whole.
Process metrics are therefore complemented by performance indicators to balance effort and impact, and to extend the deployment of pipelines through risk management and the progressive decentralization of team autonomy.
The 3 capabilities to activate and develop are :
- Performance to define and measure performance indicators
- Risks to adjust the level of tolerable risk in iterations
- Empowerment to develop team autonomy.
Ideally, measured performance should be linked to the overall dynamics of software production transformation. The MAMOS model proposes 3 relevant ones in our case: deployment frequency, CFR, automation rate.
Deployment frequency is the pivotal indicator for measuring the acceleration of changes supported by non-regression tests, which, if they become too important, slow down deployment frequency and need to be reviewed.
The Change Fail Rate (CFR) identifies pipelines requiring additional non-regression testing if the rate exceeds defined ceilings. In addition, the automation rate can be used to gradually eliminate manual activities.
An example of systemic pipelines at scale
The various stages enable the development of core and supporting production capabilities for resilient and efficient deployment pipelines. MAMOS mapping will approach the following result.
The following example is taken from a personal experience of implementing systemic deployment pipelines at scale. Shared practices have enabled the model to be replicated across the organization, even though this example focuses on a digital perimeter.
The digital perimeter covers the key applications in the hands of customers: the e-commerce site and mobile application, connected to a myriad of internal solutions (product repository, offers, customer, payment, order management) and external solutions (mainly marketing).
Applying the methodology enabled us to move from a production delivery every 12 weeks, followed by 2 weeks of patching, to a 96% successful production delivery per day, supported by a hundred tests and up to over 7000.
A forthcoming article will detail the various stages and practices involved in this approach to industrializing deployment pipelines, with their non-regression tests for greater value, robustness and speed of delivery.
Maximizing impact with a systems approach
The systems approach offers a holistic solution to complex software production challenges such as deployment pipelines, overcoming performance levels that too often remain unachieved for many companies.
The paradigm is summed up in the mantra “Build Better, Build Faster”, recognizing the need for global value creation supported by committed players and antifragile production capabilities, of which sustainable iteration speed is a consequence.
The AAA (Assess, Architect, Accelerate) shared methodology extends beyond deployment pipelines by providing a ready-to-use three-stage model, adaptable to other software production contexts and issues.
This first stage of analysis enables you to take a comprehensive look at your production system in the face of recurring quality, speed and efficiency issues that traditional approaches fail to address.
By the way, make sure that the non-regression tests of your pipelines are faced with quality requirements, as the State of Software Quality 2023 study found that 70% of defects stem from insufficient specifications, and are perhaps, your first priority.
Step back and maximize your impact with AAA.
References
2023. Software Testing Statistics – 2023. Magnitia.
Antoine Craske (2021). 96% Successful Daily Web Production Deployments. La Redoute.io.