We have to test in production.
While some testing must be done in non-production environments, other tests are only valuable when performed in production.
One critical test for teams aiming to deliver Quality at Speed software is to measure the value of their software increments, directly with their users.
But with the power to test in production comes the great responsibility. Deploying too fast a feature resulting in a loss of demand can be a disaster for a company.
Organizations can rely on the methodology of Progressive Deployment, part of the Quality Engineering framework, to balance value and risks of software changes.
Follow the QE Unit for more Quality Engineering from the community.
What is Progressive Deployment?
Progressive deployment is the act of gradually activating features in an environment, usually in production, to assess its performance for both IT and business.
The need for gradual deployments emerged from the accumulated risks of big deployments performed in big chunks, with the only option to rollback in case of issue.
The execution of progressive deployment strategies takes place in three phases:
- Deploy the software, then gradually shift the traffic
- Monitor and analyze the performance in production
- Adapt rollout or rollback depending on behavior and results.
The method gives the ability to deploy new software in a controlled way and provide safeguards and control levers to balance the risks of continually pushing changes.
Software teams accelerating the rate of software delivery rely on a continuous flow of changes with release trains to keep moving forward, not backward.
Progressive deployment aims to bring that features deployment flexibility.
The term comes from James Governor of RedMonk after hearing Sam Guckenheimer of Azure DevOps team explain Microsoft’s “Progressive Experimentation” model for features.
A series of techniques has been identified from dark launches, feature flagging, testing in production, canary launches, blue-green deployments, A/B testing, and so on.
Progressive delivery focuses on a gradual process of rollout and ownership to increase speed and reduce risks, while continuous delivery is about accelerating releases delivery.
One key point of progressive delivery is the control kept on which features will be accessible to which users, and by whom, consolidated in the two core elements:
- Release progression for users scope with canary, ring, percentage, etc.
- Progressive delegation, from software engineers to product managers.
Let’s see how Progressive deployment takes place in Quality Engineering.
Why using Progressive Deployment in Quality Engineering?
Quality Engineering is the paradigm constraining the entire software lifecycle to continuously deliver Quality at Speed software.
Its implementation relies on progressive practices across MAMOS on Methods, Architecture, Management, Organization and Skills.
Progressive deployment is an effective way to keep iterating with speed, while meeting the quality requirements by gradually assessing the value delivered in production.
How does Progressive Deployment contribute to Quality?
It is hard to guess what users will find usable, useful or even fun to interact with before we actually see concrete results from experimentation.
Teams can and must reduce that gap with research, design thinking and customer feedback, but also require to actually test to confront their ideas fast to the market.
Progressive deployment is an effective way to perform controlled tests in production to assess the value of software increments, being for incremental or disruptive innovation.
The controlled activation mechanisms enable to ensure quality until it is too late: in case a new feature is not meeting the user expectations, the change is rollback.
Progressive Deployment contributes to Quality being:
- Result-driven in measuring the value of software increments
- Systematic assessed for the need per feature or business idea
- Scalable in its deployment across teams and software components.
How does Progressive Deployment contribute to Speed?
Organizations with the pressure of the digital transformation increase their rate of software delivery to improve their value proposition faster than the competition.
The speed at which they iterate makes the difference to capture first a market opportunity, unlocking the access to economies of speed.
Software teams relying on continuous delivery practices are able to accelerate, but they need a way to reduce risks when deploying more software changes.
Progressive deployment is the methodology providing that flexibility of activating features gradually over a set of users, allowing to keep the speed of iteration and of quality.
Progressive Deployment contributes to Speed with:
- Focus on meeting the user expectations with feature increments
- Rhythm in forcing to activate and test features that are in the flow
- Asynchronicity with features test and analytics easing collaboration
- Visibility relying on data and facts to assess the value and validation to rollout.
How to start with Progressive Deployment in QE?
Progressive deployment requirements must be taken into account upstream to be a reality downstream in the software lifecycle.
Key elements have to be considered such as which technique to use and which metrics to rely on to correctly implement a gradual rollout.
Keep in mind the Progressive deployment principles:
- Focus on reducing risks
- Measure value and stability
- Use techniques with minimal effort
- Keep activation window short
- Clean the house afterward.
This last point is critical to avoid the accumulation of technical debt with many features flags left unused, polluting the code, and increasing the risk of introducing regressions.
The following steps can be followed to implement Progressive deployment:
- Define features in scope
- Define roll-out methodology
- Ensure observability elements
- Perform gradual roll-out process
- Transition the ownership.
Define features in scope
Not all features or all gradual deployment techniques must be used for all features; it’s about reducing more important risks with the least effort.
A practical way to assess the need for progressive deployment per features is to add that question inside your Definition of Ready and Definition of Done.
In the first one, you assess the need for such a mechanism; in the second one, you make sure the necessary implementation elements are present.
The key criteria related to the risks from your users and internal operations perspective. A small back-end application probably does not need it, but core applications do.
Define roll-out methodology
Multiple roll-out methodologies are available as part of the progressive deployment model to balance the risks of deployment.
The deployment types can be organized in three categories:
- Feature-flag
- Gradual roll-out (Targeted, canary, ring, percentage)
- A/B testing.
Each technique is more suitable to specific needs. Feature flags focus on features only, gradual roll-out for overall deployment and value risks, and A/B testing mainly for product.
Feature flags are useful to activate gradually features to a defined context of users, requiring tags in the code, is quite simple even for junior developers, and easy to rollback.
Gradual roll-outs specificities are:
- Targeted rollout releases for a selected group of customers
- Canary deployment allow to release based on different release version
- Ring deployment gradually releases to bigger set of users from ring to ring
- Percentage roll-out releases based on a traffic percentage
A/B testing, also known as split testing, exposes two variants of the same application to different segments of users.
By shifting specific versions of an application, teams can compare, experiment, and better understand how these changes impact their users.
Ensure observability elements
We need measurement to perform the entire roll-out strategy with confidence.
It is not enough to define the adequate progressive deployment strategy, the various elements of observability must be identified prior to implementation.
Identify the need for the following attributes:
- Customer journeys monitoring
- Feature analytics on usage
- Business metrics like transformation rate
- Traffic ratio allocation
- Golden signals of application
- Business monitor and alerts
- Technical monitor and alerts
- Logging and traces available.
Perform gradual roll-out process
You can then move to the effective rollout based on the choice of features, progressive deployment techniques, and users scope.
The quality of the prerequisites will make the difference here, as the majority of the work is required upstream within the lifecycle.
That stage is mostly integration and using the tools at your disposal that let you activate features, gradual deploy the release across a set of users or traffic.
You have to perform iterations depending on the technique and the risks involved; some A/B tests are done gradually over a set of traffic, and percentage roll-out can vary.
Transition the ownership
In addition to gradually releasing a change, transitioning the ownership of the release is equally important to effectively perform the entire rollout on time.
Some features can remain of the engineering responsibility, even if normally the product owner, product manager, customer success and even customer support can in the end.
High performing teams automate the majority of this handover process, checking metrics and events to tell the system when to switch the release’s owner.
Gradual, and especially automatic, changes of ownership help to ensure the feature is always being tracked by the most appropriate team for the job.
Progressive Deployment within MAMOS
Deploying Quality at Speed software requires mastering the risks of deploying an accelerating flow of software changes.
Progressive deployment is part of the Quality Engineering framework to effectively constrain releases risks on a continuous basis.
In good cases, the features are entirely deployed to the set of users; in others, a rollback is performed after learning that the features are not meeting its expectations.
It can be frustrating, but it’s always better than having released something negatively impacting the overall user experience at a point in time.
What’s your next gradual release?
References
Matt DeLaney, What Is Progressive Delivery All About? LaunchDarky.
Progressive Deployment Glossary at Split.io.
Debasree Panda (2021), What is Progressive Delivery and How to Implement it? Opsmx.
Harness (2021), Progressive Delivery: Everything You Need to Know. Harness.io.