I had to do something.
Software engineers were blind after the development environment. Production teams struggled with incidents with limited application knowledge.
Release managers had pressure to deploy software changes like hot potatoes to the next stage, leaving no time for reviewing, even less testing.
It was a mess.
Everyone was frustrated, including the most important ones: our users.
The initiative was led at La Redoute, a $1b e-commerce retailer that reborn from its ashes various times, now in an accelerated growth.
I was leading a 70+ back-end engineering team when I decided to focus the team on a common objective: the delivery pipeline.
This article shares one concrete experience of evolving an existing organization to deliver Quality Engineering.
Follow the QE Unit for more exclusive Quality Engineering from the community.
Create a sense of urgency on a simple common objective
Simplicity enables focus. The first thing was to rally the actors on a common objective, creating a sense of urgency.
I first looked for internal facts, such as recent incidents with customer impacts, slow resolution time, showing factual leaks in the processes.
An external benchmark using Accelerate complemented the approach to demonstrate our gap versus a known market standard, equally impacting the business.
With all releases coming, we had to change.
Focus methodologies on the limiting factors
I decided to focus the team on the pipeline being a limiting factor that accumulated too much organizational and technical debt.
The message was to deliver Quality at Speed without depending on a new technical project, following these principles:
- Measure the business outcome;
- Deploy at least once per day;
- Deliver with stability, recover fast.
The daily deployment constraint generated most of the complaints: “Impossible with manual steps”, “Too risky today”, “Business cannot test every day”.
Indeed, these problems were the ones to be solved.
I then rely on two main sets of methodologies to implement:
- Lean software factory
- Standard CI/CD pipeline;
- Trunk-based development maintaining Merge Request;
- Test automation (minimal unit, functional, integration).
- Lean software measurements
- Observability pipeline;
- Business metrics;
- Accelerate metrics.
Only then, it was time to talk about technology.
Use the architecture to sustain systematic processes
The software factory we built had CI/CD pipelines across environments but were falling short in their flow, security and testing design.
Software engineers lacked access to deploy after the first test environment; remaining environments were reserved to operational team due to legacy processes.
I believe autonomy makes the difference.
The roles were distributed for letting developers iterate until production, maintaining approval steps for the business or team lead for audit requirements.
The next step was to increase the quality gates testing requirements:
- Maximum build time for fast cycle-time & minimal unit tests;
- Minimum number of functional tests to ensure their presence;
- Systematic pre & post-deployment sanity checks for fast feedback.
We then move to Lean software measurements, starting by “Measure the business outcome”.
Use technology to support business outcomes
The operations team held most of the monitoring with technical and infrastructure-driven dashboards.
Business metrics pushed the developers towards the business impacts outside of their development environment‒and operations folks started to look at the same thing.
The achievement of “Deliver with stability, recover fast” was measured with Accelerate metrics on standard dashboards.
I relied on one motivated engineer to build a simple yet standard observability dashboard with various metrics.
It was then time to drive these changes.
Drive the team with accountability for results
From engineering to operations, teams required psychological safety to become true Quality Engineering units.
I took full accountability if something went wrong, whatever the issues were from engineering (my team), production, or whatever.
“Failure was acceptable, but failure to continuous improvement was not.”
Antoine Craske
We also clarified with team leads that the power to iterate up to production comes with the responsibility of ensuring the operations.
The next requirement was to set the rhythm.
Define the rhythm with minimal milestones
The teams had 90 days to improve their performance, aligning the status and action plans on a weekly 30 minutes meeting.
The defined scope of software components was assessed in their architecture readiness and performance assessment, both in business and accelerated metrics.
Simple objective, simple question: “Are you now deploying daily?”
We used business dashboards to link the outputs to the outcomes, ensuring the applications had the metrics available and with monitoring ratios.
Our lean approach enabled faster results.
We accepted the trade-off of not having all we wanted‒we were not measuring the change fail rate‒ lead-time, cycle and stability were sufficient.
The achievements of results enabled early wins to celebrate with the team; first in small groups, then within the organization.
Ambassadors were a way to influence the evolving organization.
Align the organizational structure for valuable interactions
The organizational structure drives the sources of power. Changes are necessary to succeed after early wins with the existing model.
Beforehand, the power was clearly for operations teams in charge of production, creating bottlenecks in the software changes flow.
The organization adaptation focused on a minimum viable collaboration:
- Domain-driven team, components and flows;
- Central to decentralized release management;
- Automation with self-service and APIs.
The visual messages are also important. See below software teams going to production with supporting platform and infrastructure teams.
A company has limited money. Changing an organizational structure requires to choose between dedicated or shared resources:
- Dedicated for stream-aligned teams per domain, requiring to iterate fast with no hand-overs, and where idle-time supports velocity;
- Shared for smaller teams either providing reusable components for other teams or isolating a specific complexity (e.g. middleware, DBA).
Part of these changes, we also clearly affected a Quality Assurance Engineer and a Platform support contact to each team.
The next stage was to measure the organizational structure flows.
Align the organization to desired system flows
Daily deployments set a constraint to streamline the activities necessary to achieve a particular objective.
Value-stream was essential to remove the limiting factors, focusing on the developer experience to achieve fast and stable changes.
The developer experience responsibility was set between the platform team to provide the necessary means, and team leads to improve continuously.
I set a global team objective for daily delivery, clarifying the shared priority. In addition, I set Quality at Speed objectives in individual engineering objectives.
The system incentives alignment is required to expand organizational changes.
Product and operations departments were not aligned on these same objectives, creating friction limiting improvements.
“Learning; Involve stakeholders from the ideation stage.”
Antoine Craske
I tried alone but failed with the HR yearly system (unfortunately, not quarterly), not taking the subject at the right time, and lacking more executive support.
The learning: Step back & involve the C-level and other stakeholders from the ideation stage.
Change is possible by leveraging the skills of the actors.
Leverage existing skills to achieve time-boxed results
Skills directly impact the capacity to change ways of working at the organizational, process, and technology levels.
The relationship using soft-skills of empathy, active listening and influence was essential to embark people in the change.
Active listening, reformulation and showing interest taking notes and sending summaries make the difference in future consideration.
The hard skills were valuable mainly on the design aspects for reliable processes flow, self-service orchestration, and data treatment.
There was no need for the last expert in observability: know what you want to build, look for standard ways, and start with motivated people.
Acting out with curiosity, perseverance, and a growth‒mindset was the most important aspect of building the change engine.
Yes, Quality Engineering requires us to act out of our comfort zone.
Building Quality Engineering focusing on the pipeline
After 90 days, we deployed a replicable way of performing controlled daily deployments in the remaining perimeter.
Passing the revelation stage, I let the team leads drive the broader expansion, remaining in support and delegating the status and improvement plan.
We get back to autonomy: giving people clear expectations and support to experiment makes the difference in the organizational performance.
“The essential element was to define the rules of the game with business measurement, daily deployment and operational stability.”
Antoine Craske
This case is an example of applying Quality Engineering, constraining the entire software lifecycle to continuous value delivery.
We covered the five pillars of MAMOS, the Quality Engineering framework: Methods, Architecture, Management, Organization, Skills.
We had other changes on-going such as business model, architecture and organizational changes.
The thing was to take one priority with short and mid-term milestones, realistically achievable in 90 days with minimal resources.
So, what’s your next focus on the pipeline?
Follow the QE Unit for more Quality Engineering.