There Is No More Quality Assurance @ Manomano & OpenClassrooms

“We no longer have Quality Assurance”.

This is what Olivier Dennemont and Vincent Dauce achieved at Manomano and OpenClassrooms, two major digital players. The word “QA” and its perception have disappeared from these organizations.

Quality Assurance has given way to the organizational model of Quality Assistance. It is about addressing software quality as early as possible, transversally and continuously on the software value chain.

In this interview, we share their experiences of respective transformations by exploring their definition, change management and results obtained. The references mentioned are available at the end of the article.

Follow the QE Unit to access more exclusive content from the community.

Can you start by introducing yourself?

blank

Vincent : I am the quality director at OpenClassrooms. I am also preparing a talk on this subject for the Paris Test Conf. This is the approach we have put in place with our quality team. I am also part of the Ministry of Testing Paris community, which generally interacts with other communities in France.

Olivier : I am Quality Manager at Manomano. I manage the “Qraft” team which represents the word game in Quality and Craftsmanship. I arrived two and a half years ago on QA to redefine the strategy. I have an engineering manager background mainly on mobile applications.

What definition do you give to Quality Assistance?

Olivier : Quality Assistance clearly contrasts with Quality Assurance. We too often associate the notion of being the guarantor of the software that goes into production in many contexts.

At Manomano this model does not work at all. Our context is that of a company in hyper-growth, with an architecture becoming more complex and distributed to meet the challenges of the organization. Having people checking that “the quality is good” or “that there are no bugs” at the end of the chain does not work.

I found the Quality Assistance model at Atlassian while looking to develop an alternative. Over time, I learned to understand and iterate on the implementation of the model. It responds to a concrete issue of the scalability of Quality practices. We teach everyone to work better rather than optimize a silo.

“Coming from engineering myself, I am convinced that software should be produced at the right quality in the first place. Acting in the second phase is not the priority. ”
Olivier Dennemont

Vincent : I agree with Olivier on many points. The big difference is that this approach involves a lot of people. You have to achieve a real change of mindset for different actors: the developer, designer, product manager, etc. It is a real organizational change.

The main objective is to accelerate delivery through collaboration ultimately. We say that we collaborate, it’s true. However, too often we find silos passing hot potatoes without really having a transverse exchange.

Why did you initiate a Quality Assistance process?

Vincent : In July 2020 we changed the way we manage our roadmaps. We had launched a joint survey during the various retrospectives. During this exercise, we obtained twice as much feedback on the organization of quality, its difficulties and its inclusion with the roadmap. This was the starting point.

We were leaving from afar. QA was “responsible” for delays in production deliveries. Shortcuts were then taken to go quickly but at the expense of quality. We often heard “it’s the QA that blocks”. And yet, 40-95% of the tasks assigned to QA actually contained upstream root causes.

We therefore decided to address the organization for these structural performance concerns.

Olivier : We have accumulated similar symptoms. Too often QA was held responsible for software delivery issues. A very questionable shortcut was made between quality issues encountered by a lack of tests by the QA team. We also needed to stop recurring errors and bugs.

The perception of QA being contradictory with the very reasons for the existence of bugs. They exist because of architecture, code issues or misunderstandings; not for lack of testing. More tests or different tests can help in detection. But what interests us are the root causes.

We really needed to get our heads above water to get our quality up. It was necessary to succeed in creating envy and on board the teams to also include oneself in the Lean, Agile or DevOps approaches. I am also often surprised by the maturity gap in software engineering for quality approaches.

What changes have you made?

Olivier : I renamed the team to “Qraft”. The strong message was to clarify that we were no longer a QA team.

We have incorporated other profiles such as developers into it. They help other developers with tools, collaboration, methods. We have also integrated craftsmen coaches to help developers do better at the root. Their goal is like Uncle Bob’s principle of “QA should find nothing”.

To achieve this, we have pooled the craftsmanship and architecture teams into this team. We turned Quality Assurance Engineers jobs into Quality Advisors. The latter are delegated to the product teams to collaborate as early as possible, as closely as possible and continuously in the software chain.

Different roles are needed. We have managers, coaches, and also ambassadors. Animating all the actors is fundamental to change.

Vincent : We have changed a lot of things iteratively. The QA team was also renamed to Quality Engineering. I myself changed my position from QA manager to Quality Director. We wanted to mark the occasion by removing the term “QA” too associated with “Quality Assurance”.

We have aligned the team’s mission to clarify our intention. Far from doing tests or just looking for bugs, our mission focused on “accelerate the achievement of shippable quality that makes education accessible”. The two important concepts are “accelerate” and “shippable quality”.

We communicated a lot throughout the organization, to the product, to the tech. I organized an official transverse kick-off to clarify where we wanted to go. We also fostered acculturation with various content such as Alan Page’s video on Adventures in Modern Testing. In 2021, we brought in an external company to help us define Quality: share perceptions, criteria and objectives.

We have broken down our mission into objectives to speed up delivery, eliminating bottlenecks while ensuring quality. No more indicators on the number of verified tickets and local focus. We look at transversal objectives such as cycle-time aligned with our mission.

The commitment of the various actors was fundamental to obtain their needs, their support and allow a real transformation towards a transversal collaboration.

What results have you obtained? For what value creation?

Vincent : We returned to the measure a year after launching the process.

Several structural improvements have been observed:

An improvement in cycle time of 34% from the end of development to production availability;
The disappearance of “QA bottleneck” complaints during various retrospectives;
An improvement in “useful” tests created by players other than the historical ones in QA.

In addition, we have also launched a survey on the perception of QA. 60% of the teams considered us essential, 30% very useful and 10% useful. It is a real marker of change in the organization after a year. We are even a little afraid of being indispensable.

This is understandable in our maturity phase and we are working on it.

Olivier : We first measured the number of known anomalies and openings in production. During the first year, we accumulated without seeing any structural changes. It was also worrying, I was almost starting to no longer believe it.

But then we saw a reversal of the trend. Our 800 defects have been reduced by half.

We have also exited the vicious circle of the broken window. Like a zero-bug policy, the presence of a single bug immediately generated responsiveness to deal with it as quickly as possible. Previously, one bug among many others was considered normal and remained open for a considerable time.

The role of the team is actually to create and maintain this constant pressure for quality without making it overwhelming. This constraint should make it possible to free up energies to proactively address the quality focused on the user experience and the improvements rather than the treatment of defects.

What brakes have you encountered? What do you learn from it?

Olivier : It was difficult to stay the course without getting results in the first few months. We had to explain that the model was viable and sustainable without being able to justify it in our context.

The actors were too accustomed to the comfortable pace of executing post-clearance checks. In reality, it was impossible to address quality in this way. We must be able to challenge the choices of architecture, design and even solutions.

“Resistance to change was the main issue to be addressed.”
Olivier Dennemont

We had to have a real force of affront and assertiveness to question certain choices. The pedagogy was essential by making references to other sectors, other organizations like Atlassian, by hooking them to internal examples.

The false equations of “quality = test” and “test = quality” also had to be dismantled in people’s minds. This seems obvious, but is necessary to position at the right place testing in the software value chain.

I really like the work of James Bach in this regard which decorrelates the tests which can be automated or not. One of the limits of the model can be reached when it is necessary to carry out tests requiring the expertise of a real tester. You have to have the right skills to perform the right tests.

Vincent : We have encountered the classic brakes for any change. Actors who ask questions, skeptical of the models, to criticize the model, to question the processes and proposals. These brakes are part of the process.

We moved forward with the people who were the most appetizing for this process. Little by little, it snowballed the rest of the organization with people getting caught up in the game.

“The main issues are about roles and responsibilities in a Quality Assistance approach.”
Vincent Dauce

We often say “quality is everyone’s business”. We can agree with this principle, but we still have to make it a reality in the rest of the organization. In Quality Assurance, the model was clear but ineffective. In Quality Assistance, a collective effort must be put in place which requires clarification which can be qualified depending on the context. Sometimes collaboration is the answer.

It was also necessary to address roles and skills. Our existing roles had to evolve for more support and coaching instead of single operational contributions. Some have evolved on code review, TestingOps.

At the same time, our organization has grown in a context of growth which has made our task difficult. We had to integrate people who had neither completely known the world before nor been involved in the process. The alignment of actors through clarity of responsibility and sustained communication were fundamental.

What are your next Quality Assistance priorities?

Olivier : We focus on Quality Management which even goes beyond Quality Assistance.

I work to educate top management on Quality issues. The objective is for Quality to become a criterion integrating structuring decisions of companies. The debate must be transversal and go up to the level of the management committee, to then allow even more structural investments.

I have a strong communication effort to achieve by using data and by linking the subject to the challenges of the company. It is a constant effort to increase this positive pressure for Quality.

For example, metrics focused on user experience are very powerful and easily deployable to the entire organization.

Vincent : I worked on a document to formalize the evolution of our Quality practices at OpenClassrooms. It is in fact the formalization of all our effort and our vision to align the actors and help us in the future.

Quality has always been a strong component in OpenClassrooms. We have quality profiles assigned to the quality of the courses directly in the content teams. It is an important cultural subject to keep alive in everyday work life.

Our next priority is to consolidate and then climb to the next level of maturity of Dan Buckland’s model. We want to move towards Phase 2 where most of the other teams carry out Quality activities.

For example, we address the practice of integration tests with developers, both on the front and on the back end. Our goal is to provide toolkits to make them autonomous on automated tests.