ROI, I love you, me neither

Benjamin Butel shares his perspective on the famous ROI of automation in the context of software quality in this QE Unit interview.

Benjamin acts as an agile and test coach with a product-oriented approach and organization. Its priority is the creation of value supported by iterative methods.

This interview is dedicated to the ROI (Return On Investment) of automation, a question that can irritate without context, like a bottle thrown into the sea. This subject nevertheless raises fundamental questions.

This sharing leads us to take a step back from a business perspective, the customers, and the organization, refocusing on the right questions and processes to follow.

Join the QE Unit to access exclusive and regular content from the community.

About Benjamin Butel

As a test coach, I enjoy developing innovative test strategies for software project problems. I also love to lead the test team and help them improve their testing skills, and give them confidence in their ability to always do their best in a fun and straightforward way.

I am also convinced that communication is the best key for improvement, personal and professional successes. Benjamin is the organizer of the Ministry of Testing Rennes meetups, a member of the MForTest, Organizer of the Paris Test Conf.

Antoine: Can you start by introducing yourself?

I am Test Coach at Klaxoon, a French company that produces software focused on business collaboration. Although one of my passions is software testing, I like participating in the Ministry of Testing of Rennes beyond my role. Since the confinement, we have been organizing online events covering the whole of France, which have allowed us to have good exchanges.

I also participate in the organization of the next edition of the Paris Test Conf. Over the past few years, I have also shared my perspective on testing through articles on the Tester’s Tavern.

Antoine: One subject tends to particularly annoy you in the field of testing and software quality. Can you tell us a little more about it?

For some time now, I have regularly seen messages passing on calculating the ROI of automation. At first, what bothers me is the lack of context, making it difficult to articulate a possible answer. But, above all, it raises for me the real question which is not asked: “Why?”.

Why would you want to calculate an ROI? What is the objective behind this ROI calculation? The question draws the why from automation. This leads to a reflection that I do not see emerging; we remain focused on this correlation of automation with a merely financial gain. I think it’s unnecessary, especially in agile dynamics.

Antoine: What concepts do you see relevant to agility? Is it the notion of value, different from that of quality or that of tests?

This is the notion of value. If we try to answer the why of automation with a “Go faster in my acceptance phase”, that’s a red flag for me. That does not make sense. The real question is, “Why do you want to go faster in the acceptance phase?”. If we are moving towards a need for acceleration, we already have the first element of response. It remains to identify what we would like to deliver faster, what features, why. Without doing the exercise until the end, questioning repeatedly the why will leads us to the meaning of the work to be done and its value. The starting point is the value we want to bring and for whom.

“The starting point is the value you want to bring and for whom.”
Benjamin Butel

If we now start thinking about creating value, we can assess how automation can help us. The relevance of an ROI is questionable for this exercise. How will it help us measure value? Is calculating the best cost the equation to ask? We take the subject backward starting with the ROI.

Antoine: The ROI approach effectively positions us at the end of the chain to calculate the time saved on manual versus automated tests, for example. Is automation at the service of creating value we will measure?

Automation, for me, is a tool, like testing. I will even go further; software production is just a tool. Apart from a few companies specializing in software edition whose software is their job, the job is normally not the tool that we develop. If we take the tools that support the banking profession, the job is to do that of a banker, backed by tools to accelerate and improve this profession. From this perspective, automation is just one tool among others to help create value.

Calculating an automation ROI of 25 out of 50 manual test cases would make us think we’re going to save at least half of our time. The prism is wrong, not to mention the supposedly “saved” time we will spend maintaining our tests. I wonder a lot about the relevance of this ROI, showing a lack of perspective of organizations. I find it hard to understand why.

“If we answer the “why” and the “for whom” correctly, the ROI is not the subject.”
Benjamin Butel

Of course, automation can bring confidence up to a specific limit. But if we answer the “why” and the “for whom” well, it is not an ROI that will support our decision-making and priorities definition. In any case, an ROI alone will not make it possible to drive the production of value. Instead, indicators must measure the additional value following the changes and iterations we carry out.

We must answer two questions to drive the value created “How to measure the production of value?” and “How do we measure that we produce it better?”. ROI or automation do not seem to me to be the correct systematic answers to these questions. You can quickly try to answer the wrong problem without the proper perspective. An example is wanting to go faster in our iterations, compensating for bad organizations upstream. We will perhaps gain a few hours downstream, to the detriment of the days lost in the design and initial alignment.

Antoine: Our priority is to build an efficient organizational system, iterating and improving value creation. Do you have any recommendations for measuring this value?

Customer satisfaction tools are a good starting point for measuring the quality of a product. We can also use the conversion rate, the satisfaction of various users of the product; we come back to the tooling supporting the business. Automation can be one of the elements to accelerate, facilitate, or better achieve the activities. However, we easily forget the human factors of comfort and usability, which may, on the contrary, require slowing down.

The indicators are, therefore, very contextual. They depend on each context, company, and business.

We can then work on the alignment of the product, the software suite, and even the information system to be built in the face of these value indicators. This is where the indicators measuring the adequacy of the organizational system for value creation come into play. It is only by starting from the purpose of the product that we can find the right organizations, tools, and processes.

“It is only by starting from the purpose of the product that we can find the right organizations, tools, and processes.”
Benjamin Butel

There is no miracle; we start from a given state, with its advantages and disadvantages. It is from this starting point that we must measure the progressive improvement of value creation by iterations. Calculating the ROI of automated tests will not bring any interest in identifying this better production.

It is not necessary to have an ROI to initiate an automation process. We can identify a minimal automation scope, explore relevance and value creation with the team. From the first iterations, we can measure the value addition. We can realize that we gain or lose time, even if the focus must remain on the value specific to each context.

“It is not necessary to have an ROI to initiate an automation process.”
Benjamin Butel

If we place ourselves in an agile context, one of the great strengths of automation is the trust brought to the team during development cycles. A calculation of the ROI comparing manual versus automated tests does not make sense in this context. The priority is to build the product allowing it to create value. The team must collectively answer the “How” question.

How to organize? How to integrate quality? Automatic, exploratory tests will ultimately only be tools for injecting quality into this production. As an output, we deliver a product with its ecosystem of solutions to measure its value and areas for improvement. We can measure the feeling, the factual, for example, the rate of backtracking, anomalies. But I have a hard time understanding the point of calculating an ROI before even having automated anything and for which purposes.

My message is to try. If the team thinks that a specific practice can improve their value production, they may decide to test their hypothesis on a small scope in the next sprint. If a value is indeed created, the team can continue, or on the contrary, without value, stop immediately to evaluate other means.

Antoine: So an approach to automation focused on testing departs from the real subject of value production?

I have the feeling that the ROI is very similar to the notion of financial gain. When I dealt with this indicator, it was to achieve completely disproportionate financial or contractual objectives.

The first case was to want to measure what gain between the automation of 80% or 100% of the existing tests. Another is 60% test coverage by default, without even a notion of the scope covered. What does this 60% mean? What is the link with value? What interests me are the production of value, the speed of obtaining feedback, and the team’s confidence.

Antoine: It goes back to a cultural subject of the perception of the value of quality by organizations. We believe that quality saves us money by automation alone, far from value creation. Do you have the same feeling?

Yes, while having the feeling that it is a rather French vision of quality. On the other hand, looking at my Anglo-Saxon counterparts or across the Atlantic, I have the feeling that the prism is not the same.

You just have to look at people like John Ferguson Smart, Lisa Crispin, or Janet Grégory. When we follow them, we find the same subjects: value, product, and human interactions. Tools are then mechanisms that can certainly speed things up and make life easier, but their first prism is that of value. None of these elements appear in an automation ROI nor allow us to draw any types of conclusions.

Antoine: What could we give in guidelines when we see these requests for ROI appear?

For me, you have to ask yourself the right questions. The fact that this demand emerges does not shock me, perhaps in consequence of a managerial reflex or an effect of a V-cycle. What interest me is back to the “Why”. We find these elements in Lisa and Janet’s book “Agile Testing Condensed”: the Why, the Who, and How. The calculation of the automation ROI is included in the How, in the third position. I would therefore start with the “Why” then the “For whom” which will lead to sound reasoning.

“Beginning with the “Why” and the “For whom” will lead to good reasoning.”
Benjamin Butel

If we want to make money and reduce the testing team, I’m sorry for you, but you’d be better off resigning. As soon as you have something programmatic, automation is still subject to maintenance, no matter what. The software lives, and the tools must accompany its development and its changes. Agility also pushes us to be more pragmatic by using the right tool at the right time to deliver value. So I have a hard time predicting any gain in value through automation.

My experience has often shown me that we rarely go beyond step 1 or 2 compared to our imagined final states.

Antoine: You can quickly make a film about the ultimate completion of an automation architecture when a just-enough approach was enough to add value.

There is regular feedback from Axa on their test infrastructure. It would be interesting to see the positioning they were able to make of this investment. I will not speak for them, but I imagine that they have taken it to step by step, measuring the value brought from the overall system to Axa’s business rather than measuring the gain of their test infrastructure. I invite you to consult the latest articles that have pushed for both the results obtained and the structure put in place.

If we project ourselves into more complex, political organizations, it will be complicated to change mentalities in this context. So the answer shouldn’t be for me not to make the ROI; you have to take a step back from the “Why” in the context to make sense of it.

Perhaps in the questioning, we will find a point that will not make us go towards automation. Instead, a more beneficial idea may emerge to better respond to the identified problem. This is often what happens in the expressions of need; we come directly with a solution. The same process should be applied to move up the chain to assess other possible solutions.

Antoine: I often see a focus of engineering profiles on purely technical and technological subjects. What you have just shared is the materialization of the need for collaboration, empathy, listening to the heart of value creation?

This is drawn by the different rituals such as Impact Mapping or Example Mapping, where the focus is on collaboration. There may be some collaborative tools to help in a remote context. It reminds me of a tweet from Lisa telling us to take a notepad, stand up and share what we imagine on a whiteboard.

I am deeply convinced that exchanges activate much stronger collaboration mechanisms than emails or tickets. We are in the agile manifesto with “interactions over tools”; we place the tools and processes supporting these mechanisms without neglecting them or being the first concern.

When we start to reach a good level in collaboration, we can move on to the next level of playing collectively.

The collective can knock down mountains; there is nothing more potent for me. We can be collaborative without being collective. Like a soccer team with the best players acting individually, they won’t get far. In software, we can succeed in 4 or 5 sprints and completely miss sprint 6 by fatigue or lack of collective. It is not the ROI or the automation that will protect us. It is human subjects, much more challenging to measure and to maintain cohesion.

Antoine: In addition to what we exchanged, do you have any content, quotes, or people who inspire you?

Internationally, the books of Lisa Crispin and Janet Grégory are references “Agile Testing” or “Agile Testing Condensed”. It contains content and references to Mapping exercises and automation pyramids. There are also some exciting things from Alan Page about Modern Testing, like data management.

The book “Le Chaînon Manquant” by Valentin Guerlesquin and Henri Bigot relates the transformation of a team within a large group. I might say something wrong, but I don’t think there is the word ROI once in this book! This is not what I remembered anyway. The story is about a team looking to start test automation and how it gets there through a transversal approach. I recommend it without wanting to “advertise” to my colleagues; it reads very well, in a romanesque format, and can help people with questions about automation.

I take this opportunity to share a small teaser that the book “Agile Testing Condensed” by Lisa and Janet will soon be available in French! We are finalizing the proofreading with my colleague Emna Ayadi for an imminent release.