We shared with the Portuguese community the theme: “How to succeed in your test automation project? (transversally)”
I thank the participants for their participation and contributions:
- Filipe Carvalho, Engineering Manager at TalkDesk,
- Joel Oliveira, Senior Project Manager & Test Evangelist,
- Ricardo Lopes, QA Lead at Feedzai,
- Marco Pedro, “Making Legacy Software Great Again”,
- Diogo Carvalho, Software QA Tester at Smart Consulting,
- João Santos, Freelance QA Tester,
- Fábio Barbosa, Test Automation Engineering at Basecone
In this event we shared the following points around test automation projects:
- What is the expected business value of the project?
- Align your test strategy with business needs, not only with a generic pyramid
- Quantity is not quality, drive by value and risk reduction
- Value is in the use of tests, speed of execution is often key
- Tests need design like any other engineering activity
- Have an incremental approach on stable foundations
- The importance of training teams in quality
- Quality is a transversal responsibility that the team must compose
Contextualization of the event
Test automation projects have a high failure rate, presenting real difficulties and particularities to be understood.
Unfortunately, they are often delegated to QA, to a trainee, or boil down to a tooling aspect.
As a warm-up we have filled in the following circle presenting traditional steps of an automation project.
We first put our ideas in the circle, the most central according to their importance.
From that base, the conversation started with Marco’s sharing a case of test automation in his professional context.
The next sections sharing the key points we have identified take height.
What is the expected value of the project for the business?
We quickly returned to the fundamentals, sometimes too quickly skimmed over: what do we really expect from the automation project?
Siloed organizations tend to obscure the real need.
Sometimes the same can also be started in a specific silo without a global link.
From my point of view, this initiative must quickly be hooked up to a stake of the transverse team and more generally in the company.
As for the event carried out with the French-speaking community, the questions “Why” and “For what” remain fundamental and must have a clear answer, preferably at the strategic level of the organization.
This meets the point of the need for a project sponsor, accompanied by an awareness of non-quality in the face of the challenges of the profession.
In summary, finding a way to identify and share business value with stakeholders will make the difference for the project.
Aligning your test strategy with business needs, not with a generic pyramid
We have addressed a second point in relation to the business need: what tests to put in place?
A first instinctive response might be to refer to a model pyramid of tests, useful in providing a framework.
The downside of models is that they are taken from more general practices out of context. In the case of the test, the business need is partly abstracted from the equation.
The test pyramids are not directly aligned with the business need.
Joel Oliveira
So we can easily find ourselves doing a lot of unit tests, on an application that actually had more of an issue of automating end-to-end critical functional tests.
Our reading was to:
- Start by aligning business needs
- Identify test techniques that are most relevant.
Only then does the question of automation arise according to the resources, the tools available, the maturity of the team.
Quantity is not quality, driving by value and risk reduction
Showing results in automation is often associated with achieving a high percentage of coverage or absolute value of automated tests.
Aside from the possible flattering effect and a fulfilled personal ego, this type of metric often turns out to be the wrong goal.
A coverage or number of tests can be a metric in support of the implementation of a process, but in no case the objective to be achieved, let alone a business performance indicator.
Rather, we shared to drive by the value of testing within the process.
For example, does you tests allow you to:
- Deploy in production with confidence for customers?
- Save precious time for people on the team?
- Reduce risks associated with particular scenarios?
The quantity does not intervene as a primary criterion, it is therefore advisable to focus on valuable indicators that are communicable within the team.
What business value can you bring even with a few tests?
Joel Oliveira
As a personal note, measure the additional time generated by automation on the false positive analysis, rework and maintenance steps.
This will give you a factual view of the automation effort and its possible optimization.
In any system, an action has a counterbalance effect necessary to understand in order to control the mechanisms put in place.
Concretely:
- If you had to choose 3 tests of higher value, what are they?
- Can you run these tests on demand, quickly and reliably?
- Is their value perceived by the team before going adding more?
The value is in the use of tests, speed of execution is often the key
Here we shared a concrete case: tests carried out at night because they are too long during the day and as a result, not used by their teams.
Are they actually really useful even at night? A priori very little in the shared context, being functional tests. The teams will tend to abandon them.
So the value of test automation is not just limited to saving time, but also to actually using it.
Speed of automated tests are part of their value for the team.
Antoine Craske
CI/CD type industrialization practices can help promote their use, but the need for speed of execution remains strong.
Pragmatically, it is therefore necessary to:
- Choose a reasonable scope of tests to be automated
- Define a solution to parallelize the most recurring tests
- Measure that the execution time remains below the fixed threshold
The other automated tests, less recurring, can for example be decoupled of the short validation cycle, while providing alerts.
Testing Needs Design Like Any Other Engineering Activity
Maintaining automated tests is often a challenge for teams.
We often relate a long set-up, in a closed chamber, of a suite of automatic tests, which following a change of the product falls completely into error.
Unfortunately, if they are bypassed at this time, most likely the test suite will be forgotten and the investment lost.
Dedicating time to test design is a medium to long-term value investment.
Antoine Craske
In the hypothesis that a reasonable scope of tests is automated, the lack of modularization and decoupling is a recurring cause.
These issues are common in other engineering practices, hence the criticality of architecture and design activities.
Often skipped over in an automation project, the design proves to be fatal at a time when testing could have shown its value.
Applying modular test tests allows for example to identify common components.
Questions you can address:
- Is there a clearly defined test plan, prioritized, and business-aligned?
- Have you defined clear steps for the design, validation, and refactoring of the tests?
- Has the modularity and decoupling of the tests been explained?
Have an incremental approach on stable foundations
We sometimes associate a test automation project with a V-cycle that can easily be rationalized and integrated once completed.
The reality is very different, the tests being linked to the validation of a product normally evolving, they must accompany this rhythm.
Maintenance issues are today mainly driven by a good test design, tomorrow improved by artificial intelligence as we hope so.
In view of these different elements and our sharing, an incremental and iterative approach seems the most suitable .
Laying the foundations early with end-to-end integration allows the rapid validation of structuring assumptions about the process.
By using the process on a daily basis, one can iterate over the process to maximize its value by quickly adapting the process.
Personally, it is advisable to keep from the start a vision of the objectives to be achieved of the design and the organization, as Stephen R. Covey reminds us in this quote.
Begin with the end in mind.
Dr. Stephen R. Covey
The iterations should get a team to the right destination, not where the wind takes us.
What you can apply in your context:
- What is your MVP applicable to the entire development cycle?
- What are the minimum foundations needed?
- Can you simplify its initial deployment?
The importance of training teams in quality
How confident would you be for an operation by an untrained anesthesiologist?
I guess not.
Yet this is the reality of most organizations that entrust the quality of their operations to untrained profiles, or leave the anesthesia to the surgeon in our metaphor.
How confident would you be for an operation by an untrained anesthetist? This is what you let happen to your digital experience.
Antoine Craske
Joel told us about the importance of quality skills, and the difficulty in obtaining relevant answers in interviews.
Unfortunately, there is no lack of examples of profiles not having perspective on the context and testing techniques.
How many times have you heard “We continue in the same logic of what existed”.
It is important to understand the past, to learn from the experience, it remains nevertheless to be sure of the relevance of the practices in the current context.
It is here that experienced profiles and with the right reflexes will have the judgment and the critical spirit adequate to adapt if necessary.
The point is to invest in these skills:
- Recognize the value of quality in the organization
- Assume QA and QE as a practice in its own
- Dedicate an investment in training and continuous improvement.
Quality is a transversal responsibility that the team must compose
The quality as a shared responsibility of the whole team with a global commitment is one of the subjects we also discussed.
Nevertheless, the point remains to be balanced between being the sole responsibility of the QA, and being that of everyone and nobody at the same time.
By sharing experiences, a role of facilitator, coach and an enabler in the team around common quality objectives seems necessary to us.
As for automation tasks, they do not necessarily have to be done by testers or developers, adaptation to the context remains key.
The presence of skills is a differentiator in a project of this type, whether internal or external, sustainable or one-off through expertise, for example.
This last point joins the first of business awareness: quality can and must be an investment that brings value to the organization.
Concretely, these steps must be evaluated:
- What testing and quality skills do you ideally need for your project?
- Can you train your teams, accelerate with external skills?
- How can you defend a medium-term budget for this need?
Cross-functional practices for the success of an automation project
Through sharing the experience of the various members of the event, many subjects relate to cross-functional practices.
The clarification of the business need with a commitment from the whole team on quality is a strong prerequisite.
An incremental approach based on a solid design and implemented over short cycles in the life cycle will make the difference.
The following iterations should then make it possible to adapt the automation so that it brings the identified benefits and that it remains of value.
For example, focusing on having few tests used and fast will be more useful than setting up 100 extra tests run at night that the team ignores.
I hope that this feedback will have been useful and concrete to you, I hope to be able to exchange with you during an event or continue the discussion in the comments.