Atlassian, Gitlab, Unity deployed Quality Engineering.
These organizations are able to grow and continuously deliver value in a changing environment. While the best tech companies have different products, they commonly deploy the elements of Quality Engineering.
We cover in previous articles how Atlassian, Manomano and OpenClassrooms do Quality Engineering. This article shares the Quality Engineering framework containing the best practices of top tech companies.
In this page, each practice has a primary pointer without backlinks or ads. You can complement the practice by consulting openly accessible content. The objective is to complement them later on.
Methodology
This guide is a work-in-progress to be adapted based on the feedback and likely to evolve within the Quality Engineering Framework, MAMOS: Methods, Architecture, Management, Organization, Skills.
Each practice has been ranked among Quality, Speed and Complexity by maximum score representing the implementation priority. This article is only an excerpt, you can access full prioritization of these practices available here.
Follow the QE Unit for more Quality Engineering.
The Methods of Quality Engineering Organizations
Quality Engineering materializes in a sum of small changes that over-time change the way the software is built and operated. The organizational ecosystem, its actors and culture need time to change their habits to become visible. Changing the processes is the most effective way to drive the transition to a Quality Engineering organization by changing the daily activities of the actors.
The methods of Quality Engineering have to enable the change, spread quality, and continuously improve. Hence, change management methodologies are essential in initiating, driving, and sustaining the change.
The spread of quality is achieved by implementing specific Quality Engineering processes in the teams. A model is provided where continuous adaptation and simplification enable to keep increasing the level of quality.
Finally, the continuous measurement of the outputs with the outcomes ensures that the activities are effectively creating value. These methodologies are done collaboratively to keep the engagement of the actors alive.
Methods Framework
The recommended Quality Engineering implementation curve per impact for Method.
Methods Practices
Change Management
1. Stakeholders Analysis
Stakeholders management is a large theme in itself. For Quality Engineering, the stakeholders’ analysis is the most important technique to interact with the most valuable influencers. By identifying, mapping out, and prioritizing the stakeholders, the leader enables the guiding coalition to leverage power in the organizations to initiate and sustain the changes across the teams.
2. Sense of urgency
All systems change faster under pressure. Take the example of COVID and the digitalization and remote work evolution. A sense of urgency is a must-have to initiate the change to Quality Engineering. It can be built using the practices “Warning signs, benchmark” of the Management domain. The guiding coalition should be ambassadors of the sense of urgency and deliver early wins.
3. Vision, mission, values
Quality Engineering requires setting a new star to reach for the organization. The vision and the mission enable a common purpose of quality across organizational boundaries. The values enable setting the rules of the game and the principles in which the activities should be performed. See the “Common mission” and “Quality principles” of the Management domain for more details.
4. Communication Plan
Good ideas left alone remain ideas. The leader of Quality Engineering must build and execute a communication plan adapted to each transition. Various models should be used like all hands, workshops, and 1to1 interactions. Different contents such as a public portal, videos, and articles can be made available in a variety of media. The important point is that communication is to be continuous, impactful, and available for the actors.
5. Tipping Point
The guiding coalition has to spread the Quality Engineering model up to 15-25% of the organization to benefit from the Tipping Point effect. From that point, the rest of the organization will adopt the practices with much less effort and guidance. The guiding coalition must therefore focus in the first transition on early wins to then deploy the methodologies on the teams with more potential of a successful adoption and value creation.
6. Transition Management
There is no overnight nor big-bang change to Quality Engineering. The leaders of Quality Engineering must be able to drive successful incremental transitions to reach visible and valuable milestones. Without this approach, there is a high risk of staying in the existing situation with a tunnel effect spreading doubts, lack of trust, and demotivation in the organization.
7. Organizational Culture
Quality Engineering requires evolving the company culture towards Quality. This requires acting on the shared codes, beliefs, and values of the different actors. The stakeholders’ analysis can be used to leverage the influencers of the organization to share key messages of change and gradually embed them in the organization. The most impactful is sharing concrete examples on a continuous basis. For example, by having regular interactions on quality, talks done by the cross-functional teams, or informal events.
8. Skills Matrix
The skills for quality needs to be affected to different persons and compose in the first place. They are then evolving on a continuous basis with learning, training, and hiring. A skills matrix is essential to map out the different skills, identify and cover the gaps by priority. This task should be initiated by the leaders and then delegated to the operational managers supported by HR partners.
9. Retrospective
The teams have to continuously improve quality in their respective context. Retrospectives are an effective mechanism for teams’ learnings and improvement in their challenges. They should be implemented as a systematic practice part of the team rituals. Valuable retrospectives are best held when having the remaining ingredients of Quality Engineering skills. Retrospectives are also part of the measurement category hereafter.
Quality Engineering
(*) Most valuable processes from top actors are marked with this asterisk symbol. They are usually kept even with a high level of Quality Engineering maturity.
10. QA kick-off
The goal is to peer team members to brainstorm about the risks attributes and associated testing requirements. This activity is in preparation of the “Testing notes”. This methodology comes from Atlassian part of their Quality process. As the peer is usually done by quality advisors, this is a transition methodology that becomes optional with more maturity to avoid any bottleneck.
11. Testing notes*
Testing notes cover the risks that need to be tested. They are not an explanation of how to test. This method enables the effective prioritization of testing in the Quality Engineering model: testing is the minimal act of verification that should ideally not exist and find nothing. This lean approach focuses the team on value and optimization of their testing effort.
12. Peer Review*
This methodology can be applied to any process and should be prioritized in your context. Peer review is the most effective way to increase quality in short feedback loops and a scalable way. The involved peers are able to interact asynchronously, learn, and rapidly adapt their deliverables.
13. Developer on Test
This process affects a different software engineer to the implementation of tests. This is a good way to increase the sensibility to tests and secure the testing notes’ implementation. However, keeping this practice over time is not recommended as the main software engineer should implement the tests right in the first place.
14. QA Demo*
Agile pushes for a working product over documentation. The logic of a QA demo is the same in Quality Engineering: the quality attributes and the product are systematically shared within the team to align the requirements. This practice also helps the team to reach viable results by removing non-necessary activities.
15. Blitz Test
Testing together with the product in a timebox session to find bugs is a Blitz Test. This practice also comes from Atlassian. It is performed after the QA demo and before the early release of the product. This is an additional safeguard for finding bugs that a previous process would have missed. It is also useful as a team-building exercise for the cross-functional team. This methodology should become optional with an increasing quality maturity.
16. Dogfooding
Decoupling the build, deployment, and release is one of twelve factors apps best practice. Dogfooding sits on the release phase as an early release for the employees. It enables internal users to trial the product before releasing it to the end customers and detect any issues harder to detect in the standard process and infrastructures. With more maturity in terms of testing, infrastructure, the teams can limit the use of this methodology. Exploratory and crowd-testing can be a good compliment in some cases.
17. Community of Practice (CoP)
The teams have to continuously improve their quality level and contribution in their context. An effective way to diffuse practical knowledge within the organization is through communities of practices. By having transversal regular interactions, individuals are able to learn from peers outside of boundaries. It supports the rapid spread of knowledge and best practices in an organization as well as creating valuable interactions between peers that otherwise do not interact much.
18. Quality Culture Guide
This guide available in draft is from Alan Page and Brent Jensen, the founder of the modern testing principles. It contains a reference to various practices a team can leverage to increase its Quality level depending on maturity stages.
Link outputs to outcomes
19. Quality Engineering NPS
The Net Promoter Score is one the most used customer satisfaction measurement. It consists in computing a score based on the ratio of promoters, neutrals, and detractors of the company. Each category depends on the option chosen in recommending the serving experience on a scale of 0 to 10. It is used by the major actors specifically on the “likelihood to recommend the Quality Engineering team to a friend or a peer”. The method enables to be in contact with the reality of the team’s perception, increase engagement, and solve the most important problems.
20. Quality Monitoring
A systematic and collaborative monitoring of quality is implemented by high-performing teams. By being regular, it forces to factually assess the performance and keep the team conscious of the gap to cover, fostering continuous improvement. The collaborative aspect enables to reinforce interactions with the actors and increase the engagement by letting the actors own their quality.
This monitor is coming from Atlassian as part of their Quality practices. It is one effective way to implement Quality Monitoring.
21. Business driven
The following metrics enable to focus of the team on the users’ interactions and business performance. While isolated actions can be hardly correlated to one indicator, the team actions have to improve these indicators over time. They can be identified through an Objective Key Results (OKR) exercise. Here’s a sample of the possibilities.
- Users registration
- Average Order Value (AOV)
- Reorder ratio
- Daily Engagement
- Referrals
22. Accelerate Capability
The Accelerate study demonstrated the link between software delivery capability and business performance. Four indicators tagged below with (Accelerate) are part of the study.
- Lead-time for changes (Accelerate)
- Availability (Accelerate)
- MTTA/MTTR (Accelerate)
- Change Failed Rate (Accelerate)
23. Developer Experience
The day-to-day flow of work of software engineers directly impacted the performance and well-being of the team. Measuring the Developer Experience (DevEx) enables us to keep in touch with the reality of building and shipping software in the organization. Additionally, the platform adoption is a key element to track to reach Quality at Speed. The following metrics can be used.
- DevEx NPS: the organization can have direct feedback of the engineering team experience by applying the NPS survey model. It is useful when applied regularly and to drive continuous improvement.
- Cycle-time: measures the elapsed time between when a work starts on an item (story, task, bug etc.) until it’s ready for delivery. Cycle time tells how long (in calendar time) it takes to complete a task and helps in streamlining software delivery by removing waste and waiting time.
- Adoption: the company platform supports a replicable and streamline software delivery process including non-functional requirements in a standard way. Adoption can be measured through ratios of adoption on projects, daily activity, non-conforming projects.
24. Value-stream
A value stream is the set of actions that take place to add value for customers from the initial request through realization of value by the customers. It enables mapping out the key processes focusing on the value to then drive continuous improvement to accelerate and increase the value delivered.
Framework
These practices will set you up for a journey in transforming to a Quality Engineering organization. The practices will support you in maintaining continuous improvement and adaptation with capabilities, systematic processes and organizational learning.
This content was limited to the pillar of Methods, letting the remaining practices of Architecture, Management, Organization & Skills in a separate document. The objective is to enrich it within the community and improve its content.
You can access the full Quality Engineering Framework containing the rank of the practices across Quality, Speed and Effort. It also contains an option to customize the priority of certain practices and already leave you with an action plan.
Follow the QE Unit for more Quality Engineering.
The Quality Engineering Definition, Manifesto and Framework are available through a Creative Common Attribution-NonCommercial-ShareAlike 4.0 International (CC BY-NC-SA 4.0).