How Atlassian does Quality Assistance

Atlassian is worth $93.25 billion.

Without debate, they know how to deliver Quality for their customers and grow a SaaS business. From a Quality Engineering perspective, their software approach is full of learnings and actionable practices.

Atlassian was a pioneer in evolving the traditional Quality Assurance model to a Quality Assistance one. They develop over the years their methodology available through various contents.

This article shares how Atlassian does Quality Assistance analysis of its management, organization, skills and methodologies. The initiating force of their model starts with the management.

Follow the QE Unit for more exclusive Quality Engineering content.

Management aligns Quality as a shared organizational mission

We can all think Quality Assurance is not enough. But someone needs to take action to actually change the status quo. The management of Atlassian has a true role in enabling the shift to Quality Assistance in three areas: defining quality, a shared mission and its principles.

People need a common ground before evolving as a group. Quality is no exception and its subjectivity requires particular attention. Quality Assistance pushes for an end-to-end quality inclusion, hence requiring to engage with the different actors in their definition of Quality.

Atlassian defines quality as enabling “Quality at Speed”.

“Quality at Speed” is Atlassian’s approach to QA, and we are constantly evolving what that means and how it translates to actual dev team processes. Our developers can confidently take on testing activities, while our QA Engineers tackle larger, harder, and bolder challenges. Teams can ship better features, faster, and reach ambitious quality improvement goals.”

A true mission is defined here. No implementation details and milestones are set. The goal of quality is to increase the performance of the organization. Different people contribute in different ways with evolving processes. At the end of the day, quality has to make a difference in accelerating value delivery.

The Atlassian teams then iterate within a set of principles.

Quality at Speed is the star to reach evolving in maturity

Atlassian did not expect to be at the top from day one. Conscious of the need of time for continuous improvement, they defined a “hierarchy of quality” inspired from the Maslow’s pyramid. It clarifies the end result to achieve and the necessary maturity steps to get there.

At the bottom layer, they are concerned about standard and reliable execution. It clarifies the need for solid foundations in the first place. By going up the chain, the value delivered increases from an internal to an external perspective of Quality at Speed.

blank — Figure 1: Atlassian hierarchy of Quality.

Then comes the principles aligned on their mission and business contribution. Their Quality Assistance is oriented on the outcomes of the development team. There is no focus on an internal perspective like bug counts or resolution metrics.

The first principles of Quality and Speed cascade from their mission. We then find the principle of independence clearly removing a potential bottleneck from a Quality Assurance team. Lastly, they expect their teams to adapt the quality practices to the context with experimentation. We can note the results and data-driven approach as expecting to “validate the results of those changes”.

The right context is then required.

Organization pivots the actors’ roles on the shared mission

Their mission and principles set a clear frame of execution. But Quality Assistance is not about adding QA roles in each team to effectively “assist” them. It requires changing the interactions of the existing actors leveraging organizational design to get out of the silo inferno.

Quality Assurance roles are usually to “guarantee the quality of the product” with assessments, testing, or bug management. The Quality Assistance model changed QA roles to Quality Engineers with a different role. Their goal is not to handle quality for others; they have to “improve the dev team” in three areas: mentor, monitor and educate.

Consequently, software engineers are addressing quality in the first place with faster feedback loops and autonomy. This avoids increasing the overhead for Quality Assistance. For example, Atlassian had 10 product managers, 90 software engineers, and 7 QA engineers for JIRA.

As a result, Quality Engineers have more valuable time to increase their impact by:

Empowering developers to confidently test by providing training, tools, and environments.
Keeping tabs on the team’s output, and identifying where improvements can be made.
Helping the team to produce high-quality software in the first place, efficiently.
Foreseeing and preventing issues before they become bigger problems

This organizational model enabled scaling the model through culture.

Culture focuses on a built-in Quality rather than Testing

The best mechanism for large human coordination is culture. Shared mission, codes, and beliefs enable a reliable and replicable diffusion of practices. This explains the importance of culture in scaling software products. For Atlassian, it was essential for Quality Assistance.

The organizational mission of Quality at Speed was the first building block. Then, they had to change the belief of Quality Assurance responsible for “quality” and “testing”. Atlassian spread the new codes through a variety of communication support in the organization. Over time, a paradigm shift happens reaching the Tipping Point.

The ways to work also had to evolve. As humans, we tend to stay in our comfort zone, even more within a group. Atlassian had to significantly invest in explicitly stating how to do quality differently. The first change was to remove the mixed association of “testing is quality”. For instance, they clearly stated how to do less testing:

Before moving to the What, the Who is essential to the transformation.

Skills are aligned to collaboratively deliver Quality at Speed

Another responsibility of management is to align the necessary skills for Quality Assistance. The difficulties lie in composing with the existing ones while improving them with training and hiring.

Atlassian defines the following skills to effectively implement Quality Assistance:

Testing expertise
Teaching
Identify problems
Influence with data
Inspiration
Facilitation

Testing expertise & teaching

Quality Engineers have to enable the engineering teams for quality. Both expertise in quality and teaching is required to effectively transfer the knowledge to the other teams. One of the skills can be completed by formal coachs or training when not having these skills.

Identify problems & influence

The broader perspective gained by Quality Engineers serves to address more complex problems of quality. They therefore need problem-management skills to properly scope problems in the first place. Then, they need influence to drive problem-solving with the different teams.

Lead with inspiration & facilitation

Quality requires a strong coordination of local contribution to the global value creation. Quality Engineers have to be the leaders of quality, inspiring the teams and continuously help them on the field. Their contributions are essential in the transition to Quality Assistance, gradually shifting quality to the teams.

Then, the challenge is to sustain the initiated changes.

Software crafting and delivery processes include Quality

The best way to change ways of working is by acting on the processes. By changing the course of systematic actions, we end up affecting the overall coordination of activities. For Atlassian, software crafting and delivery processes were the fundamental processes to evolve.

Atlassian did not set a one-size-fits-all model. A lean target model is completed with a series of variations to adapt to the team’s maturity or context. For example, a transition process contains more steps with simpler interactions and stages.

We can note core processes of Quality Assistance present in both models such as Testing Notes or QA Demo. For example, the QA kick-off disappear when the team is able to perform the quality assessment in autonomy.

As the team becomes more confident in their quality capability, they can adapt and even remove some gateways. The following processes are truly helping in implementing systematic quality actions:

Blitz testing: many people performing a time-constrained test session
QA kickoffs: pairing to brainstorm on testing notes before the coding starts
Dogfooding: internal use of new features before they are pushed to production
Developer on Test: good at introducing developers to testing responsibility

At the end of the day, they have to measure progress.

Outcomes measurement drive continuous improvement

Being busy is not a synonym of results. The continuous measurement of the outputs with the outcomes enables to follow the actual value creation. Atlassian does that through collaborative exercises for Quality Assistance: NPS and Quality Health Monitor (QHM).

The Net Promoter Score (NPS) is the first measurement of Quality Assistance. The teams are asked to rank on a scale of 0 to 10 their likelihood to recommend. Promoters are between 8 and 10, while detractors between 0 and 6, leaving the neutrals on 7. The NPS gives a good sense of the perception in the organization. It is then used to drive action plans to address bad scores and maintain positive ones.

The Quality Health Monitor is a collaborative exercise similar to a silent retrospective. An external facilitator animates a group of 8 persons maximum to rate their quality attributes over time. The workshop favours interactions, problem-solving and continuous improvement of quality as a united team.

Both measurements are essential in keeping the quality ownership and team’s focus on the overall value creation.

Atlassian leverages Quality Assistance for sustained growth

Quality Assistance is a real transformation of the organization. Atlassian completely evolved the way to build and ship software with quality. This has now become a true differentiating capability to keep delivering Quality at speed.

The changes did not happen overnight. Through the MAMOS Quality Engineering framework, we see that it took time to change the Organization, Skills, and Methodologies with the Management.

Equally important as the level they reached, the company is able to continuously maintain scale its quality practices. They demonstrate the importance of continuous adaptation and culture in maintaining an organization at the high standard.

Atlassian inspires various organizations to implement Quality Assistance. Aligned with their principles, the adaptation to the context remains the key. You can find two concrete examples at Manomano and OpenClassrooms in this article.

Are you ready to adapt your organization for Quality Assistance?