Coding can be a nightmare.
It’s not that producing code is painful—it’s about the overall experience.
Organizations focus on speed to rapidly discover what matters for their users. But this leads to more technology, more problems.
Software engineers then lose hours solving configuration, deployment or other integration problems—actually not delivering value.
Quality Engineering counter-force is required to keep the software value-chain constrained to Quality at Speed.
This article shares how Spotify enables Quality Engineering Productivity while scaling with a painful starting point.
Follow the QE Unit for more exclusive Quality Engineering.
The Why of Quality Engineering Productivity
Spotify engineers have to experiment, learn, and launch features quickly.
Speed is achieved with cross-functional teams with all the necessary skills to ship features with a high autonomy.
This is their definition of Quality at Speed:
“Quickly turn ideas into products and experiment to improve the user experience, growing into new markets, and remaining competitive as a content streaming provider.”.
While focusing on Speed can works at the beginning, each team accumulates a debt with diverging engineering practices and fragmented technology.
Quality Engineering Productivity is the requirement to keep iterating with Quality at Speed, balancing Speed and Standards.
The Platform Developer Experience team is the counterforce responsible to keep engineers able to iterate fast at scale.
Quality Engineering to accelerate Developer Experience
A successful Developer Experience is powered by Engineering Productivity.
Spotify coined their experience as “rumor-driven development” when the only way to get software working was by asking nearby and following rumors.
The on-boarding time for new developers reached 60 days.
That was far from a productive Developer Experience and impacting their scale.
Spotify focused on Quality Engineering Productivity by:
- Measuring the ratio of inputs to outputs;
- Continuously linking the outputs with the outcomes;
- Keep improving for more outputs and more outcomes.
“To understand how we can help teams at Spotify ship experiments fast we need to understand how they work.”—How We Improved Developer Productivity for Our DevOps Teams, Spotify Engineering.
The Platform Developer Experience team provides products optimizing productive cycles with repeatable components per use-cases.
Let’s see what this team provides to eliminate distractions.
Spotify’s Quality Engineering Productivity Elements
Cross-functional teams required 14 days to run a newly created application—too slow for Quality at Speed.
The infrastructure team worked at creating abstractions of rapid and stable software artifacts engineers can leverage depending on their needs.
The following products were built:
- Backstage as the one-stop shop for software engineers;
- Golden Paths providing standard bootstraps with tutorials;
- Tingle CI/CD, the build, deployment and release platform;
- Test-certification program, providing testing assessment.
These elements reduced the time to set up a service from 14 days to less than 5 minutes.
For example, a developer can get the framework for a site like Spotify wrapped with a URL, repository, CI/CD, in one day.
Golden Paths on Backstage as the one-stop shop
Reducing the number of decisions enables scalable and fast processes.
The Golden Path is the ‘opinionated and supported’ path to ‘build something’ (for example, build a backend service, put up a website, create a data pipeline).
Golden Paths are step-by-step tutorials that walk you through these paths available on the Backstage portal.
Software engineers can use Golden Paths to build a standard application following a step-by-step wizard documented with best-practices.
Over the years, Golden Path tutorials grew with client development, data engineering, data science, machine learning, audio processing, web, etc.
There was a time when engineers at Spotify couldn’t imagine life with Golden Paths; now they can’t imagine life without them.
Systematic build, deploy and release management with Tingle
Once a project works on the engineer’s laptop, integration issues appear to deploy the application up to production.
Tingle is a CI/CD system that automatically builds code pushed to GitHub and gives feedback to the developers there and in Backstage.
All of that without needing to understand how to configure a build pipeline.
Tingle automatically builds, tests, packages, and deploys changes to production if all tests pass using the normal GitHub workflow.
Spotify replaced over 200 stand-alone Jenkins machines by Tingle, running more than tens of thousands of builds every week.
But confidence to deploy was still lacking.
Test certification program for more confidence to deliver
Spotify’s engineers rely on gamification to subject their code to the appropriate tests.
Each major engineering discipline has a test-certification program that:
- Provide checks on quality controls compliance;
- Provide reports on build times, code coverage, and reliability;
- Alert teams of unreliable tests (i.e. flaky tests);
- Give badges next to the services depending on certification level.
Test certification program fostered the creation of test suites supporting the acceleration of software delivery cycles.
From 2018, they measured a drastic reduction of blocking bugs and other types of reactive work.
How can this work with so much autonomy?
Quality Engineering Productivity for Autonomous Teams
Golden Paths, Tingle, and the test-certification program helped to standardize the technology stack.
But Spotify keeps the teams free to choose the tools and methods that fit with how they operate.
Hence, their Quality Engineering Platform required modularity for a continuous adoption while it grew in maturity.
Each team can effectively decide to use none to all of the platform services, depending on their added-value.
That fosters an ecosystem where only the components allowing Quality at Speed remain in use.
Quality Engineering Productivity for Quality at Speed
Spotify accelerated towards Quality at Speed building up Quality Engineering Productivity.
Building a project requires less than 5 minutes for existing and new engineers, supporting the scale of their organization.
The certifications programs with systematic measurements and alerts keep improving their engineering productivity.
Spotify learned to keep things simple:
- Keep one Golden Path per component avoiding sub-modules;
- Document Paths as they are—even if painful—then optimize them;
- Foster on continuous usage and feedback of the platform tooling;
Backstage is available in open-source offering more than 100 integrations. It is used by players such as Zalando, Unity, Expedia.
Their current improvements are to automate compliance of projects with their Golden Paths using Golden States and keeping a good documentation level.
Golden Paths are Spotify’s most read and most used technical documentation—a true Quality Engineering achievement.
Ready to build your Quality Engineering Productivity?
Follow the QE Unit for more Quality Engineering.