This article was initially supposed to focus only on the Airbnb repo. Their story, however, contained traits of Quality Engineering.
So I decided to have a broader prism on the analysis of the evolution of their repository. The Airbnb quality engineering story is interesting, being business-driven, while the dimension reached by the company makes it useful for learning.
The choice of a mono or multirepo is a structuring subject. For a start-up, the choice of monorepo seems to be the default. On the other hand, how should the architecture of our repository, development, and deployment support the company’s growth?
So let’s take a look at the evolution of Airbnb from the perspective of Quality Engineering.
It all started with a monolith in monorepo, Monorail
Airbnb started its adventure with the first lines of code. The construction of their product focused on a monolithic architecture supported by a monorepo. Few legacy was existing with a newly created company.
The choice of language was Ruby on Rails, also giving a name to their repo, Monorail. Their front-end started in BackBone to switch to React and Redux from 2015. Java services completed the back-end codebase relying on Resque for asynchronous mechanisms.
We can note a centralized model via a monolith in monorepo, and a faster change of technologies on the front-end. Their components had to evolve quickly to support the business iteration cycles. The pipeline supporting the development and release process was therefore structuring.
A pipeline integrating quality by design
Relatively standard stages compose the software delivery chain. However, their performance depends on the choice and quality of the implementations carried out.
Airbnb has secured its development process by applying demanding practices from the start. A mechanism of pre-merge, similar to Google’s pre-submit, ensures a review by a peer in charge of the project. Once the change is validated, the build can start, including executing the project’s tests.
The repository integrates the changes that successfully passed these first stages of validation. With a monolithic architecture inside a monorepo, the complete Continuous Integration (CI) system is triggered.
Deployment in different environments can follow, accompanied by manual and automatic testing. This is the opportunity to run uniquely relevant tests in an integrated environment such as performance or end-to-end.
Note that the “All of CI is ran again” involves a complete build, with no split-repo or project selection model to deploy. Over time, the code base grows and becomes more complex, increasing the build time. We will see later that this is a subject that Airbnb had to address.
An engineering culture integrating operations
Risk awareness is fundamental for engineering teams, in particular developers who can quickly distance themselves from it. Creating a shared engineering culture is essential for the performance of the organization.
Airbnb early on fostered a transversal culture of delivery and deployment. The infrastructure teams provide developers with the tools to deploy to production. The deployment responsibility to follow-up, coordinates, and verifications is the developer’s responsibility.
Airbnb has fostered initiative and collaboration among its teams. Application hotline and out of support hours are done on a volunteer model. The coordination of deployments is carried out between engineers without relying on a siloed team of release management. By default, deliveries are during working hours, even if the team can agree on exceptions.
These practices are at the service of a quality customer experience, requiring robust operations. The mechanisms of progressive activation, feature flags, and automatic rollback are crucial to ensuring responsiveness. Even if simple in their concept, these practices require a proper end-to-end integration, more complex to achieve.
The split of the monolith becomes necessary for the company
These various iterations and improvements have supported the growth of Airbnb. This growth has expanded its codebase, creating a problem to be addressed: the slowing down of its iteration cycles.
The growing code base slows down the delivery pipeline in a monolithic architecture, in monorepo, with an “All of CI is run again”. Ultimately, Airbnb’s ability to improve its user experience was impacted, and so its survival.
There was, therefore, a good reason to review their development system, clarifying the “Why”. A Why that too often forget by engineering teams losing sight of the business objectives.
In 2019 Airbnb reviewed its overall software system, including the architecture, code, and deployment processes. Their priority was to support Airbnb’s growth through a better user experience in web and mobile while expanding the range of services.
Airbnb has therefore addressed each issue in its context in the service of the company and its customers. Their transformation kept the objectives in mind business with the phrase “we cannot impact company growth”.
The front-end and back-end contexts were isolated respectively in Hyperloop and Treehouse. You will notice that the names chosen explain the speed and robustness intentions targeted by each of them. It was the first level of separation, allowing for evolving later towards a service-based division with the SOA approach.
The front must respond to the challenges of acceleration
The front-end finds itself in harmony with its acceleration challenges, frameworks, and teams focused on improving the user experience. Airbnb has capitalized on its culture of sharing and engineering practices for this transformation.
The goal of the front-end was to support “faster, smaller and better loops”.
The keys to answers are in balance. Each context is isolated using a traditional MVC architecture. Standards APIs in JSON ensure the communication within the front. Finally, the organization and the code are aligned in Business Unit, the last cornerstone to allow independent deployments.
The back must remain reliable and expand in services
The back office is evolving to support the company backbone, which must develop more offers and services. The word “Treehouse” clearly reflects the forest of services Airbnb has to build. Java is chosen with tools facilitating the application development replicability.
The communication between the different services was structuring by the distribution of its system. The synchronous and asynchronous exchange models are set up to meet the different use cases. Thrift is retained to provide validation of the diagrams based on a standard and open-source protocol. This brick allows them to guarantee consistency of data and abstract exchanges of technological choices.
The asynchronous model promotes the decoupling and scalability of individual components via an event-oriented approach. The implementation of the event platform via Kafka integrates an ecosystem of standards wrappers supporting Thrift.
The performance of Airbnb’s development pipeline remained the main focus. Hence the importance of the Developer Experience, or DevEx.
A platform approach supporting the Developer Experience
Accelerating software delivery requires an optimization of the development process. Cycles of iterations must be faster, particularly those of code production, hence the importance of the development experience.
Using a platform approach makes it possible to industrialize the services valued by the players in the system. In our case, the users are the developers. They need to easily navigate between project creation, code, construction, testing, and deployment.
Airbnb has invested in several key stages of its process:
- A service generator providing a standard project with infrastructure libraries, logging, and testing in seconds
- Externalizing and automating the configuration of projects, leaving developers to focus on the code and less on the integration
- Standard deployment pipelines updated for their different technological stacks
The implementation of such a tool requires engineering maturity and processes in place. Teams must be able to explain complex sequences to automate and industrialize them.
A balanced approach to Quality Engineering
The evolution of the repository is fascinating given the other changes implemented at Airbnb. The choice of the repo is one of the elements to consider to build a powerful software system in a more widely organizational system.
Airbnb’s iterative and business-oriented approach provides added value while combining risk management. Based on the example of architecture, the monolith has gradually evolved in the face of real business scalability issues. We also find the business focus on the architectural principle “We cannot impact the business growth”.
Airbnb made an essential investment very early on: their engineering culture. A culture that they have firstly focused on across operations and the customer while empowering the teams. These foundations enabled them to develop a capacity to obtain accelerated development cycles.
The results achieved in automation, development experience, and pipelines are no accident. They are the result of continuous investments in their engineering system, from organization to operations.
Few shortcuts are possible to develop real capabilities of Quality Engineering.
From Monorail to Monorepo: Airbnb’s journey into microservices – GitHub Universe 2018 https://www.youtube.com/watch?v=47VOcGGm6aU