In an ultra-competitive, complex and uncertain ecosystem, organizations need to be able to adapt continuously to survive. This is where Accelerate‘s “4 key metrics” come into their own for assessing solely software performance.
Nevertheless, the speed with which companies adapt also depends on their ability to (i) respond positively to unforeseen stimuli, and (ii) make efficient use of increasingly scarce resources, both internal and external (customers, employees, budget).
Measuring software performance therefore requires a more global perspective, taking into account the systemic dimension of software production. This approach involves measuring value creation, the working environment and system efficiency.
Systemic software performance
The systems approach to software production is based on system-thinking, organizational dynamics and sociology, with software production players at the heart of a dynamic, interdependent ecosystem.
This approach considers the various components of a software production system by describing the triptych of “Process, People, Technology” through the 5 MAMOS domains: Methods, Architecture, Management, Organization, Skills.
These 5 domains form the basis for the definition of 50 software production capabilities that enable the development of a system that is increasingly resistant to stimuli and efficient in its use of resources, with a consequent ability to adapt continuously.
The better the system’s performance, the better the company’s ability to reinvent itself and stay ahead of its competitors, who are too busy resolving the impact of the latest stimuli under a mountain of inefficiency.
Evaluation of the software production system
A MAMOS software production system is first assessed on 4 levels of maturity, ranging from “fragile” to “antifragile“, with improved response to external stimuli, from breakage, resistance and adaptation to innovation.
There are 5 main categories of evaluation metrics:
- Company with market growth and customer NPS
- Organization focused on revenue per employee and internal NPS
- Change delivery and incident response workflows
- Reliability with variance between planned and actual and failure rate
- Efficiency in resource allocation and automation ratio.
This level of maturity provides the company with an overall view of the performance of its software production system, enabling it to focus improvement actions on the areas most in need of attention.
An advantage compared to the SPACE model is to define a common structure of system evaluation instead of having to define each KPIs within the 5 areas more focused on the developer productivity, while still leaving room for downstream metrics.
Company
Successful organizations in this area are able to leverage digital and technological resources to create value for the company and its customers, by being proactive and responsive in real time to customer expectations.
On the one hand, their software production system enables them to develop new offerings on the market, capturing value faster than competitors, while benefiting from network effects that have an exponential impact on growth.
On the other hand, the software enables them to maximize the satisfaction of the customer experience throughout his interactions with the company, from awareness to purchase, not forgetting the resolution of issues that are often penalizing when neglected.
One example is Uber, which has succeeded in developing a global platform combining market-leading growth (still over 10% in 2023), a customer NPS of 26, and is developing offers like UberEats (NPS still -11).
Organization
While growth in the market and customers are necessary for a company to function, they do not represent the context in which employees need to collaborate and develop, nor the real gains that software can bring.
Employee satisfaction, as measured by the internal NPS, is the most representative indicator of the working environment. In a market where 30% of positions remain vacant, with a turnover rate of over 25%, investment in human capital is a priority.
Employee performance must also be measured by their ability to create exponential value through software. Measuring revenue generated per employee complements customer NPS with a digital productivity perspective.
Zoom is positioned at level 3 “Mastering” with 3-4% growth, an internal NPS ranging from 30-55 (with 66% positive recommendations) and optimal revenue per employee at $300,000 per year, like other companies such as Samsara or Expensify.
Delivery flows
An organization’s ability to rapidly deliver software-supported business changes and resolve incidents are two strong indicators of Accelerate research, and demonstrate good collaboration across team silos.
The first measure of business change lead-time is the smooth interaction between players with different profiles, necessary for the production of software increments that are complex to manage due to their abstract and unpredictable nature.
Above underlying metrics such as deployment frequency or detection time, systemic performance of flow responsiveness is translated at the highest level by mean-time-to-repair.
For example, improving a retailer’s delivery and response flow for on-demand deployments and less than an hour’s resolution time required structural and end-to-end improvements to its software production system.
Reliability
Reliability indicators enable us to take a step back from the various software iterations, which may be individually rapid but generate high technical debt or rework rates, highly penalizing for companies undergoing reinvention.
The variance rate between planned and actual is a measure of the ability to plan well over a recommended time horizon of 3 months, focusing on major cross-functional and local themes and avoiding the creation of couplings and waiting times.
The Change Fail Rate (CFR) is a measure of our ability to deliver well, thanks to control over the upstream flows of alignment and design, and those of delivery, enabling us to detect failures as early as possible, thus drastically limiting rework costs.
Organizations with antifragile systems, for example, have a variance rate of less than 5%, enabling them to focus their efforts with minimum waste and maximum impact.
Efficiency
For most software publishers, the value of human time is 20 to 100 times greater than the value of machine time. And one effort avoided is as much time and resources saved for a higher value-added task.
Efficiency should therefore be measured in terms of resource allocation rate, ideally at 50% maximum, in order to deliver identified subjects as quickly as possible, while retaining the capacity to deal directly with improvements or unforeseen work which will often arise.
The automation rate completes the efficiency measurement by ensuring that the right tasks are allocated to the machines throughout the software production cycle, and not just for testing or deployment.
The concrete impact of one person allocated 100% per week translates directly into the inability to include inexpensive improvements that can be made directly, non-response or backlog of work for new requests, and medium-term burn-out.
The systems approach to software production
The successful companies of today and tomorrow will be those who have invested in the development of an antifragile software production system, enabling them to rapidly adapt and grow in their market.
The transition of a software production system from fragile to antifragile means moving through the different levels of maturity, so as to be able to respond more and more effectively to external stimuli through innovation at the highest level.
To achieve this, the AAA methodology (Assess Architect, Accelerate) was consolidated through various business transformations, enabling MAMOS’ 50 capabilities to be developed by optimizing investments using systemic analysis and leverage effects.
Get ready today to develop your MAMOS software production system with an assessment of your level of maturity, followed by an evaluation of your production capabilities to meet the concrete issues and metrics of your context.
Getting started with MAMOS.
References
Google Cloud. 2023. State of Devops Report, Google.
Tim Stobierski. 2022. What are network effects? Harvard Business School Online.
Nicole Forsgren, Margaret-Anne Store, The Space of Developer Productivity, ACM Queue.
Uber Technologies Revenue 2017-2023, Macrotrends.
UberEats NPS, Comparably.
Zoom Video Communications Revenue 2019-2023, Macrotrends.
Zoom Video Communications Happiness Score, Comparably.
Jason Lemkin, Dear SaaStr: How Many Employees Does a SaaS Company Have at 100M ARR? SaaSTr.
Taleb, Nassim Nicholas. 2013. Antifragile. Harlow, England: Penguin Books.
Gergely Orosz. 2022. Measuring Engineering Efficiency at LinkedIn. The Pragmatic Engineer.
CNCF Case Studies. 2022. How La Redoute leverages Kubernetes to accelerate deployment of cloud native microservices, CNCF.