What is Quality Engineering? That question raises others like the why, the what, the how. We covered these during a deep dive into Quality Engineering with João Proença.
We decided to talk to João to enrich the existing set of content aiming at clarifying Quality Engineering implementation within the industry. In this interview, we share his perspective coming from more than 10 years of experience in quality.
We focused on sharing real-world cases to materialize how Quality Engineering can live in organizations. The mentioned content is available in the references at the end of the article. We are happy to continue the discussion in the comments.
Join the QE Unit to access exclusive and regular content from the community.
About João Proença
João Proença comes from Lisbon, Portugal, and is a Principal Quality Engineer in R&D for OutSystems, a company that provides one of the leading low-code development platforms in the world.
He has assumed various roles throughout his career in the past 14 years, including quality assurance, development, customer support, and marketing. Finding innovative solutions for complex problems drives him the most, so he is always eager to talk about how professionals are overcoming testing challenges around the world.
Outside of IT, João is passionate about songwriting, movies, and football. You’ll see him tweet about all of these topics using the @jrosaproenca handle.
Antoine: Can you start by introducing yourself?
Over the years, I have done a variety of things, music being one of them at some point 🙂 I started my career as a researcher in 3D rendering when I finished university. It was interesting, I was using alternative types of rendering to find new ways of observing items. After a while, I figured out I wanted to have more impact in the real world from what I was doing in research. I decided to do something else.
I moved to Outsystems, becoming a Quality Engineer at the time. In those days, it was like regular quality assurance work. I was within an agile team handling both manual and automation testing activities. I stayed in that role for about 5 years with a broader experience than QA: I performed software development from features development to bug fixing. After this period, I did not want to come back purely to quality assurance.
I invested more time in development, but I also worked at the customer support team. It was an enriching experience, having to pick up phones, answer to customers, happy or even sometimes angry ones. I had to decide what to escalate within the company to solve particular problems. Given the company’s size, that was an elite position due to the complexity and variety of support to handle first-line.
Antoine: Where did you move after this enriching support experience?
I was then invited to join the newly created cloud operations team when Outsystems launched its related Cloud offer. I was appointed to lead the team on the Ops side. At the same time, I was also working on my side music album project, handling a lot of marketing activities. I started to gain more interest in this area and got an opportunity in a product marketing team at another company. I worked there for a while, being the engineer producing content for this software company. After some time, I realized that my vocation was to be a core engineer.
I was then back at Outsystems with a proposition to join the Quality domain. I hesitated as I did not want to join that area again. They told me “We now have a completely different view of Quality”. They did not have coined a name yet but really had a different perspective. I accepted and have been a Quality Engineer since 2017, with many different experiences since then.
Antoine: Interesting path with diverse experiences, from engineering, support, operations, music, and marketing. To share with the audience, Can you tell more about the company?
Outsystems provides a low-code software platform for the rapid development of different products, from web to mobile experiences. Over the years, it has found its market and accelerated its growth worldwide from Portugal. It is one of the major unicorns in the technology area, with hypergrowth and scaling challenges.
I remember the day when the company had less than 100 employees. Right now, we are ranging between 1500 to 2000. I have seen different stages and levels of complexity depending on the size of the team. We are currently expanding; it is hard to know how many we are precisely at a given point.
Antoine: Quality engineering is not that well defined. I wanted to dig into that theme with your professional experience and what you share at conferences. Before starting, can you share what your top priorities are as a Principal Quality Engineer?
One big challenge directly cascades from the accelerated growth of the company. We have to enable the Quality practice to scale with the team expansion. It means many new teams to align with our shared quality ownership culture, practices, and standards. We want to avoid quality silos that lack sharing within the whole organization. One key priority is to have teams sharing a common understanding of how they should address quality.
I work with a specific team while transitioning to a more transversal role, overseeing various teams. These teams may or may not have quality engineers embedded. I’m fostering teams to communicate together, make quality engineers aware of different practices and opportunities, or even take subjects broader than the ones of a single team. It is a more strategic role requiring me to be more aware of the industry trends to make sure quality is built-in to these various teams.
Antoine: The dimension you shared demonstrates a good level of quality maturity, also tied to the business objective of scaling and maintaining the product quality. To summarize, your role is to make Quality an enabler at a local level within each team, while at the same time ensuring that it contributes to a global value company-wide?
Yes exactly. A concrete aspect is to direct quality engineers to work more closely with specific teams. A lot can be done by the human aspects rather than with the technical ones. I do a lot of coaching of other engineers in the way they approach quality, interact with other teams, and try to resolve specific problems.
Quality Engineers are true facilitators and enablers. They are not the only ones doing quality activities; that’s not the goal for us. Knowing how to enable teams on risk management or testing strategy is, for example, more valuable. Sometimes, how you facilitate a specific workshop is a game-changer in the type of output generated.
Antoine: Let’s now forget for a time the specific context of Outsystems. How would you define Quality Engineering in one sentence? What is Quality Engineering from your perspective?
I would try to give my best definition where my background still influences how I will define Quality Engineering. I come from a “school of testing” rooted in the whole team approach to quality, the modern testing principles that Brent Jensen and Alan Page started. When I became a Quality Engineer back in 2017 at Outsystems, I did it while working with the new head of Quality, João Neto, who came from Microsoft, where these modern principles originated. He worked to bring that school of thought to our organization.
I could build upon one of these materials, but I can share a story first. In 2018, I went to Agile Testing Days in Germany. People were talking about similar topics, and I discovered a keynote from Anne-Marie Charrett from Australia. She shared that Quality Engineering, coming from other engineering fields, is now entering the software world. Quality Engineering is for me coming right now, linked to other trends like DevOps and Agility.
“Quality Engineering encompasses a much broader holistic perspective.”João Proença
Quality Assurance tends to focus more on validation and verification, pre- and post-implementation; sometimes very localized in the software lifecycle. Quality Engineering brings a more holistic view of Quality to be built-it in the first place by different teams at different stages. A useful visual is the one of Dan Ashby: there is the infinity scheme of DevOps where testing and quality happens at every stage. This vision of quality is also building upon the existing quality and testing practices.
Antoine: In your view, what are the key Quality Engineering practices? What are the most differentiating and valuable from the known ones?
I don’t see a one-size-fits-all recipe. I can identify several actions I did as a Quality Engineer not that common and falling under my responsibilities. First, consider quality from the design phase. Risks are for me very important. Being risk-driven is to seriously consider risks as the source of a lot of things you do, and particularly quality risks. Note that a lot of times teams talk about risks referring to “project risks” which are different in my view from “quality risks”. Project risks endanger the success of a particular project.
In quality risks, we are thinking about how the product could go wrong, not satisfy the expectations of our users. I saw a lot of discussions about risks without a clear framework of thinking about them thoroughly or systematically. Therefore, I learned about risk assessment through risk-storming, a fundamental practice to discuss quality from the start. I even discussed the value of doing it both early and continuously with their creators, Beren Van Daele and Marcel Gehlen.
“Quality Engineering is about finding the sweet spot between delivering value while keeping the ability to react.”João Proença
One thing I found about risk assessment is that it’s key to do it at the right time in a project or initiative. At the very beginning, you will be unclear about what you are trying to build and deliver to the users, so the risks you come up with are not that complete. Later when you’re already developing, you can be too far in the implementation to accommodate the risk mitigations you need in your team’s plans. You have a much better understanding of the risks but less margin to act on them. The sweet spot is right in the middle when you start to materialize the product value while not having too much commitment. Achieving this state requires an early collaborative model. And mitigating risk involves for sure a lot of testing but it can also involve other practices like testing in production techniques like shadow deployments, or Observability. These practices go beyond testing and enable us to measure risks and adapt early.
Another practice I found very impactful is by acting at the Architecture level. Looking at particular problems, testing was useless; we collaborated with architects to solve structural issues. It enables you to re-state the why, the what, and the how, making you more explicit about the most important risks and priorities. I am already talking about many different practices that are truly different from what a traditional testing or QA team would do.
Antoine: Quality Engineering takes quality into account and acts as a whole, where quality assurance teams play a fundamental role. How to manage Quality Engineering? Do you recommend specific measures to use?
Definitely, my work is to enable the teams to think that way towards a transversally built-in quality. In retrospective, what we have been trying over the last year is to make teams understand two things: what quality means to them and what quality means to the product. We have been formalizing some metrics that they can use and tailor to assess their current and target performance in their own context.
We named our model the “Quality Radar”, and it provides a set of processes, habits, and principles that can be assessed. The Radar was highly inspired in the Quality Culture Transition Guide from Alan Page. The model looks at various areas like the quality culture, testing typologies, or even maintenance. For example, we have teams actively working on tech debt and promoting its treatment in a qualified way.
Its implementation relies on drawing the radar. The goal is not to evaluate teams but to enable them to make their own assessment and action plan. One team can figure out “Hey, we suc** at Observability, what can we do?” The radar gives them the first steps they can take by level of maturity. We are continuously improving the model, some areas being experimental.
Antoine: This sharing leads us to the value of Quality Engineering. The product and team perspective help them to work towards continuously bridging the gap of value. Are there specific valuable metrics to consider?
There are two main ideas regarding measurement and metrics. Metrics that are not contextualized to your organization, product, or teams usually do not work; there are no one-size-fits-all metrics to use. That steers the discussion as to which metrics matter the most to teams in their context. One thing that we tried is something we call “Product Quality Goals” for teams to set their directions on the quality they want to see in their assets.
The Product Quality Goals were in part inspired by the concept of Northstars. Product Northstars are the key measure of success for a product team. As an example: for Facebook, at some point it was the number of likes or sharing, if I recall correctly. That truly guides the decision-making. Figuring out the key metrics that matter to you, you can then align an action to cover the gap between your starting and target levels. It is useful especially when the assets of a team change a lot over time; they are more tied to the value to deliver rather than a previously built product that could change completely.
We brought this idea of the Northstars to the quality space and I am convinced that the teams must come up with the quality metrics on their own. You can provide them the way to achieve that, but not the metrics in themselves. I also discovered a whole set of metrics’ perspectives with our quality engineers, communities, and even Lisa Crispin that worked at Outsystems. I noticed that people agree that the most valuable metrics cannot be universal, but if you really really want some “industry-standard metrics” that apply in most places, then your best bet are the DORA ones coming from the Accelerate book.
They are considered the DevOps metrics by now. In my perspective, these metrics are beneficial for the whole industry. I recommend anyone to perform the measurement exercise in their maturity model.
Antoine: Quality Engineering priorities are therefore contextual to the organizational priorities, maturity, and organization. Are there specific organizational models of Quality Engineering? How does that impact what a Quality Engineer actually does?
The activities and even the responsibilities of Quality Engineers depend on the organization’s size, context, and history. Outsystems has more than 20 years and has seen different levels of scale and challenges during its existence. We saw that their definition and how they address quality evolve, reflecting its priority and maturity changes.
Antoine: I feel that it is more a question of mindset and actions you enable, tolerate, and prioritize. A CTO could be the sole Quality Engineer in a start-up?
Exactly! The nature of the company, product, and organization are essential elements of context to consider. I remember talking with a friend of mine, Melissa Eaden, working at Unity where Alan Page works now. From an organizational perspective, they were also pushing the teams to develop their most valuable metrics. It is an excellent example of how teams must be accountable for their quality metrics and action plans.
Metrics can be dangerous but at the same time empowering. Done right, they empower the team to take control of their improvement plans, defining and learning from experiments. This collaborative and shared alignment is truly beneficial for organizations to act on a larger goal.
Antoine: That’s making me think about when companies should start to have Quality Engineering positions. From our sharing we understand that it must be present at every stage of the company’s maturity. The need for dedicated positions appears from a specific size or types of subjects to address? When does it become a requirement and not only an option? (A subsidiary question is what would happen to larger companies not already addressing Quality Engineering)
Quality Engineering makes sense in most of the scenarios. It’s not only about the size of the company. If you want to kick-off a brand new start-up tomorrow, you probably don’t start with a dedicated Quality Assurance practice, so thinking about Quality Engineering may be a good bet. What is important is to implement the ideas and the principles behind these domains.
I sometimes find myself thinking that quality assurance made more sense back in the old days. It was very hard to act once the software was deployed to production for a variety of reasons. There were practical limitations like delivering software in physical CDs that, once shipped, releasing updates was a real problem. The continuous software adaptation was not that simple, reinforcing the need for very preventive quality and risk management.
Nowadays, the dynamic is very different. DevOps is one example of a drastic change in how we build, deliver and operate our software. Quality Engineering makes a lot of sense in the current engineering software and relating to our users.
Antoine: What are the key steps to start exploring the world of Quality Engineering?
I would recommend having a deep look at the whole team approach to quality and the modern testing principles. Some people are doubtful about their practical implementation. I can assure you that done well, they work. In a lot of places in the world, the practices are implemented without directly referring to them.
Many emerging practices in our fields are sometimes commented with “It looks nice in theory, but in practice, I have never seen it working”. I faced a practical and recent example with contract testing. I went into workshops, learned more about it, even assisted in talks at conferences to implement it. I found a lot of detractors failing to implement it. I eventually spoke to a friend of mine, Rob Meaney, who gave me the first examples of places where it was in fact successful.
A lot depends on the capability to translate the concept into practice. It requires a level of understanding, excellence, and commitment actually to make it a priority. It is like TDD. It is a precious practice, but it is tough to “do the TDD right” in an organization. So if people are doubtful that the whole team approach to quality is a theory, I would contradict them.
Quality Engineering build-up a lot of these principles with a more transversal and holistic perspective, giving us more options.
Antoine: Closing up with a personal note, do you have specific inspirational content, like quotes, books, even outside of Quality?
Modern testing is beneficial for me. You can find the AB Testing podcast with hundreds of episodes on their site and join their slack to engage with the community.
I mentioned Anne-Marie Charrett’s keynote on Quality Engineering, “Is your quality on the road to nowhere?”. She also shares regular content in her blog.
In terms of books, I highly recommend reading Accelerate and Leading Quality, setting essential knowledge and perspectives. I will also start to read the Unicorn project that follows the Phoenix project, two other great titles.
Antoine: Thanks a lot João for this Quality Engineering interview. I wish you an impactful journey of Quality Engineering at Outsystems, and more insightful talks at conferences like the ones on cognitive biases. You can follow João at @jrosaproenca.
Brent Jensen & Alan Page, Modern Testing Principles, AB Testing Podcast https://moderntesting.org/
Dan Ashby, Continuous Testing in DevOps. https://danashby.co.uk/2016/10/19/continuous-testing-in-devops/
Brent Jensen & Alan Page, The Quality Culture Transition Guide https://testingpodcast.com/ab-testing-episode-93-the-quality-culture-transition-guide/
Alan Page, The Quality Culture Transition Guide https://docs.google.com/spreadsheets/d/1kan20hYsdbvk7HW4si-X6Ve1fLtCeTI2H_PjiniKsxY/edit#gid=1897633328
Lisa Crispin, The Whole Team Approach in Practice https://lisacrispin.com/2011/04/26/the-whole-team-approach-in-practice/
Janet Grégory, Holistic Testing https://janetgregory.ca/testing-from-a-holistic-point-of-view/
Anne-Marie Charrett, Keynote “Anne-Marie Charrett: Is your Quality on the Road to Nowhere?” https://www.youtube.com/watch?v=46QFPrs5dT0. Agile Testing Days
Anne-Marie Charrett, personal blog at https://www.annemariecharrett.com/
Beren Van Daele and Marcel Gehlen, Riskstorming
Nicole Forsgren; Jez Humble; Gene Kim (2018), Accelerate: The Science of Lean Software and DevOps: Building and Scaling High Performing Technology Organizations https://www.amazon.com/Accelerate-Software-Performing-Technology-Organizations/dp/1942788339
John Ronald Cummings, Owais Peer (2019), Leading Quality: How Great Leaders Deliver High Quality Software and Accelerate Growth https://www.amazon.com/Leading-Quality-Leaders-Software-Accelerate/dp/1916185800
Google, The Accelerate Report https://cloud.google.com/devops/state-of-devops
Every Product Needs a North Star Metric https://amplitude.com/blog/product-north-star-metric