How To Select a Tool For Continuous API Testing

APIs are everywhere.

Digital businesses are becoming more dependent on APIs for their customer experience and operations.

The delivery of APIs is therefore a critical area to secure.

But there are so many tools to consider for API Testing that we can get lost: Is Postman sufficient? How about integration test framework? etc.

This article shares the steps to select the tools enabling Continuous API Testing.

Disclaimer: ¹This article does not focus on products nor aims to be fully exhaustive nor to compare them; the focus is on the decision-process. ²I contribute to the open-source product Cerberus Testing listed with the other ones. ³If you think an important solution is missing, comment on this post or contact me here.

Content

Identify your Continuous API Testing requirements

Start by defining what you want to focus on your needs, independently of the possible solutions.

Identify which part of Continuous API Testing is more valuable to you:

Validates API design with use-cases, data dictionary and models;
Fast verification of API stubs and contracts while coding;
Early testing of API in non-production environments;
Integration testing of APIs in distributed systems;
Systematic verification after a deployment in an environment;
Ensuring specific levels of security, performance, reliability;
Measuring the performance in production, in different setups.

You may want all of them, but order your requirements into a list of priorities.

You will start to see patterns to answer these questions:

Are you searching for unit, integration, functional tests?
Is your focus more on functional or non-functional requirements?
Who are the main users of your solutions, the entire team or specific roles?
Are your needs upstream, in the middle, or at the end of the software lifecycle?

Then you can start identifying your criteria.

Clarify the important criteria of your API testing

We can imagine complex decision-matrices when talking about criteria—already bootstrapping a spreadsheet if you are analytical like me.

We usually don’t need that level of complexity.

Clarify these important points to quickly make a short-list:

Which criteria can make you directly discard a solution?
What are the top 3 must-have requirements you are looking for?
Which use-cases and value am I expected to create versus today?

Your answers will already provide a good summary of what you are looking for.

You can then check a systematic list of common API requirements for the sake of verification:

Usability, possibly low-code and scriptless for collaboration;
Supports test design, modules and reuse;
Support definition of various applications and environments;
Ease of API stubs, definition and contract definition;
Built-in library for verification like assertions, security, error codes;
External integrations and public APIs;
Collaboration features preferably through a SaaS/web interface;
Maturity, support, on-boarding and documentation;
Security, scalability and performance requirements.

The type of solutions required starts to emerge.

Decide the typology of API Testing solutions

Different problems have different solutions. Separating the concerns enables you to see more clearly the options you have.

There are 3 types of solutions relevant to distinguish:

Focused API product: they are tailored to support API testing automation with monitoring capabilities. They are usually not provided with other types of web/mobile testing for example, their focus being APIs. More comparable to Postman. They usually include API discovery and documentation.

Testing automation platform: they are usually providing a whole set of test automation capabilities for web, mobile, APIs, desktop, database with a broad set of capabilities. APIs are one set of possible testing, therefore they can be less deep in terms of ease of use, library of tests, etc.

Full monitoring solution: they are focused at monitoring IT components including APIs but less on automation testing on the chain of development. They do not necessarily provide CI/CD capabilities but can for example execute API tests around the globe and provide other types of metrics.

Each of them having strengths and weaknesses that would fit depending on your requirements.

You can decide the types of solutions from your requirements and criteria:

Pick a focused API product if your needs are narrowed to API-only along the software development chain, accepting the trade-offs such as not being able to combine actions with web, mobile.
Use a test automation platform if you want to capitalize on an end-to-end test automation practice for web, mobile, API etc, usually with more integrations and features, accepting you may lack advanced API features.
Go for an API monitoring solution if your concern is focused on performance, dynamic security test and measurement over the world; other products are not made for that and is not their focus.

What is important is to take the best decision per context.

Compose with one or more API Testing products

We cannot be good at everything. It’s the same for technology products; you have to narrow the core problem they are solving.

Judge solutions across the 3 criteria for Quality Engineering choices:

Quality
- Does it answer your main functional and non-functional requirements?
- Are you going to create more value for your team using it?
Speed
- Can you use and maintain the solution with minimal effort?
- Will you accelerate the delivery of valuable software for your team?
Complexity
- Are you able to deploy the solution with ease and clear interfaces?
- Will all your users rate the product above 8 on a scale from 0 to 10?

Once you challenge that, make a Proof of Value.

You may conclude that you need more than one solution. In that case, the capability of integration becomes important to keep rapid cycles of API implementation.

If you start to have various APIs repositories not sync between themselves, you will end up losing time replicating data, leading to manual issues and rework.

Finally select the tools to support Quality Engineering

Here’s a list of known products you can consider per category. Without being completely exhaustive, it clarifies the main options.

Focused API product

RapidAPI: Reuse API test, built-in scheduler, CI/CD, you can schedule and run tests at intervals from different locations around the globe. This lets you manage the differences for users depending on location and request size, detailed analytics.

API Fortress: This platform focuses on testing APIs and microservices. It provides analytics, upload of specs CI/CD integration.

Gravity: Allows you to discover your customer journeys from native logs connectors such as Dynatrace, Elastic Search, among others. It is then integrates with test management to keep the bi-directional view.

APIMetrics, RunScope, Assertible, Rapidspike, Checkly.

Testing automation platform

Katalon: Toolbox based on an Eclipse IDE now providing reportings with a web platform. The scheduler is via licensing or custom external integration and trigger.

Cerberus Testing: Open-source fully available on github, for API in REST/SOAP/GraphQL and Kafka, built in java, support web/mobile/api/desktop/database testing, with CI/CD, scheduling, continuous testing.

TestSigma: A test automation platform letting you automate a variety of automated tests including APIs. The interface is mainly web and to ease the journey. Scheduler requires integration.

TestProject: Open-source by design available in Cloud, with web/mobile/api toolbox available. Basic scheduler.

Robot Framework: A complete toolbox widely known in the market. It is a powerful software with a lot of capabilities. The trade-offs are the complexity to open to more business teams, collaboration and integration in more modern cloud environments. Scheduler is usually external.

Hoppscotch: Formerly Postwoman, very popular, almost 30k on github focused on providing an open source API development ecosystem.

Full Monitoring solution

Uptrends: Handle the case of API monitoring part of other monitoring use-cases, less deep than RapidAPI for example.

AlertSite: from Smartbear, cloud-based, therefore, it’s accessed in your browser. You can create payloads, add assertions, and run tests on demand. Lot of notification mechanisms, CI/CD.

Loggly: Monitoring of APIs around the globe with standard features. The product is initially designed to monitor logs.

Dotcom-Monitor: A cloud-based testing platform for websites and web services that includes availability checks for REST and SOAP APIs.

Pinghut, API Science, Apps Dynamics, Bearer, New Relic, CloudWatch, ManageEngine, Datadog, Dynatrace, Site24x7 Website Monitoring (Zoho), Rigor.

Continuous API Testing for Quality at Speed

Testing your APIs is part of a larger effort of accelerating the delivery of valuable software.

Your goal is therefore not to pick the best API testing product—it’s to select the one enabling more Quality at Speed with minimal Complexity.

You may need more than one solution to fully cover your needs. I am convinced that the convergence of technology creates the opportunities.

Continuous API Testing requires acting along the software value-chain, composing with solutions acting on each space.

More solutions like Gravity will appears. That one focuses on customer journeys clustering from API logs that then integrates with test management and automation solutions.

What you need is not a tool.

It’s Quality Engineering for Quality at Speed.

Follow the QE Unit for more exclusive Quality Engineering from the community.