API Advice

Contract testing with OpenAPI

Brian Flad

September 30, 2024

We’ve all heard that infernal phrase, “It works on my machine.” Scaling any solution to work across many machines can be a challenge. The world of APIs with disparate consumers is no different. What if there was a well-defined contract between them?

But APIs and consumers change over time, which begs the question: How do we ensure our systems stick to their agreements? The answer is contract testing.

In this article, we’ll explore contract testing, how it fits into the testing pyramid, and why it’s important. We’ll also walk through how to implement contract testing using OpenAPI, a popular API specification format.

Finally, we’ll discuss how you can generate contract tests automatically using OpenAPI and Speakeasy.

What is contract testing?

Contract testing verifies that two parties who interact via an API stick to the contract they agreed upon. We’ll call these parties a consumer and a provider, based on the direction of the API calls between them. The consumer is the system that makes API calls, and the provider is the system that receives and responds to the API call.

In most cases, the contract between our parties is defined by an API specification, such as OpenAPI.

Even if there is no API specification, there is always, at the very least, an implicit agreement between the consumer and the provider. The consumer expects the provider to respond in a certain way, and the provider expects the consumer to send requests in a certain way.

This explicit or implicit agreement is the contract.

Contract testing, from the consumer’s perspective, is about verifying that the provider sticks to the contract. From the provider’s perspective, it’s about verifying that the consumer sticks to the contract.

How contract testing differs from unit testing, integration testing, and end-to-end testing

Testing strategy is often represented as a pyramid, with unit testing at the base, followed by other testing methodologies in ascending order of complexity and scope. The idea is that the majority of tests should be at the base of the pyramid, with fewer tests at each subsequent level.

The testing pyramid is a useful model for thinking about how different types of tests fit together in a testing strategy. It helps to ensure that the right types of tests are used in the appropriate proportions, balancing the need for comprehensive testing with the need for fast feedback.

Let’s discuss each level of the pyramid in more detail, starting from the base.

Unit testing: Does this function work as expected?

Unit testing forms the base of the testing pyramid. These tests focus on individual components or functions in isolation, typically mocking any dependencies. They are fast to run and easy to maintain, but don’t test how components work together.

In the consumer’s context, a unit test might verify whether the function that deserializes a JSON object into a Python object works correctly, without making any external API calls.

Unit tests are essential for catching bugs early in the development process and providing fast feedback to developers. They are also useful for ensuring that code behaves as expected when refactoring or adding new features. However, they don’t provide much confidence that the system as a whole works correctly.

Contract testing: Do we honor the API specification?

Moving up the pyramid, we have contract tests. Contract testing sits between unit testing and integration testing. It focuses specifically on the interactions between the provider and consumer for a given call, ensuring that the API contracts are honored. Contract tests are more complex than unit tests but less complex than integration tests.

A contract test verifies that a consumer can create requests with specific data and correctly handle the provider’s expected responses or errors. This might be accomplished with mocked request or response data based on the contract. For example, an API contract test for an order creation endpoint might verify that request data correctly maps integer item IDs to quantities and that the response decodes to an expected success with an integer order ID.

Contract tests are useful for catching issues that arise when the consumer or provider strays from the agreed-upon contract. They provide a level of confidence that the system works as expected when the consumer and provider interact. They are also useful for catching breaking changes early in the development process.

Integration testing: Do systems work together?

Further up, we find integration tests. These verify that different components of a system work together correctly. They are more complex than unit tests and contract tests, and may involve multiple components or services.

An integration test might verify that the consumer can successfully make an API call to the provider and receive a valid response. This test would involve both the consumer and provider, and would typically run in a test environment that mirrors the production environment.

Because integration tests involve multiple systems, they are useful for catching issues that arise when components interact, such as network issues between two services.

End-to-end testing: Does the user flow work as expected?

At the top of the pyramid are end-to-end tests. These test the entire user flow and supporting systems from start to finish. They provide the highest level of confidence but are also the slowest to run and most difficult to maintain.

In our API context, an end-to-end test might involve making a series of API calls that represent a complete user journey, verifying that the system behaves correctly at each step. This could include creating a resource, updating it, retrieving it, and finally deleting it, all through the API.

When end-to-end tests fail, it can be challenging to identify the root cause of the failure, as the problem could be in any part of the system. Due to their complexity and cost, they are often used as a final check before deploying to production, rather than as part of the regular development process.

Testing pyramid summary

Here’s a summary of the different types of tests and how they compare:

Why contract testing is important

Over time, APIs change in response to changing requirements, sometimes in subtle and imperceptible ways. For example, a provider may change the format of a field in a certain response from a string to an integer. This change may seem innocuous to the provider but could have catastrophic effects for the consumer.

Contract testing mitigates this risk by ensuring that any changes to the API contract are detected early in the development process. When the provider updates the API, corresponding contract tests will fail if the update is not backward compatible. This failure acts as an immediate signal that the change needs to be reviewed, preventing breaking changes from reaching production.

Consider this example: A subscription management platform (the provider) has an endpoint /plan/{id} that returns a subscription plan based on the plan ID. The consumer expects the response to include an amount field, which is an integer representing the cost of the plan. If the provider changes the amount field from an integer to a string, the consumer’s contract test will fail, alerting the consumer to the breaking change.

In this example, a contract test would catch the breaking change early in the development process, before it reaches production. The consumer and provider can then work together to resolve the issue, ensuring that the API contract is honored.

How to implement contract testing with OpenAPI

Let’s walk through the process of implementing contract testing using OpenAPI, focusing on both the consumer and provider perspectives.

Step 1: Create a new project

We’ll start by creating a new project for our consumer and provider code. We’ll use a simple subscription management API as an example.

In the terminal, run:

mkdir contract-example
cd contract-example

Let’s create a new TypeScript project for our SDK and tests:

In the terminal, run:

npm install --save-dev typescript
npx tsc --init

Select the default options when prompted.

Step 2: Define the OpenAPI specification

We’ll start by writing an OpenAPI specification for our API. In most cases, the provider will define the OpenAPI specification, as they are responsible for the implementation of the API.

openapi: 3.1.0
info:
  title: Subscription Management API
  version: 1.0.0
servers:
  - url: http://127.0.0.1:4010
    description: Local server
tags:
  - name: subscription
    description: Subscription management
security:
  - api_key: []
paths:
  /plan/{id}:
    get:
      operationId: getPlanById
      tags:
        - subscription
      summary: Get a subscription plan by ID
      parameters:
        - name: id
          in: path
          required: true
          schema:
            type: integer
          examples:
            basic:
              value: 1
            premium:
              value: 2
      responses:
        "200":
          description: Successful response
          content:
            application/json:
              schema:
                $ref: "#/components/schemas/Subscription"
              examples:
                basic:
                  value:
                    id: 1
                    name: "Basic"
                    amount: 100
                    currency: "USD"
                premium:
                  value:
                    id: 2
                    name: "Premium"
                    amount: 200
                    currency: "USD"
components:
  schemas:
    Subscription:
      type: object
      properties:
        id:
          type: integer
        name:
          type: string
        amount:
          type: integer
        currency:
          type: string
      example:
        id: 1
        name: "Basic"
        amount: 100
        currency: "USD"
  securitySchemes:
    api_key:
      type: apiKey
      name: X-API-Key
      in: header

This example OpenAPI specification only defines the happy path for the /plan/{id} endpoint. In a real-world scenario, you would define additional paths, operations, and error responses to cover all possible scenarios.

Step 3: Create an SDK with Speakeasy

We’ll use Speakeasy to create a TypeScript SDK from the OpenAPI specification. If you don’t have Speakeasy installed, you can install it from the Introduction to Speakeasy guide.

With Speakeasy installed, run:

speakeasy quickstart

When prompted, select the subscriptions.yaml file and choose TypeScript as the target language. We decided on Billing as the SDK name, and billing as the package name.

Step 4: Add tests

Now that we have an SDK, we can write tests to verify that the SDK handles the API responses correctly.

Let’s install the necessary dependencies:

npm i --save-dev vitest

Create a new tests directory in the root of your project. Then, create a new file, tests/subscription.test.ts:

import { expect, test } from "vitest";
import { Billing } from "../billing-typescript/src/index.ts"

test("Subscription Get Plan By Id Basic", async () => {
  const billing = new Billing({
    apiKey: process.env["BILLING_API_KEY"] ?? "",
  });
  const result = await billing.subscription.getPlanById({
    id: 1,
  });
  expect(result).toBeDefined();
  expect(result).toEqual({
    id: 1,
    name: "Basic",
    amount: 100,
    currency: "USD",
  });
});

Add the following script to your package.json:

{
  "scripts": {
    "test": "vitest run"
  }
}

Now you can run the tests:

npm run test

This should run the test and verify that the SDK correctly handles the API responses, but since we haven’t started a server yet, the test will fail.

Step 5: Start a mock server

We’ll use Prism to start a mock server that serves responses based on the OpenAPI specification.

Add Prism as a dev dependency:

npm install --save-dev @stoplight/prism-cli

Then, add a new script to your package.json:

{
  "scripts": {
    "test": "vitest run",
    "mock": "prism mock subscriptions.yaml"
  }
}

In a new terminal window, run:

npm run mock

This will start a mock server at http://127.0.0.1:4010.

Step 6: Run the tests

Now that the mock server is running, you can run the tests again:

npm run test

This time, Prism returns a 401 status code because we haven’t provided an API key. Let’s run the test with the BILLING_API_KEY set to test:

export BILLING_API_KEY=test
npm run test

$ vitest run

 RUN  v2.1.1 /Users/speakeasy/contract-example

 ✓ tests/subscription.test.ts (1)
   ✓ Subscription Get Plan By Id Basic

 Test Files  1 passed (1)
      Tests  1 passed (1)
   Start at  07:28:38
   Duration  630ms (transform 191ms, setup 0ms, collect 199ms, tests 150ms, environment 0ms, prepare 86ms)

The test should now pass, verifying that the SDK correctly handles the API response.

Step 7: Test for correctness

We’ve validated that the SDK can correctly handle an API response by interacting with a mock server, but we haven’t confirmed whether the response conforms to the contract. To make this a true contract test, let’s verify that both the consumer and provider behaviors align with the agreed-upon OpenAPI specification.

We’ll add a contract-validation step to the test, then use Ajv, a JSON Schema validator, to validate the response against the OpenAPI schema.

npm install --save-dev ajv ajv-errors ajv-formats yaml

Create a new file, validateSchema.ts:

import Ajv from "ajv";
import addFormats from "ajv-formats";
import addErrors from "ajv-errors";
import { readFileSync } from "fs";
import yaml from "yaml";

// Load and parse the OpenAPI specification
const openApiSpec = yaml.parse(readFileSync("./subscriptions.yaml", "utf8")) as any;

// Initialize Ajv with formats and error messages
const ajv = new Ajv({ allErrors: true, strict: false });
addFormats(ajv);
addErrors(ajv);

// Compile the schema for the Subscription response
const subscriptionSchema = {
  ...openApiSpec.components.schemas.Subscription
};

const validate = ajv.compile(subscriptionSchema);

export const validateSubscription = (data: any) => {
  const isValid = validate(data);
  if (!isValid) {
    console.error(validate.errors);
    throw new Error("Validation failed");
  }
};

Update the test to include the contract-validation step:

import { expect, test, expectTypeOf } from "vitest";
import { Billing } from "../billing-typescript/src/index.ts";
import { validateSubscription } from "../validateSchema.ts";

test("Subscription Get Plan By Id Basic", async () => {
  const billing = new Billing({
    apiKey: process.env["BILLING_API_KEY"] ?? "",
  });
  const result = await billing.subscription.getPlanById({
    id: 1,
  });
  expect(result).toBeDefined();
  expect(result).toEqual({
    id: 1,
    name: "Basic",
    amount: 100,
    currency: "USD",
  });
  validateSubscription(result); // Contract validation
});

Now when you run the tests, the contract validation will ensure that the response from the mock server matches the OpenAPI specification.

npm run test

Generating contract tests automatically with OpenAPI

Manually writing contract tests can be a time-consuming and error-prone process. If you’re starting with an OpenAPI document as your contract, you may be able to automatically generate tests that conform to your contract.

By generating contract tests, you reduce the risk of human error, save significant development time, and ensure that tests are always kept up to date.

The biggest advantage of automated test generation is the assurance that your tests are based on the API specification. This means that all aspects of the API contract, from endpoint paths and methods to data types and constraints, are accurately represented in the generated tests.

A drawback of basing tests on OpenAPI documents is that the OpenAPI Specification does not currently have built-in support for test generation. Although examples of requests and responses allow test case generation, there are still challenges in linking request and response pairs to each other. These are problems we’re working hard to overcome at Speakeasy.

Speakeasy test generation

EARLY ACCESS

Speakeasy API test generation is in beta. Join the early access program to give it a try.

At Speakeasy, we enable developers to automatically test their APIs and SDKs by creating comprehensive test suites. Shipping automated tests as part of your SDKs will enable your team to make sure that the interfaces your users prefer, your SDKs, are always compatible with your API. We ensure your APIs and SDKs stick to the contract so that you can focus on shipping features and evolving your API with confidence.

The process of adding tests with Speakeasy is straightforward: Add detailed examples to your OpenAPI document, or describe tests in a simple and well-documented YAML specification that lives in your SDK project. Speakeasy will regenerate your tests when they need to change, and you can run the tests as part of development or CI/CD workflows.

Speakeasy’s new automated API testing platform is in early access and currently supports Python, Go, TypeScript, and Java.

Speakeasy test generation roadmap

Looking ahead, Speakeasy’s testing roadmap includes broader language support, advanced server mocking, ability to run contract tests on past versions of the API and SDK, and using the Arazzo specification to string together multiple contract tests. With these features, you’ll be able to monitor the health of all your SDKs and APIs in one place.

We’re also working on support for behavior-driven development (BDD) and end-to-end (E2E) testing by embracing OpenAPI and the recently published Arazzo specification for complex testing workflows.

To join the Speakeasy automated API testing early access program, follow the signup link on the testing page.