Product Updates

Introducing: API contract test generation

Brian Flad

December 2, 2024

Today we’re excited to announce the beta launch of API contract test generation, a powerful new addition to our API development platform that uses your OpenAPI spec to automatically create comprehensive test suites for your API. This release continues our mission of automating the tedious parts of API development, enabling your team to stay focused on building great products.

The Hidden Cost of API Testing

API contract testing is crucial for maintaining reliability and preventing breaking changes, but creating and maintaining test suites is a significant burden on engineering teams. Today, developers face two major challenges:

First, they must manually write test code using HTTP frameworks, carefully crafting requests and validation logic for each endpoint. This involves tedious work like handling authentication, managing data structures, and writing assertions.

Second, they need to create and maintain realistic test data. While simple endpoints might only need basic examples, real-world APIs often have complex data structures with numerous fields and edge cases. Missing even a single field in your test coverage can lead to broken integrations down the line.

The result? Teams either invest significant engineering resources in testing or, more commonly, settle for incomplete test coverage that leaves them vulnerable to breaking changes.

Generating test suites

Speakeasy Contract Testing approaches this problem from both sides. We don’t just generate the test code – we also create the test data needed to validate your API’s behavior.

Native Test Generation: Tests are generated in your favorite language’s native testing framework (pytest (opens in a new tab) for Python, vitest (opens in a new tab) for TypeScript, etc.), ensuring they integrate seamlessly with your existing development workflow. We know that debugging impenetrable auto-generated tests is a nightmare, so we’ve put a lot of work into making the tests we generated look and feel like they were written by your team.

test("Users Create User", async () => {
  const petstore = new Petstore({
    serverURL: process.env["TEST_SERVER_URL"] ?? "http://localhost:18080",
    httpClient: createTestHTTPClient("createUser"),
    apiKey: process.env["PETSTORE_API_KEY"] ?? "",
  });
  const result = await petstore.users.create({
    id: 10,
    username: "theUser",
    firstName: "John",
    lastName: "James",
    email: "john@email.com",
    password: "12345",
    phone: "12345",
    userStatus: 1,
  });
  expect(result).toBeDefined();
  expect(result).toEqual({
    id: 10,
    username: "theUser",
    firstName: "John",
    lastName: "James",
    email: "john@email.com",
    password: "12345",
    phone: "12345",
    userStatus: 1,
  });
});

Solving the testing workflow

Generating testing code is only part of the solution. Testing is a workflow, and we know every team’s testing needs are different, so we’ve built flexibility and out of the box completeness into the core of our testing platform:

Comprehensive Data Coverage: Beyond just testing happy paths, our platform generates test data that covers edge cases and optional fields. If you have examples in your OpenAPI spec, we’ll use those. If not, we’ll generate realistic mock data that validates your API’s full schema.

Flexible testing environment: Have your own sandbox environment? We’ve got you covered. You can run your tests against any server URL you’d like. Don’t have a sandbox? We’ve got you covered there too. Our platform will create a mock server for you to run your tests against.

Flexible Configuration: We are ready to support your preferred workflow. For teams managing their OpenAPI spec, simply add the x-speakeasy-test annotation to your OpenAPI spec to enable testing for specific endpoints or your entire API. Alternatively, you can use our external configuration support to define tests without modifying the underlying OpenAPI document.

CI/CD Integration: Tests can be run locally during development or automated in your CI/CD pipeline. We provide GitHub Actions workflows out of the box, making it easy to validate API changes on every pull request.

Testing end-to-end

In addition to contract testing, we’re also rolling out the beginnings of end-to-end testing support. This will allow you to validate your API’s behavior across multiple endpoints, ensuring that your API behaves as expected in complex workflows. For example, you might want to test a workflow where a user logs in, creates an order, and then verifies the order appears in their order history. This kind of multi-step testing is crucial because it validates not just individual endpoints, but the connections between them.

To enable end-to-end testing, we’re leveraging the Arazzo specification, which allows you to arrange API calls into multi-step workflows. The Arazzo specification is fully compatible with the OpenAPI specification, so you can use your existing OpenAPI documents to generate end-to-end tests with a fully open source framework.

We’ll be rolling out more support for bootstrapping Arazzo workflows in the coming months, so stay tuned for more updates!

Getting Started

Contract Testing is available for early access. You can generate tests in TypeScript, Python, and Go today, with more languages (Java, C#, etc.) coming soon. Existing customers can reach out to their account manager to get access. If you’re new to Speakeasy, you can request access to the testing module by filling out this form.

We’re excited to see how this helps teams build more reliable APIs with less effort. This is just the beginning – we have lots more planned for testing, including deeper workflow validation, end-to-end testing support, and enhanced mock data generation.

Ready to automate your API testing? Sign up now (opens in a new tab) or check out our documentation (opens in a new tab) to learn more.