Custom Contract and End-to-End Tests with Arazzo

Arazzo is a simple, human-readable, and extensible specification for defining API workflows. Arazzo powers test generation, allowing you to define custom tests for any use case and define rich tests capable of:

Testing multiple operations.
Testing different inputs.
Validating the correct response is returned.
Run against a real API or mock server.
Configure setup and teardown routines for complex E2E tests.

The Arazzo Specification allows you to define sequences of API operations and their dependencies for contract testing, enabling you to validate that your API behaves correctly across multiple interconnected endpoints and complex workflows.

When a .speakeasy/tests.arazzo.yaml file is found in your SDK repo, the Arazzo workflow will be used to generate tests for each of the workflows defined in the file.

Prerequisites

The following are requirements for generating tests:

Testing feature prerequisites are met.

Test Suite Example

The following is an example Arazzo document defining a simple E2E test for the life cycle of a user resource in the example API.

arazzo: 1.0.0
info:
  title: Test Suite
  summary: E2E tests for the SDK and API.
  version: 0.0.1
sourceDescriptions:
  - name: The API
    url: https://example.com/openapi.yaml
    type: openapi
workflows:
  - workflowId: user-lifecycle
    steps:
      - stepId: create
        operationId: createUser
        requestBody:
          contentType: application/json
          payload: {
            "email": "Trystan_Crooks@hotmail.com",
            "first_name": "Trystan",
            "last_name": "Crooks",
            "age": 32,
            "postal_code": 94110,
            "metadata": {
              "allergies": "none",
              "color": "red",
              "height": 182,
              "weight": 77,
              "is_smoking": true
            }
          }
        successCriteria:
          - condition: $statusCode == 200
          - condition: $response.header.Content-Type == application/json
          - condition: $response.body#/email == Trystan_Crooks@hotmail.com
          - condition: $response.body#/postal_code == 94110
        outputs:
          id: $response.body#/id
      - stepId: get
        operationId: getUser
        parameters:
          - name: id
            in: path
            value: $steps.create.outputs.id
        successCriteria:
          - condition: $statusCode == 200
          - condition: $response.header.Content-Type == application/json
          - condition: $response.body#/email == Trystan_Crooks@hotmail.com
          - condition: $response.body#/first_name == Trystan
          - condition: $response.body#/last_name == Crooks
          - condition: $response.body#/age == 32
          - condition: $response.body#/postal_code == 94110
        outputs:
          user: $response.body
          age: $response.body#/age
      - stepId: update
        operationId: updateUser
        parameters:
          - name: id
            in: path
            value: $steps.create.outputs.id
        requestBody:
          contentType: application/json
          payload: $steps.get.outputs.user
          replacements:
            - target: /postal_code
              value: 94107
            - target: /age
              value: $steps.get.outputs.age
        successCriteria:
          - condition: $statusCode == 200
          - condition: $response.header.Content-Type == application/json
          - condition: $response.body#/email == Trystan_Crooks@hotmail.com
          - condition: $response.body#/first_name == Trystan
          - condition: $response.body#/last_name == Crooks
          - condition: $response.body#/age == 32
          - condition: $response.body#/postal_code == 94107
        outputs:
          email: $response.body#/email
          first_name: $response.body#/first_name
          last_name: $response.body#/last_name
          metadata: $response.body#/metadata
      - stepId: updateAgain
        operationId: updateUser
        parameters:
          - name: id
            in: path
            value: $steps.create.outputs.id
        requestBody:
          contentType: application/json
          payload: {
            "id": "$steps.create.outputs.id",
            "email": "$steps.update.email",
            "first_name": "$steps.update.first_name",
            "last_name": "$steps.update.last_name",
            "age": 33,
            "postal_code": 94110,
            "metadata": "$steps.update.metadata"
          }
        successCriteria:
          - condition: $statusCode == 200
          - condition: $response.header.Content-Type == application/json
          - condition: $response.body#/email == Trystan_Crooks@hotmail.com
          - condition: $response.body#/first_name == Trystan
          - condition: $response.body#/last_name == Crooks
          - condition: $response.body#/age == 33
          - condition: $response.body#/postal_code == 94110
      - stepId: delete
        operationId: deleteUser
        parameters:
          - name: id
            in: path
            value: $steps.create.outputs.id
        successCriteria:
          - condition: $statusCode == 200

The above workflow defined 4 steps that each feed into the next, representing the creation of a user, retrieving that user via its new ID, updating the user, and finally deleting the user. Outputs have been defined for certain steps that are then used as inputs for the following steps.

It will generate the test shown below:

// src/__tests__/sdk.test.ts

import { assert, expect, it, test } from "vitest";
import { SDK } from "../index.js";
import { assertDefined } from "./assertions.js";
import { createTestHTTPClient } from "./testclient.js";

test("Sdk User Lifecycle", async () => {
  const sdk = new SDK({
    serverURL: process.env["TEST_SERVER_URL"] ?? "http://localhost:18080",
    httpClient: createTestHTTPClient("user-lifecycle"),
  });

  const createResult = await sdk.createUser({
    email: "Trystan_Crooks@hotmail.com",
    firstName: "Trystan",
    lastName: "Crooks",
    age: 32,
    postalCode: "94110",
    metadata: {
      allergies: "none",
      additionalProperties: {
        "color": "red",
        "height": "182",
        "weight": "77",
        "is_smoking": "true",
      },
    },
  });
  expect(createResult.httpMeta.response.status).toBe(200);
  expect(createResult.user?.email).toEqual("Trystan_Crooks@hotmail.com");
  expect(createResult.user?.postalCode).toBeDefined();
  expect(createResult.user?.postalCode).toEqual("94110");

  const getResult = await sdk.getUser(assertDefined(createResult.user?.id));
  expect(getResult.httpMeta.response.status).toBe(200);
  expect(getResult.user?.email).toEqual("Trystan_Crooks@hotmail.com");
  expect(getResult.user?.firstName).toBeDefined();
  expect(getResult.user?.firstName).toEqual("Trystan");
  expect(getResult.user?.lastName).toBeDefined();
  expect(getResult.user?.lastName).toEqual("Crooks");
  expect(getResult.user?.age).toBeDefined();
  expect(getResult.user?.age).toEqual(32);
  expect(getResult.user?.postalCode).toBeDefined();
  expect(getResult.user?.postalCode).toEqual("94110");

  const user = assertDefined(getResult.user);
  user.postalCode = "94107";
  user.age = getResult.user?.age;
  const updateResult = await sdk.updateUser(
    assertDefined(createResult.user?.id),
    assertDefined(getResult.user),
  );
  expect(updateResult.httpMeta.response.status).toBe(200);
  expect(updateResult.user?.email).toEqual("Trystan_Crooks@hotmail.com");
  expect(updateResult.user?.firstName).toBeDefined();
  expect(updateResult.user?.firstName).toEqual("Trystan");
  expect(updateResult.user?.lastName).toBeDefined();
  expect(updateResult.user?.lastName).toEqual("Crooks");
  expect(updateResult.user?.age).toBeDefined();
  expect(updateResult.user?.age).toEqual(32);
  expect(updateResult.user?.postalCode).toBeDefined();
  expect(updateResult.user?.postalCode).toEqual("94107");

  const updateAgainResult = await sdk.updateUser(
    assertDefined(createResult.user?.id),
    {
      id: assertDefined(createResult.user?.id),
      email: assertDefined(updateResult.user?.email),
      firstName: updateResult.user?.firstName,
      lastName: updateResult.user?.lastName,
      age: 33,
      postalCode: "94110",
      metadata: updateResult.user?.metadata,
    },
  );
  expect(updateAgainResult.httpMeta.response.status).toBe(200);
  expect(updateAgainResult.user?.email).toEqual("Trystan_Crooks@hotmail.com");
  expect(updateAgainResult.user?.firstName).toBeDefined();
  expect(updateAgainResult.user?.firstName).toEqual("Trystan");
  expect(updateAgainResult.user?.lastName).toBeDefined();
  expect(updateAgainResult.user?.lastName).toEqual("Crooks");
  expect(updateAgainResult.user?.age).toBeDefined();
  expect(updateAgainResult.user?.age).toEqual(33);
  expect(updateAgainResult.user?.postalCode).toBeDefined();
  expect(updateAgainResult.user?.postalCode).toEqual("94110");

  const deleteResult = await sdk.deleteUser(
    assertDefined(createResult.user?.id),
  );
  expect(deleteResult.httpMeta.response.status).toBe(200);
});

# tests/test_sdk.py

import io
import openapi
from openapi import SDK
import os
import pytest
from tests.test_client import create_test_http_client


def test_sdk_user_lifecycle():
    with SDK(
        server_url=os.getenv("TEST_SERVER_URL", "http://localhost:18080"),
        client=create_test_http_client("user-lifecycle"),
    ) as sdk:
        assert sdk is not None

        create_res = sdk.create_user(
            request=openapi.BaseUser(
                email="Trystan_Crooks@hotmail.com",
                first_name="Trystan",
                last_name="Crooks",
                age=32,
                postal_code="94110",
                metadata=openapi.Metadata(
                    allergies="none",
                    **{
                        "color": "red",
                        "height": "182",
                        "weight": "77",
                        "is_smoking": "true",
                    },
                ),
            )
        )
        assert create_res is not None
        assert create_res.email == "Trystan_Crooks@hotmail.com"
        assert create_res.postal_code is not None
        assert create_res.postal_code == "94110"

        get_res = sdk.get_user(id=create_res.id)
        assert get_res is not None
        assert get_res.email == "Trystan_Crooks@hotmail.com"
        assert get_res.first_name is not None
        assert get_res.first_name == "Trystan"
        assert get_res.last_name is not None
        assert get_res.last_name == "Crooks"
        assert get_res.age is not None
        assert get_res.age == 32
        assert get_res.postal_code is not None
        assert get_res.postal_code == "94110"

        user = get_res
        user.postal_code = "94107"
        user.age = get_res.age

        update_res = sdk.update_user(id=create_res.id, user=user)
        assert update_res is not None
        assert update_res.email == "Trystan_Crooks@hotmail.com"
        assert update_res.first_name is not None
        assert update_res.first_name == "Trystan"
        assert update_res.last_name is not None
        assert update_res.last_name == "Crooks"
        assert update_res.age is not None
        assert update_res.age == 32
        assert update_res.postal_code is not None
        assert update_res.postal_code == "94107"

        update_again_res = sdk.update_user(
            id=create_res.id,
            user=openapi.User(
                id=create_res.id,
                email=update_res.email,
                first_name=update_res.first_name,
                last_name=update_res.last_name,
                age=33,
                postal_code="94110",
                metadata=update_res.metadata,
            ),
        )
        assert update_again_res is not None
        assert update_again_res.email == "Trystan_Crooks@hotmail.com"
        assert update_again_res.first_name is not None
        assert update_again_res.first_name == "Trystan"
        assert update_again_res.last_name is not None
        assert update_again_res.last_name == "Crooks"
        assert update_again_res.age is not None
        assert update_again_res.age == 33
        assert update_again_res.postal_code is not None
        assert update_again_res.postal_code == "94110"

        sdk.delete_user(id=create_res.id)

// tests/sdk_test.go

package sdk_test

import (
	"context"
	"github.com/stretchr/testify/assert"
	"github.com/stretchr/testify/require"
	"openapi"
	"openapi/internal/utils"
	"os"
	"testing"
)

func TestSDK_UserLifecycle(t *testing.T) {
	ctx := context.Background()

	s := openapi.New(
		openapi.WithServerURL(utils.GetEnv("TEST_SERVER_URL", "http://localhost:18080")),
		openapi.WithClient(createTestHTTPClient("user-lifecycle")),
	)

	createRes, err := s.CreateUser(ctx, openapi.BaseUser{
		Email:      "Trystan_Crooks@hotmail.com",
		FirstName:  openapi.String("Trystan"),
		LastName:   openapi.String("Crooks"),
		Age:        openapi.Float64(32),
		PostalCode: openapi.String("94110"),
		Metadata: &openapi.Metadata{
			Allergies: openapi.String("none"),
			AdditionalProperties: map[string]string{
				"color":      "red",
				"height":     "182",
				"weight":     "77",
				"is_smoking": "true",
			},
		},
	})
	require.NoError(t, err)
	assert.Equal(t, 200, createRes.HTTPMeta.Response.StatusCode)
	assert.Equal(t, "Trystan_Crooks@hotmail.com", createRes.User.Email)
	assert.NotNil(t, createRes.User.PostalCode)
	assert.Equal(t, openapi.String("94110"), createRes.User.PostalCode)

	getRes, err := s.GetUser(ctx, createRes.User.ID)
	require.NoError(t, err)
	assert.Equal(t, 200, getRes.HTTPMeta.Response.StatusCode)
	assert.Equal(t, "Trystan_Crooks@hotmail.com", getRes.User.Email)
	assert.NotNil(t, getRes.User.FirstName)
	assert.Equal(t, openapi.String("Trystan"), getRes.User.FirstName)
	assert.NotNil(t, getRes.User.LastName)
	assert.Equal(t, openapi.String("Crooks"), getRes.User.LastName)
	assert.NotNil(t, getRes.User.Age)
	assert.Equal(t, openapi.Float64(32), getRes.User.Age)
	assert.NotNil(t, getRes.User.PostalCode)
	assert.Equal(t, openapi.String("94110"), getRes.User.PostalCode)

	user := *getRes.User
	user.PostalCode = openapi.String("94107")
	user.Age = getRes.User.Age

	updateRes, err := s.UpdateUser(ctx, createRes.User.ID, user)
	require.NoError(t, err)
	assert.Equal(t, 200, updateRes.HTTPMeta.Response.StatusCode)
	assert.Equal(t, "Trystan_Crooks@hotmail.com", updateRes.User.Email)
	assert.NotNil(t, updateRes.User.FirstName)
	assert.Equal(t, openapi.String("Trystan"), updateRes.User.FirstName)
	assert.NotNil(t, updateRes.User.LastName)
	assert.Equal(t, openapi.String("Crooks"), updateRes.User.LastName)
	assert.NotNil(t, updateRes.User.Age)
	assert.Equal(t, openapi.Float64(32), updateRes.User.Age)
	assert.NotNil(t, updateRes.User.PostalCode)
	assert.Equal(t, openapi.String("94107"), updateRes.User.PostalCode)

	updateAgainRes, err := s.UpdateUser(ctx, createRes.User.ID, openapi.User{
		ID:         createRes.User.ID,
		Email:      updateRes.User.Email,
		FirstName:  updateRes.User.FirstName,
		LastName:   updateRes.User.LastName,
		Age:        openapi.Float64(33),
		PostalCode: openapi.String("94110"),
		Metadata:   updateRes.User.Metadata,
	})
	require.NoError(t, err)
	assert.Equal(t, 200, updateAgainRes.HTTPMeta.Response.StatusCode)
	assert.Equal(t, "Trystan_Crooks@hotmail.com", updateAgainRes.User.Email)
	assert.NotNil(t, updateAgainRes.User.FirstName)
	assert.Equal(t, openapi.String("Trystan"), updateAgainRes.User.FirstName)
	assert.NotNil(t, updateAgainRes.User.LastName)
	assert.Equal(t, openapi.String("Crooks"), updateAgainRes.User.LastName)
	assert.NotNil(t, updateAgainRes.User.Age)
	assert.Equal(t, openapi.Float64(33), updateAgainRes.User.Age)
	assert.NotNil(t, updateAgainRes.User.PostalCode)
	assert.Equal(t, openapi.String("94110"), updateAgainRes.User.PostalCode)

	deleteRes, err := s.DeleteUser(ctx, createRes.User.ID)
	require.NoError(t, err)
	assert.Equal(t, 200, deleteRes.HTTPMeta.Response.StatusCode)
}

Input and Outputs

Inputs

Inputs can be provided to steps in a number of ways, either via inputs defined in the workflow, references from previous steps, or via values defined inline .

Workflow Inputs

Workflow inputs are a way to provide input parameters to the workflow that can be used by any step defined in the workflow. The inputs field of a workflow is a JSON Schema object that defines a property for each input the workflow wants to expose.

Test Generation can use any examples defined for a property in the inputs json schemas as literal values to use as inputs for the test. As tests are none interactive, if no examples are defined, the test generation will just randomly generate values for the inputs, as it can’t ask the user for input.

arazzo: 1.0.0
# ....
workflows:
  - workflowId: some-test
    inputs: # This is the JSON Schema for the inputs each property in the inputs object represents a workflow input
      type: object
      properties:
        email:
          type: string
          examples:
            - Trystan_Crooks@hotmail.com # Examples defined will be used as literal values for the test
        firstName:
          type: string
          examples:
            - Trystan
        lastName:
          type: string
          examples:
            - Crooks
    steps:
      - stepId: create
        operationId: createUser
        requestBody:
          contentType: application/json
          payload: {
            "email": "$inputs.email", # The payload will be populated with the literal value defined in the inputs
            "first_name": "$inputs.firstName",
            "last_name": "$inputs.lastName",
          }
        successCriteria:
          - condition: $statusCode == 200

Step References

Parameters and request body payloads can reference values via Runtime Expressions from previous steps in the workflow. This allows for the generation of tests that are more complex than a simple sequence of operations. Speakeasy’s implementation currently only allows the referencing of a previous step’s outputs, which means you will need to define what values you want to expose to future steps.

arazzo: 1.0.0
# ....
workflows:
  - workflowId: some-test
    steps:
      - stepId: create
        operationId: createUser
        requestBody:
          contentType: application/json
          payload: #....
        successCriteria:
          - condition: $statusCode == 200
          - condition: $response.header.Content-Type == application/json
          - condition: $response.body#/email == Trystan_Crooks@hotmail.com
        outputs:
          id: $response.body#/id # The id field of the response body will be exposed as an output for the next step
      - stepId: get
        operationId: getUser
        parameters:
          - name: id
            in: path
            value: $steps.create.outputs.id # The id output from the previous step will be used as the value for the id parameter
        successCriteria:
          - condition: $statusCode == 200

Inline Values

For any parameters or request body payloads a step defines, literal values can be provided inline to populate the tests if static values are suitable for the test.

arazzo: 1.0.0
# ....
workflows:
  - workflowId: some-test
    steps:
      - stepId: update
        operationId: updateUser
        parameters:
          - name: id
            in: path
            value: "some-test-id" # A literal value can be provided inline for parameters that matches the json schema of the parameter as defined in the associated operation
        requestBody:
          contentType: application/json
          payload: { # literals values that match the content type of the request body can be provided inline
            "email": "Trystan_Crooks@hotmail.com",
            "first_name": "Trystan",
            "last_name": "Crooks",
            "age": 32,
            "postal_code": 94110,
            "metadata": {
              "allergies": "none",
              "color": "red",
              "height": 182,
              "weight": 77,
              "is_smoking": true
            }
          }
        successCriteria:
          - condition: $statusCode == 200

Payload Values

When using the payload field of a request body input, the value can be a static value to use, a value with interpolated Runtime Expressions or a Runtime Expression by itself.

The payload value can then be overlayed using the replacements field which represents a list of targets within the payload to replace with the value of the replacements, which themselves can be a static valuue or a Runtime Expression.

arazzo: 1.0.0
# ....
workflows:
  - workflowId: some-test
    steps:
      - stepId: get
        # ...
        outputs:
          user: $response.body
      - stepId: update
        operationId: updateUser
        parameters:
          - name: id
            in: path
            value: "some-test-id"
        requestBody:
          contentType: application/json
          payload: $steps.get.outputs.user # use the response body of the previous step as the payload for this step
          replacements: # overlay the payload with the below replacements
            - target: /postal_code # overlays the postal_code field with a static value
              value: 94107
            - target: /age # overlays the age field with the value of the age output of a previous step
              value: $steps.some-other-step.outputs.age
        successCriteria:
          - condition: $statusCode == 200

Outputs

As shown above, outputs can be defined for each step in a workflow allowing values from things such as the response body to be used as values in following steps.

Current Speakeasy supports only referencing values from a response body, using the Runtime Expressions syntax and json-pointers.

Any number of outputs can be defined for a step.

arazzo: 1.0.0
# ....
workflows:
  - workflowId: some-test
    steps:
      - stepId: create
        operationId: createUser
        requestBody:
          contentType: application/json
          payload: #....
        successCriteria:
          - condition: $statusCode == 200
          - condition: $response.header.Content-Type == application/json
          - condition: $response.body#/email == Trystan_Crooks@hotmail.com
        outputs: # Outputs are a map of an output id to a runtime expression that will be used to populate the output
          id: $response.body#/id # json-pointers are used to reference fields within the response body
          email: $response.body#/email
          age: $response.body#/age
          allergies: $response.body#/metadata/allergies

Success Criteria

The successCriteria field of a step is a list of Criterion Objects that are used to validate the success of the step. For test generation these will form the basis of the test assertions.

successCriteria can be as simple as a single condition testing the status code of the response, or as complex as testing multiple individual fields within the response body.

Speakeasy’s implementation currently only supports simple criteria and the equality operator == for comparing values, and testing status codes, response headers and response bodies.

For testing values within the response body, criteria for testing the status code and content type of the response are also required, to help the generator determine which response schema to validate against due to the typed nature of the SDKs.

arazzo: 1.0.0
# ....
workflows:
  - workflowId: some-test
    steps:
      - stepId: create
        operationId: createUser
        requestBody:
          contentType: application/json
          payload: #....
        successCriteria:
          - condition: $statusCode == 200
          - condition: $response.header.Content-Type == application/json
          - condition: $response.body#/email == Trystan_Crooks@hotmail.com
          # or
          - context: $response.body
            type: simple
            condition: |
              {
                "email": "Trystan_Crooks@hotmail.com",
                "first_name": "Trystan",
                "last_name": "Crooks",
                "age": 32,
                "postal_code": 94110,
                "metadata": {
                  "allergies": "none",
                  "color": "red",
                  "height": 182,
                  "weight": 77,
                  "is_smoking": true
                }
              }

Testing operations requiring binary data

Some operations will required providing binary data to test uploading or downloading files etc. In these cases test files can be provided to the test using the x-file directive in the example for that field.

arazzo: 1.0.0
# ....
workflows:
  - workflowId: postFile
    steps:
      - stepId: test
        operationId: postFile
        requestBody:
          contentType: multipart/form-data
          payload:
            file: "x-file: some-test-file.txt"
        successCriteria:
          - condition: $statusCode == 200
          - condition: $response.header.Content-Type == application/octet-stream
          - context: $response.body
            condition: "x-file: some-other-test-file.dat"
            type: simple

The files will be sourced from the .speakeasy/testfiles directory in the root of your SDK repo, where the path provided in the x-file directive is relative to the testfiles directory.

The contents of the sourced file will be used as the value for the field being tested.

Next Steps

Setup testing in GitHub Actions for your SDK.
Advanced test configuration for your tests.