End-to-end API testing with Arazzo, TypeScript, and Deno

Brian Flad

October 30, 2024

We’ve previously written about the importance of building contract & integration tests to comprehensively cover your API’s endpoints, but there’s still a missing piece to the puzzle. Real users don’t consume your API one endpoint at a time - they implement complex workflows that chain multiple API calls together.

That’s why reliable end-to-end API tests are an important component of the testing puzzle. For your APIs most common workflows, you need to ensure that the entire process works as expected, not just the individual parts.

In this tutorial, we’ll build a test generator that turns Arazzo specifications into executable end-to-end tests. You’ll learn how to:

• Generate tests that mirror real user interactions with your API
• Keep tests maintainable, even as your API evolves
• Validate complex workflows across multiple API calls
• Catch integration issues before they reach production

We’ll use a simple “Build-a-bot” API as our example, but the principles and code you’ll learn apply to any REST API.

Arazzo? What & Why

Arazzo is a specification that describes how API calls should be sequenced to achieve specific outcomes. Think of it as OpenAPI for workflows - while OpenAPI describes what your API can do, Arazzo describes how to use it effectively.

Arazzo was designed to bridge the gap between API reference documentation and real-world usage patterns. Fortunately for us, it also makes a perfect fit for generating end-to-end test suites that validate complete user workflows rather than isolated endpoints.

By combining these specifications, we can generate tests that validate not just the correctness of individual endpoints, but the entire user journey.

Arazzo?

Arazzo roughly translates to “tapestry” in Italian. Get it? A tapestry of API calls “woven” together to create a complete user experience. We’re still undecided about how to pronounce it, though. The leading candidates are “ah-RAT-so” (like fatso) and “ah-RAHT-zoh” (almost like pizza, but with a rat). There is a minor faction pushing for “ah-razzo” as in razzle-dazzle. We’ll let you decide.

Let’s look at a simplified (and mostly invalid) illustrative example. Imagine a typical e-commerce API workflow:

arazzo: 1.0.0
workflowId: purchaseProduct
sourceDescriptions:
  - url: ./openapi.yaml
steps:
  - stepId: authenticate
    operationId: loginUser
    # post login details
    # response contains auth token
    # if successful, go to checkInventory
  - stepId: checkInventory
    operationId: getProductsStock
    # with auth token from previous step:
    # get stock levels of multiple products
    # response contains product IDs and stock levels
    # if stock levels are sufficient, go to createOrder
  - stepId: createOrder
    operationId: submitOrder
    # with auth token from first step
    # and product IDs and quantities from previous step
    # post an order that is valid based on stock levels
    # response contains order ID
    # if successful, go to getOrder
# ...

Arazzo allows us to define these workflows, and specify how each step should handle success and failure conditions, as well as how to pass data between steps and even between workflows.

From specification to implementation

The example above illustrates the concept, but let’s dive into a working implementation. We’ll use a simplified but functional example that you can download and run yourself. Our demo implements a subset of the Arazzo specification, focusing on the most immediately valuable features for E2E testing.

We’ll use the example of an API called Build-a-bot, which allows users to create and manage their own robots. You can substitute this with your own OpenAPI document, or use the Build-a-bot API to follow along.

Arazzo deep dive

Let’s examine the key components of an Arazzo specification using our Build-a-bot API example. The specification describes a workflow for creating, assembling, and activating a robot - a perfect example of a complex, multi-step process that would be difficult to test with traditional approaches.

arazzo: 1.0.0
info:
  title: Build-a-Bot Workflow
  version: 1.0.0
  description: This workflow guides you through the process of creating, assembling, and activating a robot using the Build-a-Bot API.

sourceDescriptions:
  - name: buildABotAPI
    url: ./openapi.yaml
    type: openapi

workflows:
  - workflowId: createAndActivateRobot
    description: Create, assemble, and activate a new robot
    inputs:
      type: object
      properties:
        BUILD_A_BOT_API_KEY:
          type: string
          description: The API key for the Build-a-Bot API
    steps:
      - stepId: createRobot
        description: Create a new robot design session
        operationId: createRobot
        parameters:
          - name: x-api-key
            in: header
            value: $inputs.BUILD_A_BOT_API_KEY
        requestBody:
          payload:
            model: humanoid
            name: MyFirstRobot
        successCriteria:
          - condition: $statusCode == 201
          - condition: /^[0-9A-F]{8}-[0-9A-F]{4}-[4][0-9A-F]{3}-[89AB][0-9A-F]{3}-[0-9A-F]{12}$/i
            context: $response.body#/robotId
            type: regex
          - condition: $response.body#/model == "humanoid"
          - condition: $response.body#/name == "MyFirstRobot"
          - condition: $response.body#/status == "designing"
          - context: $response.body#/links
            condition: $.length == 5
            type: jsonpath
        onSuccess:
          - name: gotoAddParts
            type: goto
            stepId: addParts
        outputs:
          robotId: $response.body#/robotId

      - stepId: addParts
        description: Add parts to the robot
        operationId: addRobotParts
        parameters:
          - name: x-api-key
            in: header
            value: $inputs.BUILD_A_BOT_API_KEY
          - name: robotId
            in: path
            value: $steps.createRobot.outputs.robotId
        requestBody:
          payload:
            parts:
              - type: arm
                name: Hydraulic Arm
                quantity: 2
              - type: sensor
                name: Infrared Sensor
                quantity: 1
        successCriteria:
          - condition: $statusCode == 200
          - condition: $response.body#/robotId == $steps.createRobot.outputs.robotId
        onSuccess:
          - name: gotoAssembleRobot
            type: goto
            stepId: assembleRobot

      - stepId: assembleRobot
        description: Assemble the robot
        operationId: assembleRobot
        parameters:
          - name: x-api-key
            in: header
            value: $inputs.BUILD_A_BOT_API_KEY
          - name: robotId
            in: path
            value: $steps.createRobot.outputs.robotId
        successCriteria:
          - condition: $statusCode == 200
          - condition: $response.body#/robotId == $steps.createRobot.outputs.robotId
          - condition: $response.body#/status == "assembled"
        onSuccess:
          - name: gotoConfigureRobotFeatures
            type: goto
            stepId: configureRobotFeatures

      - stepId: configureRobotFeatures
        description: Configure robot features
        operationId: configureRobotFeatures
        parameters:
          - name: x-api-key
            in: header
            value: $inputs.BUILD_A_BOT_API_KEY
          - name: robotId
            in: path
            value: $steps.createRobot.outputs.robotId
        requestBody:
          payload:
            features:
              - name: Voice Recognition
                description: Enables voice command recognition.
              - name: Obstacle Avoidance
                description: Navigates around obstacles.
        successCriteria:
          - condition: $statusCode == 200
          - condition: $response.body#/robotId == $steps.createRobot.outputs.robotId
        onSuccess:
          - name: gotoActivateRobot
            type: goto
            stepId: activateRobot

      - stepId: activateRobot
        description: Activate the robot
        operationId: activateRobot
        parameters:
          - name: x-api-key
            in: header
            value: $inputs.BUILD_A_BOT_API_KEY
          - name: robotId
            in: path
            value: $steps.createRobot.outputs.robotId
        successCriteria:
          - condition: $statusCode == 200
          - condition: $response.body#/robotId == $steps.createRobot.outputs.robotId
          - condition: $response.body#/status == "activated"
        outputs:
          activationTime: $response.body#/activationTime
          capabilities: $response.body#/capabilities
        onSuccess:
          - name: gotoGetRobot
            type: goto
            stepId: getRobot

      - stepId: getRobot
        description: Get the robot details
        operationId: getRobot
        parameters:
          - name: x-api-key
            in: header
            value: $inputs.BUILD_A_BOT_API_KEY
          - name: robotId
            in: path
            value: $steps.createRobot.outputs.robotId
        successCriteria:
          - condition: $statusCode == 200
          - condition: $response.body#/robotId == $steps.createRobot.outputs.robotId
          - condition: $response.body#/status == "activated"
          - condition: $response.body#/activationTime == $steps.activateRobot.outputs.activationTime
          - condition: $response.body#/capabilities == $steps.activateRobot.outputs.capabilities
        outputs:
          robot: $response.body

components: {}

This combination of validation patterns, data extraction, and flow control creates the foundation for testing complex API workflows.

Each success criterion serves a specific purpose in ensuring the API behaves as expected, while the outputs and success actions enable smooth workflow progression.

We’ll explore the concepts we covered in more detail in the next sections, but first, let’s set up a development environment so we can see Arazzo in action.

Setting up the development environment

First, clone the demo repository:

git clone https://github.com/speakeasy-api/e2e-testing-arazzo.git
cd e2e-testing-arazzo

You’ll need Deno v2 (opens in a new tab) installed. On macOS and Linux, you can install Deno using the following command:

curl -fsSL https://deno.land/install.sh | sh

The repository contains:

• A simple API server built with @oak/acorn (opens in a new tab) that serves as the Build-a-bot API (in packages/server/server.ts)
• An Arazzo specification file (arazzo.yaml)
• An OpenAPI specification file (openapi.yaml)
• The test generator implementation (packages/arazzo-test-gen/generator.ts)
• Generated E2E tests (tests/generated.test.ts)
• An SDK created by Speakeasy to interact with the Build-a-bot API (packages/sdk)

Running the Demo

To run the demo, start the API server:

deno task server

Deno will install the server’s dependencies, then start the server on http://localhost:8080. You can test the server by visiting http://localhost:8080/v1/robots, which should return a 401 Unauthorized error:

{
  "status": 401,
  "error": "Unauthorized",
  "message": "Header x-api-key is required"
}

Next, in a new terminal window, generate the E2E tests:

deno task dev

After installing dependencies, this command will generate the E2E tests in tests/generated.test.ts and watch for changes to the Arazzo specification file.

You can run the tests in a new terminal window:

deno task test

This command will run the generated tests against the API server:

> deno task test
Task test deno test --allow-read --allow-net --allow-env --unstable tests/
running 1 test from ./tests/generated.test.ts
Create, assemble, and activate a new robot ...
  Create a new robot design session ... ok (134ms)
  Add parts to the robot ... ok (2ms)
  Assemble the robot ... ok (1ms)
  Configure robot features ... ok (2ms)
  Activate the robot ... ok (3ms)
  Get the robot details ... ok (1ms)
Create, assemble, and activate a new robot ... ok (143ms)
 
ok | 1 passed (6 steps) | 0 failed (147ms)

Beautiful, everything works! Let’s see how we got here.

Building an Arazzo test generator

Let’s start with the overall structure of the test generator.

Project structure

The test generator is a Deno project that consists of several modules, each with a specific responsibility:

• generator.ts: The main entry point that orchestrates the test generation process. It reads the Arazzo and OpenAPI specifications, validates their compatibility, and generates test cases.

• readArazzoYaml.ts and readOpenApiYaml.ts: Handle parsing and validation of the Arazzo and OpenAPI specifications respectively. They ensure the specifications are well-formed and contain all required fields.

• expressionParser.ts: A parser for runtime expressions like $inputs.BUILD_A_BOT_API_KEY and $steps.createRobot.outputs.robotId. These expressions are crucial for passing data between steps and accessing workflow inputs.

• successCriteria.ts: Processes the success criteria for each step, including status code validation, regex patterns, direct comparisons, and JSONPath expressions.

• generateTestCase.ts: Takes the parsed workflow and generates the actual test code, including setup, execution, and validation for each step.

• security.ts: Handles security-related aspects like API key authentication and other security schemes defined in the OpenAPI specification.

• utils.ts: Contains utility functions for common operations like JSON pointer resolution and type checking.

The project also includes a runtime-expression directory containing the grammar definition for runtime expressions:

• runtimeExpression.peggy: A Peggy grammar file that defines the syntax for runtime expressions
• runtimeExpression.js: The generated parser from the grammar
• runtimeExpression.d.ts: TypeScript type definitions for the parser

Let’s dive deeper into each of these components to understand how they work together to generate effective E2E tests.

Parsing until you parse out

While our project says “test generator” on the tin, the bulk of our work will go into parsing different formats. To generate tests from an Arazzo document, we need to parse:

1. The Arazzo document
2. The OpenAPI document
3. Conditions in the Arazzo success criteria
4. Runtime expressions in the success criteria, outputs, and parameters
5. Regular expressions in the success criteria
6. JSONPath expressions in the success criteria
7. JSON pointers in the runtime expressions

We won’t cover all of these in detail, but we’ll touch on each to get a sense of the complexity involved and the tools we use to manage it.

Parsing the Arazzo specification

The first step in our test generator is parsing the Arazzo specification in readArazzoYaml.ts. This module reads the Arazzo YAML file and should ideally validate its structure against the Arazzo specification.

For our demo, we didn’t implement full validation, instead parsing the YAML file into a JavaScript object. We then use TypeScript interfaces to define the expected structure of the Arazzo document:

export interface ArazzoDocument {
  arazzo: string;
  info: ArazzoInfo;
  sourceDescriptions: Array<ArazzoSourceDescription>;
  workflows: Array<ArazzoWorkflow>;
  components: Record<string, unknown>;
}

export interface ArazzoWorkflow {
  workflowId: string;
  description: string;
  inputs: {
    type: string;
    properties: Record<string, { type: string; description: string }>;
  };
  steps: Array<ArazzoStep>;
}

export interface ArazzoStep {
  stepId: string;
  description: string;
  operationId: string;
  parameters?: Array<ArazzoParameter>;
  requestBody?: ArazzoRequestBody;
  successCriteria: Array<ArazzoSuccessCriterion>;
  outputs?: Record<string, string>;
}

These TypeScript interfaces help with autocompletion, type checking, and documentation, making it easier to work with the parsed Arazzo document in the rest of our code.

The real complexity comes in validating that the parsed document follows all the rules in the Arazzo specification. For example:

• Each workflowId must be unique within the document
• Each stepId must be unique within its workflow
• An operationId must reference a valid operation in the OpenAPI document
• Runtime expressions must follow the correct syntax
• Success criteria must use valid JSONPath or regex patterns

We don’t validate all these rules in our demo, but in production, we’d use Zod (opens in a new tab) or Valibot (opens in a new tab) to enforce these constraints at runtime and provide helpful error messages when the document is invalid.

The OpenAPI team hasn’t finalized the Arazzo specification’s JSON Schema yet, but once they do, we can use it to validate the Arazzo document against the schema with tools like Ajv (opens in a new tab).

Speakeasy also provides a command-line interface for linting Arazzo documents:

# Expects arazzo.yaml in the current directory
speakeasy lint arazzo

Parsing the OpenAPI specification

The OpenAPI specification’s path is gathered from the Arazzo document. In our test, we simply use the first sourceDescription to find the OpenAPI document path. But in a production generator, we’d need to handle multiple sourceDescriptions and ensure the OpenAPI document is accessible.

We parse the OpenAPI document in readOpenApiYaml.ts and use TypeScript interfaces from the npm:openapi-types (opens in a new tab) package to define the expected structure of the OpenAPI document.

We won’t cover the OpenAPI parsing in detail, but it’s similar to the Arazzo parsing: Read the YAML file, parse it into a JavaScript object, and type check it against TypeScript interfaces.

For OpenAPI, writing a custom validator is more complex due to the specification’s size and complexity. We recommend validating against the official OpenAPI 3.1.0 JSON Schema (opens in a new tab) using Ajv (opens in a new tab), or Speakeasy’s own OpenAPI linter:

speakeasy lint openapi --schema openapi.yaml

Parsing success criteria

This is where things get interesting. Success criteria in Arazzo are a list of conditions that must be met for a step to be considered successful. Each criterion can be one of the following types:

• simple: Selects a value with a runtime expression and compares it to an expected value
• jsonpath: Selects a value using a JSONPath expression and compares it to an expected value
• regex: Validates a value against a regular expression pattern
• xpath: Selects a value using an XPath expression and compares it to an expected value, used for XML documents

In our demo, we don’t implement the xpath type, but we do cover the other three. Here’s an example of a success criterion in the Arazzo document:

successCriteria:
  - condition: $statusCode == 201
  - condition: /^[0-9A-F]{8}-[0-9A-F]{4}-[4][0-9A-F]{3}-[89AB][0-9A-F]{3}-[0-9A-F]{12}$/i
    context: $response.body#/robotId
    type: regex
  - condition: $response.body#/model == "humanoid"
  - condition: $response.body#/name == "MyFirstRobot"
  - condition: $response.body#/status == "designing"
  - context: $response.body#/links
    condition: $.length == 5
    type: jsonpath

The condition field is required, and contains the expression to evaluate, while the context field specifies the part of the response to evaluate. The type field indicates the type of validation to perform.

If no type is specified, the success criterion is treated as a simple comparison, where the condition is evaluated directly.

Evaluating simple criteria

Here’s an example of how we parse a simple success criterion:

condition: $statusCode == 201

We split this simple condition into:

• Left-hand side: $statusCode - Runtime expression to evaluate
• Operator: == - Comparison operator or assertion
• Right-hand side: 201 - Expected value

We’ll evaluate the runtime expression $statusCode and compare it to the expected value 201. If the comparison is true, the criterion passes; otherwise, it fails.

Runtime expressions can also reference other variables, like $inputs.BUILD_A_BOT_API_KEY or $steps.createRobot.outputs.robotId, or fields in the response body, like $response.body#/model.

We’ll cover runtime expressions in more detail later.

Evaluating JSONPath criteria

For JSONPath criteria, we use the jsonpath type and a JSONPath expression to select a value from the response:

context: $response.body#/links
condition: $.length == 5
type: jsonpath

Let’s break down the JSONPath criterion:

• context: $response.body#/links - Runtime expression to select the links array from the response body
• condition: $.length == 5 - JSONPath expression compared to an expected value
• type: jsonpath - Indicates the criterion type

We further need to break down the condition into:

• Left-hand side: $.length - JSONPath expression to evaluate
• Operator: == - Comparison operator
• Right-hand side: 5 - Expected value

We evaluate the JSONPath expression $.length and compare it to the expected value 5. If the comparison is true, the criterion passes.

Evaluating regex criteria

For regex criteria, we use the regex type and a regular expression pattern to validate a value:

context: $response.body#/robotId
condition: /^[0-9A-F]{8}-[0-9A-F]{4}-[4][0-9A-F]{3}-[89AB][0-9A-F]{3}-[0-9A-F]{12}$/i
type: regex

Let’s break down the regex criterion:

• context: $response.body#/robotId - Runtime expression to select the robotId field from the response body
• condition: /^[0-9A-F]{8}-[0-9A-F]{4}-[4][0-9A-F]{3}-[89AB][0-9A-F]{3}-[0-9A-F]{12}$/i - Regular expression pattern to validate the value as a UUID v4
• type: regex - Indicates the criterion type

We evaluate the runtime expression $response.body#/robotId against the regular expression pattern. If the value matches the pattern, the criterion passes.

In our implementation, we use TypeScript’s factory.createRegularExpressionLiteral to create a regular expression literal from the pattern string. This ensures that the pattern is properly escaped and formatted as a valid JavaScript regular expression.

The generated test code looks something like this:

assertMatch(
  response.body.robotId,
  new RegExp(/^[0-9A-F]{8}-[0-9A-F]{4}-[4][0-9A-F]{3}-[89AB][0-9A-F]{3}-[0-9A-F]{12}$/i),
  'robotId should be a valid UUID v4'
);

This code uses Deno’s built-in assertMatch function to validate that the robotId matches the UUID v4 pattern. If the value doesn’t match, the test fails with a helpful error message.

Parsing runtime expressions

Runtime expressions are used throughout the Arazzo specification to reference variables, fields in the response body, or outputs from previous steps. They follow a specific syntax defined in the Arazzo specification as an ABNF (augmented Backus–Naur form) grammar.

To parse runtime expressions, we use a parser generated from the ABNF grammar. In our demo, this is a two-step process. First, we use the abnf (opens in a new tab) npm package to generate a Peggy grammar file from the ABNF grammar:

cd packages/arazzo-test-gen
abnf_gen runtime-expression/runtimeExpression.abnf

This generates a runtime-expression/runtimeExpression.peggy file that defines the syntax for runtime expressions. We then use the peggy (opens in a new tab) npm package to generate a parser from the Peggy grammar:

cd packages/arazzo-test-gen
peggy --dts --output runtime-expression/runtimeExpression.js --format es runtime-expression/runtimeExpression.peggy

This generates a runtime-expression/runtimeExpression.js file that contains the parser for runtime expressions. We also generate TypeScript type definitions in runtime-expression/runtimeExpression.d.ts.

The parser reads a runtime expression like $response.body#/robotId and breaks it down into tokens. We then evaluate the tokens to resolve the expression at runtime.

Evaluating runtime expressions

Once we’ve parsed a runtime expression, we need to evaluate it to get the value it references. For example, given the expression $response.body#/robotId, we need to extract the robotId field from the response body.

The evaluateRuntimeExpression function in utils.ts handles this evaluation. Here’s an example of how it works:

switch (root) {
  case "$statusCode": {
    // Handle $statusCode expressions
    result = factory.createPropertyAccessExpression(
      factory.createIdentifier("response"),
      factory.createIdentifier("status"),
    );
    break;
  }
  case "$response.": {
    // Handle $request and $response expressions
    const data = factory.createIdentifier("data");
    // use parseRef to handle everything after $response.body
    const pointer = parsePointer(expression.slice(15));
    result = pointer.length > 0
      ? factory.createPropertyAccessExpression(data, pointer.join("."))
      : data;
    break;
  }
  // Handle other cases ...
}

Here, we handle two types of runtime expressions: $statusCode and $response.body. We extract the status field from the response object for $statusCode, and the body object from the response object for $response.body.

We use the TypeScript compiler API to generate an abstract syntax tree (AST) that represents the expression. This AST is then printed to a string and saved as a source file that Deno can execute.

Supported runtime expressions

In our demo, we support a limited set of runtime expressions:

• $statusCode: The HTTP status code of the response
• $steps.stepId.outputs.field: The output of a previous step
• $response.body#/path/to/field: A field in the response body selected by a JSON pointer

Arazzo supports many more runtime expressions, for example:

Parsing regular expressions

Regular expressions in Arazzo are used to validate string values against patterns. They’re particularly useful for validating IDs, dates, and other structured strings.

In our implementation, we handle regex patterns in the parseRegexCondition function:

function parseRegexCondition(
  condition: string,
  usedAssertions: Set<string>,
  context: string,
): Expression {
  usedAssertions.add("assertMatch");
  return factory.createCallExpression(
    factory.createIdentifier("assertMatch"),
    undefined,
    [
      evaluateRuntimeExpression(context),
      factory.createNewExpression(
        factory.createIdentifier("RegExp"),
        undefined,
        [factory.createRegularExpressionLiteral(condition)],
      ),
      factory.createStringLiteral(condition),
    ],
  );
}

This function takes three parameters:

• condition: The regex pattern to match against
• usedAssertions: A set to track which assertion functions we’ve used
• context: The runtime expression that selects the value to validate

The function generates code that:

1. Evaluates the context expression to get the value to validate
2. Creates a new RegExp object from the pattern
3. Uses Deno’s assertMatch function to validate the value against the pattern

The generated code looks like this:

assertMatch(
  response.body.robotId,
  new RegExp(/^[0-9A-F]{8}-[0-9A-F]{4}-[4][0-9A-F]{3}-[89AB][0-9A-F]{3}-[0-9A-F]{12}$/i),
  'robotId should be a valid UUID v4'
);

This approach has several advantages:

• It preserves the original pattern’s flags (like i for case-insensitive matching).
• It provides clear error messages when validation fails.
• It integrates well with Deno’s testing framework.

In a production implementation, we’d want to add:

• Validation of the regex pattern syntax
• Support for named capture groups
• Error handling for malformed patterns
• Performance optimizations like pattern caching

But for our demo, this simple implementation is sufficient to show how regex validation works in Arazzo.

Parsing JSONPath expressions

JSONPath expressions are a powerful way to query JSON data. In Arazzo, we use them in success criteria to select objects or values from complex response structures. While JSON Pointer (which we’ll cover next) is great for accessing specific values, JSONPath shines when you need to:

• Validate arrays (for example, checking array length)
• Filter elements (for example, finding items matching a condition)
• Access deeply nested data with wildcards
• Aggregate values (for example, counting matches)

Here’s how our test generator handles JSONPath expressions:

function parseJsonPathExpression(path: string, context: string): Expression {
  return factory.createCallExpression(
    factory.createIdentifier("JSONPath"),
    undefined,
    [
      factory.createObjectLiteralExpression(
        [
          factory.createPropertyAssignment(
            factory.createIdentifier("wrap"),
            factory.createFalse(),
          ),
          factory.createPropertyAssignment(
            factory.createIdentifier("path"),
            factory.createStringLiteral(path),
          ),
          factory.createPropertyAssignment(
            factory.createIdentifier("json"),
            evaluateRuntimeExpression(context),
          ),
        ],
        true,
      ),
    ],
  );
}

This function generates code that evaluates a JSONPath expression against a context object (usually the response body). For example, given this success criterion:

successCriteria:
  - context: $response.body#/links
    condition: $.length == 5
    type: jsonpath

Our generator creates a test that:

1. Extracts the links array from the response body using a JSON pointer
2. Evaluates the JSONPath expression $.length against this array
3. Compares the result to the expected value 5

The generated test code looks something like this:

assertEquals(
  JSONPath({
    wrap: false,
    path: "$.length",
    json: response.body.links
  }),
  5,
  "links array should contain exactly 5 elements"
);

JSONPath is particularly useful for validating:

• Array operations: $.length, $[0], $[(@.length-1)]
• Deep traversal: $..name (all name properties at any depth)
• Filtering: $[?(@.status=="active")] (elements where status is active)
• Wildcards: $.*.name (name property of all immediate children)

A few things to keep in mind when using JSONPath:

1. JSONPath isn’t well standardized, so different implementations vary widely. Arazzo makes provisions for this by allowing us to specify the JSONPath version in the test specification.
2. Even though we can specify a version, we still need to be cautious when using advanced features. Some features might not be supported by the chosen JSONPath library.
3. Check the JSONPath comparison (opens in a new tab) page to see how different libraries handle various features, and decide which features are safe to use.

Parsing JSON Pointers

While JSONPath is great for complex queries, JSON Pointer (RFC 6901) (opens in a new tab) is perfect for directly accessing specific values in a JSON document. In Arazzo, we use JSON Pointers in runtime expressions to extract values from responses and pass them to subsequent steps.

Here’s how our test generator handles JSON Pointers:

function evaluateRuntimeExpression(expression: string): Expression {
  // ...
  case "$response.": {
    const data = factory.createIdentifier("data");
    // Parse everything after $response.body
    const pointer = parsePointer(expression.slice(15));
    result = pointer.length > 0
      ? factory.createPropertyAccessExpression(data, pointer.join("."))
      : data;
    break;
  }
  // ...
}

This function parses runtime expressions that use JSON Pointers. For example, given this output definition:

outputs:
  robotId: $response.body#/robotId

Our generator creates code that:

1. Takes the part after # as the JSON Pointer (/robotId)
2. Converts the pointer segments into property access expressions
3. Generates code to extract the value

The generated test code looks something like this:

// During test setup
const context = {};
// ...
// In the first test
const data = response.json();
// Generated because of outputs: { robotId: $response.body#/robotId } in the Arazzo document
// highlight-next-line
context["createRobot.outputs.robotId"] = data.robotId;
// ...
// In a subsequent test
const robotId = context["createRobot.outputs.robotId"];
const response = await fetch(`${serverUrl}/v1/robots/${robotId}/assemble`, {
  // ...
});

Generating end-to-end tests

Now that we understand how to parse Arazzo documents, let’s look at how we generate executable tests from them. Our generator creates type-safe test code using TypeScript’s factory methods rather than string templates, providing better error detection and maintainability.

Test structure

The generator creates a test suite for each workflow in the Arazzo document. Each step in the workflow becomes a test case that executes sequentially.

Let’s explore the structure of a generated test case.

/**
 * This test file was generated from an arazzo.yaml document.
 * Do not edit this file manually.
 */
import "jsr:@std/dotenv/load";
import { JSONPath } from "npm:jsonpath-plus";
import { assertEquals, assertMatch } from "jsr:@std/assert";

Deno.test("Create, assemble, and activate a new robot", async (test) => {
  const serverUrl = "http://localhost:8080";
  const apiKey = Deno.env.get("BUILDABOT_API_KEY_AUTH");
  const context: Record<string, unknown> = {};
  await test.step("Create a new robot design session", async () => {
    const response = await fetch(`${serverUrl}/v1/robots`, {
      method: "POST",
      headers: {
        "X-API-Key": `${apiKey}`,
        "Content-Type": "application/json",
      },
      body: '{"model":"humanoid","name":"MyFirstRobot"}',
    });
    const data = await response.json();
    assertEquals(response.status, 201, "$statusCode == 201");
    assertMatch(
      data.robotId,
      new RegExp(
        /^[0-9A-F]{8}-[0-9A-F]{4}-[4][0-9A-F]{3}-[89AB][0-9A-F]{3}-[0-9A-F]{12}$/i,
      ),
      "/^[0-9A-F]{8}-[0-9A-F]{4}-[4][0-9A-F]{3}-[89AB][0-9A-F]{3}-[0-9A-F]{12}$/i",
    );
    assertEquals(data.model, "humanoid", '$response.body#/model == "humanoid"');
    assertEquals(
      data.name,
      "MyFirstRobot",
      '$response.body#/name == "MyFirstRobot"',
    );
    assertEquals(
      data.status,
      "designing",
      '$response.body#/status == "designing"',
    );
    assertEquals(
      JSONPath({
        wrap: false,
        path: "$.length",
        json: data.links,
      }),
      5,
      "$.length == 5",
    );
    context["createRobot.outputs.robotId"] = data.robotId;
  });

  await test.step("Add parts to the robot", async () => {
    const robotId = context["createRobot.outputs.robotId"];
    const response = await fetch(`${serverUrl}/v1/robots/${robotId}/parts`, {
      method: "POST",
      headers: {
        "X-API-Key": `${apiKey}`,
        "Content-Type": "application/json",
      },
      body:
        '{"parts":[{"type":"arm","name":"Hydraulic Arm","quantity":2},{"type":"sensor","name":"Infrared Sensor","quantity":1}]}',
    });
    const data = await response.json();
    assertEquals(response.status, 200, "$statusCode == 200");
    assertEquals(
      data.robotId,
      context["createRobot.outputs.robotId"],
      "$response.body#/robotId == $steps.createRobot.outputs.robotId",
    );
  });

  await test.step("Assemble the robot", async () => {
    const robotId = context["createRobot.outputs.robotId"];
    const response = await fetch(`${serverUrl}/v1/robots/${robotId}/assemble`, {
      method: "POST",
      headers: {
        "X-API-Key": `${apiKey}`,
        "Content-Type": "application/json",
      },
      body: null,
    });
    const data = await response.json();
    assertEquals(response.status, 200, "$statusCode == 200");
    assertEquals(
      data.robotId,
      context["createRobot.outputs.robotId"],
      "$response.body#/robotId == $steps.createRobot.outputs.robotId",
    );
    assertEquals(
      data.status,
      "assembled",
      '$response.body#/status == "assembled"',
    );
  });

  await test.step("Configure robot features", async () => {
    const robotId = context["createRobot.outputs.robotId"];
    const response = await fetch(`${serverUrl}/v1/robots/${robotId}/features`, {
      method: "POST",
      headers: {
        "X-API-Key": `${apiKey}`,
        "Content-Type": "application/json",
      },
      body:
        '{"features":[{"name":"Voice Recognition","description":"Enables voice command recognition."},{"name":"Obstacle Avoidance","description":"Navigates around obstacles."}]}',
    });
    const data = await response.json();
    assertEquals(response.status, 200, "$statusCode == 200");
    assertEquals(
      data.robotId,
      context["createRobot.outputs.robotId"],
      "$response.body#/robotId == $steps.createRobot.outputs.robotId",
    );
  });

  await test.step("Activate the robot", async () => {
    const robotId = context["createRobot.outputs.robotId"];
    const response = await fetch(`${serverUrl}/v1/robots/${robotId}/activate`, {
      method: "POST",
      headers: {
        "X-API-Key": `${apiKey}`,
        "Content-Type": "application/json",
      },
      body: null,
    });
    const data = await response.json();
    assertEquals(response.status, 200, "$statusCode == 200");
    assertEquals(
      data.robotId,
      context["createRobot.outputs.robotId"],
      "$response.body#/robotId == $steps.createRobot.outputs.robotId",
    );
    assertEquals(
      data.status,
      "activated",
      '$response.body#/status == "activated"',
    );
    context["activateRobot.outputs.activationTime"] = data.activationTime;
    context["activateRobot.outputs.capabilities"] = data.capabilities;
  });

  await test.step("Get the robot details", async () => {
    const robotId = context["createRobot.outputs.robotId"];
    const response = await fetch(`${serverUrl}/v1/robots/${robotId}`, {
      method: "GET",
      headers: {
        "X-API-Key": `${apiKey}`,
        "Content-Type": "application/json",
      },
      body: null,
    });
    const data = await response.json();
    assertEquals(response.status, 200, "$statusCode == 200");
    assertEquals(
      data.robotId,
      context["createRobot.outputs.robotId"],
      "$response.body#/robotId == $steps.createRobot.outputs.robotId",
    );
    assertEquals(
      data.status,
      "activated",
      '$response.body#/status == "activated"',
    );
    assertEquals(
      data.activationTime,
      context["activateRobot.outputs.activationTime"],
      "$response.body#/activationTime == $steps.activateRobot.outputs.activationTime",
    );
    assertEquals(
      data.capabilities,
      context["activateRobot.outputs.capabilities"],
      "$response.body#/capabilities == $steps.activateRobot.outputs.capabilities",
    );
    context["getRobot.outputs.robot"] = data;
  });
});

This structure repeats for each step in the workflow, creating a series of test cases that execute the workflow sequentially. The generated tests validate the API’s behavior at each step, ensuring that the workflow progresses correctly.

Future development and improvements

Our generated tests are a good start, but they might not be truly end-to-end if we don’t consider the interfaces our users interact with to access the API.

Testing with SDKs

In our demo, we use the fetch API to interact with the Build-a-bot API. While this is a common approach, it’s not always the most user-friendly. Developers often prefer SDKs that provide a more idiomatic interface to the API.

To make our tests more end-to-end, we could use the SDK Speakeasy created from the OpenAPI document to interact with the API.

Since the SDK is generated from the OpenAPI document, with names and methods derived from the API’s tags and operation IDs, we could use Arazzo to validate the SDK’s behavior against the API’s capabilities.

For example, we could:

1. Get the operationId from the Arazzo step and derive the corresponding SDK method import.
2. Call the SDK method with the required parameters.
3. Validate the response against the success criteria.
4. Extract the outputs from the response and store them in the context object.
5. Repeat for each step in the workflow.

This approach would provide a more realistic end-to-end test, validating the SDK’s behavior against the API’s capabilities.

Handling authentication

In our demo, we use a hard-coded API key for authentication. In a real-world scenario, we’d likely need to authenticate with the API to get a valid API key.

OpenAPI also supports more advanced authentication schemes like OAuth 2.0, JWT, and API key in headers, query parameters, or cookies. Our test generator should handle these schemes to ensure the tests are realistic and cover all authentication scenarios.

Arazzo can point to the security schemes in the OpenAPI document, allowing us to extract the required authentication parameters and set them up in the test suite.

Hardening the parsers against vulnerabilities

Our parsers are simple and work well for the demo, but they lack robust error handling and edge case coverage.

For example, JSONPath-plus, the library we use for JSONPath, recently fixed a remote code execution vulnerability. We should ensure our parser is up to date and secure against similar vulnerabilities, or limit the JSONPath features we support to reduce the attack surface.

This applies to parsers in general, and the risk is even higher when parsing user input and generating code from it.

Deno provides some protection by limiting access to the filesystem and network by default, but the nature of API testing means we need to access the network and read files.

Where to next?

The Arazzo specification, although released as v1.0.0, is in active development. The OpenAPI team is working on a JSON Schema for Arazzo, which will provide a formal definition of the specification’s structure and constraints.

We found the specification slightly ambiguous in places, but the team is active on GitHub (opens in a new tab) and open to feedback and contributions. If you’re interested in API testing, Arazzo is a great project to get involved with.

At Speakeasy, we’re building tools to make API testing easier and more effective. Our TypeScript, Python, and Go SDK generators can already generate tests from OpenAPI documents, and we’re working on integrating Arazzo support. Our CLI can already lint Arazzo documents, and we’ll have more to share soon.

We’re excited to see how Arazzo evolves and how it can help developers build robust, end-to-end tests for their APIs.