API Advice

How To Build A Best In Class Python SDK

Tristan Cartledge

April 29, 2024

This tutorial demonstrates one way to code and package a Python SDK.

Why Are We Building an SDK?

Since the 2000s, HTTP APIs have enabled businesses to be more connected than ever and, as a result, produce higher utility and profit — no wonder APIs have exploded in popularity.

So, your business has created an API? Great! At first, developers might use curl or create a Postman collection to experiment with your API. However, the true power of an API is only unleashed when developers can interact with it in an automated and programmatic way. To get some real work done, you’ll want to wrap the HTTP API in your language of choice, so that it can be testable and have the most important aspects abstracted for your convenience and productivity.

What Are We Building?

We’ll build a Python SDK that wraps an HTTP API.

Fictional e-commerce startup ECOM enables the selling of products through the creation of online stores and associated products on its website. The ECOM HTTP API enables developers to create stores and manage products in an automated way. We want to give Python programmers easy access to the ECOM API with a robust, secure, and user-friendly SDK that handles the underlying HTTP API service in a type-safe, programmatic way.

Example SDK Repository

You can follow along without downloading our complete example repository, but if you get stuck or want to see the final result, you can find the code on GitHub at speakeasy-api/ecom-sdk (opens in a new tab).

Initialize the Project

Before we begin coding the SDK, let’s set up our project.

Install pyenv

While not strictly necessary in all cases, we prefer to manage Python versions with pyenv. If you’re building the SDK with us, install pyenv (opens in a new tab) for your operating system.

On macOS, you can install pyenv with Homebrew:

brew install pyenv

Decide on a Python Version

For this tutorial, we’ll use Python 3.8, because it is the oldest version that still receives security updates.

Install Python 3.8

Install Python 3.8 with pyenv:

pyenv install 3.8.19

Set Up a Python Virtual Environment

We’ll set up a Python virtual environment to isolate our project dependencies from the system Python installation.

Create a new Python 3.8 virtual environment:

pyenv virtualenv 3.8.19 ecom-sdk

Activate the virtual environment:

pyenv activate ecom-sdk

After activating the virtual environment, depending on the shell you’re using, your shell prompt might change to indicate that you are now using the virtual environment. Any Python packages you install will be installed in the virtual environment and won’t affect the system Python installation.

Upgrade pip

When you create a new virtual environment, it’s good practice to upgrade pip to the latest version:

python3.8 -m pip install --upgrade pip

Decide on a Build Backend

In the past, Python used a setup.py or setup.cfg configuration file to prepare Python packages for distribution, but new standards (like PEPs 517, 518, 621, and 660) and build tools are modernizing the Python packaging landscape. We want to configure a modern build backend for our SDK and we have many options to choose from:

• Hatchling (opens in a new tab)
• Poetry (opens in a new tab)
• PDM (opens in a new tab)
• Setuptools (opens in a new tab)
• Flit (opens in a new tab)
• Maturin (opens in a new tab)

Some factors to consider when selecting a build backend include:

• Python version support: Some build backends only support a subset of Python versions, so ensure your chosen build backend supports the correct versions.
• Features: Determine the features you need and choose a backend that meets your requirements. For example, do you need features like project management tools, or package uploading and installation capabilities?
• Extension support: Do you need support for extension modules in other languages?

We’ll use Hatchling (opens in a new tab) in this tutorial for its convenient defaults and ease of configurability.

Set Up a Project With Hatchling

First, install Hatch:

pip install hatch

Now, create a new project with Hatch:

hatch new -i

The -i flag tells Hatch to ask for project information interactively. Enter the following information when prompted:

Project name: ECOM SDK
Description: Python SDK for the ECOM HTTP API

Hatch will create a new project directory with the name ecom-sdk and initialize it with a basic pyproject.toml file.

Change into the project directory:

cd ecom-sdk

See the Project Structure

Run tree to see the project structure:

tree .

The project directory should now contain the following files:

.
├── LICENSE.txt
├── README.md
├── pyproject.toml
├── src
│   └── ecom_sdk
│       ├── __about__.py
│       └── __init__.py
└── tests
    └── __init__.py

4 directories, 6 files

The LICENSE.txt file contains the MIT license, the README.md file has a short installation hint, and the pyproject.toml file contains the project metadata.

Update the README File

If you continue to support and expand this SDK, you’ll want to keep the README.md file up to date with the latest documentation, providing installation and usage instructions, and any other information relevant to your users. Without a good README, developers won’t know where to start and won’t use the SDK.

For now, let’s add a short description of what the SDK does:

# ECOM SDK

Python SDK for the ECOM HTTP API

Create a Basic SDK

Add a ./src/ecom_sdk/sdk.py file containing Python code, for example:

def sdkFunction():
    return 1

We’ll use the ./src/ecom_sdk/sdk.py file to ensure our testing setup works.

Create a Test

As we add functionality to the SDK, we will populate the ./tests directory with tests.

For now, create a ./src/ecom_sdk/test_sdk.py file containing test code:

import src.ecom_sdk.sdk

def test_sdk():
    assert src.ecom_sdk.sdk.sdkFunction() == 1

Later in this guide, we’ll run the test with scripts provided by Hatch.

Inspect the Build Backend Configuration

The pyproject.toml file contains the project metadata and build backend configuration.

Be sure to take a peek at the pyproject.toml guide (opens in a new tab) and specification (opens in a new tab) for details on all the possible metadata fields available for the [project] table. For example, you may want to enhance the discoverability of your SDK on PyPI by specifying keyword metadata or additional classifiers.

Test Hatch

Here’s the Hatch test script that we’ll use to run tests:

hatch run test

The first time you run the test script, Hatch will install the necessary dependencies and run the tests. Subsequent runs will be faster because the dependencies are already installed.

After running the test script, you should see output similar to the following:

========================= test session starts ==========================
platform darwin -- Python 3.12.3, pytest-8.1.1, pluggy-1.5.0
rootdir: /speakeasy/ecom-sdk
configfile: pyproject.toml
collected 1 item

tests/test_sdk.py .                                              [100%]

========================== 1 passed in 0.00s ===========================

Code the SDK

Now that we have the project set up, let’s start coding the SDK.

Add an SDK Method To Fetch a List of Stores

In the ./src/ecom_sdk/sdk.py file, define a class, constructor, and list_stores() method for fetching a list of stores from our API:

import requests

HTTP_TIMEOUT_SECONDS = 10


class EComSDK:
    def __init__(self, api_url, api_key):
        self.api_url = api_url
        self.api_key = api_key

    def list_stores(self):
        r = requests.get(
            self.api_url + "/store",
            headers={"X-API-KEY": self.api_key},
            timeout=HTTP_TIMEOUT_SECONDS,
        )
        if r.status_code == 200:
            return r.json()
        else:
            raise Exception("Invalid response status code: " +
                            str(r.status_code))

Now we can begin writing tests in the ./tests/test_sdk.py file to test that the newly added list_stores() method works as intended.

In requests.get(), we use HTTP_TIMEOUT_SECONDS to set a maximum bound for waiting for the request to finish. If we don’t set a timeout, the requests library will wait for a response forever.

Add the following to ./tests/test_sdk.py:

from src.ecom_sdk.sdk import EComSDK
import responses
from responses import matchers

api_url = "https://example.com"
api_key = "hunter2"


def test_sdk_class():
    sdk = EComSDK(api_url, api_key)
    assert sdk.api_url == api_url
    assert sdk.api_key == api_key


@responses.activate
def test_sdk_list_stores():
    responses.add(
        responses.GET,
        api_url + "/store",
        json=[
            {"id": 1, "name": "Lidl", "products": 10},
            {"id": 2, "name": "Walmart", "products": 15},
        ],
        status=200,
        match=[matchers.header_matcher({"X-API-KEY": api_key})],
    )

    sdk = EComSDK(api_url, api_key)
    stores = sdk.list_stores()

    assert len(stores) == 2
    assert stores[0]["id"] == 1
    assert stores[0]["name"] == "Lidl"
    assert stores[1]["id"] == 2
    assert stores[1]["name"] == "Walmart"

Here we use the responses library to mock API responses since we don’t have a test or staging version of the ECOM HTTP API. However, even with the ECOM API, we might choose to have some part of the testing strategy mock the API responses as this approach can be faster and tightly tests code without external dependencies.

The test_sdk_class() test function checks that we can instantiate the class and the correct values are set internally.

The test_sdk_list_stores() test function makes a call to sdk.list_stores() to test that it receives the expected response, which is a JSON array of stores associated with the user’s account.

Add Dependencies

We need to add the requests and responses libraries to the project dependencies. Add the following to the pyproject.toml file:

dependencies = [
  "requests",
  "responses",
  "pydantic",
]

Run the Tests

Let’s run the Hatch test script we configured earlier to check that everything is working correctly:

hatch run test

Once Hatch has installed our new dependencies, you should see output similar to the following:

=============================== test session starts ================================
platform darwin -- Python 3.12.3, pytest-8.1.1, pluggy-1.5.0
rootdir: /speakeasy/ecom-sdk
configfile: pyproject.toml
collected 2 items

tests/test_sdk.py ..                                                         [100%]

================================ 2 passed in 0.09s =================================

Exception Handling

To create a pleasant surface for our SDK users, we’ll hide details of HTTP implementation behind exception handling and provide helpful error messages should things go wrong.

Let’s modify the list_stores() function to catch some more common errors and print helpful information for the user:

import requests

HTTP_TIMEOUT_SECONDS = 10


class EComSDK:
    def __init__(self, api_url, api_key):
        self.api_url = api_url
        self.api_key = api_key

    def list_stores(self):
        try:
            r = requests.get(
                self.api_url + "/store",
                headers={"X-API-KEY": self.api_key},
                timeout=HTTP_TIMEOUT_SECONDS,
            )
            r.raise_for_status()
        except requests.exceptions.ConnectionError as err:
            raise ValueError("Connection error, check `EComSDK.api_url` is set correctly") from err
        except requests.exceptions.HTTPError as err:
            if err.response.status_code == 403:
                raise ValueError("Authentication error, check `EComSDK.api_key` is set correctly") from err
            else:
                raise

        return r.json()

When an exception is raised from the call to requests.get(), we catch the exception, add useful information, and then re-raise the exception for the SDK user to handle in their code.

The call to r.raise_for_status() modifies the requests library behavior to raise an HTTPError if the HTTP response status code is unsuccessful. Read more about raise_for_status() in the Requests library documentation (opens in a new tab).

Now we’ll add tests for a 403 response and a connection error to the test_sdk.py file. We’ll also import the requests library into the test_sdk.py file:

import requests

# ...

@responses.activate
def test_sdk_list_stores_connection_error():
    responses.add(
        responses.GET,
        api_url + "/store",
        body=requests.exceptions.ConnectionError(),
    )

    sdk = EComSDK(api_url, api_key)
    try:
        sdk.list_stores()
    except ValueError as err:
        assert "Connection error" in str(err)
    else:
        assert False, "Expected ValueError"


@responses.activate
def test_sdk_list_stores_403():
    responses.add(
        responses.GET,
        api_url + "/store",
        status=403,
    )

    sdk = EComSDK(api_url, api_key)
    try:
        sdk.list_stores()
    except ValueError as err:
        assert "Authentication error" in str(err)
    else:
        assert False, "Expected ValueError"

This test ensures that the helpful error message from our SDK is displayed when a connection error or 403 response occurs.

Check that the new code passes the test:

hatch run test

Type Safety With Enums

To make development easier and less error-prone for our SDK users, we’ll define Enums to use with the list_products endpoint.

Add the following Enums to the EComSDK class:

import requests
from enum import Enum

HTTP_TIMEOUT_SECONDS = 10

class EComSDK:
    # ...

    class ProductSort(str, Enum):
        PRICE = "price"
        QUANTITY = "quantity"

    class ProductSortOrder(str, Enum):
        DESC = "desc"
        ASC = "asc"

    # ...

The ProductSort and ProductSortOrder Enums inherit from the str class and can be used with the list_products() method we’ll define next – ProductSort specifies the sort category, and ProductSortOrder determines the sort order of the product list.

Now define the list_products() function at the bottom of the EComSDK class:

# ...
class EComSDK:
    # ...
    def list_products(self, store_id, sort_by=ProductSort.PRICE, sort_order=ProductSortOrder.ASC):
        try:
            r = requests.get(
                self.api_url + f"/store/{store_id}/product",
                headers={"X-API-KEY": self.api_key},
                params={"sort_by": sort_by, "sort_order": sort_order},
                timeout=HTTP_TIMEOUT_SECONDS,
            )
            r.raise_for_status()
        except requests.exceptions.ConnectionError as err:
            raise ValueError("Connection error, check `EComSDK.api_url` is set correctly") from err
        except requests.exceptions.HTTPError as err:
            if err.response.status_code == 403:
                raise ValueError("Authentication error, check `EComSDK.api_key` is set correctly") from err
            else:
                raise

        return r.json()

The list_products() function accepts named parameters sort_by and sort_order. If the list_products() function is called with named parameters, the parameters are added to the params dictionary to be included in the request’s HTTP query parameters.

An SDK user can call the list_products() function with the string list_products(sort_by="quantity") or with the safer equivalent using Enums: list_products(sort_by=sdk.ProductSort.QUANTITY). By encouraging the user to use Enums in this way, we prevent HTTP error requests that don’t clearly identify what went wrong should a user mistype a string, for example, “prise” instead of “price”.

Now add another test at the bottom of the test_sdk.py file:

# ...

@responses.activate
def test_sdk_list_products_sort_by_price_desc():
    store_id = 1
    responses.add(
        responses.GET,
        api_url + f"/store/{store_id}/product",
        json=[
            {"id": 1, "name": "Banana", "price": 0.5},
            {"id": 2, "name": "Apple", "price": 0.3},
        ],
        status=200,
        match=[matchers.header_matcher({"X-API-KEY": api_key})],
    )

    sdk = EComSDK(api_url, api_key)
    products = sdk.list_products(store_id, sort_by=EComSDK.ProductSort.PRICE, sort_order=EComSDK.ProductSortOrder.DESC)

    assert len(products) == 2
    assert products[0]["id"] == 1
    assert products[0]["name"] == "Banana"
    assert products[1]["id"] == 2
    assert products[1]["name"] == "Apple"

Check that the new test passes:

hatch run test

Output Type Safety With Pydantic

To make the SDK even more user-friendly, we can use Pydantic to create data models for the responses from the ECOM API.

First, add Pydantic to the project dependencies in the pyproject.toml file:

dependencies = [
  "requests",
  "responses",
  "pydantic",
]

Now, create a Product model in the ./src/ecom_sdk/models.py file:

from pydantic import BaseModel

class Product(BaseModel):
    id: int
    name: str
    price: float

Next, import the Product model into the ./src/ecom_sdk/sdk.py file:

from .models import Product

Modify the list_products() function to return a list of Product models:

# ...
class EComSDK:
    # ...
    def list_products(self, store_id, sort_by=ProductSort.PRICE, sort_order=ProductSortOrder.ASC):
        try:
            r = requests.get(
                self.api_url + f"/store/{store_id}/product",
                headers={"X-API-KEY": self.api_key},
                params={"sort_by": sort_by, "sort_order": sort_order},
                timeout=HTTP_TIMEOUT_SECONDS,
            )
            r.raise_for_status()
        except requests.exceptions.ConnectionError as err:
            raise ValueError("Connection error, check `EComSDK.api_url` is set correctly") from err
        except requests.exceptions.HTTPError as err:
            if err.response.status_code == 403:
                raise ValueError("Authentication error, check `EComSDK.api_key` is set correctly") from err
            else:
                raise

        return [Product(**product) for product in r.json()]

The list_products() function now returns a list of Product models created from the JSON response.

Now update the test_sdk.py file to test the new Product model:

# ...
from src.ecom_sdk.models import Product

# ...

@responses.activate
def test_sdk_list_products_sort_by_price_desc():
    store_id = 1
    responses.add(
        responses.GET,
        api_url + f"/store/{store_id}/product",
        json=[
            {"id": 1, "name": "Banana", "price": 0.5},
            {"id": 2, "name": "Apple", "price": 0.3},
        ],
        status=200,
        match=[matchers.header_matcher({"X-API-KEY": api_key})],
    )

    sdk = EComSDK(api_url, api_key)
    products = sdk.list_products(store_id, sort_by=EComSDK.ProductSort.PRICE, sort_order=EComSDK.ProductSortOrder.DESC)

    assert len(products) == 2
    assert products[0].id == 1
    assert products[0].name == "Banana"
    assert products[1].id == 2
    assert products[1].name == "Apple"
    assert isinstance(products[0], Product)
    assert isinstance(products[1], Product)

We’ll also need to update the test_sdk_list_products() test to check that the Product models are returned from the list_products() function.

# ...

    assert len(products) == 2
    assert products[0].id == 1
    assert products[0].name == "Banana"
    assert products[0].price == 0.5
    assert products[1].id == 2
    assert isinstance(products[0], Product)
    assert isinstance(products[1], Product)

Check that the new tests pass:

hatch run test

We’ve now added type safety to the SDK using Pydantic, ensuring that the SDK user receives a list of Product models when calling the list_products() function. The same principle can be applied to other parts of the SDK to ensure that the user receives the correct data types.

Add Type Safety to the SDK Constructor

To ensure that the SDK user provides the correct data types when instantiating the EComSDK class, we can use Pydantic to create a Config model for the SDK constructor.

First, create a Config model in the ./src/ecom_sdk/models.py file:

from pydantic import BaseModel

class Config(BaseModel):
    api_url: str
    api_key: str

Next, import the Config model into the ./src/ecom_sdk/sdk.py file:

from .models import Config

Now, modify the EComSDK class to accept a Config model in the constructor:

# ...
class EComSDK:
    def __init__(self, config: Config):
        self.api_url = config.api_url
        self.api_key = config.api_key

    # ...

The EComSDK class now accepts a Config model in the constructor, ensuring that the SDK user provides the correct data types when instantiating the class.

Let’s update the test_sdk.py file to test the new Config model:

# ...

from src.ecom_sdk.models import Config

# ...

def test_sdk_class():
    config = Config(api_url="https://example.com", api_key="hunter2")
    sdk = EComSDK(config)
    assert sdk.api_url == config.api_url
    assert sdk.api_key == config.api_key

Follow the same pattern to update the other tests in the test_sdk.py file to use the new Config model. Replace sdk = EComSDK(api_url, api_key) with:

# ...
    config = Config(api_url=api_url, api_key=api_key)
    sdk = EComSDK(config)
# ...

Check that the new tests pass:

hatch run test

Things to Consider

We’ve described some basic steps for creating a custom Python SDK, but there are many facets to an SDK project besides code that contribute to its success or failure. For any publicly available Python library, you should consider such aspects as documentation, linting, and licensing.

Documentation

To focus on the nuts and bolts of a custom Python SDK, this guide does not cover developing an SDK’s documentation. But documentation is critical to ensure your users can easily pick up your library and use it productively. Without good documentation, developers may opt not to use your SDK at all.

All SDK functions, parameters, and behaviors should be documented, and creating beautiful, functional documentation is an essential part of making your SDK usable. You can roll your own documentation and add Markdown to the project README.md file or use a popular library like Sphinx (opens in a new tab) that includes such features as automatic highlighting, themes, and HTML or PDF outputs.

Linting

Consistently formatted code improves readability and encourages contributions that are not jarring in the context of the existing codebase. While this guide doesn’t cover linting, linting your SDK code should form part of creating an SDK.

For best results, linting should be as far-reaching and opinionated as possible, with little to zero default configuration required. Some popular options for linting include Flake8 (opens in a new tab), Ruff (opens in a new tab), and Black (opens in a new tab).

Licensing

A project’s license can significantly influence the type and number of potential open-source contributors. Choosing the right license for your SDK is key. Do you want your SDK’s license to be community-orientated? Simple and permissive? To preserve a particular philosophy on enforcing the sharing of code modifications? In this example, we selected the MIT license for simplicity and ease of use. If you’re unsure which license would best suit your needs, consider using a tool like Choose a license (opens in a new tab).

Other Endpoints

This tutorial covered a basic SDK for two endpoints. Consider that real-world SDKs can have many more endpoints and features. You can use the same principles to add more endpoints to your SDK.

Authentication Methods

If APIs use OAuth 2.0 or other authentication methods, you’ll need to add authentication methods to your SDK. You can use libraries like Authlib (opens in a new tab) to handle OAuth 2.0 authentication.

Pagination

If the API returns paginated results, you’ll need to handle pagination in your SDK.

Retries and Backoff

If the API is rate-limited or has intermittent failures, you’ll need to add retry and backoff logic to your SDK.

Build the Package

Now that we have a working SDK, we can build the package for distribution.

hatch build

Hatch will build the package and output the distribution files:

────────────────────────────────────── sdist ───────────────────────────────────────
dist/ecom_sdk-0.0.1.tar.gz
────────────────────────────────────── wheel ───────────────────────────────────────
dist/ecom_sdk-0.0.1-py3-none-any.whl

You should now have a ./dist directory containing your source (.tar.gz) and built (.whl) distributions:

.
├── dist
│   ├── ecom_sdk-0.0.1-py3-none-any.whl
│   └── ecom_sdk-0.0.1.tar.gz

Upload the SDK to the Distribution Archives

We now have just enough to upload the SDK to the PyPI test repo at test.pypi.org.

To upload the build to TestPyPI, you need:

1. A test PyPI account. Go to https://test.pypi.org/account/register/ (opens in a new tab) to register.

2. A PyPI API token. Create one at https://test.pypi.org/manage/account/#api-tokens (opens in a new tab), setting the “Scope” to “Entire account”.

If you plan to automate SDK publishing in your CI/CD pipeline, you should create per-project tokens. For automated releasing using CI/CD, it is recommended that you create per-project API tokens.

Finally, publish the package distribution files by executing:

hatch publish -r test

The -r test switch specifies that we are using the TestPyPI repository.

Hatch will ask for a username and credentials. Use __token__ as the username (to indicate that we are using a token value rather than a username) and paste your PyPI API token in the credentials field.

hatch publish -r test                                                                                                                                            ⇐ [15:13]═
Enter your username [__token__]: __token__
Enter your credentials: (paste your token here)
dist/ecom_sdk-0.0.1-py3-none-any.whl ... success
dist/ecom_sdk-0.0.1.tar.gz ... success

[ecom-sdk]
https://test.pypi.org/project/ecom-sdk/0.0.1/

Your shiny new package is now available to all your new Python customers!

The ecom-sdk python package can now be installed using:

pip install --index-url https://test.pypi.org/simple/ --no-deps ecom-sdk

We use the --index-url switch to specify the TestPyPI repo instead of the default live repo. We use the --no-deps switch because the test PyPI repo doesn’t have all dependencies (because it’s a test repo) and the pip install command would fail otherwise.

Automatically Create Language-Idiomatic SDKs From Your API Specification

We’ve taken the first few steps in creating a Python SDK but as you can see, we’ve barely scratched the surface. It takes a good deal of work and dedication to iterate on an SDK project and get it built right. Then comes the maintenance phase, dealing with pull requests, and triaging issues from contributors.

You might want to consider automating the creation of SDKs with a managed service like Speakeasy. You can quickly get up and running with SDKs in nine languages with best practices baked in and maintain them automatically with the latest language and security updates.