API Advice

Python HTTP Clients: Requests vs. HTTPX vs. AIOHTTP

Georges Haidar

August 24, 2024

Anyone who’s been using Python for more than a minute has come across the Requests library. It is so ubiquitous, some may have thought it was part of the standard library. Requests is so intuitive that writing r = requests.get has become muscle memory. In contrast, any script using Python’s built-in urllib (opens in a new tab) starts with a trip to the Python docs.

But Python has evolved, and simply defaulting to Requests is no longer an option. While Requests remains a solid choice for short synchronous scripts, newer libraries like HTTPX and AIOHTTP are better suited for modern Python, especially when it comes to asynchronous programming.

Let’s compare these three popular HTTP clients for Python: Requests (opens in a new tab), HTTPX (opens in a new tab), and AIOHTTP (opens in a new tab). We’ll explore their strengths, weaknesses, and ideal use cases to help you choose the right tool for your next project.

In The Beginning, Guido Created Urllib

Before we dive into our comparison of modern HTTP libraries, it’s worth taking a brief look at where it all began: Python’s built-in urllib module.

urllib has been part of Python’s standard library since the early days. It was designed to be a comprehensive toolkit for URL handling and network operations. However, its API is notoriously complex and unintuitive, often requiring multiple steps to perform even simple HTTP requests.

Here’s a basic example of making a GET request with urllib:

from urllib.request import urlopen

with urlopen('https://api.github.com') as response:
    body = response.read()
    print(body)

While this might seem straightforward for a simple GET request, things quickly become more complicated when dealing with headers, POST requests, or authentication. For instance, here’s how you might make a request with authentication:

import urllib.request
import json

url = 'http://httpbin.org/basic-auth/user/passwd'
username = 'user'
password = 'passwd'

# Create an opener with authentication handler
password_mgr = urllib.request.HTTPPasswordMgrWithDefaultRealm()
password_mgr.add_password(None, url, username, password)
auth_handler = urllib.request.HTTPBasicAuthHandler(password_mgr)
opener = urllib.request.build_opener(auth_handler)

# Make the request
with opener.open(url) as response:
    raw_data = response.read()
    encoding = response.info().get_content_charset('utf-8')
    data = json.loads(raw_data.decode(encoding))

print(data)

In this example, we create an authentication handler and an opener to make the request. We then read the response, decode it, and parse the JSON data.

The verbosity and complexity of urllib led to the creation of third-party libraries that aimed to simplify HTTP requests in Python.

Requests: HTTP For Humans™️

In 2011 (on Valentine’s day, no less), Kenneth Reitz released the Requests (opens in a new tab) library, designed to make HTTP requests as human-friendly as possible. After only two years, by July 2013, Requests had been downloaded more than 3,300,000 times (opens in a new tab), and as of August 2024 (opens in a new tab), it gets downloaded around 12 million times a day.

It turns out devex is important after all (opens in a new tab)!

To install Requests, use pip:

pip install requests

Let’s compare the previous urllib examples with their Requests equivalents:

import requests

# GET request
response = requests.get('https://api.github.com')
print(response.text)

# request with auth
url = 'http://httpbin.org/basic-auth/user/passwd'
username = 'user'
password = 'passwd'

response = requests.get(url, auth=(username, password))
data = response.json()

print(data)

The simplicity and readability of Requests code compared to urllib is immediately apparent. Requests abstracts away much of the complexity, handling things like authentication headers and JSON responses with ease.

Some key features that made Requests the de facto standard include:

1. Automatic content decoding: Requests automatically decodes the response content based on the Content-Type header.
2. Session persistence: The Session object allows you to persist certain parameters across requests.
3. Elegant error handling: Requests raises intuitive exceptions for network problems and HTTP errors.
4. Automatic decompression: Requests automatically decompresses gzip-encoded responses.

However, as Python evolved and the use cases for Python expanded, new needs arose that Requests wasn’t designed to address. In particular, Asynchronous rose as a need which led to the introduction of asyncio in Python 3.4.

AIOHTTP: Built for Asyncio

AIOHTTP (opens in a new tab), first released in October 2014, was one of the first libraries to fully embrace Python’s asyncio framework. Designed from the ground up for asynchronous operations, it’s an excellent choice for high-performance, concurrent applications. Today, AIOHTTP is widely used, with around six million downloads per day (opens in a new tab) as of August 2024.

AIOHTTP has several key features that set it apart from Requests:

1. Purely asynchronous: All operations in AIOHTTP are async, allowing for efficient handling of many concurrent connections.
2. Both client and server: AIOHTTP can be used to create both HTTP clients and servers.
3. WebSocket support: It offers full support for WebSocket connections.

Install AIOHTTP using pip:

pip install aiohttp

Here’s a basic example of using AIOHTTP:

import aiohttp
import asyncio

async def fetch(session, url):
    async with session.get(url) as response:
        return await response.text()

async def main():
    async with aiohttp.ClientSession() as session:
        html = await fetch(session, 'https://api.github.com')
        print(html)

asyncio.run(main())

To really test AIOHTTP’s capabilities, you need to run multiple requests concurrently. Here’s an example that fetches multiple URLs concurrently:

import asyncio
import aiohttp
import time

urls = [
    "https://httpbin.org/delay/1",
    "https://httpbin.org/delay/2",
    "https://httpbin.org/delay/3",
    "https://httpbin.org/delay/1",
    "https://httpbin.org/delay/2",
]

async def fetch(session, url, i):
    try:
        start_time = time.perf_counter()
        async with session.get(url) as response:
            await response.text()
            elapsed = time.perf_counter() - start_time
            print(f"Request {i} completed in {elapsed:.2f}s")
    except asyncio.TimeoutError:
        print(f"Request {i} timed out")

async def async_requests():
    start_time = time.perf_counter()
    async with aiohttp.ClientSession(timeout=aiohttp.ClientTimeout(total=10)) as session:
        tasks = [fetch(session, url, i) for i, url in enumerate(urls, 1)]
        await asyncio.gather(*tasks)
    
    total_time = time.perf_counter() - start_time
    print(f"\nTotal time: {total_time:.2f}s")

if __name__ == "__main__":
    asyncio.run(async_requests())

In this example, we’re fetching five URLs concurrently, each with a different server-side delay. The script will output the time taken for each request to complete, as well as the total time taken:

Request 1 completed in 2.22s
Request 4 completed in 2.22s
Request 5 completed in 3.20s
Request 2 completed in 3.20s
Request 3 completed in 4.30s

Total time: 4.31s

For comparison, here’s how you might achieve the same thing using Requests:

import requests
import time

urls = [
    "https://httpbin.org/delay/1",
    "https://httpbin.org/delay/2",
    "https://httpbin.org/delay/3",
    "https://httpbin.org/delay/1",
    "https://httpbin.org/delay/2",
]

def sync_requests():
    start_time = time.time()
    with requests.Session() as session:
        session.timeout = 10.0
        for i, url in enumerate(urls, 1):
            try:
                response = session.get(url)
                print(f"Request {i} completed in {response.elapsed.total_seconds():.2f}s")
            except requests.Timeout:
                print(f"Request {i} timed out")
    
    total_time = time.time() - start_time
    print(f"\nTotal time: {total_time:.2f}s")

if __name__ == "__main__":
    sync_requests()

The output will be similar to the AIOHTTP example, but the total time taken will be significantly longer due to the synchronous nature of Requests.

Request 1 completed in 3.04s
Request 2 completed in 2.57s
Request 3 completed in 3.26s
Request 4 completed in 1.23s
Request 5 completed in 2.49s

Total time: 12.61s

As you can see, AIOHTTP’s asynchronous nature allows it to complete all requests in roughly the time it takes to complete the slowest request, while Requests waits for each request to complete sequentially.

While AIOHTTP is a powerful library for asynchronous operations, it doesn’t provide a synchronous API like Requests. This is where HTTPX comes in.

HTTPX: The Best of Both Worlds

HTTPX (opens in a new tab), released by Tom Christie (the author of Django REST framework) in August 2019, aims to combine the best features of Requests and AIOHTTP. It provides a synchronous API similar to Requests but also supports asynchronous operations.

Key features of HTTPX include:

1. Familiar Requests-like API: HTTPX maintains a similar API to Requests, making it easy for developers to transition.
2. Both sync and async support: Unlike Requests or AIOHTTP, HTTPX supports both synchronous and asynchronous operations.
3. HTTP/2 support: HTTPX natively supports HTTP/2, allowing for more efficient communication with modern web servers.
4. Type annotations: HTTPX is fully type-annotated, which improves IDE support and helps catch errors early.

To install HTTPX, use pip:

pip install httpx

Here’s a basic example of using HTTPX synchronously:

import httpx

response = httpx.get('https://api.github.com')
print(response.status_code)
print(response.json())

The code is almost identical to the Requests example, making it easy to switch between the two libraries. However, HTTPX also supports asynchronous operations:

import asyncio
import httpx
import time

urls = [
    "https://httpbin.org/delay/1",
    "https://httpbin.org/delay/2",
    "https://httpbin.org/delay/3",
    "https://httpbin.org/delay/1",
    "https://httpbin.org/delay/2",
]

async def fetch(client, url, i):
    response = await client.get(url)
    print(f"Request {i} completed in {response.elapsed.total_seconds():.2f}s")

async def async_requests():
    start_time = time.time()
    async with httpx.AsyncClient(timeout=10.0) as client:
        tasks = [fetch(client, url, i) for i, url in enumerate(urls, 1)]
        await asyncio.gather(*tasks)
    
    total_time = time.time() - start_time
    print(f"\nTotal time: {total_time:.2f}s")

if __name__ == "__main__":
    asyncio.run(async_requests())

The output will be similar to the AIOHTTP example:

Request 1 completed in 1.96s
Request 4 completed in 1.96s
Request 5 completed in 3.00s
Request 2 completed in 3.30s
Request 3 completed in 4.42s

Total time: 4.44s

HTTPX’s ability to switch seamlessly between synchronous and asynchronous operations makes it a versatile choice for a wide range of applications. It’s especially useful when you need to interact with both synchronous and asynchronous code within the same project.

This brings us to the question: which library should you choose for your next Python project? That depends on your specific requirements.

Choosing the Right HTTP Client for Your Project

Here’s a quick comparison of the key features of Requests, AIOHTTP, and HTTPX:

Recommendations

1. If you’re working on a simple script or a project that doesn’t require asynchronous operations, stick with Requests. Its simplicity and wide adoption make it an excellent choice for straightforward HTTP tasks.

2. For high-performance asyncio applications, especially those dealing with many concurrent connections or requiring WebSocket support, AIOHTTP is your best bet. It’s particularly well-suited for building scalable web services.

3. If you need the flexibility to use both synchronous and asynchronous code, or if you’re looking to future-proof your application with HTTP/2 support, go with HTTPX. It’s also a great choice if you’re familiar with Requests but want to start incorporating async operations into your project.

How Speakeasy Uses HTTPX

When creating Python SDKs, Speakeasy includes HTTPX as the default HTTP client. This choice allows developers to use our SDKs for synchronous and asynchronous operations.

For example, here’s how you might use the Mistral Python SDK (opens in a new tab) created by Speakeasy to make requests.

First, install the SDK:

pip install mistralai

Set your Mistral API key (opens in a new tab) as an environment variable:

export MISTRAL_API_KEY="your-api-key"

Here’s how you might use the SDK to make a synchronous request:

from mistralai import Mistral
import os

s = Mistral(
    api_key=os.getenv("MISTRAL_API_KEY", ""),
)

res = s.chat.complete(model="mistral-small-latest", messages=[
    {
        "content": "Who is the best French painter? Answer in one short sentence.",
        "role": "user",
    },
])

if res is not None and res.choices:
    print(res.choices[0].message.content)

And here’s the same SDK and request using the asynchronous API:

import asyncio
from mistralai import Mistral
import os

async def main():
    s = Mistral(
        api_key=os.getenv("MISTRAL_API_KEY", ""),
    )
    res = await s.chat.complete_async(model="mistral-small-latest", messages=[
        {
            "content": "Who is the best French painter? Answer in one short sentence.",
            "role": "user",
        },
    ])
    if res is not None:
        print(res.choices[0].message.content)

asyncio.run(main())

Note how the asynchronous version uses the _async suffix for the method name, but otherwise the code is almost identical. This consistency makes it easy to switch between synchronous and asynchronous operations as needed.

You’ll also notice that there is no need to instantiate a different client object for the asynchronous version. SDKs created by Speakeasy allow developers to use the same client object for both synchronous and asynchronous operations. By abstracting away the differences between the modes of operation in HTTPX, Speakeasy reduces boilerplate code and makes your SDKs more user-friendly.

To illustrate the value of mixing synchronous and asynchronous operations, consider a scenario where you need to make a synchronous request to fetch some data, then use that data to make multiple asynchronous requests. HTTPX’s unified API makes this kind of mixed-mode operation straightforward.

import asyncio
from mistralai import Mistral
import os

# Initialize Mistral client
s = Mistral(
    api_key=os.getenv("MISTRAL_API_KEY", ""),
)

def sync_request():
    res = s.chat.complete(model="mistral-small-latest", messages=[
        {
            "content": "Who is the best French painter? Answer with only the name of the painter.",
            "role": "user",
        },
    ])
    if res is not None and res.choices:
        print("Sync request result:", res.choices[0].message.content)
        return res.choices[0].message.content

async def async_request(question):
    res = await s.chat.complete_async(model="mistral-small-latest", messages=[
        {
            "content": question,
            "role": "user",
        },
    ])
    if res is not None and res.choices:
        return res.choices[0].message.content
    return None

async def main():
    # Make a sync request
    painter = sync_request()
    
    # Make two async requests
    tasks = [
        async_request(f"Name the most iconic painting by {painter}. Answer in one short sentence."),
        async_request(f"Name one of {painter}'s influences. Answer in one short sentence."),
    ]
    results = await asyncio.gather(*tasks)
    
    # Print the results of async requests
    print("Async request 1 result:", results[0])
    print("Async request 2 result:", results[1])

if __name__ == "__main__":
    asyncio.run(main())

In this example, we first make a synchronous request to get the name of a painter. We then use that information to make two asynchronous requests to get more details about the painter. The SDK is only instantiated once, and the same client object is used for both synchronous and asynchronous operations.

Using a Different HTTP Client in Speakeasy SDKs

While HTTPX is the default HTTP client in SDKs created by Speakeasy, you can easily switch to Requests for synchronous operations (opens in a new tab) if needed. For example, to use Requests in the Mistral SDK, you can set the client parameter when initializing the client:

import os
import requests
from mistralai import Mistral, HttpClient

# Define a custom HTTP client using Requests
class RequestsHttpClient(HttpClient):
    def __init__(self):
        self.session = requests.Session()

    def send(self, request, **kwargs):
        return self.session.send(request.prepare())

    def build_request(
        self,
        method,
        url,
        *,
        content = None,
        headers = None,
        **kwargs,
    ):
        return requests.Request(
            method=method,
            url=url,
            data=content,
            headers=headers,
        )

# Initialize the custom client
client = RequestsHttpClient()

# Initialize Mistral with the custom client
s = Mistral(
    api_key=os.getenv("MISTRAL_API_KEY", ""),
    client=client,
)

# Use the Mistral client
res = s.chat.complete(model="mistral-small-latest", messages=[
    {
        "content": "Who is the best French painter? Answer in one short sentence.",
        "role": "user",
    },
])

if res is not None and res.choices:
    print(res.choices[0].message.content)

In this example, we define a custom RequestsHttpClient class that extends HttpClient from the Mistral SDK. This class uses the Requests library to send HTTP requests. We then initialize the Mistral client with this custom client, allowing us to use Requests for synchronous operations.

Conclusion

To learn more about how we use HTTPX in our SDKs, see our post about Python Generation with Async & Pydantic Support.

You can also read more about our Python SDK design principles in our Python SDK Design Overview.