Skip to content

Don’t mock Python’s HTTPX

Published on: December 8, 2023    Categories: Django, Python

This is part of a series of posts I’m doing as a sort of Python/Django Advent calendar, offering a small tip or piece of information each day from the first Sunday of Advent through Christmas Eve. See the first post for an introduction.

Moving on from requests

For quite a long time, the standard recommendation for making HTTP requests in Python was the aptly-named requests package. And you can still use requests if you want to, but there are a couple things about it that are showing their age:

  1. It didn’t set up any sort of timeout by default, which meant you had to remember to do it explicitly (and quite a few linters will flag use of requests without an explicit timeout, if you’re worried about it).
  2. It’s synchronous/blocking-only, which doesn’t play as nicely with the current wave of async Python libraries, frameworks, and other tools.

Which led to the rise of HTTPX (the Python one — there’s also an HTTPX in Ruby). HTTPX provides both sync and async support, a default timeout, and a pile of other features, and with a largely requests-compatible API requiring only minimal code changes to adopt. So I’ve been gradually switching my own projects over to it for a while now.

Making a mockery

But there’s something I wish I’d learned a lot earlier, and so I’ll share it with you now in case it’s also something you will end up wishing you’d learned earlier, or glad that you did learn early: don’t mock HTTPX.

By which I mean: Python ships with mock support in the standard library, and many third-party tools exist to build on top of that or provide alternatives — “mock” in the sense of providing drop-in replacements for a given module or function or class that you can use during testing (there is also a whole tree of terminology and distinctions when you go deep into the testing world, but I’ve never personally needed to understand the difference between mocks and stubs and doubles and all the other similar-but-apparently-distinct things, so I’ll just use “mock” as a catch-all, especially because the unittest.mock library is the basis for all of them in Python). The basic idea behind mocks is to provide a replacement that you can control and observe, so you can make it behave in specific ways, or check that it was used in specific ways, and so on.

Which, in the case of mocking a request library like requests or HTTPX, usually means patching the get() or post() handlers to replace them with mocked versions that behave the way you need them to in your tests. For example, you might want to check that your code behaves correctly when some external service returns an error like an HTTP 404 — you could mock the request library to return that 404 and use it to verify the behavior.

Except you shouldn’t do that with HTTPX. I know there are tools and libraries and packages out there which do this, but I don’t think you should use them. Instead, I think you should do two specific things:

  1. Refactor your own code to receive an HTTPX client as an argument or configuration option — either the synchronous httpx.Client or the asynchronous httpx.AsyncClient (these are the equivalent of requests.Session).
  2. Use HTTPX’s own built-in mock transport instead of mocking HTTPX or its constituent parts.

The first suggestion, of refactoring code to expect to be passed an HTTPX client instance (and maybe creating a default instance as a fallback), has a lot of benefits. It makes testing a little bit easier, but it also gives flexibility to users of your code. If someone needs to, say, deploy your code in an environment which enforces the use of an HTTP proxy, they can do that by setting up an HTTPX client configured with the correct proxy. This kind of flexibility is very nice to have, and you often don’t realize how nice it is until you have it, so I highly recommend trying it out.

The second suggestion is just to make use of the HTTPX mock transport feature to avoid the need to mock and patch so much. And this is what I wish I’d known earlier — even though it’s documented, I didn’t discover it until fairly recently. So suppose, as mentioned above, you need to test how your code behaves when it receives an HTTP 404. You can do that:

from http import HTTPStatus
import httpx

my_test_client = httpx.Client(
    transport=httpx.MockTransport(
        lambda request: httpx.Response(
            HTTPStatus.NOT_FOUND, content="Not Found"
        )
    )
)

This client object will always return the specified response, which makes it great for testing and, best of all, doesn’t require you to build multiple layers of mocks (one for the client, one for each of its request methods, etc.) or build mocks and then patch them in to replace HTTPX’s own request functions/methods during testing. This also helps you start adhering to the principle of “Don’t Mock What You Don’t Own”, reducing your reliance on mock objects, and probably saving yourself a lot of complexity and trouble.