Skip to content

Response Cache

A catalog endpoint reads the same rows for every caller and answers the same bytes every time. Under load it runs that query thousands of times a minute, and every replica runs its own copy.

@cached is not the answer. It caches what a function returns, and what a route answers is not that: the framework still has to serialize it, and a header the handler set is not part of the return value. Cache the response.

from fastapi import FastAPI
from pydantic import BaseModel

from grelmicro import Grelmicro
from grelmicro.cache import Cache
from grelmicro.http import CachedResponses
from grelmicro.integrations.fastapi import CachedResponse
from grelmicro.providers.redis import RedisProvider

redis = RedisProvider("redis://localhost:6379/0")
micro = Grelmicro(uses=[Cache(redis), CachedResponses()])
app = FastAPI()

micro.install(app)


class Product(BaseModel):
    id: int
    name: str


@app.get("/products", dependencies=[CachedResponse(ttl=60)])
async def list_products() -> list[Product]:
    return await load_products()


async def load_products() -> list[Product]:
    return [Product(id=1, name="Anvil")]

Declare it on the route and the second caller is answered without reaching the handler. It rides the registered Cache, so a response one replica computed answers the callers of every other one.

CachedResponse is declared rather than called, because the middleware has to answer before the app is routed, and a handler body runs long after that. It is the same shape as Conditional: the component holds the rules every route shares, and the route says it wants them.

What a hit looks like

A hit carries Age, which says how many seconds it has been in the store. That is the standard header a cache answers with, so a client, a CDN, and a proxy all read it without knowing anything about grelmicro.

Every stored response carries an ETag. When the client already holds it and sends If-None-Match, the answer is 304 Not Modified with no body at all:

sequenceDiagram
    autonumber
    participant C as Client
    participant S as Your service
    participant H as Handler

    C->>S: GET /products
    S->>H: the cache is cold
    H-->>S: 200, 40 KB
    S-->>C: 200, ETag "a1b2", Age 0

    C->>S: GET /products, If-None-Match "a1b2"
    Note over S: the entry is still fresh
    S-->>C: 304 Not Modified

The handler ran once. The second request cost a cache read and 100 bytes on the wire.

One cold key runs the handler once

The moment an entry expires, every request for it misses at the same time. A plain cache answers that by running the handler once per caller, which is the load spike the cache was there to prevent.

Here the first request runs the handler and every other one waits for it, in the process and across replicas when a Coordination backend is configured. It is the same stampede protection TTLCache already gives @cached.

A request's own Cache-Control is not read. This answers for the resource rather than for one caller, so honouring no-cache from an unauthenticated caller would let anybody spend the handler at will.

A path whose responses are never storable, a stream above all, stops taking the lock once the first one has shown that nothing is kept for it, so it is never queued behind itself. A HEAD takes it never: it reads what a GET stored and fills nothing, so folding it would hold up the read it shares a key with.

What is never stored

A response cache is dangerous in exactly one way: answering one caller with another caller's response. These rules are not configurable, because every one of them is that mistake.

Not stored Why
A request carrying Authorization or Cookie it was answered for one caller
A request whose ASGI scope carries an authenticated user or authentication scopes an outer authentication middleware identified one caller
A response carrying Set-Cookie it is one caller's session
A response carrying Cache-Control: no-store, no-cache or private it said so
A response carrying Cache-Control: max-age=0 or s-maxage=0 it is stale already
Anything but 200 a failure is not an answer to hand on
A response carrying Content-Encoding compression sits outside this middleware
A HEAD response its body is empty, and would answer the GET after it with nothing
A request carrying Range what is stored is the whole resource, not the part asked for

A HEAD still reads what a GET stored, and is answered with the same headers and no body.

Only GET is cached. CachedResponse() on a route that answers anything else is refused when micro.install(app) reads it, naming the path. A FastAPI GET with another dependency is left uncached: a hit would answer before that dependency ran, and an ordinary Depends may read a custom credential or otherwise vary the response without declaring Vary.

Vary

Vary is where naive response caches leak. A response that says Vary: Accept-Language is only an answer to a client that asked for the same language, and a cache that stores it under the URL alone will hand the French page to the next caller who wanted German.

Declare what the key reads:

CachedResponses(vary_by_headers=("accept-language",))

A response whose own Vary names a header outside that set is not stored, and the refusal is logged. So a handler that starts varying on something new stops being cached rather than starting to answer the wrong callers.

What the key reads is written into the stored response's Vary, joined with whatever the handler set. The cache in front of yours has to be told too: a CDN, a corporate proxy, or the browser would otherwise hand one caller's copy to the next one who sent a different value.

Vary: * is never stored.

A response naming its own freshness is kept no longer than it says. Cache-Control: max-age=30 caps the entry at 30 seconds whatever the TTL is, and s-maxage wins over max-age where both are named, because it is the one written for a shared cache. Named twice, the smaller one decides.

Every occurrence of a header counts. A response carrying two Vary lines, or two Cache-Control lines, says all of what they say, and reading only the last of them is how the one that refused the store goes missing.

The key

By default the key is the scheme, the host, the prefix the app is served under, the path, and the whole parsed query. Distinct parameter names are put in one canonical order, so page=1&sort=name and sort=name&page=1 share an entry. Repeated values keep their request order: role=admin&role=user is not the same resource as role=user&role=admin. Percent-encoded names are compared after decoding. The key also carries every occurrence of each header named in vary_by_headers, and distinguishes an absent selected query parameter or header from one present with an empty value.

The host and the prefix are in it so an app answering for two hostnames, and two services behind one gateway sharing one store, never hand out each other's responses.

Name the parameters that matter and the rest is ignored, so a tracking parameter does not turn one resource into a thousand:

CachedResponses(vary_by_query=("page", "size"))

key= replaces the whole thing. It takes the ASGI scope and returns the key, or None to leave that request uncached.

Keying on the caller

The scope carries the client address, and building it into the key turns a shared cache into a per-caller one: the hit rate collapses, the store grows with the number of callers, and a mistake in the key answers one of them with another one's response. A per-user result belongs in @cached, on the data rather than the response.

Naming paths instead of routes

CachedResponse() is a FastAPI dependency, on a route or on the whole router it is included with:

app.include_router(products, dependencies=[CachedResponse(ttl=60)])

APIRouter(dependencies=[CachedResponse(ttl=60)]) and FastAPI(dependencies=[CachedResponse(ttl=60)]) say the same thing, for a router and for a whole app.

A router holds more than reads, so what a cache cannot answer for is left to its handler rather than refused: a write under it is simply not cached. Declared on one route, the same thing is a mistake, and it is refused where it is written. The nearest declaration decides, so a route beats the router it sits in, an inner router beats the one that includes it, and a route that declared one is not overridden by an include pattern naming it.

A middleware around a mounted application, or configured on that application or its router, is a cache boundary. A CachedResponse() declaration behind it is not consumed by a cache on the parent, because a parent hit would answer before the mounted middleware ran. An explicit parent include rule stops at the same boundary. Install CachedResponses inside that application when its own routes should be cached.

Middleware configured on an individual Starlette route is an exact boundary for that route as well, so an explicit cache rule cannot answer before it runs.

Starlette and Litestar resolve no dependencies to hang it on, and a router you did not write cannot be changed either, so name the URLs and how long each is kept:

from fastapi import FastAPI

from grelmicro import Grelmicro
from grelmicro.cache import Cache
from grelmicro.http import CachedResponses
from grelmicro.providers.redis import RedisProvider

redis = RedisProvider("redis://localhost:6379/0")
micro = Grelmicro(
    uses=[
        Cache(redis),
        CachedResponses(include={"/catalog": 300, "/products/*": 60}),
    ]
)
app = FastAPI()

micro.install(app)

Exact match, unless the pattern ends with *, which matches as a prefix. It is the matching every grelmicro middleware uses, the same as ConditionalRequests(include=...). The most specific pattern decides, so {"/products/*": 60, "/products/hot": 300} keeps the hot one for 300 seconds.

A tuple says the same thing when every path is kept for the same time, and reads like every other middleware:

CachedResponses(ttl=60, include=("/products/*", "/catalog"))

exclude= carves a path out again, whatever a route or a pattern says.

A pattern naming a read that sits behind a security scheme is refused where it is written, the same as declaring it on the route would be. A read with an ordinary FastAPI dependency is left uncached too, whether the dependency was declared on the route, router, app, or include. On a framework grelmicro cannot read the routes of, nothing can check that for you: name paths that answer everybody the same.

Invalidating

The TTL is the floor, not the only lever. A write that changes what a read answers drops what the cache would otherwise go on serving:

@app.post("/products")
async def create(product: ProductIn) -> Product:
    created = await save(product)
    await cached_responses.purge()
    return created

purge() deletes every response that component stored, and nothing else in the cache, because each entry carries its tag.

When the store is down

A cache that cannot be reached is a cache miss. A read that fails is logged and answered by the handler, and a response that cannot be written still goes out to the caller who waited for it. Adding the cache never makes a path less available than it was without it.

Where it sits

Register it before ConditionalRequests(), so a hit is answered without entering it. Both are added inside whatever middleware the app itself installed, so a request still passes middleware authentication before either can answer.

A hit answers before the app is routed

A route's own Depends never runs on a hit, because the response is already on its way back by then. A route carrying any dependency besides the CachedResponse() marker is therefore left uncached, including a plain Depends that reads Request or Header. Dependencies on the app, router, and include count too.

A FastAPI security scheme, such as APIKeyHeader or HTTPBearer, remains a configuration error and is refused when micro.install(app) reads it, naming the path. Cache what answers everybody the same, and use @cached on the data behind the ones that do not.

Changing it while it runs

Every option below except the store and the two callables is tuned from a mounted ConfigMap, so a TTL is raised under load without a redeploy:

grel:
  cached_responses:
    ttl: 60
    include:
      "/products/*": 60
      "/products/hot": 300

A pattern arriving that way is checked against the app's routes the same way micro.install(app) checks one, so a file cannot start caching a write or a read behind a security scheme. Read Where a rule applies.

Options

Every option of CachedResponsesMiddleware is taken by CachedResponses and forwarded, so a registered component and a hand-added middleware answer the same.

Option What it does
ttl seconds a response is kept when its route names none
include path patterns and the seconds each is cached for
exclude paths never cached, whatever else says
vary_by_headers request headers the key reads
vary_by_query query parameters the key reads
key builds the key itself
skip leaves one response unstored
max_body_size largest body stored, 1 MB by default
cache the TTLCache to store in
namespace, name keep two sets of rules apart on one app