Rate Limit

A RateLimiter hands back a RateLimitResult and stops. Every service then writes the same 429 by hand, and most write it wrong: no Retry-After, no quota, a body that does not match what the rest of the app answers with, and a bucket keyed on the socket peer, which behind an ingress is the ingress.

Take the decision to the edge instead:

from fastapi import FastAPI

from grelmicro import Grelmicro
from grelmicro.http import ErrorResponses, RateLimitedRequests
from grelmicro.providers.redis import RedisProvider
from grelmicro.resilience import RateLimiter, RateLimiterComponent
from grelmicro.security import TrustedProxies

redis = RedisProvider("redis://localhost:6379/0")

burst = RateLimiter.sliding_window("burst", limit=100, window=60)
daily = RateLimiter.sliding_window("daily", limit=10000, window=86400)

micro = Grelmicro(
    uses=[
        RateLimiterComponent(redis),
        ErrorResponses(),
        RateLimitedRequests(
            burst,
            daily,
            trusted=TrustedProxies(["10.0.0.0/8"]),
            exclude=("/livez", "/readyz"),
        ),
    ]
)
app = FastAPI()

micro.install(app)

Every request spends one token of every limiter listed. A burst limit stands beside a daily one, and a request passes both or is turned away by the first that says no.

With AuthenticatedRequests, a request spends its tokens once its route admitted it. A request the route refuses, and one no route answers, spends none. A flood limit covers those.

What the caller is told

Allowed or refused, the response states what is left, in the two fields of the RateLimit header specification:

RateLimit: "burst";r=97;t=42, "daily";r=9713;t=51840
RateLimit-Policy: "burst";q=100;w=60, "daily";q=10000;w=86400

r= is what is left, t= the seconds until it resets, q= the quota and w= the window. Both fields are lists, which is why two policies fit in one answer.

A refusal adds Retry-After and travels the same AdmissionError path every other rejection takes, so it is rendered by the registered ErrorResponses as application/problem+json, and one except still covers every way a caller is turned away.

legacy_headers=True adds X-RateLimit-Limit, -Remaining and -Reset, the superseded fields, for a client that reads only those. They carry whichever limiter is closest to being spent.

A token bucket states no policy

RateLimit-Policy describes a quota over a window. A token bucket refills continuously, so its reset is the wait for the next token rather than the edge of a window, and a policy built from it would promise a reset that never comes. A window shorter than a second is left out for the same reason: written as the whole second the header takes, it would publish half the rate it enforces. Both state RateLimit and nothing else.

Who is metered

The bucket is the resolved client address: the socket peer, unless a proxy you trust vouched for another one.

RateLimitedRequests(burst, trusted=TrustedProxies(["10.0.0.0/8"]))

That is not a detail. A limiter keyed on the peer meters the ingress, so one caller spends everybody's budget. A limiter keyed on a raw X-Forwarded-For meters whatever the caller wrote there, so nobody spends anything. TrustedProxies is what tells the two apart, and it is required: without it, and without a key of your own, the component is refused where it is written rather than metering the wrong thing.

The address is resolved once per request and left where ClientAddressMiddleware leaves it, so a route that meters itself reads the same caller.

A bucket is only as good as its key

A key a caller controls is a key a caller can rotate. Behind a trusted proxy the address is vouched for and cannot be, which is the point of the trusted set. In front of one, a spoofed source address is a spoofed bucket, and rate limiting is not the tool that stops that.

key= replaces the whole thing, for a service that meters by tenant or by API key. It takes the ASGI scope and returns the bucket, or None to leave a request unmetered, on the component and on a route alike.

Metering one route

RateLimitedRequests meters the app. A route that costs more than the rest, or has a quota of its own, declares it:

from grelmicro.integrations.fastapi import RateLimited


@app.get("/search", dependencies=[RateLimited(search, cost=5)])
async def do_search(query: str) -> list[Hit]: ...

A route that returns a Response of its own needs the component registered for those fields to reach the wire, because a framework merges what a dependency states into the response it builds itself and not into one a handler already built. The tokens are spent either way.

Both budgets are spent, and both are stated in the answer: the standard fields are lists, so the route's policy and the app's join into one, and two declarations on one route join the same way. One limiter metered twice is one policy, stated with the smaller count, which is the one a client has to pace itself off. The superseded X-RateLimit-* fields are single integers that cannot be read twice, so the meter with less left answers for both: a client reading only those is told the budget that refuses it first.

cost= is how many tokens the call takes, for an endpoint worth more than one. A cost no limiter could ever serve is refused where it is written rather than failing every request, on the route and on the app alike. A route that finds no caller to meter says so once rather than metering nothing in silence.

Limiting floods before routing

With AuthenticatedRequests, the route limits run at the route, so a scan of URLs no route answers spends nothing of them. Give a flood limit to cap it:

from fastapi import FastAPI

from grelmicro import Grelmicro
from grelmicro.http import (
    AuthenticatedRequests,
    ErrorResponses,
    RateLimitedRequests,
)
from grelmicro.providers.redis import RedisProvider
from grelmicro.resilience import RateLimiter, RateLimiterComponent
from grelmicro.security import JWTVerifier, TrustedProxies

redis = RedisProvider("redis://localhost:6379/0")

burst = RateLimiter.sliding_window("burst", limit=100, window=60)
flood = RateLimiter.sliding_window("flood", limit=600, window=60)

micro = Grelmicro(
    uses=[
        RateLimiterComponent(redis),
        ErrorResponses(),
        AuthenticatedRequests(
            JWTVerifier.discover(
                "https://auth.example.com/", audience="orders-api"
            ),
            exclude=("/livez", "/readyz"),
        ),
        RateLimitedRequests(
            burst,
            flood=flood,
            trusted=TrustedProxies(["10.0.0.0/8"]),
            exclude=("/livez", "/readyz"),
        ),
    ]
)
app = FastAPI()

micro.install(app)

Every request spends one token of flood before routing, a URL no route answers included. burst is still spent at the route, so the two budgets never mix: a request a route refuses spends flood and leaves burst whole.

The flood limit runs once authentication passed the request on. A token that does not verify is refused before it, and so is a request without a token, unless the app has a public route and has to route it to know. Authentication goes first because it is the cheaper check: a token verifies in about 12 microseconds, and a limiter over Redis takes a round trip. What each flood meets says what stops the others.

Without AuthenticatedRequests, both run before routing, flood first, so a request spends both wherever it goes.

The flood limit is keyed like the route limits, by trusted= or key=. Size it as a ceiling for one address, since every caller behind it shares it. A key= is then called before routing too, so build it from the caller, never from the route: path parameters are not there yet.

A flood refusal is the same 429, stating the flood policy alone. An allowed response states the route policies and leaves the flood one out, since a client paces itself off the route's budget.

Only exclude= applies to it, since a flood is made of the URLs include= does not name. It spends one token and never waits, whatever cost= and max_wait= say. Like the route limits, it meters HTTP requests and leaves websocket handshakes alone.

An app that adds its own RateLimitMiddleware keeps it, and the registered one is not added beside it. install then refuses a registered flood=, which would never run. On Starlette and FastAPI, pass flood= to the one the app added instead.

On Litestar, register both components with Grelmicro(uses=[...]). A middleware passed to Litestar(middleware=[...]) runs behind the router, so install refuses flood= when either one is passed there.

Waiting instead of refusing

max_wait=0.0 refuses as soon as the budget is spent, which is the default, because waiting at the edge holds the connection open and a client that retries is cheaper than a server that queues.

Give a budget where a short wait is better than a retry:

RateLimitedRequests(burst, trusted=..., max_wait=0.5)

A budget that runs out is still a refusal: the caller is answered 429 with the same headers, not an error.

Where it sits

Register it before CachedResponses() and IdempotentRequests(). Registration order is wrapping order, and a cache hit or an idempotent replay answers without reaching what sits inside it: put the limiter outside, or a caller spends no tokens on the requests it repeats.

Which endpoints it applies to

include names the paths metered and exclude names the ones left alone, the same two words every grelmicro middleware takes. Where a rule applies covers the matching and what wins.

What is never metered

exclude= names the paths that pass through, and takes the same patterns every grelmicro middleware takes. Put the probes there: Kubernetes polls /livez and /readyz every few seconds forever, and that is not a caller's budget to spend.

A request whose caller cannot be read at all is let through rather than metered under a bucket that is not theirs, and the reason is logged once.

So is one whose caller the walk could only take as far as your own proxy. That means trusted= does not describe this deployment, and metering every caller behind that proxy as one would let any of them spend the budget of all of them. It is logged once, as a configuration to fix.

When a setting stops describing the deployment

Two of them are read again on every request, and both are answered by serving the caller and saying so once, because a number that changed must not take the service down with it: a quota reconfigured below the cost a request spends, and a trusted= set that resolves nothing but your own proxy.

When the backend is down

RateLimiter decides that, not this middleware. A limiter built with fail_open=True serves the request and reports a full quota, so the headers say a budget nobody counted. A limiter built without it raises, and the app answers as it does for any dependency failure. One setting, on the limiter, wherever it is used.

In the schema

A registered RateLimitedRequests documents the 429, and the RateLimit, RateLimit-Policy and Retry-After fields it carries, on every operation in the OpenAPI schema FastAPI or Litestar builds. So a generated client has a branch for it.

Every operation, not the metered ones. Which paths are metered is tuned while the service runs, and the schema is built once when the app starts, so naming the current set would publish a document that stops being true the first time an operator narrows it. A 429 says only what a client may be answered with, never what it must send, so stating it everywhere stays true whichever paths are metered.

Pass openapi=False to leave the schema alone.

Options

Every option of RateLimitMiddleware is taken by RateLimitedRequests and forwarded, so a registered component and a hand-added middleware answer the same.

Option	What it does
*limiters	the limiters every request spends
flood	the limiter every request spends before routing
trusted	the proxies whose forwarded entries may be believed
key	builds the bucket key itself
cost	tokens one request spends of each
max_wait	seconds a throttled request waits before it is refused
include	paths metered, empty means every path
exclude	paths never metered, whatever include says
legacy_headers	also send the superseded X-RateLimit-* fields
openapi	describe the 429 on every operation in the schema
name	keep two sets of rules apart on one app