Retry

A Retry policy repeats a failing call with backoff and jitter. Use it for transient errors that are safe to retry: network blips, brief overloads, deadline-related failures.

Why

• Survive flaky network paths without changing the call site.
• Apply exponential backoff with jitter so retries do not stampede a recovering dependency.
• Stay safe by default: retries trigger only on the exception classes you allow.

Usage

Decorate an async or sync function. Pass when= to say which outcomes should retry:

import asyncio

import httpx

from grelmicro.resilience import retry


def api(request: httpx.Request) -> httpx.Response:
    """Stand in for a remote API, so the example runs offline."""
    return httpx.Response(200, content=b"hello")


@retry(when=httpx.HTTPError, attempts=3)
async def fetch(client: httpx.AsyncClient, url: str) -> bytes:
    response = await client.get(url)
    response.raise_for_status()
    return response.content


async def main() -> None:
    async with httpx.AsyncClient(transport=httpx.MockTransport(api)) as client:
        content = await fetch(client, "https://api.example/greeting")
        print(content)


if __name__ == "__main__":
    asyncio.run(main())

The decorator auto-detects whether it wraps an async or sync function.

For inline retries that span multiple statements, use the block form:

import asyncio

import httpx
from pydantic import BaseModel

from grelmicro.resilience import retrying


class Payment(BaseModel):
    amount: int


class Receipt(BaseModel):
    id: str


def payments_api(request: httpx.Request) -> httpx.Response:
    """Stand in for the payment API, so the example runs offline."""
    return httpx.Response(200, json={"id": "pay_1"})


async def submit(
    client: httpx.AsyncClient, url: str, payment: Payment
) -> Receipt:
    async for attempt in retrying(when=httpx.HTTPError, attempts=3):
        async with attempt:
            response = await client.post(url, json=payment.model_dump())
            response.raise_for_status()
            return Receipt.model_validate(response.json())
    msg = "retrying returns or raises, never falls through"
    raise AssertionError(msg)


async def main() -> None:
    transport = httpx.MockTransport(payments_api)
    async with httpx.AsyncClient(transport=transport) as client:
        receipt = await submit(
            client, "https://payments.example/charges", Payment(amount=100)
        )
        print(receipt)


if __name__ == "__main__":
    asyncio.run(main())

when= is required. There is no default. Pass an exception class, a tuple of classes, a predicate callable, or a Match instance.

Choose a backoff

Retry ships five backoff algorithms. The default is exponential with full jitter. Pick by purpose.

Algorithm	Use when
ExponentialBackoff (default)	Network and HTTP retries. Doubling delay with jitter avoids retry storms.
ConstantBackoff	Polling-style retries (waiting for a job). Fixed interval is predictable.
LinearBackoff	Steady, predictable growth without the early-attempt cluster of exponential.
FibonacciBackoff	Smoother growth than exponential, faster than linear.
RandomBackoff	Uniform random delay in a fixed range. Maximum spread, no growth.

The factory classmethods build the right config for you:

policy = Retry.exponential("payments", when=httpx.HTTPError, attempts=5)
polling = Retry.constant("wait_job", when=NotReady, attempts=20, delay=1.0)

Add max_seconds= to cap the total time spent retrying. Retrying stops when either attempts or the time budget is reached, whichever comes first:

# Up to 5 attempts, but never more than 30 seconds in total.
policy = Retry.exponential("payments", when=httpx.HTTPError, max_seconds=30.0)

How backoff and jitter work

Exponential. The raw wait before retry N is min(base_delay * 2 ** (N - 1), max_delay). It doubles each attempt until it reaches the cap. With the defaults (base_delay=0.1, max_delay=30.0), the raw wait is 0.1s, 0.2s, 0.4s, 0.8s, 1.6s, ..., capped at 30.0s.

Jitter then transforms each raw wait into the actual sleep. The actual sleep may be smaller than the raw value.

Jitter is randomness added to the wait so concurrent clients do not retry at the same instant and overwhelm the recovering server. Pick a mode by how much spread you want. Default: full.

Jitter	Spread	When to use
none	none	Single client. Never with concurrency.
full (default)	maximum	The safe default.
equal	half	When you need predictable timing.
decorrelated	adaptive	High contention on a shared dependency.

Constant. A fixed delay between retries. One field: delay (seconds, default 1.0).

Filtering outcomes with Match

Match is the DSL every resilience strategy uses to decide whether an outcome (an exception OR a return value) should engage the strategy. The when= parameter on Retry accepts any Match instance, plus the bare-class shorthand for the simple case.

Here is the common case: retry on an exception type, or a tuple, or a predicate.

import httpx
from pydantic import BaseModel

from grelmicro.resilience import Match, retry


class Job(BaseModel):
    status: str


# Compose exception and result matchers with `|`.
# Retry on transient HTTP errors OR when the response carries a
# server-side soft-fail marker.
@retry(
    when=Match.exception(httpx.HTTPError)
    | Match.result(lambda r: r.headers.get("X-Soft-Fail") == "true"),
    attempts=5,
)
async def fetch(client: httpx.AsyncClient, url: str) -> httpx.Response:
    return await client.get(url)


# Polling-style: retry until the result is no longer ``None``.
@retry(when=Match.result(None), attempts=20)
async def poll_job(client: httpx.AsyncClient, job_id: str) -> Job | None:
    response = await client.get(f"/jobs/{job_id}")
    job = Job.model_validate(response.json())
    return job if job.status == "ready" else None

Full Match DSL

Exception filtering

from grelmicro.resilience import Match, Retry

Retry("api", when=Match.exception(httpx.HTTPError))
Retry("api", when=Match.exception(httpx.HTTPError, OSError))
Retry("api", when=Match.exception(lambda e: e.status >= 500))
Retry("api", when=Match.exception_message(contains="timeout"))
Retry("api", when=Match.exception_message(regex=r"^5\d\d "))
Retry("api", when=Match.exception_cause(KeyError))

Result filtering

Retry("polling", when=Match.result(None))
Retry("polling", when=Match.result(False))
Retry("polling", when=Match.result(lambda r: r.status_code >= 500))

Match.result(callable) always treats the argument as a predicate. To match a function literal exactly, wrap with lambda r: r is my_fn.

Composition

# OR
Retry("api", when=Match.exception(httpx.HTTPError) | Match.result(None))

# AND
Retry("api", when=Match.exception(httpx.HTTPError) & Match.exception(lambda e: e.status >= 500))

# NOT (one symmetric `not_*` per primitive)
Retry("api", when=Match.not_exception(ValueError))
Retry("api", when=Match.not_result(None))
Retry("api", when=Match.not_exception_message(contains="ok"))
Retry("api", when=Match.not_exception_cause(KeyError))

Use | for OR and & for AND. Each primitive (exception, result, exception_message, exception_cause) has a not_* twin for the negated form.

When a predicate misbehaves

A matcher is called from inside the except block that is handling your error, so it never raises. A predicate that raises reads as no match. A predicate that returns something other than a bool is read for its truth value. Either way grelmicro logs one warning naming the matcher, once per predicate rather than once per attempt, and the error you were handling reaches you unchanged.

A bad argument is a different case: Match raises ValueError when you build it, not when it runs.

What never retries

asyncio.CancelledError, KeyboardInterrupt, and SystemExit are BaseException subclasses outside Exception. They always propagate, regardless of the Match you pass. This is required for correct asyncio shutdown.

When the client disconnects

What happens to a retry inside a request handler depends on whether the server cancels the handler when the client goes away.

• The server cancels the handler. The cancellation reaches the retry during an attempt or during the wait before the next one. It propagates at once and no further attempt runs.
• The server lets the handler run. Uvicorn does this. The retry runs to its end as if the client were still there, and the response is then dropped.

Bound the work either way with attempts and max_seconds, or wrap the call in a Timeout.

Behavior on exhaustion

When attempts is exhausted, the underlying exception is re-raised with a PEP 678 note attached:

fragment

try:
    await fetch(url)
except httpx.ConnectError as exc:
    print(exc.__notes__)
    # ['retry: 3/3 attempts exhausted in 1.40s (exponential backoff)']

Callers catch the underlying error type, unchanged. There is no RetryError wrapper class.

Composition with Circuit Breaker

Retry and Circuit Breaker compose by intent. When the breaker is OPEN, it raises CircuitBreakerError. Pick a narrow when= allowlist so the retry loop does not swallow that signal:

import asyncio

import httpx
from pydantic import BaseModel

from grelmicro import Grelmicro
from grelmicro.providers.memory import MemoryProvider
from grelmicro.resilience import CircuitBreaker, CircuitBreakerComponent, retry

# Memory keeps the breaker state in this process.
micro = Grelmicro(uses=[CircuitBreakerComponent(MemoryProvider())])

cb = CircuitBreaker("payments")


class Payment(BaseModel):
    amount: int


class Receipt(BaseModel):
    id: str


def payments_api(request: httpx.Request) -> httpx.Response:
    """Stand in for the payment API, so the example runs offline."""
    return httpx.Response(200, json={"id": "pay_1"})


# A narrow allowlist that excludes CircuitBreakerError. When the
# breaker is open it raises CircuitBreakerError, which is not in
# `when`, so the retry loop aborts immediately.
@retry(when=(httpx.ConnectError, httpx.TimeoutException), attempts=3)
async def call_payments(
    client: httpx.AsyncClient, url: str, payment: Payment
) -> Receipt:
    async with cb:
        response = await client.post(url, json=payment.model_dump())
        response.raise_for_status()
        return Receipt.model_validate(response.json())


async def main() -> None:
    transport = httpx.MockTransport(payments_api)
    async with micro, httpx.AsyncClient(transport=transport) as client:
        receipt = await call_payments(
            client, "https://payments.example/charges", Payment(amount=100)
        )
        print(receipt)


if __name__ == "__main__":
    asyncio.run(main())

A broad allowlist (when=Exception) would retry through the open breaker. The narrow allowlist lets the breaker do its job.

Configuration

Build the policy with keyword arguments. Set when to choose which outcomes retry, attempts for the budget, and backoff for the wait strategy.

import httpx

from grelmicro.resilience import Retry

policy = Retry.exponential(
    "payments",
    when=httpx.HTTPError,
    attempts=5,
    base_delay=0.2,
    max_delay=10.0,
    jitter="full",
)


@policy
async def call_payments():
    pass

Environment variables

Prefix: GREL_RETRY_{NAME_UPPER}_. The default instance drops the name segment and reads GREL_RETRY_*.

Environment variables are read by default

A GREL_* variable that fills a component field is read unless GREL_ENV_LOAD=0. With GREL_ENV_LOAD=0, a variable that is set is ignored, the default applies, and startup warns with its name. Passing env_load=False to a component turns its reads off and stays quiet. See How a value is resolved for the three ways to configure a component, including a local .env.

Env var	Field	Type	Default
GREL_RETRY_{NAME_UPPER}_ATTEMPTS	attempts	int (>= 1)	3
GREL_RETRY_{NAME_UPPER}_WHEN	when	CSV or JSON list of FQN strings (e.g. httpx.HTTPError). Coerced to Match.exception(...). Predicate forms cannot come from env.	required
GREL_RETRY_{NAME_UPPER}_BACKOFF	backoff	JSON object with a kind field (see below)	{"kind":"exponential"}

The full backoff config is a discriminated Pydantic union, so the env value is parsed as one JSON object. Each algorithm accepts the same fields it takes in code:

kind	Fields
exponential	base_delay, max_delay, jitter (none / full / equal / decorrelated)
constant	delay
linear	base_delay, max_delay
fibonacci	base_delay, max_delay
random	min_delay, max_delay

import httpx

from grelmicro.resilience import Retry

# Reads the config from environment variables. The backoff field is
# a discriminated union, pass it as a single JSON object.
#
# - GREL_RETRY_PAYMENTS_ATTEMPTS=5
# - GREL_RETRY_PAYMENTS_WHEN=httpx.HTTPError
# - GREL_RETRY_PAYMENTS_BACKOFF={"kind":"exponential","base_delay":0.2}
policy = Retry("payments", when=httpx.HTTPError)

The callable form of when cannot come from env. Use the FQN list for env-driven configs.

Advanced

For the from_config declarative path and pydantic-settings composition, see Declarative configuration.

Live reconfiguration

Retry inherits Reconfigurable[RetryConfig]. Calling policy.reconfigure(new_config) swaps the snapshot for future loops. An in-flight async for attempt in policy: keeps its snapshot until it completes. See Live reconfiguration.

Reference

See the API reference for every option.