Skip to content

Tracing

Unified instrumentation. Use it to enrich logs with structured context and create distributed traces from one decorator.

  • One decorator: @instrument captures function arguments as context.
  • Logs always enriched: context flows into every log record, no OpenTelemetry needed.
  • Traces when installed: with OpenTelemetry present, the same fields become OTel span attributes.

Quick Start

quickstart.py
import asyncio
import logging

from grelmicro.log import configure
from grelmicro.trace import add_context, instrument, span

configure(format="json")
logger = logging.getLogger(__name__)


@instrument
async def process_order(order_id: str, user_id: str) -> None:
    logger.info("started")

    add_context(payment_status="pending")
    logger.info("payment initiated")

    with span("db_query", table="orders"):
        logger.info("querying")

    # `table` is gone with the span, `payment_status` is still here.
    logger.info("done")


if __name__ == "__main__":
    asyncio.run(process_order(order_id="ORD-1", user_id="USR-1"))

Output:

output
{"order_id":"ORD-1","user_id":"USR-1","time":"2026-04-01T10:30:00.123456+00:00","level":"INFO","msg":"started","logger":"__main__"}
{"order_id":"ORD-1","user_id":"USR-1","payment_status":"pending","time":"2026-04-01T10:30:00.123556+00:00","level":"INFO","msg":"payment initiated","logger":"__main__"}
{"order_id":"ORD-1","user_id":"USR-1","payment_status":"pending","table":"orders","time":"2026-04-01T10:30:00.123656+00:00","level":"INFO","msg":"querying","logger":"__main__"}
{"order_id":"ORD-1","user_id":"USR-1","payment_status":"pending","time":"2026-04-01T10:30:00.123756+00:00","level":"INFO","msg":"done","logger":"__main__"}

Each argument of process_order becomes a field on every record the call writes. add_context adds one for the rest of the call, and a span adds one for its block alone.

API

@instrument

Decorator that captures function arguments as structured context. Works with sync and async functions.

# Bare decorator: captures all arguments
@instrument
async def process(order_id: str, user_id: str): ...

# Skip sensitive arguments
@instrument(skip={"password", "token"})
async def login(username: str, password: str): ...

# Skip all arguments
@instrument(skip_all=True)
async def bulk_process(payload: bytes): ...

# Custom span name
@instrument(name="db.query")
async def fetch_user(user_id: str): ...

span()

Context manager for mid-function instrumentation. Creates a nested context that is automatically removed on exit.

@instrument
async def handle_request(request_id: str):
    logger.info("received")  # has request_id

    with span("auth", method="jwt"):
        logger.info("authenticating")  # has request_id + method

    with span("db", table="users"):
        logger.info("querying")  # has request_id + table

    logger.info("done")  # back to request_id only

add_context()

Add fields to the current context. Updates both log records and the active OTel span (if tracing is configured).

@instrument
async def process(order_id: str):
    result = charge()
    add_context(payment_id=result.id, status=result.status)
    logger.info("charged")  # includes payment_id and status

A request or message handler needs no span. micro.install(app) opens a context for each request and each message, so fields added in a handler reach every log record until the response, and the next request starts empty:

fragment
@app.post("/orders/{order_id}")
async def create_order(order_id: str) -> None:
    add_context(order_id=order_id)
    logger.info("order received")  # includes order_id

A WebSocket gets one context for the whole connection, not one per message. With micro.install(app, ambient=False), no middleware opens the context, so open a span() in the handler. Outside a span and outside a handler, add_context does nothing.

Configuration

The tracing context enriches log records regardless of how logging is configured. No additional configuration is needed.

When OpenTelemetry is installed, @instrument and span() also create OTel spans and add the same fields as span attributes. A single decorator produces both structured logs and distributed traces.

Install

OpenTelemetry integration needs the opentelemetry extra: pip install "grelmicro[opentelemetry]". See the installation guide for uv and poetry.

Standalone

# Logging only (no OTel dependency needed)
configure()

# Logging + OTel: install opentelemetry and configure your exporter separately.
# @instrument and span() will automatically create OTel spans when opentelemetry
# is installed and a TracerProvider is configured.

Via Grelmicro app

Trace() owns the TracerProvider lifecycle when registered on a Grelmicro app:

import asyncio

from grelmicro import Grelmicro
from grelmicro.log import Log
from grelmicro.trace import Trace, instrument


@instrument
async def process(order_id: str) -> None:
    pass


micro = Grelmicro(
    uses=[
        Log(),
        Trace(
            service_name="orders",
            exporter="console",
        ),
    ]
)


async def main() -> None:
    async with micro:
        await process(order_id="ORD-1")


asyncio.run(main())

Off until an endpoint is configured

Trace() defaults to exporter="auto". It exports over OTLP HTTP when an endpoint is configured (the endpoint argument, OTEL_EXPORTER_OTLP_TRACES_ENDPOINT, or OTEL_EXPORTER_OTLP_ENDPOINT) and otherwise auto-disables into a true no-op: it installs no tracer provider, leaves the global provider untouched, and runs no auto-instrumentation. So you register Trace() unconditionally and it stays silent in dev, test, and CI instead of falling back to localhost:4318, and it never conflicts with a second app. A bounded shutdown_timeout (default 5.0 seconds) caps the flush on exit, so a slow or unreachable collector cannot hang shutdown.

For local development, set exporter="console" to print spans to the console. An explicit exporter="none" is different from the auto-disable above: it still installs the provider so spans are created (and auto-instrumentation runs), they are just not exported. Leave the exporter on auto for the unconditional no-op.

Basic auth in one line

Backends like OpenObserve want an Authorization: Basic header. Pass basic_auth=(username, password) and Trace builds and attaches it to the exporter:

Trace(
    service_name="orders",
    endpoint="https://obs.example.com/api/default/v1/traces",
    basic_auth=("me@example.com", password),
)

From the environment, set GREL_TRACE_BASIC_AUTH_USERNAME and GREL_TRACE_BASIC_AUTH_PASSWORD instead. The header is attached on the exporter directly, so it bypasses the OTEL_EXPORTER_OTLP_HEADERS encoding where base64 padding (=) can be mangled or dropped.

Environment variables are read by default

A GREL_* variable that fills a component field is read unless GREL_ENV_LOAD=0. With GREL_ENV_LOAD=0, a variable that is set is ignored, the default applies, and startup warns with its name. Passing env_load=False to a component turns its reads off and stays quiet. See How a value is resolved for the three ways to configure a component, including a local .env.

The config never prints your credentials

An endpoint can embed credentials in its userinfo or query (https://usr:token@collector/v1), and header values are API keys. On TraceConfig the endpoint field is a SecretUrl and each headers value is a SecretStr, so repr(), model_dump(), and model_dump_json() show *** in their place. The endpoint keeps its scheme, host, and path readable, so you can still see which collector is configured. Read either back with get_secret_value(). What grelmicro sends to the collector is unchanged.

Provider lifecycle and exporters

The provider is installed on enter and restored to the prior global on exit, so sequential apps in tests do not stack providers.

Trace() reads GREL_TRACE_* environment variables (see TraceConfig for the full field set) or accepts the same fields as keyword arguments. The OTLP HTTP and gRPC exporters require their own packages (opentelemetry-exporter-otlp-proto-http or opentelemetry-exporter-otlp-proto-grpc) and are imported only when selected.

auto resolves to OTLP HTTP or to the no-op, never to gRPC. It does not read the endpoint URL scheme and does not check which exporter packages are installed, so the rule stays the same however you deploy. Set exporter="otlp-grpc" to send over gRPC. Because auto already means HTTP whenever an endpoint is present, pinning "otlp-http" yourself for that case changes nothing.

Works with all logging backends

The tracing context is injected into all three logging backends (stdlib, loguru, structlog). Use whichever logger you prefer:

# stdlib
import logging
logger = logging.getLogger(__name__)
logger.info("message", extra={"key": "value"})

# loguru
from loguru import logger
logger.info("message", key="value")

# structlog
import structlog
log = structlog.get_logger()
log.info("message", key="value")

All produce the same JSON output with tracing context included.

Automatic instrumentation

@instrument traces your own functions. To trace incoming HTTP requests and database or cache calls without touching every handler, Trace exports the request spans of your FastAPI, Starlette or Litestar app and instruments the providers it manages and every other library you use that ships an OpenTelemetry instrumentor.

Install the instrumentor packages alongside the OpenTelemetry SDK:

pip install "grelmicro[opentelemetry,instrumentation]"

The instrumentation extra bundles the Redis and asyncpg instrumentors. Requests need none: FastAPI traces its own, and grelmicro traces Starlette and Litestar. The set of installed opentelemetry-instrumentation-* packages defines what gets traced : add opentelemetry-instrumentation-sqlalchemy or opentelemetry-instrumentation-httpx and Trace wires them too, with no code change. This matters when your app uses its own database client (a SQLAlchemy or asyncpg engine) instead of a grelmicro PostgresProvider : the spans appear all the same.

Then Trace does the rest. Request spans wrap each handler, and database, cache, and outbound HTTP spans nest under them, all bound to the app's tracer provider:

from fastapi import FastAPI

from grelmicro import Grelmicro
from grelmicro.providers.postgres import PostgresProvider
from grelmicro.providers.redis import RedisProvider
from grelmicro.trace import Trace

micro = Grelmicro(
    uses=[
        Trace(service_name="payments-api"),  # instrument=True is the default
        RedisProvider("redis://cache"),
        PostgresProvider("postgresql://db/payments"),
    ]
)

app = FastAPI()
micro.install(app)  # the FastAPI app is instrumented too

instrument is on by default and degrades to a no-op when an instrumentor package is absent, so it does nothing until you install the extras. A grelmicro-managed Redis client is traced precisely (per-client). Every other installed instrumentor is attached process-wide, so a library the app uses through its own client is traced without a grelmicro provider.

Select what to instrument by OpenTelemetry instrumentor name (the same names as OTEL_PYTHON_DISABLED_INSTRUMENTATIONS):

Trace(instrument=False)                      # nothing (the @instrument decorator still works)
Trace(instrument=["fastapi", "asyncpg"])     # only the named targets, an unknown name raises
Trace(instrument={"sqlalchemy": False})      # every installed target except the named ones

What is covered:

Target Spans Notes
FastAPI, Starlette, Litestar Incoming HTTP requests and WebSocket connections see Requests
Redis Cache and lock commands per-client when grelmicro owns the client, cluster included
asyncpg Queries covers a grelmicro PostgresProvider and any app-owned asyncpg or SQLAlchemy-on-asyncpg engine
Any other installed instrumentor Per that library e.g. sqlalchemy, httpx, psycopg : install the package and it is wired
Valkey Commands grelmicro ships a first-party instrumentor (valkey-py has no OpenTelemetry package). Spans match the Redis ones
SQLite None aiosqlite has no OpenTelemetry package. Use @instrument.

asyncpg and SQLAlchemy together

If both the asyncpg and SQLAlchemy instrumentors are installed they would double-span the same queries (SQLAlchemy runs through asyncpg). Trace keeps asyncpg and drops SQLAlchemy with a warning. Pass instrument={"asyncpg": False} to trace at the SQLAlchemy layer instead.

Requests

A FastAPI, Starlette or Litestar app gets a span for every request and WebSocket connection, named by its route, such as GET /v1/items/{item_id}. micro.install(app) exports those spans through Trace and the request metrics through Metrics. FastAPI records them itself. On Starlette and Litestar, grelmicro records the same spans and metrics, with the same names and attributes.

  • Trace(instrument={"fastapi": False}) turns the request spans off, and so do "starlette" and "litestar". The request metrics stay. A Trace with no endpoint turns them off too.
  • An exception the app does not handle is an exception event on the request span. Set OTEL_SEMCONV_EXCEPTION_SIGNAL_OPT_IN=logs to leave it off the span.
  • OTEL_PYTHON_FASTAPI_EXCLUDED_URLS, OTEL_PYTHON_STARLETTE_EXCLUDED_URLS or OTEL_PYTHON_LITESTAR_EXCLUDED_URLS, or OTEL_PYTHON_EXCLUDED_URLS when it is unset, lists comma-separated regular expressions. A request whose full URL, such as http://api.example/healthz, matches one is neither traced nor measured.
  • A request to a mounted app is one span, recorded by the app you installed, and its route keeps the mount prefix, such as GET /tenants/{tenant}/items/{item_id}. The installed app's instrument name decides for the mounted app too.

On FastAPI, with Trace or Metrics registered, FastAPI sets up no exporters of its own from OTEL_EXPORTER_OTLP_*. Trace exports the spans and Metrics the metrics. With Metrics alone, register Trace to export the spans. FastAPI's own log records are not exported. Settings you pass to FastAPI(telemetry={...}) apply. grelmicro only turns things off, and runs your exclude before its own.

Under a FastStream app, micro.install(app) also wires the broker's OpenTelemetry telemetry middleware, so consumed and published messages get spans, on top of the provider and library spans. It is selected by the same directive (faststream) and is a no-op when the broker's faststream telemetry support is not installed.