Skip to content

Deployment

What a container image and a Kubernetes manifest need before they run a grelmicro application in production.

Declare the environment

Set GREL_ENVIRONMENT=production in the manifest of every deployed environment:

GREL_ENVIRONMENT=production

Four values name a tier: development, test, staging, and production. They match in any case, so Production reads as production. They are the well-known values of the OpenTelemetry deployment.environment.name attribute, and grelmicro writes the declared value into that attribute on the tracer resource, so the same variable names your environment in every trace.

Declaring production or staging turns on the backend check below. This is the one place to set it per environment, so it belongs in the manifest rather than the image.

Any other value names no tier grelmicro can act on, so it neither gates the check nor reaches the trace attribute. It reads as undeclared, and says so:

GrelmicroConfigWarning: GREL_ENVIRONMENT='preprod' is not one of development,
test, staging, production, so the backend check runs as if it were
undeclared.

A fleet that calls its tiers qa and preprod keeps booting, and prodution is loud instead of silent.

The value names a tier, not the name your organisation gave the environment. Map yours to the closest of the four: a deployed environment that runs more than one replica is staging, a throwaway CI run is test. You know which one an environment called integration is, and grelmicro does not.

The backend check

A Lock on a memory backend is not a lock. It excludes nothing once a second replica runs, and without a check it says nothing about it. So in production and staging, a pattern that promises a guarantee across replicas refuses to start on a backend that cannot deliver it:

BackendScopeError: Coordination('default') is bound to MemoryLockAdapter,
which provides scope 'process', but requires scope 'cluster' in environment
'production'. Use a Redis, Valkey, Postgres, or Kubernetes backend, or pass
requires= to say what reach you want.

The check runs before the first connection opens, so a misconfigured pod fails at boot instead of on the request that needed the lock.

A backend provides a scope, and a component requires one. The message names both sides, so the fix is in the sentence that reports the problem.

Backend scope Backends State is shared by
process Memory One process
host SQLite The processes on one host
cluster Redis, Valkey, Postgres, Kubernetes Every process that connects

cluster means every process connected to that backend, wherever those processes run. One Redis serving two Kubernetes clusters still holds one lock.

Coordination and Outbox require cluster, because a lock, a leader, a distributed cron and an outbox all promise something across replicas. Cache, RateLimiterComponent and CircuitBreakerComponent require process: a per-replica cache and a per-replica circuit breaker are the standard shape, not a mistake.

IdempotentRequests and Idempotency require cluster too, because a replay has to find the stored response wherever the retry lands. They store through a Cache, so the check reads the backend of that Cache:

BackendScopeError: IdempotentRequests('default') rides Cache('default'), which
is bound to MemoryCacheAdapter and provides scope 'process', but requires
scope 'cluster' in environment 'production'. Use a Redis, Valkey, or
Postgres backend, or pass requires= to say what reach you want.

Say what you want with requires= and the default no longer applies:

from grelmicro import Grelmicro
from grelmicro.coordination import Coordination
from grelmicro.providers.memory import MemoryProvider
from grelmicro.providers.redis import RedisProvider
from grelmicro.resilience import RateLimiterComponent

memory = MemoryProvider()
redis = RedisProvider("redis://localhost:6379/0")

micro = Grelmicro(
    uses=[
        # One replica owns this schedule, and a restart may repeat a run.
        Coordination(memory, name="cron", requires="process"),
        # A budget shared by every replica, or the app does not start.
        RateLimiterComponent(redis, requires="cluster"),
    ]
)

It reads in both directions. Lower the bar to accept the scope you have, or raise it to make the wiring you meant a startup condition: a RateLimiterComponent(redis, requires="cluster") fails at boot the day someone points it at memory.

Only a bound backend is checked. A component you never registered is the documented way to say you want local behavior: a @cron with no gate fires on every replica on purpose. Wiring a memory schedule backend instead says you expected the fleet to agree, so that one is reported.

A pattern that holds a backend of its own is checked too, even though no app registers it. Lock("cart", backend=MemoryLockAdapter()) is reported by the next app that opens, and by micro.check_backends(). Built inside an open app, it is reported where it is built. The same goes for TaskLock, ReadWriteLock, LeaderElection and a gated @cron.

A lock takes no requires= of its own. To say a per-process lock is what you want, register its backend with the reach you accept, and that component answers for every pattern holding the same backend:

from grelmicro import Grelmicro
from grelmicro.coordination import Coordination, Lock
from grelmicro.coordination.memory import MemoryLockAdapter

backend = MemoryLockAdapter()
micro = Grelmicro(uses=[Coordination(lock=backend, requires="process")])
lock = Lock("cart", backend=backend)

A Bulkhead(uses=[...]) holds components the app never registers, so those are checked on first entry to the scope instead of at startup.

When the environment is not declared

An undeclared environment still reports a backend that cannot keep its promise. The report is a GrelmicroConfigWarning and a WARNING on the grelmicro logger, once at startup, on both channels like the ignored-variable report:

GrelmicroConfigWarning: Coordination('default') is bound to MemoryLockAdapter,
which provides scope 'process', but requires scope 'cluster'. Set
GREL_ENVIRONMENT to declare where this runs, or pass requires='process' to
say that is the reach you want.

So the three states differ in severity, not in what they find:

GREL_ENVIRONMENT A backend that cannot keep the promise
production, staging BackendScopeError at startup
unset Warning on both channels, once
development, test Nothing

Declare test in your test suite and the memory backends every test wires up go quiet. Testing shows where.

Check it before it deploys

micro.check_backends() asks the question the deployed app will ask, from a process that declares something else, so a test catches the wiring before a pod does. It raises the same BackendScopeError, naming every binding that does not hold:

def test_backends_hold_across_replicas() -> None:
    micro.check_backends()

It checks against production by default. Pass environment= to ask about another one.

Environment configuration

Every GREL_* variable that fills a component field is read by default. A pod that sets GREL_LOG_LEVEL=DEBUG logs at DEBUG, and a pod that sets GREL_LOCK_CART_LEASE_DURATION=120 holds the lock for 120 seconds.

GREL_ENV_LOAD=0 turns these reads off for the whole process. GREL_ENVIRONMENT and the flag itself are still read, because neither fills a component field.

Provider variables are never gated. REDIS_URL, VALKEY_URL, POSTGRES_URL and SQLITE_PATH are read whatever GREL_ENV_LOAD says, because those names belong to your environment rather than to grelmicro. A missing one fails at startup and names the variable it wanted.

When env reads are off

With GREL_ENV_LOAD=0, grelmicro names each ignored variable on two channels. It warns with EnvLoadOffWarning, which fails a test suite running -W error. It also logs a WARNING on the grelmicro logger, so on the default backend the report is a normal record in the log stream:

{"variable":"GREL_LOG_LEVEL","time":"2026-08-06T13:44:04.156560+00:00","level":"WARNING","msg":"GREL_LOG_LEVEL is set but was not applied: GREL_ENV_LOAD turns environment-driven configuration off. Unset it or set GREL_ENV_LOAD=1 to read the variable, or pass the value directly.","logger":"grelmicro"}

The variable name is in the variable field, so an alert can match on the field instead of the message text. Each name is reported once per process, at startup, and only for variables a component actually declares.

Passing env_load=False to a component is a decision, so it stays quiet.

Logs

A container needs no log configuration. The default auto format writes text to a terminal and json everywhere else, so the same image gives a readable stream in development and a parseable one in a pod. Uvicorn's own lines take the same format.

Set the level per environment, once the flag above is on:

GREL_LOG_LEVEL=INFO

Kubernetes polls the probe endpoints every few seconds for the life of the pod, and the access log reports each one. Attach ProbeFilter to drop them:

import logging

from fastapi import FastAPI

from grelmicro import Grelmicro
from grelmicro.health import HealthChecks
from grelmicro.integrations.fastapi import health_router
from grelmicro.log import ProbeFilter, configure

health = HealthChecks()
micro = Grelmicro(uses=[health])

app = FastAPI()
micro.install(app)
app.include_router(health_router())

configure()
logging.getLogger("uvicorn.access").addFilter(ProbeFilter())

Logging covers the formats, the backends and the other filters.

Health probes

Register an OpsServer and point the probes at its port. It serves /livez, /readyz, /healthz and /metrics on port 8080, next to the app on 8000. A pod that serves no HTTP, a consumer or a scheduler, uses it the same way:

from fastapi import FastAPI

from grelmicro import Grelmicro
from grelmicro.health import HealthChecks
from grelmicro.http import OpsServer
from grelmicro.metrics import Metrics

micro = Grelmicro(
    uses=[
        OpsServer(),
        HealthChecks(provider_checks="require"),
        Metrics(exporter="prometheus"),
    ]
)

app = FastAPI()
micro.install(app)
# Your routes on the app port (8000 with Uvicorn)
# GET /livez, /readyz, /healthz and /metrics on :8080

Point the liveness probe at /livez, which stays 200 while the process is alive and its liveness checks pass. Point the readiness probe at /readyz, which turns 503 as soon as a critical check fails and takes the pod out of the Service.

OpsServer binds its port once per process. Run one worker per container, as below, because a second worker in the same container cannot bind the same port.

When the platform probes only the app port, include health_router() in the app instead. The probes then answer on the public port, so read Secure the ops endpoints for what to gate.

Keep the readiness period short and the liveness period long. Readiness reacts to a lost backend, liveness only to a process that is gone or stuck. Use a startupProbe on /livez instead of a long initialDelaySeconds, so a slow first connection never counts as a liveness failure. Health checks covers the component and the endpoint behavior.

One worker per container

Run one server worker per container and scale with replicas. The liveness probe then reaches the only event loop there is, so a loop blocked by a long synchronous call stops answering and the container restarts.

With several workers in one container, the probe reaches whichever worker accepts the connection, so an idle worker answers for a blocked one. And a deadlock between coroutines or threads leaves the loop free, so no probe sees it. Liveness catches both: a stuck worker exits, and whatever runs it starts another.

Failure Without Liveness With Liveness
Loop blocked, one worker per container The liveness probe The probe, or the watchdog
Loop blocked, Gunicorn with uvicorn-worker Gunicorn, after --timeout Gunicorn, or the watchdog
Loop blocked, uvicorn --workers N Nothing The watchdog
Coroutines or threads deadlocked, loop free Nothing A liveness check

Secure the ops endpoints

The ops endpoints describe your service to whoever reaches them:

Endpoint Reveals
/livez, /readyz Whether the process is alive and ready. Nothing more.
/healthz The name of every check, and the error string of each failing one.
/metrics Your route names and their errors by exception type, every circuit breaker's state, and the name of every task, lock, cache and outbox topic.

OpsServer has no authentication, no IP allowlist and no TLS. The network in front of it is its only protection. Keep its port reachable by the kubelet, the load balancer and Prometheus, and by nothing else.

Keep the default host

host="127.0.0.1" looks safe and breaks every caller that matters. The kubelet probes the pod IP. A load balancer probes from outside the container. Prometheus scrapes from another pod or container. None of them reaches the container's loopback, so the pod never turns Ready and the liveness probe restarts it. Loopback fits only a sidecar that scrapes from inside the same pod. Keep the default, which binds every interface, and close the port at the network instead.

Kubernetes

Name the ops port in the container, as the complete manifest does, and leave it out of the Service your Ingress routes:

apiVersion: v1
kind: Service
metadata:
  name: cart
spec:
  selector:
    app.kubernetes.io/name: cart
  ports:
    - name: http
      port: 80
      targetPort: http

The kubelet probes the pod directly and needs no Service. Prometheus scrapes the pod directly too, through a PodMonitor that names the ops port. If your setup scrapes through a Service, give the ops port a separate Service of type ClusterIP, never LoadBalancer or NodePort.

Kubernetes also sets variables for every Service in the namespace. A Service named grel-ops sets GREL_OPS_PORT=tcp://10.0.0.5:8080, and one named redis sets REDIS_PORT. grelmicro ignores these service links and logs each one it would have read. Set enableServiceLinks: false on the pod, as the complete manifest does, to keep them out of its environment.

Then admit the ops port from the Prometheus namespace alone:

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: cart
spec:
  podSelector:
    matchLabels:
      app.kubernetes.io/name: cart
  policyTypes: [Ingress]
  ingress:
    - ports:
        - port: http
    - from:
        - namespaceSelector:
            matchLabels:
              kubernetes.io/metadata.name: monitoring
      ports:
        - port: ops

The first rule keeps the app port open to the Ingress controller. Replace monitoring with the namespace Prometheus runs in. The kubelet probes from the pod's own node, and a NetworkPolicy always allows connections from the pod's node, so the probes need no rule. A cloud load balancer that health-checks the pod directly probes from its own addresses, so add an ipBlock rule for them on the ops port. A NetworkPolicy takes effect only on a network plugin that enforces it.

Docker, Podman and Compose

Publish the app port only: -p 8000:8000, never -p 8080:8080. Containers on the same network still reach the ops port, so Prometheus scrapes app:8080 without it being published:

services:
  app:
    image: registry.example.com/cart:1.4.0
    ports:
      - "8000:8000"
    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost:8080/livez"]
  prometheus:
    image: prom/prometheus

The healthcheck runs inside the container, so it reaches the ops port with no port published.

Caddy and other reverse proxies

Proxy the app port only:

api.example.com {
    reverse_proxy app:8000
}

Nothing routes to app:8080, so /metrics never reaches the internet.

On the app port

health_router() and metrics_router() serve the same endpoints on the app port, where every caller reaches them. Gate what reveals something:

  • Gate /metrics with metrics_router(dependencies=[...]).
  • Gate /healthz with healthz_dependencies, and keep show_details off or behind Depends(...). Read Verbose Details.
  • Leave /livez and /readyz open. They reveal up or down, and most probes carry no credentials.

Shutdown

Kubernetes sends SIGTERM and waits terminationGracePeriodSeconds before SIGKILL. Keep Tasks(shutdown_timeout=...) at or below that window. Both default to 30 seconds, so the defaults already line up.

Removing the pod from the Service and sending SIGTERM happen in parallel, so a request in flight can arrive after the server has started to stop. A short preStop sleep holds the signal back until the endpoint change has spread.

Draining matters for the locks. A task that finishes its iteration releases its lock through async with, and LeaderElection releases the leadership lock so a standby takes over at once. A task cancelled in its body at the end of the window still runs its release. A cancel that lands during the release itself leaves the lease to expire on the backend. Graceful shutdown has the full contract.

Replicas

Lock, TaskLock and LeaderElection coordinate through the backend, so every replica has to point at the same one. Give them a provider and let the environment carry the connection:

REDIS_URL=redis+sentinel://sentinel-0:26379,sentinel-1:26379/mymaster/0
REDIS_PASSWORD=...
REDIS_SENTINEL_PASSWORD=...

The composition root then holds no connection code at all:

from grelmicro import Grelmicro
from grelmicro.providers.redis import RedisProvider

redis = RedisProvider()
micro = Grelmicro(uses=[redis])

Pods sharing a namespace with another grelmicro application need a prefix so the two do not collide on key or lease names. Providers covers the URL forms and the backend options.

Reconfigure without a restart

Mount a ConfigMap or a Secret and grelmicro re-resolves the live components when the file changes, with no rollout. See Live reconfiguration.

A complete manifest

apiVersion: apps/v1
kind: Deployment
metadata:
  name: cart
spec:
  replicas: 3
  selector:
    matchLabels:
      app.kubernetes.io/name: cart
  template:
    metadata:
      labels:
        app.kubernetes.io/name: cart
    spec:
      enableServiceLinks: false
      terminationGracePeriodSeconds: 30
      securityContext:
        runAsNonRoot: true
        seccompProfile:
          type: RuntimeDefault
      containers:
        - name: cart
          image: registry.example.com/cart:1.4.0
          securityContext:
            allowPrivilegeEscalation: false
            readOnlyRootFilesystem: true
            capabilities:
              drop: ["ALL"]
          resources:
            requests:
              cpu: 100m
              memory: 256Mi
            limits:
              memory: 512Mi
          ports:
            - name: http
              containerPort: 8000
            - name: ops
              containerPort: 8080
          env:
            - name: GREL_ENVIRONMENT
              value: production
            - name: GREL_LOG_LEVEL
              value: INFO
            - name: GREL_LOCK_CART_LEASE_DURATION
              value: "120"
            - name: REDIS_URL
              value: redis://redis:6379/0
            - name: REDIS_PASSWORD
              valueFrom:
                secretKeyRef:
                  name: cart-redis
                  key: password
          startupProbe:
            httpGet:
              path: /livez
              port: ops
            periodSeconds: 2
            failureThreshold: 30
          livenessProbe:
            httpGet:
              path: /livez
              port: ops
            periodSeconds: 20
          readinessProbe:
            httpGet:
              path: /readyz
              port: ops
            periodSeconds: 5
          lifecycle:
            preStop:
              sleep:
                seconds: 5

Checklist

  • [ ] GREL_ENVIRONMENT is set in every deployed environment.
  • [ ] micro.check_backends() is asserted in a test.
  • [ ] The startup logs carry no variable field.
  • [ ] GREL_LOG_LEVEL is set per environment, and the format is left to auto.
  • [ ] ProbeFilter is attached to uvicorn.access.
  • [ ] /livez and /readyz are served by OpsServer, with readiness the faster of the two.
  • [ ] The ops port is in no public Service, no published port and no proxy route.
  • [ ] terminationGracePeriodSeconds is at or above Tasks(shutdown_timeout=...).
  • [ ] Every replica points at the same backend, through a provider.