Distributed tracing

Sulu can emit OpenTelemetry traces for every authenticated HTTP request, propagating trace context through the V2 RabbitMQ ingest pipeline so a single trace spans:

POST /api/v2/launches → controller → publisher → exchange → queue → consumer → IdempotentIngestService → DB write → WebSocket publish.

The feature is opt-in. Disabled by default to keep small installations lean.

Enable

Set in .env:

SULU_TRACING_ENABLED=true
SULU_TRACING_SAMPLE_RATE=0.1
SULU_TRACING_OTLP_ENDPOINT=http://tempo:4318/v1/traces

Then start the observability overlay (which now bundles Tempo):

docker compose -f docker-compose.yml -f docker-compose.prod.yml \
    -f docker-compose.observability.yml up -d tempo grafana

Grafana auto-provisions a Tempo datasource at http://tempo:3200. Open Grafana → Explore → choose Tempo → enter a service name (sulu-backend) and a time range to find traces.

Sampling

SULU_TRACING_SAMPLE_RATE is head-based — at trace root we decide 1-in-N to keep. Recommended settings:

Environment	Rate
Local development	1.0 (capture all)
Staging	0.5
Production (low traffic)	1.0
Production (high traffic)	0.05–0.1

Errors are NOT preferentially kept at this sampling layer — tail-based sampling with error-bias is a future feature.

Loki ↔ Tempo correlation

When tracing is on, every log line carries traceId=<hex> in MDC. Grafana's Loki datasource is configured with derivedFields so the traceId value renders as a clickable link in the log view, jumping directly to the trace.

TLS for remote Tempo

Default SULU_TRACING_OTLP_ENDPOINT=http://tempo:4318/v1/traces is plain HTTP — safe inside a Docker network. For a Tempo instance on a different host:

1. Set SULU_TRACING_OTLP_ENDPOINT=https://tempo.example.com:4318/v1/traces
2. Configure Spring Boot SSL bundle (see Spring Boot 4.1 docs: spring.ssl.bundle.jks.*).

A first-class SULU_TRACING_OTLP_TLS_* env var set may ship in a future release.

Troubleshooting

• No traces appear in Grafana → check that the OTLP endpoint is reachable from the backend container; check SULU_TRACING_SAMPLE_RATE > 0; verify Tempo is healthy (docker logs sulu-tempo).
• Traces appear but no Loki links → verify Logback layout includes %X{traceId}; Spring Boot 4.1 with tracing on adds it to MDC automatically. Check the regex in datasources.yml matches the actual log line format.
• Performance impact → at 10% sampling, the overhead is single-digit microseconds per request. Bump to 100% only in local dev.

See also: Observability, Backup and restore.