Tracing with OpenTelemetry

Trace a boundary you can verify

A span represents an operation with timing and context. Related spans form a trace when context is propagated. OpenTelemetry provides instrumentation and collection; a Collector is not itself a durable trace-query database.

MicroBank has a Python Accounts service, a Java Ledger service, and asynchronous SNS/SQS processing. Adding HTTP instrumentation does not automatically prove that the outbox, message envelope, and consumer preserve trace context. Demonstrate each boundary separately.

Read a trace as relationships, not a stopwatch sum

Each span has a span ID, trace ID, start/end times, attributes, and optionally a parent or links. A parent-child relationship means the operations are related through propagated context; it does not mean their durations should be added. Instrumentation creates spans in the application. An exporter sends them. The Collector receives/processes/forwards telemetry. A trace backend stores and queries it. OTLP is the telemetry transport format/protocol, not the business API.

Imagine a deliberately invented trace with a 240ms HTTP parent, a database child from 20–100ms, and a broker child from 110–150ms. The known children account for 120ms of non-overlapping time within the 240ms parent; adding all three gives 360ms and double-counts work. The remaining interval needs more evidence. With overlapping children, even summing child durations can overcount elapsed time.

Guided paper exercise: draw those three bars on one time axis, then label a later SQS consumer span that has a different trace ID. Can you call it the same distributed trace? No. A matching transaction ID may help correlate logs, but trace continuity needs context carried through the message path. Decide where the producer should inject and consumer extract that context, including the outbox record if it delays publication. This is proposed instrumentation until tested in MicroBank.

Sampling chooses which traces are retained; a missing trace may therefore be a coverage issue. Propagated context can also cross trust boundaries, so validate what enters attributes and never use a trace ID as authorization. Start with a single verified request before extending across asynchronous processing.

Incremental implementation

First add compatible OpenTelemetry Python instrumentation to a separate Accounts image and lock the selected dependencies. Enable FastAPI instrumentation, set a clear service name, and send OTLP to a local Collector. Instrument Ledger separately with a reviewed Java agent or SDK setup. Start by tracing a single HTTP request, then work toward the full event flow.

This minimal Collector configuration receives OTLP/HTTP and prints spans for local inspection. Save it as infra/study/observability/otel.yaml; use a distribution containing these components:

receivers:
  otlp:
    protocols:
      http:
        endpoint: 127.0.0.1:4318
processors:
  batch: {}
exporters:
  debug:
    verbosity: basic
service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [batch]
      exporters: [debug]

The loopback bind is for applications sharing that host network. Containerized applications need an explicitly reachable private endpoint; configure the bind address and network mapping for that private path while keeping the Collector off the public network. Validate with the selected Collector's validate --config command.

Evidence and limits

Capture service name, span name, duration, and parent/trace relationship for one request. Then document the work needed to carry context through the outbox and queue. Do not put customer data into span attributes. Add Tempo or Jaeger for persistent search only after measuring available resources. Debug output is enough to learn the first instrumentation boundary.

Evidence answer: the same service name and nearby timestamps do not prove parentage. Check the trace IDs and parent/link relationships in the exported spans. With the debug exporter you can learn this before installing a storage backend. Continue to the observability lab with a clear separation between its implemented saved-case metrics and these optional logging/tracing extensions.

Sources

Python automatic instrumentation↗, Java agent↗, context propagation↗, and Collector configuration↗.