Logs with Loki and Alloy

Start with the logs you actually have

Use kubectl logs for Accounts and Ledger before adding a log platform. Confirm which service emits a transaction identifier, whether timestamps are usable, and whether sensitive values or verbose SQL appear. A dashboard cannot reconstruct fields the application never emitted.

Alloy can collect Kubernetes Pod logs and forward them to Loki. Loki indexes stream labels and stores log contents. Begin with one namespace and a short retention policy; do not collect every host and system log by default. Use current Alloy guidance rather than introducing the retired Promtail agent.

Read one event and choose its labels

A useful log event answers when, where, and what happened. For example, this is an invented structured record, not output from the current MicroBank source:

{"time":"2026-10-01T10:00:00Z","level":"ERROR","service":"ledger","event":"db_timeout","transaction_id":"fixture-7","duration_ms":3000}

service=ledger is a plausible bounded stream label; transaction_id belongs in the log content. level describes severity chosen by the application, not necessarily user impact. An ERROR can be retried successfully, while a failed user journey may have no ERROR line at all. Timestamps and an operation identifier make correlation possible, but do not establish causality by themselves.

Alloy discovers and reads a source, attaches selected labels, and sends records to Loki. Loki stores them for querying; Grafana provides a view. Diagnose those hops in order: if kubectl logs has the event but Loki does not, check discovery scope, read permissions, collector errors, write endpoint, labels, time range, and retention. Restarting the application is unlikely to repair a wrong collector label mapping.

Small practice without a log stack: copy the invented record into a private scratch note and choose exactly three fields needed to investigate its timeout. A reasonable choice is timestamp, service, and transaction ID, with the event and duration explaining the failure. Remove any credential or raw request body from your proposed format. Then inspect 20 actual local Ledger lines and record which of those fields are really available. This separates the logging design you want from instrumentation you have.

A bounded implementation exercise

Use the official Alloy Kubernetes collection guide, selecting a pinned chart/release compatible with your cluster. Scope discovery to microbank. Review the rendered ServiceAccount and read-only Pod/log permissions before installation. Configure the write endpoint to your local Loki service, then check collection errors and Loki ingestion.

Use low-cardinality labels such as namespace, application, and container. Keep transaction IDs in log fields, not stream labels. If your logs are plain text, search the text directly. Use structured-field queries only when those fields are present.

A starting LogQL query after assigning the corresponding labels is:

{namespace="microbank", app="ledger"} |= "ERROR"

The labels must match your collector's actual mapping. Follow a synthetic transaction through the available logs and list where correlation is missing. Stop this optional stack before adding tracing if host memory pressure rises.

Checkpoint

Capture one actionable event, its source Pod, timestamp, relevant context, and the query used. Record the retention and storage limits, along with the cleanup steps. Do not publish database credentials, authorization headers, email addresses, or entire request bodies as observability evidence.

Query answer: the example LogQL selector searches only streams whose configured labels match; |= "ERROR" then filters their text. It is not a JSON field parser. If the collector uses a different application label, adjust the query to the observed mapping rather than assume ingestion failed. Keep a small matched event and its query; tracing next adds timing relationships that logs alone may not contain.

Sources

Alloy Kubernetes logs↗, Loki labels↗, and Promtail lifecycle↗.