Promotion and rollback records

Recovery has more than one clock

Git records what you want, the controller records what it last observed, and the workload runs some configuration now. They can disagree while a change is pending or failing. A rollback is a deliberate move to a known earlier behavior; it needs the same identity checks as a forward release.

A release needs more than a green build

Keep four identities together: source revision, image identity, desired-state revision, and observed deployment. If a portal shows only the build SHA, it cannot tell whether Argo CD applied that revision or whether an admission rule rejected it.

Record the actual time and outcome of each transition. A failed sync is useful evidence. A stale dashboard is not proof of a current failure or success; include a last-observed time and a source link.

Know the rollback boundary

Reverting a desired-state change can restore an older image or configuration. It cannot reverse new database writes, recover a deleted volume, or guarantee compatibility with a migrated schema. Before promotion, decide whether both old and new application versions understand the current schema.

With automated reconciliation, an imperative image rollback can be overwritten by Git's desired state. Fix the authoritative declaration and reconcile it. The current platform lab keeps manual sync while learning this process. Direct emergency actions require a follow-up Git change and an incident record.

Walk through a failed change

Consider this illustrative timeline, not a report of a real outage:

1. Revision R1 requests image A, and the saved transaction check passes.
2. Revision R2 requests image B. The manifest is accepted, but the transaction check fails.
3. You review whether B changed database schema or wrote incompatible data. If it did, returning to A may not be safe.
4. For a configuration-only regression with verified compatibility, prepare a revert that restores the intended image/configuration. Keep the reason and affected identities in the change record.
5. With the lab's pinned Application, point it at the reviewed recovery revision and request manual sync. A Git revert alone does not change a pinned target.
6. Compare the observed revision and image, then repeat the same transaction check used at R1.

Stop and reason: at step 5, Argo reports Synced but the transaction is still missing. Has recovery succeeded? No. The desired resources may match while application behavior or data remains wrong. Follow the data/application runbook instead of repeatedly changing Git.

For a low-risk rehearsal, use the environment lab's harmless annotation change and its reversal. It teaches revision selection and observation; it is not a substitute for database recovery testing.

Try it

Use a harmless overlay annotation change as the first promotion rehearsal. Retain the old revision, create and review the new commit, sync it, and observe the annotation. Then restore the previous value through another reviewable commit. Confirm the images and saved transaction stayed unchanged.

Create an evidence row for each attempt: request ID, source, artifacts, config commit, policy result, sync result, application check, and recovery result. Do not store private database identifiers in the public catalog; link to access-controlled evidence instead.

Checkpoint and revision

Can someone reconstruct what was intended, what ran, and why recovery was chosen? Distinguish rollback of configuration, restoration of data, and a forward fix. Make these choices visible in the platform so an operator knows which kind of recovery is being requested.

Compare your reasoning

Keep configuration rollback, application rollback and data restore separate in your notes. They may be used together, but each has different evidence and risk. Carry the exact reviewed base into the GitOps ownership lab next.

Sources

Argo CD automated sync and rollback↗, Git revert↗.