Build, Break & Operate

Ship Boring is the project. Build, Break & Operate is its ongoing engineering journal, with MicroBank as the application. The aim is to grow from deploying software to providing a platform that other developers can understand, use, and recover when something fails.

Each roadmap stage needs a working system and evidence before I move on. Local experiments come first. Cloud exercises follow once the deployment requirements, recovery procedure, and cost boundaries are understood.

The application: MicroBank

MicroBank gives the work a concrete transaction flow instead of a collection of disconnected infrastructure demos. Its three logical tiers are a browser interface, backend services, and persistent data; the backend itself spans multiple services and asynchronous messaging.

The reviewed public source contains a React and TypeScript frontend, a Python/FastAPI Accounts API, a Java/Spring Boot Ledger service, separate PostgreSQL databases, and SNS/SQS integration. Dockerfiles, Compose configuration, and a LocalStack initializer provide a starting point. Source review establishes what is present; it does not establish that every transaction works end to end.

The first milestone is to verify account creation, deposit submission, eventual ledger settlement, withdrawal, rejected transactions, and transaction history with synthetic data. Record API contracts, readiness checks, database migrations, retries, and idempotency behaviour. Fix and document any gaps before treating that flow as the baseline for another environment.

Some repository descriptions are broader than the implementation: notifications currently include mock frontend data, and an initialized S3 bucket does not demonstrate working document storage. Those capabilities need their own implementation and verification before appearing as completed work here.

Local deployment tracks

Virtual machines. Start with Linux services for the application and its databases, with a small, explicit load profile. Use Terraform to provision infrastructure where the chosen platform has a suitable provider. Use Ansible for OS configuration, runtimes, service units, application configuration, and repeatable deployment. Make clear which settings each tool owns so Terraform and Ansible do not both manage the same thing.

Containers on Mac. Run the same application through Docker Compose. Exercise image creation, health checks, local networking, persistent volumes, restart behaviour, and cleanup. Measure the footprint before carrying the workload into an orchestrator. ARM64 compatibility must be checked for every image and VM guest used on the local Mac.

Two-node kubeadm. Use one control-plane node and one worker for the MicroBank deployment track. Learn bootstrapping, CNI networking, storage choices, scheduling, certificate maintenance, upgrades, and recovery. The published cluster chapter is earlier foundation work; it is not evidence that MicroBank has been deployed to it. This topology has no redundant control plane. Follow the kubeadm prerequisites and record actual resource use.

kind. Use disposable kind clusters for manifest validation, integration checks, and release experiments. Recreate a cluster from code rather than letting it become an undocumented permanent environment. kind and kubeadm serve different learning exercises and need not run together.

Three-VM Nomad. Use one Nomad server and two clients as a small learning topology. Compare job specifications, health checks, service discovery choices, scheduling, and rolling deployment with the Kubernetes path. A single server provides no server redundancy; three VMs do not make this a highly available server quorum. Consult the Nomad architecture before extending it.

Cloud emulation and portability

Use Floci for the learning profiles. Start with the SNS, SQS, and S3 API operations that MicroBank actually calls. Record the emulator version, supported operations, known differences, and endpoint configuration. See Floci documentation and the Floci project. An emulator is useful for local integration checks; it does not prove cloud IAM, managed-service behaviour, performance, or complete API compatibility.

EKS is a sensible first cloud target because the application already uses AWS messaging APIs. GKE and other platforms become additional deployment targets through explicit infrastructure and application configuration. Moving the containers to GKE does not automatically replace SNS and SQS: either retain those AWS dependencies as a documented hybrid deployment, or implement and test an alternative messaging adapter.

Keep a common application contract for configuration, health, telemetry, and packaging. Keep platform-specific networking, identities, storage classes, load balancers, and cloud services explicit. Database backup, restore, migration, and cutover remain separate work; switching an overlay does not move application data.

GitOps and environment promotion

For Kubernetes, use Argo CD to reconcile declared application state from Git. Structure Kustomize around a shared application base, environment overlays for dev, QA, preprod, and production, and separate platform patches where the infrastructure differs. Avoid copying whole manifest trees for each combination. The Argo CD integration and Kustomize documentation define the supported mechanisms.

Build each container image once, record its immutable digest, and promote that same digest through the container environments using reviewed Git changes. Each gate should capture its test results, configuration change, policy result, and recovery decision. Native VM packages are different artifacts, linked to the same source revision; they are not the container image.

On the 16 GB lab, these environments are logical deployment configurations exercised in sequence. Four named overlays do not imply four permanent clusters. Start with a dev-to-QA promotion exercise, then add the stricter gates for preprod and production. Keep credentials out of Git and document the secret-delivery mechanism for each target.

Treat database changes as part of release compatibility. Prefer changes that allow the old and new application versions to coexist during rollout. Reverting a Git commit or rolling back an image does not undo a database migration or reverse a completed banking transaction.

Policy-as-code with Kyverno

Introduce Kyverno policies in audit mode, inspect the reports, and then enforce a small, documented baseline. Initial candidates are ownership labels, resource requests and limits, permitted registries, and workload security settings. Scope policies so platform components have deliberate, reviewable exceptions instead of broad bypasses.

Add image signing and verification only after the build identity and signing workflow exist. Include both a policy rejection and the corrected deployment in a chapter. A policy should give the developer a clear error and a repeatable fix.

Progressive delivery with Argo Rollouts

Start Argo Rollouts canaries with an HTTP-facing component such as the frontend or Accounts API. Define an observable success condition before automating promotion: request success, latency, and a minimum number of meaningful samples. Low local traffic must not produce a misleading green result.

Precise traffic percentages require a supported traffic-routing integration. Without one, canary weights are approximated through replica counts, which is particularly coarse in a small lab. Document which mechanism each experiment uses rather than labelling one extra pod as an exact percentage of traffic.

Use analysis runs to promote, pause, or abort based on defined measurements. Missing or insufficient data should pause the experiment for investigation. Demonstrate a deliberately faulty release and its recovery. An abort concerns the rollout; it is not a database rollback. Queue consumers need a separate design for message ownership, compatibility, and processing guarantees.

Synthetic users: local only

Add a small TypeScript/Playwright runner that acts like a user against the local MicroBank instance. Its proposed schedule is every five minutes, configurable after measuring cost on the laptop. Permit one run at a time, use bounded waits for settlement, and skip overlapping runs. Waking the Mac should not trigger a backlog of missed tests.

Use synthetic accounts and a controlled journey: create or select an account, submit a deposit, wait for the expected settlement, submit a withdrawal, and verify balances and history. Exercise API idempotency separately. Use the actual local application, databases, and chosen emulator. A local test identity does not validate production authentication or an external identity provider.

Run the probe outside the cluster being tested so an application-cluster failure can still be reported. Keep its destination allowlist local, restrict outbound access, provide no cloud credentials or cloud endpoint fallback, and exclude it from cloud deployment profiles. These are implementation requirements to verify, not protections already delivered by this roadmap.

Report the last successful run, failure reason, duration, and freshness. A stopped runner or sleeping host should become stale or unknown, never indefinitely healthy. Keep failure screenshots and traces for a bounded period, remove synthetic data predictably, and avoid recording credentials. Playwright assertions provide the basis for bounded browser checks.

This synthetic-user canary checks whether a journey works over time. It is distinct from an Argo Rollouts canary, which exposes a new version progressively. Both can contribute evidence, but neither replaces the other. Load testing with k6 is a separate, explicitly started local experiment; periodic probes and load generators will not run against paid cloud environments.

Observability and operating evidence

Begin with Prometheus, Grafana, and Alertmanager. Track request rate, errors and latency, service readiness, queue backlog, transaction settlement delay, database availability, and synthetic-user freshness. Define what an operator should do for each alert and demonstrate the response. Keep Prometheus retention bounded and use Alertmanager for routing and grouping.

Add OpenTelemetry and its Collector when cross-service tracing answers a real diagnostic question. Evaluate Tempo for traces and Loki for logs in later, measured profiles. Do not deploy every telemetry component by default on the laptop. Avoid account or transaction identifiers as metric labels; use appropriately protected logs and trace correlation instead.

Use k6 to create controlled load and compare a baseline with a changed system. Queue-based autoscaling, including KEDA, comes after understanding workload and metrics. Evidence should include the test shape, resource budget, result, and limits of the experiment rather than an unsupported performance claim.

Failure exercises include a stopped service, an unavailable dependency, an incompatible deployment, a policy rejection, and a restore from backup. Each chapter records the symptom, diagnosis, recovery, and a runbook improvement. Do not destroy the only copy of useful data to demonstrate a failure.

The 16 GB home lab

The starting machine has 16 GB RAM and 512 GB storage. Run one environment profile at a time: VM services, Compose, kubeadm, kind, or Nomad. Reserve memory for the host OS, editor, browser, and test runner, then measure the application before allocating the remainder to platform services.

The full GitOps, policy, rollout, and observability stack has not yet been proven to fit alongside MicroBank. Add components incrementally, record requests and limits, and stop inactive clusters and VMs. Use short telemetry retention, bounded build caches, and regular image cleanup. A working allowance of 100–150 GB of free disk is an initial planning estimate, not a measured requirement or a claim about current free space.

Multiple VMs on one machine still share one physical failure domain. The two-node kubeadm and three-VM Nomad tracks teach operations within this constraint; they do not demonstrate host-level high availability. Keep recovery artifacts outside disposable cluster disks and verify ARM64 image and guest compatibility where applicable.

Cost and platform engineering

Local-first work limits cloud spend, but local power, storage, and any software licences still have a cost. For each cloud exercise, declare the account or project, region, resources, ownership labels, expected duration, and cleanup steps. Review the actual bill and resources remaining after teardown. Budget notifications help visibility; they are not a universal hard spending cap.

Reuse what proves useful: a service template, deployment contract, environment configuration, policy baseline, dashboard, and recovery runbook. Measure whether another developer can deploy and diagnose a service with these building blocks. The test is whether self-service, feedback, and clear ownership make the work easier for another developer.

Final capstone: an internal developer portal

Once Ship Boring has working deployment, promotion, policy, observability, and recovery workflows, implement Backstage or Port as its internal developer portal (IDP). This is the final planned capstone: make the proven MicroBank platform usable through developer self-service, then add bounded autonomy across the software development lifecycle (SDLC).

Choose one portal after a small integration trial. Compare the required integrations, maintenance effort, operating cost, and footprint alongside the 16 GB lab. Backstage's software catalog models software ownership and metadata, and its Software Templates scaffold components from parameterized templates. Port's self-service actions can trigger existing automation and report execution results. The project must still implement and verify the connections to its workflows.

The capstone will bring together:

• A service catalog: MicroBank components, APIs, owners, dependencies, environments, documentation, dashboards, and runbooks, with links to their source of truth.
• Reusable development paths: templates drawn from the working application patterns, including tests, delivery configuration, telemetry, and ownership metadata.
• Self-service delivery: request a change or environment, open a reviewable pull request, build and test, publish an immutable artifact, and promote it through dev, QA, preprod, and production gates.
• Operating feedback: show the deployed revision, pipeline status, Kyverno policy results, Argo Rollouts analysis, health signals, and recovery actions. Every action returns a run identifier, progress, errors, and links to evidence.

Portal actions will invoke versioned workflows that already work independently. Kubernetes changes go through the existing GitOps path; VM and Nomad deployments invoke their respective automation. Credentials remain with the authorized execution backend. The portal adds a common entry point while retaining the deployment differences and ownership boundaries established earlier.

Start with deterministic automation. Then experiment with an agent that uses catalog context to propose changes and invoke a limited set of approved actions. Require explicit approval for production promotion, destructive operations, or actions that expand cloud spend. Bound retries, record decisions and results, and stop on missing evidence; retrying a banking transaction blindly is not a recovery strategy. Validate these controls with the selected portal and workflow backend. Port documents tool approval controls; selecting a portal alone does not make the SDLC autonomous.

Periodic synthetic-user runs remain local only, including when the portal is hosted elsewhere. The portal may display their results and freshness; cloud deployment profiles must not acquire a scheduled probe or load generator through a template or action.

Completion means a developer can use the portal to deliver a MicroBank test change, follow its approval and promotion history, inspect policy and rollout evidence, and exercise a documented failure and recovery path without undocumented cluster commands. Capture the complete audit trail, cleanup result, resource use, and remaining manual steps. This capstone is planned; no portal or autonomous workflow is claimed as implemented yet.

Learn alongside the project

The Focused Study Workspace organises supporting concepts into System engineer foundations, DevOps foundations, Advanced DevOps, and Platform engineering. The learning paths contain short lessons and lab guides; this series records implementation decisions and evidence from Ship Boring. Available material and planned modules are labelled separately.

How each chapter works

1. State the problem, prerequisites, versions, and resource budget.
2. Build the smallest reproducible implementation and verify the user journey.
3. Break a specific assumption through a controlled experiment.
4. Explain what fought back, using observations rather than an idealised account.
5. Recover, capture operating evidence, and improve the runbook.
6. Link the code and record what is complete, what remains planned, and what the experiment cannot prove.

The published chapters retain their original URLs and context as the project evolves. New work will extend the series when the evidence is ready.

Related learning journey

From Systems to Agentic DevOps connects the same application to focused concepts and the existing study paths. This build journal continues to own implementation milestones and their evidence.