Building CI/CD Pipelines for Enterprise E-Commerce Deployments

1. The Deploy We Could Not Undo

A janitorial supplies retailer I worked with — around 12,000 SKUs, three locales, self-hosted Magento 2 behind a CDN — shipped a customer-account refactor on a Thursday afternoon in October 2024. Part of it split a single customer_phone column into phone_country_code and phone_national. The migration copied the data across, then dropped the old column, because leaving it felt untidy.

Ninety minutes later the checkout's address validation started rejecting about one order in six. The bug was in the new splitting logic and it only affected numbers stored without an international prefix, which was most of the German customers. The on-call engineer did exactly the right thing and hit rollback.

The application rolled back in forty seconds. It was a container image swap and it worked perfectly. And then the old code went looking for customer_phone, which no longer existed, and the site returned 500s on every authenticated page instead of failing one checkout in six. We had turned a partial outage into a total one by using the recovery mechanism we had built and tested.

Getting out took two hours and eleven minutes: restore the column from the pre-deploy snapshot, replay the writes that had landed in the new columns during the ninety minutes, verify, then roll forward with the fix rather than back. Nobody involved had done anything careless. The pipeline was good. The tests were green. The rollback button worked.

The problem was that we had built a pipeline that could deploy and a pipeline that could revert code, and we had never asked what "revert" means when the deploy included a change to state. Almost everything difficult about ecommerce deployment lives in that gap. Code is trivially reversible. Data is not. Customer sessions are not. A confirmation email that has already been sent is not.

This article is about the whole pipeline, but that question — what happens when the undo does not work — is the one I would build the rest around. I have kept it deliberately vendor-neutral: the stages, the gates, and the failure modes are the same whether you run GitHub Actions, GitLab CI, Jenkins, Buildkite, or something your platform team wrote in 2019. If you specifically want the Azure DevOps shape of this, with YAML templates, environments, approvals and service connections spelled out, I have written that up separately in the Azure DevOps pipeline guide and I will not duplicate it here.

2. "We Have CI/CD" Is Not a Statement About Anything

Nearly every merchant I meet says they have CI/CD. What they usually have is a job that runs when someone pushes to main, and it does two things: run some tests, and copy files to a server.

That is continuous integration in the sense of "a computer runs the tests", and it is deployment automation in the sense of "a computer does the copying". Neither is the thing that makes deployment safe. The questions I ask instead:

Can you name, right now, exactly which commit is running in production? Can you deploy that same commit to staging and be confident it is byte-identical? If the deploy you did an hour ago is wrong, what is the sequence to undo it and how long does it take? What in the pipeline would have stopped the last incident you had? And: how many deploys did you do last month, and how many needed manual intervention afterwards?

That last pair is the one that reveals the most. A team deploying twice a month with a two-hour change window has a pipeline that is technically automated and organisationally manual. The automation is not the point; the point is whether shipping a small change is boring.

Deploy frequency is worth taking seriously as a design goal rather than a vanity metric, because batch size drives risk almost linearly. A release containing forty commits that has been baking for three weeks has forty candidate causes when it breaks, and the person who wrote commit eleven has moved on to something else. A release containing one commit has one candidate. Everything below is easier at small batch size, and several things below only work at all at small batch size.

3. The Stages, and What Each Gate Is Actually For

A pipeline is a sequence of increasingly expensive checks against an increasingly realistic environment. Order them by cost so the cheap ones fail first.

Validate. Lint, static analysis, dependency audit, config schema check. Seconds. This stage should be fast enough that developers run it locally without being told to.

Build. Compile, bundle, assemble the artefact, tag it with the commit SHA. This is the stage that produces the thing every later stage tests, and it must run exactly once per commit. More on that below because it is the single most important structural rule in the whole pipeline.

Unit and integration test. Against the built artefact where possible, not against a fresh checkout. Minutes.

Deploy to a staging environment. The same artefact, the same deployment mechanism, different configuration. If staging is deployed by a different process than production, staging is testing the wrong thing.

Acceptance and smoke tests. A small number of end-to-end journeys against the deployed staging environment. Browse, search, add to basket, checkout with a test card, confirm the order lands in the ERP feed.

Quality gates. Performance budget, accessibility check, security scan, bundle size diff. Some block, some warn — the distinction matters and I will come back to it.

Approval. A human decision, if you need one. Increasingly I argue teams do not, for most changes.

Deploy to production. Progressively, with health checks between increments.

Verify. Post-deploy assertions against the live environment, and an automatic decision to continue, hold, or revert.

Nine stages sounds heavy. In practice validate-through-acceptance is one pipeline run of eight to fifteen minutes, and if yours is longer than about twenty minutes people will start finding ways around it. Pipeline duration is a cultural variable, not just a technical one.

4. Build Once, Deploy Many

The rule: one build per commit, producing one immutable artefact, promoted unchanged through every environment. Staging tests artefact a3f9c21. Production runs artefact a3f9c21. Not a rebuild of the same commit — the same bytes.

The reason is that a rebuild is not deterministic in any real-world stack. Between your staging build on Tuesday and your production build on Thursday, a transitive dependency published a patch, a base image moved, a CDN cached a different version of a font, or the build machine had a different Node minor. Each of those is individually unlikely and collectively routine. I have watched a team spend a day on a bug that existed only in production because the production build ran on an agent with a different libvips, which changed how one image was resized, which changed a layout, which broke a selector in a third-party script.

The pattern in practice:

#!/usr/bin/env bash
# build.sh — runs once per commit. Everything downstream consumes the digest.
set -euo pipefail

SHA="$(git rev-parse HEAD)"
IMAGE="registry.example.com/storefront"

# Reproducibility inputs: pin the base image by digest, not by tag.
# "node:22-alpine" is a moving target; the digest is not.
docker build \
  --build-arg BASE="node:22-alpine@sha256:3f8a...c19d" \
  --build-arg SOURCE_DATE_EPOCH="$(git log -1 --pretty=%ct)" \
  --label "org.opencontainers.image.revision=${SHA}" \
  -t "${IMAGE}:${SHA}" .

docker push "${IMAGE}:${SHA}"

# The digest — not the tag — is what every later stage deploys. A tag can be
# moved by anyone with push access; a digest cannot.
DIGEST="$(docker inspect --format='{{index .RepoDigests 0}}' "${IMAGE}:${SHA}")"
echo "artifact=${DIGEST}" >> "$PIPELINE_OUTPUT"

Deploying by digest rather than tag closes a real hole. storefront:v2.4.1 is a mutable pointer; someone can push a different image to it and your "rollback to v2.4.1" gets something else entirely. storefront@sha256:... cannot be moved.

The same logic applies without containers. A PHP application deployed by rsync should be built into a versioned directory — vendor installed, assets compiled, autoloader optimised — and then activated by moving a symlink. The build happens once; each environment activates the same tree.

# Release-directory pattern for a non-containerised stack.
# The build ran once, upstream; here we only unpack and switch.
RELEASE="/var/www/releases/${BUILD_SHA}"
mkdir -p "${RELEASE}"
tar -xzf "artifact-${BUILD_SHA}.tar.gz" -C "${RELEASE}"

# Shared, environment-specific state lives outside the release tree.
ln -sfn /var/www/shared/media   "${RELEASE}/pub/media"
ln -sfn /var/www/shared/env.php "${RELEASE}/app/etc/env.php"

php "${RELEASE}/bin/magento" setup:db:status   # expansion-only check, see below

# Atomic switch: ln -sfn on the same filesystem is a rename under the hood,
# so no request ever sees a half-updated document root.
ln -sfn "${RELEASE}" /var/www/current.new && mv -Tf /var/www/current.new /var/www/current
systemctl reload php8.3-fpm

# Keep five releases. Rollback is then a symlink move, measured in milliseconds.
ls -1dt /var/www/releases/* | tail -n +6 | xargs -r rm -rf

What must not be inside the artefact

Anything that differs per environment. API endpoints, feature flag defaults, database credentials, the payment provider's mode, the CDN hostname, the locale set. If any of those are baked in at build time you have destroyed build-once-deploy-many without noticing, because you now need one build per environment and the thing you tested is not the thing you shipped.

This bites hardest with front-end bundles, where the temptation to inline the API base URL at build time is strong and the tooling actively encourages it. The fix is to serve runtime configuration as a small document the application fetches or reads from a template-injected global, so the same bundle behaves correctly everywhere.

// Runtime config, not build-time. The bundle is environment-agnostic; the
// server injects this on the document before the app boots.
window.__APP_CONFIG__ = {
  apiBase: '{{ env.API_BASE }}',
  paymentMode: '{{ env.PAYMENT_MODE }}',   // 'test' | 'live'
  flagsEndpoint: '{{ env.FLAGS_URL }}',
  buildSha: '{{ build.sha }}'              // for correlating errors to a release
};

The buildSha line is small and repays itself constantly. Every error report, every performance sample, every log line should carry the release identifier, so "did this start with the deploy at 14:20" is a query rather than an argument.

5. Secrets, and the Boring Way to Handle Them

Secrets are injected at deploy time from a secret store, never committed, never in the image, never in the CI job's plain-text variables if the platform offers anything better.

Two rules I would treat as non-negotiable. Production secrets must not be readable by a pipeline running on a branch — that is how a pull request from a fork exfiltrates your payment keys. And rotation must be a routine operation you have actually performed, not a runbook nobody has executed, because a secret you cannot rotate quickly is a secret you cannot respond to a leak about.

The thing teams get wrong more subtly: secrets that end up in build logs. A deploy script that echoes its environment for debugging, a test that prints a failing request including headers, a stack trace that serialises a config object. Scrub at the log shipper as well as being careful in the scripts, because being careful does not scale across a team.

6. Which Tests Belong Where

The pipeline's job is to catch problems as early and as cheaply as possible. That means a shape, not a pile.

Unit tests for logic with branches: tax calculation, discount stacking, shipping rules, stock allocation. Fast, numerous, no I/O. For ecommerce specifically, the highest-value unit tests are around money — rounding, VAT-inclusive versus exclusive pricing, multi-currency conversion, discount interaction. Those are the bugs that are expensive and quiet.

Integration tests for the boundaries: does the ERP adapter produce the payload the ERP expects, does the payment webhook handler cope with a duplicate delivery, does the search indexer handle a product with no price. These need real dependencies or faithful fakes and they are the tests most worth investing in on an ecommerce estate, because integration boundaries are where the incidents actually come from.

Contract tests where you own both sides. If your storefront calls your own order service, pin the shape of that call in a test both repositories run, so a field rename in the service breaks the service's pipeline rather than the storefront at 3am.

End-to-end tests for a small number of journeys. Guest checkout, registered checkout, one discount code, one payment method. Five to ten scenarios, run against a deployed environment. Not fifty. Every E2E test is a maintenance liability and a flake source, and the marginal value drops off a cliff after the critical path.

Flaky E2E tests deserve a paragraph of their own because they destroy pipelines quietly. A test that fails 5% of the time in a suite of ten is a pipeline that fails 40% of the time for no reason, and the human response is to re-run it reflexively, which means the suite has stopped conveying information. Quarantine flaky tests aggressively — move them out of the blocking suite the day they are identified, fix them or delete them within a sprint. A smaller suite everyone trusts is worth far more than a larger one everyone re-runs.

// Post-deploy smoke test: not a substitute for E2E, but the thing that runs
// against production after every increment and decides whether to continue.
const CHECKS = [
  { name: 'homepage',   url: '/',                          expect: 200, maxMs: 1500 },
  { name: 'category',   url: '/womens/dresses',            expect: 200, maxMs: 2000 },
  { name: 'pdp',        url: '/p/linen-shirt-navy-m',      expect: 200, maxMs: 2000 },
  { name: 'search',     url: '/search?q=linen',            expect: 200, maxMs: 2500 },
  { name: 'cart-api',   url: '/api/cart',                  expect: 200, maxMs: 800  },
  { name: 'health',     url: '/health/deep',               expect: 200, maxMs: 3000 },
];

export async function smoke(base, buildSha) {
  const results = [];
  for (const c of CHECKS) {
    const t0 = performance.now();
    const res = await fetch(base + c.url, { redirect: 'manual' });
    const ms = performance.now() - t0;
    // Assert we are actually testing the release we just shipped, not a
    // cached response or an instance that has not rolled yet.
    const served = res.headers.get('x-build-sha');
    results.push({
      ...c, status: res.status, ms: Math.round(ms),
      pass: res.status === c.expect && ms < c.maxMs && served === buildSha,
      served,
    });
  }
  return { ok: results.every(r => r.pass), results };
}

The x-build-sha assertion is the part people leave out, and it is the reason a smoke test can pass against the previous release and tell you everything is fine.

7. Gates That Block Versus Gates That Warn

A gate that blocks stops the pipeline. A gate that warns writes a comment and continues. Choosing wrong in either direction damages the pipeline's credibility.

Block on: failing tests, a security scan finding a known-exploited vulnerability in a direct dependency, a migration that fails to apply on a copy of the production schema, a smoke test failure after deploy. These are unambiguous and actionable.

Warn on: coverage percentage, most linting opinions, bundle size increases under a threshold, minor-severity dependency findings, accessibility rule violations on pages nobody changed. These are worth surfacing and terrible as blockers, because the failure mode is a team that learns to bypass gates.

The interesting middle is performance. A bundle-size or Core Web Vitals budget should block, but only on regression relative to the current production build and only outside a tolerance band, because absolute thresholds produce a pipeline that is permanently red for reasons unrelated to today's change. Measuring properly in CI is its own discipline and I have written about the mechanics in the Core Web Vitals monitoring guide; the pipeline-design point is simply that a performance gate must compare like with like or it will be disabled within a month.

#!/usr/bin/env python3
"""Bundle budget gate: blocks on regression, not on absolute size."""
import json, sys

TOLERANCE_PCT = 3.0          # noise floor for build-to-build variation
HARD_CEILING  = 260_000      # bytes, gzipped, main entry — a line we do not cross

base = json.load(open("baseline/bundle-stats.json"))    # from the live release
head = json.load(open("dist/bundle-stats.json"))

failures, warnings = [], []
for name, size in head.items():
    prev = base.get(name)
    if prev is None:
        warnings.append(f"new bundle {name}: {size:,}B")
        continue
    delta_pct = (size - prev) / prev * 100
    if name == "main" and size > HARD_CEILING:
        failures.append(f"{name} {size:,}B exceeds ceiling {HARD_CEILING:,}B")
    elif delta_pct > TOLERANCE_PCT:
        failures.append(f"{name} +{delta_pct:.1f}% ({prev:,} -> {size:,}B)")
    elif delta_pct < -TOLERANCE_PCT:
        warnings.append(f"{name} {delta_pct:.1f}% smaller — nice")

for w in warnings:
    print(f"::warning::{w}")
for f in failures:
    print(f"::error::{f}")
sys.exit(1 if failures else 0)

On approval gates

The manual approval before production is the stage teams add first and question least. It is worth questioning.

An approval is valuable when the approver has information the pipeline does not — a marketing freeze, a known ERP outage, a sale starting in ten minutes. It is theatre when the approver is clicking a button on a change they have not read, which is what happens by week three on any pipeline that requires approval for everything.

What I would do instead: no approval for changes that touch only application code and pass every gate; approval required for anything touching a migration, a payment integration, or infrastructure; and a hard deployment freeze window that the pipeline enforces automatically rather than relying on someone remembering. A freeze that is a calendar entry gets violated; a freeze that returns "deployments are blocked until Tuesday 09:00, override requires two approvals" does not.

8. Database Migrations Are the Actual Hard Part

Everything above is solvable with good practice and available tooling. Migrations are where judgement is required, because a migration is a change you cannot take back by moving a symlink.

The framing that makes this tractable: schema changes and code changes must be deployable independently, and every intermediate state must work. During a rolling deploy, old code and new code run simultaneously against one database. That is not an edge case — it is every deploy on any system that does not take an outage.

Which means the honest question about any migration is not "does it work" but "does the old code still work after it, and does the new code work before it".

Expand and contract

The technique is well known and inconsistently applied. Split every destructive change into an expansion that is backwards-compatible, a period where both shapes are maintained, and a contraction that happens only after the old code is definitively gone.

Renaming customer_phone to phone_national — the change that caused the incident I opened with — should have been four deploys over about two weeks.

-- DEPLOY 1 (expand). Additive only. Old code is entirely unaffected.
ALTER TABLE customer_entity
  ADD COLUMN phone_country_code VARCHAR(4)  NULL,
  ADD COLUMN phone_national     VARCHAR(32) NULL;

-- Backfill in batches, outside the deploy, with a bounded loop so a long
-- transaction never holds locks across a peak-traffic window.
-- (run repeatedly until zero rows affected)
UPDATE customer_entity
SET    phone_country_code = split_country(customer_phone),
       phone_national     = split_national(customer_phone)
WHERE  phone_national IS NULL
  AND  customer_phone IS NOT NULL
LIMIT  5000;

-- DEPLOY 2 (dual-write). New code writes BOTH shapes and reads the new one.
-- Old code, if any is still running, still reads and writes the old column.
-- Application-side; shown here as the trigger equivalent for systems where
-- writes come from more than one codebase (ERP sync, admin, import jobs).
CREATE TRIGGER customer_phone_sync
BEFORE INSERT OR UPDATE ON customer_entity
FOR EACH ROW EXECUTE FUNCTION sync_phone_columns();

-- DEPLOY 3 (read switch). New code reads only the new columns. The old
-- column is still populated, so a rollback to deploy 2 is still safe.

-- DEPLOY 4 (contract). Only after the old code is provably gone —
-- days later, verified by traffic logs, not by assumption.
ALTER TABLE customer_entity DROP COLUMN customer_phone;

The discipline is entirely in the gap between deploy 3 and deploy 4. Everyone understands expand-contract in principle; what actually happens is that the contraction gets bundled into the same release as the expansion because it is the same ticket and it feels wasteful to leave a dead column lying around for a fortnight. That is precisely the decision that cost us two hours and eleven minutes.

Make it structural. A migration linter in the pipeline that refuses to run a DROP COLUMN or a NOT NULL addition in the same release as anything else, and requires an explicit annotation naming the release that introduced the corresponding expansion.

#!/usr/bin/env python3
"""Migration safety gate. Runs on every PR touching db/migrations/."""
import re, sys, pathlib

DESTRUCTIVE = re.compile(
    r"\b(DROP\s+(COLUMN|TABLE)|RENAME\s+(COLUMN|TO)|ALTER\s+COLUMN\s+\S+\s+TYPE"
    r"|SET\s+NOT\s+NULL|DROP\s+DEFAULT)\b", re.I)
BLOCKING = re.compile(r"\bCREATE\s+INDEX\b(?!\s+CONCURRENTLY)", re.I)
UNBOUNDED = re.compile(r"\b(UPDATE|DELETE)\b(?![\s\S]{0,400}\bLIMIT\b)", re.I)

fail = False
for path in pathlib.Path("db/migrations").glob("*.sql"):
    sql = path.read_text()
    head = sql[:600]

    if DESTRUCTIVE.search(sql) and "-- contract-of:" not in head:
        print(f"::error file={path}::destructive change without a "
              f"'-- contract-of: <release>' annotation naming the expansion")
        fail = True

    if BLOCKING.search(sql):
        # A non-concurrent index build locks writes for the duration. On a
        # 40M-row order table that is an outage, not a migration.
        print(f"::error file={path}::CREATE INDEX without CONCURRENTLY")
        fail = True

    if UNBOUNDED.search(sql):
        print(f"::warning file={path}::unbounded UPDATE/DELETE — batch it")

sys.exit(1 if fail else 0)

Test the migration against production-shaped data

A migration that runs in 200ms against a seeded dev database can take forty minutes against 40 million order rows, and you will discover this during the deploy.

The gate worth building: restore the most recent production snapshot into a scratch database, run the pending migrations against it, record the wall-clock time and the locks taken, and fail the pipeline if either exceeds a budget. It is not free — you need an anonymised snapshot and somewhere to put it — and it is the highest-value thing on this page for anyone with a large orders table.

#!/usr/bin/env bash
# migration-rehearsal.sh — run in the pipeline, not during the deploy.
set -euo pipefail
BUDGET_SECONDS=90

createdb rehearsal
pg_restore -d rehearsal /snapshots/prod-anonymised-latest.dump --jobs 4

START=$(date +%s)
# Statement timeout is the safety net: better to fail the rehearsal than to
# discover a 40-minute exclusive lock on the live database.
PGOPTIONS="-c statement_timeout=${BUDGET_SECONDS}s -c lock_timeout=5s" \
  ./bin/migrate up --database rehearsal
ELAPSED=$(( $(date +%s) - START ))

echo "migration wall clock: ${ELAPSED}s (budget ${BUDGET_SECONDS}s)"
[ "$ELAPSED" -le "$BUDGET_SECONDS" ] || {
  echo "::error::migration too slow for an online deploy — batch it or run it out of band"
  exit 1
}

# Then prove the PREVIOUS release still functions against the new schema.
./bin/test-suite --tag backwards-compat --database rehearsal --app-version "$PREVIOUS_SHA"

That last line is the one that would have saved us. Running the previous release's test suite against the new schema is a direct test of "is rollback safe", and it is a thing almost nobody does.

9. Feature Flags: What They Fix, and What They Cost

Flags decouple deployment from release. You ship the code dark, turn it on for 1% of traffic, watch, and expand. Turning something off is a config change measured in seconds rather than a deploy measured in minutes, and it does not touch the database at all.

For ecommerce this is genuinely transformative on the risky changes: a new checkout step, a different shipping calculator, a redesigned PDP, a new payment method. Each is something you want to expose to a fraction of real traffic with real money before committing.

Three properties a flag system needs before I would rely on it. Evaluation must be local and fast — a network call per flag per request is a new single point of failure in your critical path, so poll the ruleset into memory and evaluate against a cached copy. It must fail to a defined default when the flag service is unreachable, and that default must be the old behaviour. And bucketing must be sticky per user or per session, because a customer whose checkout flips between two variants between page loads will produce a support ticket you cannot reproduce.

// Local evaluation against a periodically-refreshed ruleset.
// No network call in the request path; unreachable service means last-known
// rules, and an unknown flag means the safe default.
import crypto from 'node:crypto';

export class Flags {
  constructor(loader, refreshMs = 30_000) {
    this.rules = {};
    this.loader = loader;
    setInterval(() => this.refresh(), refreshMs).unref();
  }

  async refresh() {
    try { this.rules = await this.loader(); }
    catch (e) { /* keep the last good ruleset; never fall back to empty */ }
  }

  enabled(key, ctx = {}) {
    const rule = this.rules[key];
    if (!rule) return false;                    // unknown flag = old behaviour
    if (rule.kill) return false;                // global kill switch wins

    for (const o of rule.overrides ?? []) {
      if (o.customerIds?.includes(ctx.customerId)) return o.value;
      if (o.countries?.includes(ctx.country))      return o.value;
    }

    // Sticky bucketing: the same subject always lands in the same bucket for
    // a given flag, so a customer's experience does not flicker mid-journey.
    const subject = ctx.customerId ?? ctx.sessionId ?? 'anon';
    const h = crypto.createHash('sha1').update(`${key}:${subject}`).digest();
    return (h.readUInt32BE(0) % 10_000) < Math.round(rule.percent * 100);
  }
}

Now the cost, which flag advocates undersell. Every flag doubles the number of code paths, and n flags in the same subsystem produce 2^n combinations of which you have tested perhaps three. Flags that live for months become permanent conditional complexity that nobody dares remove, and I have opened codebases with sixty flags of which four were still being evaluated for anything other than "always on".

The discipline that works: every flag gets an owner and an expiry date at creation, a flag past expiry fails the build, and removing the flag is a scheduled task in the same sprint that fully enables it. Treat a flag as a temporary scaffold, because a permanent flag is just a badly documented configuration option.

And do not flag a migration. A flag can switch which code path runs; it cannot switch which columns exist. Flags and expand-contract solve adjacent problems and neither substitutes for the other.

10. Getting the New Code Into Production

Three patterns, and the choice is mostly determined by your infrastructure and your state.

Rolling. Replace instances in batches. Simple, no extra capacity, and it guarantees a window where both versions serve traffic — which is fine if you have done the compatibility work above and dangerous if you have not.

Blue/green. Two full environments, traffic switched at the load balancer. Instant cutover, instant switch back, and double the infrastructure during the deploy. The catch nobody mentions: the database is usually shared, so blue/green gives you fast application rollback and does nothing at all for schema changes. Sessions are the other trap — if sessions live in instance memory rather than a shared store, a cutover logs everyone out mid-basket.

Canary. A small percentage of traffic to the new version, watched, then expanded. The most controlled and the most work, because it only means anything if the analysis is automated. A canary that a human is supposed to eyeball is a rolling deploy with extra steps.

I would default to canary for anything customer-facing on a site with enough traffic to produce a signal within minutes, and rolling with a good smoke test below that. Blue/green is worth it specifically when you need a fast, certain revert of application code and you have already handled state separately.

#!/usr/bin/env python3
"""Automated canary analysis. Compares the canary against the stable pool on
the metrics that actually correlate with revenue, then decides."""
import sys, time

STEPS = [1, 5, 25, 50, 100]      # traffic percentage
SOAK  = {1: 300, 5: 600, 25: 900, 50: 900, 100: 0}   # seconds at each step

def gate(canary, stable) -> tuple[bool, str]:
    # Error rate: absolute floor plus a relative comparison, so a stable pool
    # that is already unhealthy does not make a bad canary look acceptable.
    if canary["error_rate"] > 0.01:
        return False, f"canary error rate {canary['error_rate']:.2%}"
    if canary["error_rate"] > stable["error_rate"] * 2 + 0.001:
        return False, "canary error rate materially worse than stable"
    if canary["p95_ms"] > stable["p95_ms"] * 1.25:
        return False, f"p95 {canary['p95_ms']}ms vs stable {stable['p95_ms']}ms"
    # The business metric. Latency and errors can both look fine while the
    # add-to-basket button silently does nothing.
    if canary["orders_per_1k_sessions"] < stable["orders_per_1k_sessions"] * 0.85:
        return False, "conversion materially down on canary"
    return True, "ok"

def run(deployer, metrics):
    for pct in STEPS:
        deployer.shift(pct)
        time.sleep(SOAK[pct])
        if pct == 100:
            break
        ok, why = gate(metrics.canary(), metrics.stable())
        if not ok:
            deployer.shift(0)                 # drain the canary first
            deployer.rollback()
            sys.exit(f"canary halted at {pct}%: {why}")
    print("canary promoted to 100%")

The conversion check in there is the one I would fight for. Error rate and latency are proxies; orders per thousand sessions is the thing. It is noisier and needs a longer soak at low percentages, which is why the soak times increase rather than decrease as traffic grows.

11. Rollback: Design the Undo Before the Do

Rollback is not a button. It is a property of a change, decided when the change is designed.

The taxonomy I use, and I would genuinely put this on a wall:

Rows one to three are the comfortable ones and they are where most people's mental model of deployment lives. Rows four onward are where incidents happen, and the point of the table is to force the conversation before the release rather than during it.

Practical requirements for the reversible cases: keep the previous five artefacts deployable and verify that quarterly by actually rolling back in a drill; make rollback a first-class pipeline action rather than "deploy the old commit", because deploying an old commit re-runs migrations and that is the wrong thing; and rehearse it. A rollback path nobody has exercised in six months is a hypothesis.

12. The Deploy That Cannot Be Rolled Back

Now the question I actually built this article around, because every guide I have read handles it by pretending it does not exist.

Some deploys are one-way. A destructive migration after the contraction step. A payment provider migration where tokens have been re-vaulted. A backfill that normalised a column and threw away the original text. A pricing change that has already sent 40,000 marketing emails. An integration cutover where the old ERP endpoint has been decommissioned by the other party.

For these, "roll back" is not an option, and the plan must be different in kind — not a better version of the same plan.

First: minimise the set. Most irreversible deploys are irreversible by choice, not by necessity. The column drop did not have to be in that release. The backfill could have written to a new column and kept the old. The token re-vaulting could have run in parallel with the old vault for a fortnight. When someone tells me a change cannot be made reversible, roughly two thirds of the time what they mean is that making it reversible is untidy. Untidy is a very cheap price.

Second: separate the irreversible act from the risky one. If a release contains a schema contraction and a new checkout flow, split them. Ship the contraction on its own, on a quiet Tuesday morning, where the only thing that can go wrong is the contraction. Ship the checkout flow behind a flag, separately. Never combine an unreversible change with an untested one; the whole point of the taxonomy above is that you get to choose which risks travel together.

Third: build a roll-forward path before you deploy, not after. If you cannot go back you must be able to go forward fast, which means: the fix branch is cut and the pipeline is warm, someone who understands the change is at a keyboard rather than on a train, and the deploy happens at a time when a two-hour recovery is survivable. Deploying an irreversible change at 16:45 on a Friday is not a scheduling error; it is a decision to have no recovery capacity.

Fourth: take the snapshot, and verify the restore. Not "we have nightly backups". An explicit snapshot taken immediately before the change, with a tested restore procedure and a known restore duration. If restoring your production database takes four hours and you have never measured it, your recovery time objective is fiction. I ask for the restore to have been performed, into a scratch environment, within the last quarter. The number of teams that can produce that evidence is small and the ones that can are noticeably calmer during incidents.

Fifth: preserve the inputs. If a backfill transforms data, write the original into an archive table in the same transaction. Storage is measured in pennies; the ability to reconstruct is measured in hours of outage avoided.

-- Irreversible-by-default becomes reversible for the price of one table.
BEGIN;

CREATE TABLE archive_customer_phone_20251014 AS
SELECT entity_id, customer_phone, now() AS archived_at
FROM   customer_entity
WHERE  customer_phone IS NOT NULL;

-- Prove the archive is complete BEFORE the destructive step, in the same
-- transaction, so a mismatch rolls the whole thing back.
DO $$
DECLARE src BIGINT; arch BIGINT;
BEGIN
  SELECT count(*) INTO src  FROM customer_entity WHERE customer_phone IS NOT NULL;
  SELECT count(*) INTO arch FROM archive_customer_phone_20251014;
  IF src <> arch THEN
    RAISE EXCEPTION 'archive incomplete: % source rows vs % archived', src, arch;
  END IF;
END $$;

ALTER TABLE customer_entity DROP COLUMN customer_phone;

COMMIT;
-- Drop the archive on a scheduled ticket 90 days later, not today.

Sixth: define the abort criteria in advance and in numbers. Before you start, write down what "this is going badly" looks like: error rate above X, conversion below Y for Z minutes, more than N support contacts about the affected flow. Then write down what you will do, given that you cannot roll back. Usually it is one of: disable the affected feature by flag, put the site into a degraded but functional mode, or take the deliberate outage while you roll forward. Deciding that at 02:00 with an audience is how you get the two-hour version instead of the twenty-minute version.

The honest summary of the irreversible case is that the engineering is mostly refusal. You reduce the number of one-way doors, you isolate the ones that remain, and you accept that for the residue the plan is preparation and speed rather than reversal. Anyone selling you a tool that makes destructive database changes reversible is selling you a restore procedure with better branding.

13. Environments, Drift, and Why Staging Lies

Staging exists to answer one question: will this work in production. It answers it accurately only to the extent it resembles production.

The differences that actually cause missed bugs, roughly in order of how often they burn people. Data volume — a category page with 40 products behaves nothing like one with 4,000, and no amount of unit testing surfaces that. Traffic patterns — cache hit ratios, connection pool saturation, and lock contention are all emergent and all absent from a staging environment with three users. Third-party integrations — a sandboxed payment gateway does not have the live gateway's latency, its timeout behaviour, or its occasional duplicate webhook. And configuration drift, where somebody changed a PHP setting on production during an incident eighteen months ago and never propagated it.

What is worth fixing: define every environment in code, deploy all of them with the same mechanism, and periodically diff the running configuration against the definition. What is not worth fixing: making staging a full-scale replica. It is expensive, it drifts anyway, and it produces false confidence. I would rather have a modest staging environment everyone knows is modest, plus a canary deploy that tests against real production conditions, than an expensive replica that people believe.

The related discipline is deploying without an outage in the first place, which has its own set of constraints around session handling, cache warming and connection draining. Those are covered properly in the zero-downtime deployment article, and they sit underneath everything in this one — a pipeline that requires a maintenance window will be run less often, and less often is what makes releases large and dangerous.

14. Deploys Are the Most Common Cause of Incidents, So Instrument Them

The single highest-value observability feature for a deployment pipeline is a release marker on every metric and every error. Not a dashboard someone builds later — a build SHA attached at the source.

With that in place, the question "did this start with the deploy" takes ten seconds. Without it, an incident begins with twenty minutes of establishing what changed, and that twenty minutes is entirely avoidable.

The metrics I would gate a deploy on, in order of usefulness: orders per thousand sessions, checkout step completion rates, HTTP 5xx rate, p95 latency on the four templates that matter, and payment authorisation success rate. Note that three of those five are business metrics, not infrastructure metrics. An infrastructure-only view will happily report a perfectly healthy system that has stopped selling anything.

Also worth tracking about the pipeline itself: how long it takes end to end, how often it fails and why, how often a deploy is followed by another deploy within an hour — a proxy for "we shipped something broken" — and what proportion of failures are flakes. That last number is the health of your gates. Above about 5%, the gates have stopped being information.

15. The Fashion Retailer, Twelve Months On

Numbers from the rebuild that followed the incident I opened with, because I think worked examples without figures are just anecdotes.

Before. Deploys every two to three weeks, in a Thursday evening window, averaging 40 to 60 commits per release. Pipeline duration 34 minutes, of which 19 was an E2E suite of 61 scenarios with a 22% flake rate. Rollbacks attempted roughly once a quarter; the phone-column incident was the second one in eighteen months where rollback made things worse. Change failure rate — deploys needing an unplanned follow-up fix — around 30%.

What we changed. Build once with digest-pinned artefacts. E2E suite cut from 61 scenarios to 9, with the removed ones either deleted or rewritten as integration tests; flake rate fell to under 2%. Migration rehearsal against an anonymised production snapshot, with a 90-second budget. The migration linter above, which blocks destructive changes without an annotation. Feature flags with mandatory expiry. Canary deploys at 1/5/25/50/100 with automated analysis on the five metrics listed above.

After twelve months. Deploys averaging 9 a week, 3 to 6 commits each. Pipeline duration 11 minutes. Change failure rate 8%. Mean time to recovery from 74 minutes to 9, mostly because the common recovery is now a flag flip rather than a deploy. Two canary halts in the year, both caught at the 5% step by the conversion gate — one was a broken discount code field that returned HTTP 200 while silently failing, which no error-rate check would ever have caught.

What did not work

The migration rehearsal was skipped for four months. The anonymised snapshot pipeline broke in January and nobody noticed, because the rehearsal step was written to skip gracefully when the snapshot was missing. Graceful skipping is the wrong default for a safety gate; it should have failed loudly. That is a one-line fix I should have got right first time and it is the mistake I am most annoyed about, because it recreated exactly the class of silent failure the whole project was about.

Canary analysis at 1% was too noisy for the conversion metric. At their traffic — around 90,000 sessions a week — 1% of traffic over five minutes is a handful of sessions and perhaps zero orders. We now run the 1% step on errors and latency only, and the conversion gate begins at 5%. I would design it that way from the start next time: not every gate makes sense at every step.

Flag expiry was resisted for a quarter. Developers found the build failure on an expired flag annoying, and the first few were extended rather than removed. It took the tech lead publicly deleting four flags and their dead branches, and a visible drop in the checkout module's complexity, before the practice stuck. The tooling was never the hard part.

16. Questions That Come Up

"How often should we deploy?" As often as you have changes worth shipping, which for most teams is several times a week. The instinct that less frequent deploys are safer has it exactly backwards: less frequent means larger, and larger means harder to diagnose and riskier to reverse. If deploying more often feels dangerous, the danger is in the pipeline, and that is what to fix.

"Can we deploy during business hours?" Yes, and you should want to, because the alternative is that every incident happens when the fewest people are awake. The prerequisites are real: zero-downtime deploys, canary or flag-gated releases, and automated verification. Get those and daytime deployment is less risky than the 22:00 version, not more.

"What about the peak trading freeze?" A November-to-January freeze is common and mostly sensible, but "no deploys" is the wrong shape. Freeze feature releases; keep deploying fixes. A team that has not exercised its pipeline for eight weeks and then needs an emergency fix on Boxing Day is in a worse position than one that has been shipping small changes throughout. Enforce the freeze in the pipeline with a documented override, so it is a rule rather than an honour system.

"Do we need Kubernetes for this?" No. Every pattern here — build once, canary, flags, expand-contract, automated verification — works with virtual machines, a load balancer, and shell scripts. Orchestration makes some of it more convenient and adds a substantial operational surface of its own. If you already run containers the patterns in the Docker and Kubernetes guide map onto this directly; if you do not, adopting an orchestrator to improve your deployments is solving the wrong problem first.

"Our platform is SaaS — does any of this apply?" Most of it. On a hosted platform you do not control the runtime, but you still have themes, apps, scripts, configuration, and integrations, all of which are changes that can break checkout. Version them, test them against a development store, promote the same artefact, and keep the ability to revert. The database migration section is the part that mostly does not apply, and that is a genuine advantage of the hosted model that people rarely name as one.

"How do we handle third-party extensions?" Pin exact versions, never a range. Read the changelog of every update, because ecommerce extensions have a long history of shipping schema changes in patch releases. Have a staging environment where you install the update and run the full checkout journey before it goes anywhere near production. And keep a list of which extensions touch checkout or payment, because those get a canary and everything else does not need one.

"Is trunk-based development required?" Not required, strongly recommended. Long-lived branches accumulate divergence, and merging them is the reintroduction of exactly the large-batch risk the pipeline exists to avoid. Short-lived branches merged daily, with flags for anything not finished, get you most of the benefit without the cultural fight over whether anyone may push to main.

"What is the minimum viable version of all this?" One build per commit tagged with the SHA; the same artefact to staging and production; a smoke test after deploy that asserts the served build SHA; a one-command rollback that does not re-run migrations; and additive-only migrations by default. That is a weekend of work and it removes most of the incidents on this page.

17. Where I'd Start

In this order, because each step makes the next one easier.

Make the build produce one immutable artefact per commit and deploy it by digest. Until that is true, nothing else you measure means anything, because the thing you tested is not provably the thing you shipped.

Put the build SHA into every response header, every log line, and every error report. It costs an afternoon and it changes how every future incident starts.

Write the post-deploy smoke test, including the assertion that the served build matches the one you deployed. Wire it to halt the deploy automatically.

Audit your last twenty migrations against the reversibility table. Count how many were destructive and how many of those were bundled with a feature change. That number is your current exposure and it is usually higher than the team expects.

Add the migration linter. Block destructive changes that are not annotated as the contraction half of a completed expansion. It will be unpopular for two weeks.

Build the migration rehearsal against a production-shaped snapshot, and make it fail loudly when the snapshot is missing. Then run the previous release's tests against the new schema, because that is the actual test of whether rollback works.

Cut your E2E suite to the journeys that would lose money if they broke. Quarantine every flaky test the day you find it.

Then, and only then, add canary deployment with automated analysis, and put a business metric in the gate rather than only error rate.

One framing to leave you with. The pipeline is not there to make deployment fast — deployment was always fast, it is copying files. It is there to make the consequences of a wrong deployment small and short. Every technique above is really an answer to the same question, which is how much of the blast radius you can decide in advance rather than discover at the time. The teams who are calm during incidents are not the ones with the most sophisticated tooling. They are the ones who worked out, before they shipped, exactly what they would do if it went wrong, and then made sure that answer was never "restore the database and hope".