Zero-Downtime Deployments for High-Traffic E-Commerce

1. Ninety Seconds, On A Tuesday

A distributor I look after deploys on Tuesday mornings, which is a choice I recommend and will defend. In October 2023 one of those deploys took the storefront down for ninety-four seconds. Not a hard 503 — worse. The site stayed up and served pages, and every one of those pages was broken: unstyled HTML, JavaScript 404s, and an "add to cart" button that posted to a route that no longer existed.

The deploy itself was atomic. Symlink swap, sub-millisecond, exactly as designed. What was not atomic was everything around it. The CDN was still serving the previous release's HTML from a 60-second cache, and that HTML referenced /static/version1697... asset paths that the new release had just replaced. Meanwhile PHP-FPM's realpath cache was still resolving current/ to the old release directory on six of the eight app nodes, so those nodes were executing old code against a database that had already been migrated.

Ninety-four seconds of a store that looked online and was not. Roughly £3,100 in orders that did not happen, which we know because the following Tuesday at the same hour, without a deploy, we did £3,100 more.

The lesson I took, and the one this article is built around: zero-downtime deployment is not a deployment technique. The symlink swap is the easy fifteen percent. The hard part is that a running ecommerce system has at least five kinds of state — in-flight HTTP requests, database schema, cached HTML, session data, and browser-side assets already delivered — and a deploy has to be simultaneously correct for the old code and the new code across all of them, for as long as any old thing is still alive.

2. What Zero Downtime Actually Has To Mean

The definition people use is "no 5xx responses during a deploy". That is far too weak, and it is why teams declare victory and then bleed conversions on deploy days without knowing it.

The definition I use: no request that a customer initiated before the deploy started may fail or behave incorrectly, and no request initiated during or after the deploy may see an inconsistent mixture of old and new behaviour.

That second clause is the demanding one. It rules out a whole category of deploys that pass every health check. A customer who loaded a product page from release N and clicks "add to cart" ninety seconds later is talking to release N+1. If the cart endpoint changed its expected payload, that customer gets a silent failure — no error page, just a button that does nothing. Your monitoring will show 200s.

Practically, this gives you four properties to engineer for:

Atomicity. At any instant, a request is served entirely by one release. No half-copied directories, no partially written files.

Backward compatibility. Release N+1's database schema must work with release N's code, because release N's code is still running somewhere for the duration of the rollout, and will still be running in the customer's browser tab for far longer.

Forward compatibility, briefly. Release N's code must tolerate release N+1's data. If N+1 starts writing a new enum value into a column N reads, N will throw.

Reversibility. You can go back. Which is a schema constraint, not a deploy-script constraint, and it is the one that gets designed away first.

3. The Five Kinds Of State, And What Each One Breaks

Before any tooling, it is worth being explicit about what a deploy actually disturbs. Every zero-downtime failure I have investigated maps to one of these.

In-flight requests. A PHP request that started 400ms ago against release N is still executing when you swap the symlink. If it does an include after the swap, it includes a file from N+1. Half the request is old code, half is new. This is the failure mode that produces genuinely baffling stack traces.

Database schema. The database is shared and singular. There is exactly one of it and it cannot be two versions at once. Everything about zero-downtime migration follows from that single fact.

Server-side caches. Full page cache, block cache, config cache, opcache, realpath cache. Each has its own invalidation semantics and its own TTL, and they do not clear in lockstep. Varnish holding release N's HTML while the app serves release N+1 is the ninety-four-second story above.

Sessions and carts. Serialised objects in Redis. If N+1 changes the shape of a session object — renames a property on a quote item, moves a value into a new container — then N's deserialisation of N+1's data, or vice versa, produces a fatal error or a silently empty cart.

Client-side assets. The longest-lived state of all. A customer's browser holds release N's JavaScript bundle for as long as that tab is open, which could be days. That bundle will keep calling your API.

Notice the timescales. Atomic swap: microseconds. In-flight requests: hundreds of milliseconds. Caches: seconds to minutes. Sessions: hours. Open browser tabs: days. Your compatibility window is set by the longest of those, not the shortest, and almost everyone budgets for the shortest.

4. Atomic Releases: Getting The Easy Part Right

The standard layout is release directories and a symlink. On PHP stacks Deployer is the tool I reach for; Capistrano is the same idea for Ruby, and the pattern is old enough that every ecosystem has a version of it.

/var/www/shop/
  releases/
    20260112143022/     # previous
    20260119091544/     # current
    20260126101233/     # being built
  shared/
    var/log/
    pub/media/
    app/etc/env.php
  current -> releases/20260119091544

The critical detail, and it is a single character: the symlink swap must be atomic. ln -s onto an existing link fails; rm then ln -s leaves a window of tens of milliseconds where current does not exist, and on a busy node that window is hundreds of failed requests. The correct incantation creates a temporary link and renames it, because rename(2) is atomic on POSIX filesystems:

# Wrong: there is a window with no symlink at all.
rm -f /var/www/shop/current
ln -s /var/www/shop/releases/20260126101233 /var/www/shop/current

# Right: rename(2) replaces the link in a single atomic operation.
ln -sfn /var/www/shop/releases/20260126101233 /var/www/shop/current.tmp
mv -Tf /var/www/shop/current.tmp /var/www/shop/current

Deployer's deploy:symlink task already does this correctly. I mention it because I have twice found hand-rolled deploy scripts doing the first version, in both cases written by someone competent who did not know the difference existed.

A Deployer recipe for Magento 2, with the parts that matter annotated:

<?php
namespace Deployer;

require 'recipe/magento2.php';

set('application', 'shop');
set('repository', '[email protected]:client/shop.git');
set('keep_releases', 6);   // enough to roll back past a bad week

// Anything that must survive a release swap lives in shared/.
set('shared_dirs', ['var/log', 'var/report', 'pub/media', 'pub/sitemap']);
set('shared_files', ['app/etc/env.php']);

// Directories the web server must be able to write into.
set('writable_dirs', ['var', 'pub/static', 'pub/media', 'generated']);

host('app-01.shop.internal')->set('labels', ['role' => 'web']);
host('app-02.shop.internal')->set('labels', ['role' => 'web']);
host('cron-01.shop.internal')->set('labels', ['role' => 'cron']);

// Build once, on one machine, and ship the artifact. Compiling DI and
// static content independently on eight nodes produces eight subtly
// different builds and takes eight times as long.
task('build:artifact', function () {
    runLocally('composer install --no-dev --prefer-dist --optimize-autoloader');
    runLocally('bin/magento setup:di:compile');
    runLocally('bin/magento setup:static-content:deploy en_GB en_US -f --jobs=4');
    runLocally('tar -czf build.tar.gz --exclude=.git .');
});

// Deregister from the load balancer BEFORE touching anything on the node.
task('lb:out', function () {
    $id = get('hostname');
    runLocally("aws elbv2 deregister-targets --target-group-arn {{tg_arn}} \
        --targets Id=$id");
    // Wait out the deregistration delay plus a margin. If you skip this,
    // the LB is still routing to a node you are mid-deploy on.
    runLocally('sleep 35');
});

task('lb:in', function () {
    $id = get('hostname');
    runLocally("aws elbv2 register-targets --target-group-arn {{tg_arn}} \
        --targets Id=$id");
    runLocally("aws elbv2 wait target-in-service --target-group-arn {{tg_arn}} \
        --targets Id=$id");
});

after('deploy:failed', 'deploy:unlock');

Two opinions embedded in that file. Build once and ship an artifact, rather than running composer install on each node — I have seen a Composer registry blip produce two nodes on different minor versions of a dependency, which took a day to diagnose. And take the node out of the load balancer before you touch it, rather than relying on a health check to notice, because the health check's polling interval is your outage window.

5. The Realpath Cache, Which Ate Ninety Seconds Of My Life

This is the specific mechanism behind the incident I opened with, and it catches nearly every PHP team the first time.

PHP caches the resolution of paths to inodes in the realpath cache, controlled by realpath_cache_ttl, which defaults to 120 seconds. When your document root is /var/www/shop/current/pub and current is a symlink, PHP resolves that symlink once and caches the answer. Swap the symlink and PHP-FPM keeps resolving to the old release for up to two minutes.

Opcache compounds it. Opcache keys compiled scripts by resolved path, so if realpath still says the old directory, opcache serves the old compiled bytecode, and opcache_reset() does not help because the path resolution happened before opcache was consulted.

There are two fixes and only one of them is good.

The bad fix, which is everywhere on the internet: reload PHP-FPM after each deploy. systemctl reload php8.3-fpm does a graceful restart of workers and clears both caches. It works. It also means every deploy restarts your PHP pool, which drops warm opcache and produces a burst of slow requests while it recompiles — on a large Magento codebase that is fifteen to thirty seconds of elevated latency per node.

The good fix: do not put a symlink in the document root at all. Point nginx at the resolved release path and change the nginx config instead.

# /etc/nginx/conf.d/shop-release.conf — written by the deploy, then reloaded.
# $realpath_root gives nginx's own resolution, and passing it to PHP means
# PHP never has to resolve the symlink itself.
map $host $release_root {
    default /var/www/shop/current;
}

server {
    root $release_root/pub;

    location ~ ^/(index|get|static|errors/report|errors/404|errors/503|health)\.php$ {
        fastcgi_pass php-fpm;
        include fastcgi_params;

        # These two lines are the whole fix. nginx resolves the symlink on
        # every request (cheaply, with its own open_file_cache) and hands
        # PHP an already-resolved absolute path.
        fastcgi_param SCRIPT_FILENAME $realpath_root$fastcgi_script_name;
        fastcgi_param DOCUMENT_ROOT   $realpath_root;
    }
}

With $realpath_root, PHP is given a path like /var/www/shop/releases/20260126101233/pub/index.php and there is no symlink for it to cache. A deploy becomes visible on the very next request. No FPM reload, no opcache cold start.

Set open_file_cache in nginx thoughtfully if you do this — nginx has its own caching of stat results, and a long open_file_cache_valid reintroduces the same problem one layer up:

open_file_cache          max=20000 inactive=60s;
open_file_cache_valid    10s;   # not 300s; this is your deploy propagation delay
open_file_cache_min_uses 2;
open_file_cache_errors   off;   # or a 404 during deploy gets cached as a 404

That last line is not optional. With open_file_cache_errors on, a file that momentarily does not exist during a deploy gets its non-existence cached for the validity period, and you serve 404s for assets that are demonstrably present on disk. It is a wonderfully confusing bug.

6. Draining In-Flight Requests

Taking a node out of rotation is not the same as the node being idle. Between deregistration and quiescence you have requests still executing, and on a checkout path those can run for seconds.

The sequence that actually drains cleanly:

#!/usr/bin/env bash
# drain.sh — take one node out of service and wait for genuine quiescence.
set -euo pipefail

NODE="$1"
TG="${TARGET_GROUP_ARN}"

# 1. Stop the LB sending new work. Deregistration delay on the target group
#    should be set to at least your p99.9 request duration; 30s is typical.
aws elbv2 deregister-targets --target-group-arn "$TG" --targets "Id=$NODE"

# 2. Flip the local health endpoint to unhealthy immediately, so anything
#    that bypasses the LB (internal callers, a second LB) also backs off.
ssh "$NODE" 'touch /var/www/shop/shared/var/.draining'

# 3. Wait for in-flight work to finish. Polling the actual FPM status is
#    far better than sleeping a fixed number of seconds and hoping.
for i in $(seq 1 60); do
  active=$(ssh "$NODE" "curl -sf localhost/status?json | python3 -c \
    'import sys,json; print(json.load(sys.stdin)[\"active processes\"])'")
  # 1 is the status request itself.
  if [ "$active" -le 1 ]; then
    echo "drained after ${i}s"
    exit 0
  fi
  sleep 1
done

echo "WARNING: still $active active after 60s; proceeding anyway" >&2

Requiring pm.status_path to be enabled in your FPM pool is a small price for knowing rather than guessing. The fixed sleep 30 that most deploy scripts use is either too short on a bad day or wasteful on a good one, and it never tells you which.

For long-running work — queue consumers, report generation — draining means something different. Send SIGTERM, let the consumer finish its current message, and do not send SIGKILL until you have waited longer than your longest message takes. Magento's queue:consumers:start respects --max-messages, and running consumers with a bounded message count under a supervisor is a simpler way to get the same property: they exit on their own schedule and come back on the new release.

7. Database Migrations: Expand, Migrate, Contract

This is the part that actually decides whether zero-downtime is achievable, and it is a schema design discipline rather than a deploy tool.

The rule: every schema change must be deployable while the previous release is still running. That eliminates, immediately and permanently, renaming a column in one step, dropping a column in the same release that stops using it, adding a NOT NULL column without a default, and changing a column type in place.

What you do instead is split every destructive change across three releases.

Expand

Release N+1 adds the new structure. It does not remove anything. Old code continues to work because nothing it depends on has changed.

-- Release N+1. Additive only. Old code ignores the new column entirely.
ALTER TABLE sales_order_grid
  ADD COLUMN fulfilment_channel VARCHAR(32) NULL DEFAULT NULL,
  ADD INDEX IDX_FULFILMENT_CHANNEL (fulfilment_channel);

-- New code writes BOTH the old and the new field. Old code reads the old
-- one and is none the wiser. This dual-write phase is what makes rollback
-- possible: revert to N and nothing is missing.

Migrate

Backfill the new structure for existing rows, in batches, outside the deploy. This is not a deploy step and should never be one. A backfill that locks a 40-million-row table for eleven minutes is an outage regardless of how atomic your symlink was.

<?php
// A backfill worker. Batched, resumable, and rate-limited by replica lag.
final class BackfillFulfilmentChannel
{
    private const BATCH = 2000;

    public function run(\PDO $db): void
    {
        $lastId = (int) ($this->checkpoint() ?? 0);

        while (true) {
            $rows = $db->prepare(
                'SELECT entity_id, shipping_method FROM sales_order
                 WHERE entity_id > :id AND fulfilment_channel IS NULL
                 ORDER BY entity_id LIMIT ' . self::BATCH
            );
            $rows->execute([':id' => $lastId]);
            $batch = $rows->fetchAll(\PDO::FETCH_ASSOC);
            if (!$batch) break;

            $db->beginTransaction();
            $upd = $db->prepare(
                'UPDATE sales_order SET fulfilment_channel = :c WHERE entity_id = :id'
            );
            foreach ($batch as $row) {
                $upd->execute([
                    ':c'  => $this->deriveChannel($row['shipping_method']),
                    ':id' => $row['entity_id'],
                ]);
                $lastId = (int) $row['entity_id'];
            }
            $db->commit();
            $this->checkpoint($lastId);

            // Back off if replicas are falling behind. A backfill that
            // outruns replication turns into a read-path incident on a
            // system where nothing was deployed at all.
            while ($this->replicaLagSeconds($db) > 5) {
                usleep(500_000);
            }
        }
    }
}

The replica-lag check is the line people leave out and then regret. A tight backfill loop on the primary generates binlog faster than replicas can apply it, and if your read traffic goes to replicas you have just given yourself stale carts and missing orders without deploying a single line of application code.

Contract

Release N+2, or N+5, or the following month — after you are certain no running code touches the old structure — removes it. Verify rather than assume. A query against your slow log, or a temporary trigger that logs writes to the deprecated column, is cheap insurance.

-- Release N+3, once the old column has had zero reads for a fortnight.
ALTER TABLE sales_order DROP COLUMN shipping_method_legacy;

The uncomfortable truth about expand/migrate/contract is that the contract step never gets scheduled. Teams do expand, do migrate, ship the feature, and the deprecated column sits there for four years growing an ever-more-elaborate mythology about whether anything still needs it. Put the contract migration in the backlog with a date, attached to the same ticket, or accept that you are building a schema you cannot change.

Online schema change for large tables

MySQL 8.0 does far more ALGORITHM=INPLACE operations than 5.7 did, but "inplace" is not the same as "non-blocking", and adding an index to a large sales_order_grid will still hold a metadata lock at the start and end of the operation. On tables above a few million rows I use pt-online-schema-change or gh-ost.

# gh-ost: builds a shadow table, tails the binlog to keep it in sync,
# then does an atomic cut-over. The throttle flags are the important part.
gh-ost \
  --host=db-primary.internal --database=shop --table=sales_order \
  --alter="ADD COLUMN fulfilment_channel VARCHAR(32) NULL, \
           ADD INDEX IDX_FC (fulfilment_channel)" \
  --max-load='Threads_running=40' \
  --critical-load='Threads_running=120' \
  --chunk-size=1000 \
  --max-lag-millis=1500 \
  --throttle-control-replicas='db-replica-01.internal,db-replica-02.internal' \
  --allow-on-master \
  --postpone-cut-over-flag-file=/tmp/ghost.postpone \
  --execute

The postpone-cut-over-flag-file is the flag I would not run without. It lets gh-ost do all its work and then wait, indefinitely, until you delete that file. You do the copy at 3am on a Sunday and the cut-over at 10am on a Monday with the whole team watching, rather than having a cut-over fire unattended.

8. Magento's setup:upgrade Is The Enemy Of All This

Every Magento deploy guide tells you to run bin/magento setup:upgrade. On a zero-downtime deploy this command is a problem, and it is worth being precise about why.

setup:upgrade does three unrelated things: it applies schema changes from every installed module, it applies data patches, and — this is the killer — it puts the application into maintenance mode for the duration unless you tell it not to. On a large installation with a lot of third-party modules it takes anywhere from forty seconds to several minutes.

The split you want:

# 1. Find out whether anything is actually pending. On most deploys the
#    answer is nothing, and the whole step can be skipped.
bin/magento setup:db:status
# echo $? -> 0 = up to date, 1 = upgrade needed, 2 = manual action required

# 2. If schema changes ARE pending, generate the SQL rather than applying
#    it blindly, and review it. --dry-run writes the statements to a file.
bin/magento setup:db-schema:upgrade --dry-run

# 3. Apply schema separately from data patches. Schema first, on one node.
bin/magento setup:db-schema:upgrade

# 4. Data patches after the code is live everywhere, because patches often
#    depend on new code being present.
bin/magento setup:db-data:upgrade

My working rule on any Magento project of size: a deploy that requires a schema change is a different kind of deploy and gets a different runbook. Ninety percent of deploys touch no schema at all, and setup:db:status returning 0 lets you skip the expensive step entirely. Reserve the schema deploy for a scheduled slot, with the expand/contract discipline above applied to whatever the modules want to do.

The uncomfortable part is that third-party modules do not follow expand/contract. A vendor extension will happily drop a column in the same release that stops using it, and you find out when you try to roll back. Reading the db_schema.xml diff of every module update before you take it is tedious and I do it anyway, because the alternative is discovering the problem at the point where rollback was the plan.

9. Blue/Green, Rolling, Canary: Picking One

These get discussed as though they were interchangeable. They have genuinely different cost and risk profiles, and the right answer depends mostly on how expensive your infrastructure is and how confident you are in your tests.

Read the last row of the middle column carefully, because blue/green is the technique people ask for by name and it is usually the wrong one. Doubling your application tier for the duration of a deploy is fine when the tier is stateless containers costing pennies per minute. It is not fine when it is eight provisioned instances with warm caches, because your green environment starts cold and cutting over to it hands 100% of traffic to a fleet with an empty opcache and an empty Redis. I have seen a textbook-correct blue/green cut-over produce a worse customer experience than the rolling deploy it replaced, purely from cache cold-start.

And note what blue/green does not solve: the database. There is still one of it. Green talks to the same primary as blue. Every schema compatibility constraint in this article applies unchanged. The people who believe blue/green removes the need for expand/contract are the people who have not yet had to roll back.

My default recommendation for a mid-size Magento or headless commerce stack is rolling with a canary step: deploy to one node, hold it at 5% of traffic for ten minutes with automated metric comparison, then roll the rest. You get the containment benefit without the duplicate-fleet cost, and the ten minutes catches the class of bug that only appears under real traffic patterns.

10. Sessions, Carts, And Serialised Objects

Sessions in Redis are the state people forget until it breaks in a way that looks like nothing else.

The failure: release N+1 changes the shape of something stored in the session. Perhaps a quote item gains a property, or a custom module renames a key. Release N reads that session and either throws on deserialisation or — more often, and worse — silently returns null for the property it expected. Customers see empty carts. Nothing appears in the error log because nothing errored.

Three rules I hold to:

Never change the shape of a serialised session object in the same release that starts writing it. Add the new field, tolerate its absence for a release, then rely on it.

Never store objects in the session where a scalar will do. PHP's serialisation of objects is tied to class definitions. Storing an array or a JSON blob decouples the data from the code that produced it, and makes version skew a data problem rather than a fatal error.

Version the session namespace when you have no choice. If you genuinely must break compatibility, change the Redis key prefix. Every customer gets a new empty session, which is bad, but it is bad in an obvious way you can announce rather than bad in a way that silently drops carts.

// app/etc/env.php — the prefix is the escape hatch. Bumping it invalidates
// every session at once. Use it deliberately, never as routine hygiene.
'session' => [
    'save' => 'redis',
    'redis' => [
        'host' => 'redis-session.internal',
        'port' => '6379',
        'database' => '2',
        'disable_locking' => '0',
        'max_concurrency' => '20',
        'break_after_frontend' => '5',
        'prefix' => 'sess_v4_',
    ],
],

Keep sessions in a different Redis database — or a different instance entirely — from your cache. When you flush cache during a deploy, and you will, a shared instance means you flushed every cart on the site. I have watched this happen on a Friday afternoon. The FLUSHALL was in the deploy script, had been for two years, and had only ever been run against a Redis where sessions did not live.

11. Static Assets And Version Skew In The Browser

The longest-lived state is the one you have least control over. A customer opened your product page an hour ago and the tab is still open. Their browser holds release N's JavaScript. When they interact, that code hits your API — which is now release N+1.

Two distinct problems.

Assets disappearing. Release N's HTML references /static/version1706/js/bundle.js. If your deploy deletes the old static directory, that request 404s and the page breaks in place. The fix is to keep the previous release's static assets served for as long as any HTML referencing them can plausibly be in circulation. Content-hashed filenames plus keeping N-1 and N-2 on disk covers it — this is what keep_releases is really for, and why I set it to 6 rather than 3.

If assets go to S3 or a CDN bucket rather than the app nodes, this is easier: upload new assets, never delete old ones on deploy, and prune with a lifecycle rule at 30 days.

# Sync new assets. Note the absence of --delete; that flag is what turns a
# deploy into an outage for anyone holding older HTML.
aws s3 sync pub/static/ "s3://shop-static/${RELEASE}/" \
  --cache-control 'public, max-age=31536000, immutable' \
  --exclude '*.map' \
  --size-only

# Prune separately, on a schedule, well outside any deploy.
aws s3 rm "s3://shop-static/" --recursive \
  --exclude '*' --include "*/" --dryrun   # review before removing --dryrun

API contract drift. Old JavaScript calling a changed endpoint. The mitigations, in order of how much I like them: do not break endpoints (add fields, never remove or repurpose them); version the endpoint if you must; and, as a last line, have the client send its release identifier and let the server respond with a "please reload" signal when the gap is too wide.

// Injected into the page at build time.
const CLIENT_RELEASE = '20260126101233';

async function api(path, init = {}) {
  const res = await fetch(path, {
    ...init,
    headers: { ...init.headers, 'X-Client-Release': CLIENT_RELEASE }
  });

  // The server sets this when the client build is older than the oldest
  // release it still supports. Prompt rather than force-reload: a forced
  // reload mid-checkout loses form state and is worse than the skew.
  if (res.headers.get('X-Release-Skew') === 'unsupported') {
    showBanner('A new version is available. Refresh to continue.');
  }
  return res;
}

I am deliberately not auto-reloading there. Blowing away a half-filled checkout form to fix a compatibility problem the customer cannot see is a bad trade. Tell them, let them choose.

12. Caches, And The Thundering Herd After A Flush

The instinct after a deploy is to flush everything. Resist it. A cold full-page cache on a busy storefront means every request goes to PHP simultaneously, and PHP capacity is sized for a warm cache.

What I actually do, in order of aggressiveness:

Flush nothing if the release changed no templates, no layout XML, and no config. Most releases. Use the git diff to decide programmatically rather than flushing out of superstition.

Invalidate by tag where the change is scoped. Varnish and Fastly both support surrogate-key purging, and Magento already emits the tags.

Soft-purge where your CDN supports it. Fastly's soft purge marks content stale rather than removing it, so the first request after the purge gets the stale copy immediately while a background fetch revalidates. Your cache hit ratio does not fall off a cliff.

# Fastly soft purge by surrogate key — stale-while-revalidate semantics
# rather than a hole in the cache.
curl -X POST "https://api.fastly.com/service/${SERVICE_ID}/purge/cms-block-homepage-hero" \
  -H "Fastly-Key: ${FASTLY_TOKEN}" \
  -H "Fastly-Soft-Purge: 1"

Warm before you cut over if you must flush hard. A crawler over your top 500 URLs, run against the new release before it takes traffic, costs two minutes and removes the herd entirely.

#!/usr/bin/env bash
# warm.sh — hit the top N URLs against a specific node before it rejoins
# the load balancer. Concurrency deliberately modest: this is meant to
# populate the cache, not to load-test the node you are about to depend on.
set -euo pipefail
NODE="$1"
xargs -a top-500-urls.txt -P 8 -I{} \
  curl -s -o /dev/null -w '%{http_code} %{time_total} {}\n' \
    --resolve "shop.example.com:443:${NODE}" "https://shop.example.com{}" \
  | awk '$1 != "200" { print "WARN", $0 }'

The --resolve flag is what makes this target one specific node rather than whatever the load balancer feels like. Without it you warm the fleet you were already serving from and learn nothing.

13. Health Checks That Tell The Truth

The default health check is a request to / that returns 200. It is worse than nothing, because it gives you confidence you have not earned. A Magento node with a dead Redis connection will happily serve a cached homepage.

A health check should assert the dependencies the node actually needs, and it should be able to say "not yet" during a deploy without saying "broken".

<?php
// pub/health.php — deliberately outside the framework so a broken DI
// container still produces a useful answer rather than a 500.
header('Content-Type: application/json');
header('Cache-Control: no-store');

$release = trim(@file_get_contents(__DIR__ . '/../RELEASE') ?: 'unknown');
$checks  = [];
$ok      = true;

// Explicit drain flag: the deploy touches this before taking work away.
if (file_exists(__DIR__ . '/../var/.draining')) {
    http_response_code(503);
    echo json_encode(['status' => 'draining', 'release' => $release]);
    exit;
}

try {
    $env = require __DIR__ . '/../app/etc/env.php';
    $db  = new PDO(
        sprintf('mysql:host=%s;dbname=%s', $env['db']['connection']['default']['host'],
                $env['db']['connection']['default']['dbname']),
        $env['db']['connection']['default']['username'],
        $env['db']['connection']['default']['password'],
        [PDO::ATTR_TIMEOUT => 2]
    );
    // A real query, not just a connection. Connecting proves the socket
    // opened; it does not prove the schema this release expects is there.
    $db->query('SELECT 1 FROM setup_module LIMIT 1')->fetch();
    $checks['db'] = 'ok';
} catch (Throwable $e) {
    $checks['db'] = 'fail: ' . $e->getMessage();
    $ok = false;
}

try {
    $r = new Redis();
    $r->connect($env['cache']['frontend']['default']['backend_options']['server'], 6379, 2.0);
    $r->ping();
    $checks['redis'] = 'ok';
} catch (Throwable $e) {
    $checks['redis'] = 'fail';
    $ok = false;
}

// Cheap proof that the code and the schema agree, which is exactly the
// thing a mid-rollout mixed-version fleet gets wrong.
$checks['schema_version'] = $db->query(
    "SELECT schema_version FROM setup_module WHERE module = 'Magento_Sales'"
)->fetchColumn() ?: 'unknown';

http_response_code($ok ? 200 : 503);
echo json_encode(['status' => $ok ? 'ok' : 'fail', 'release' => $release] + $checks);

Two design points. The drain flag is checked first and returns 503 without touching any dependency, so a node you are deliberately removing does not page anyone. And the health check reports the release identifier, which means your load balancer's target health page becomes a live view of which nodes are on which version during a rollout — genuinely useful, and free.

Keep it out of the full page cache and out of any WAF rate limit. Both mistakes are common and both produce a fleet that marks itself unhealthy under load, which is precisely the wrong time.

14. Rollback: The Step Nobody Rehearses

Every deployment document has a rollback section. Almost none of them have been executed under pressure.

Code rollback is easy — repoint the symlink at the previous release directory, reload nothing if you took the $realpath_root approach. Deployer gives you dep rollback and it works.

Everything else is where it falls apart.

Schema is not rollback-able. If N+1 dropped a column, going back to N means restoring it, and the data is gone. This is the entire argument for expand/contract stated as an operational consequence rather than a principle. Your rollback capability is exactly as good as your schema discipline.

Data written by the new release may be unreadable by the old one. New enum values, new serialisation formats, new columns the old code will not populate. Dual-writing during the expand phase is what protects you.

Static assets and CDN state. If you purged the CDN on deploy and the old assets are gone from origin, rolling back the code leaves you serving HTML that references nothing.

The practice that actually helps: roll back on a schedule, in production, deliberately, when nothing is wrong. Once a quarter, deploy a release and then immediately roll it back, at a quiet hour, with someone watching the graphs. It takes twenty minutes. It finds the broken assumption while the stakes are low, which is the only time you want to find it.

I started doing this after a rollback attempt during a genuine incident failed because keep_releases was set to 2 and the release we needed had already been pruned. Twenty minutes of quarterly practice would have surfaced that a year earlier.

15. Queues, Cron, And The Things That Are Not HTTP

Deploy discussions focus on web nodes because that is where customers are. The background tier breaks in quieter and more expensive ways.

Cron overlapping a deploy. Magento's cron runs every minute and its jobs can run for many minutes. A deploy that swaps code underneath a running indexer produces class-not-found fatals mid-reindex, and depending on the indexer, a partially rebuilt index that looks fine to every health check. Disable cron before the deploy and re-enable after, and make that part of the deploy, not a wiki page.

// deploy.php — bracket the whole deploy for cron hosts.
task('cron:pause', function () {
    run('crontab -l | sed "s|^\([^#]\)|#\1|" | crontab -');
    // Wait for anything already running to finish rather than yanking it.
    run('while pgrep -f "bin/magento cron:run" > /dev/null; do sleep 2; done');
})->select('role=cron');

task('cron:resume', function () {
    run('crontab -l | sed "s|^#\(\* \)|\1|" | crontab -');
})->select('role=cron');

before('deploy:symlink', 'cron:pause');
after('deploy:success', 'cron:resume');

Message format changes. A queue is a durable buffer of messages written by one version and read by another, which makes it a distributed-systems compatibility problem wearing a friendly hat. If N+1 changes a message schema, N+1's consumers must still handle N's messages sitting in the queue, and N's consumers — still running for the next few minutes — must not choke on N+1's. Same expand/contract rule, same three-release cadence.

Consumer restarts. Run consumers with --max-messages under a supervisor so they exit and restart naturally onto the new code. A consumer that runs forever is a consumer running last month's release, and I have found consumers eleven releases behind on sites that deploy weekly.

16. What Actually Happened On The Black Friday Freeze

The distributor from the opening. Turnover around £14m, Magento 2.4.6, eight app nodes behind an ALB, Varnish, MySQL 8.0 with two replicas, roughly 3,000 orders on a normal day and 11,000 on Black Friday.

They had a code freeze from 1 November to 5 December. Five weeks. The freeze existed because deploys were scary, and deploys were scary because of the ninety-four-second incident. The business hated it — five weeks of no fixes, and a backlog that then went out in one enormous release on 6 December, which is the single riskiest thing you can do.

The work, over about seven weeks:

Week 1-2, the realpath fix. Switched nginx to $realpath_root, removed the PHP-FPM reload from the deploy, dropped open_file_cache_valid from 300s to 10s. Deploy propagation went from "up to 120 seconds and unpredictable" to "next request". This alone removed the mixed-version window on a fleet of eight.

Week 2, drain properly. Replaced sleep 20 with the FPM status poll. Median observed drain time was 1.8 seconds; p99 was 6.4 seconds. The old fixed sleep was both wasteful and, twice in the logs I went back through, insufficient.

Week 3, the CDN skew. Static assets moved to S3 behind CloudFront with content hashes, and the --delete flag came out of the sync. This was the actual cause of the unstyled-page symptom, and it took an embarrassingly long time to see because everyone was focused on the PHP layer.

Week 4, the schema audit. I went through eighteen months of migrations and found four destructive changes that had shipped as single-step deploys. None had caused an incident. All four meant that a rollback across those releases would have destroyed data, so the "we can always roll back" assumption in the runbook had been false for a year and a half. That finding did more to change behaviour than anything else in the project.

Week 5, canary. Added an ALB weighted target group. One node takes 5% of traffic for ten minutes on the new release, with an automated comparison of error rate, p95 latency and add-to-cart conversion against the other seven. Below threshold, the rollout proceeds; above, it halts and pages.

Week 6, what went wrong. The canary comparison fired a false positive on its first real use and halted a perfectly good release. The cause: the canary node was cold, so its p95 latency was legitimately worse for the first four minutes while opcache and Redis filled. I had compared the wrong window. Fixed by warming the node before it takes canary traffic and only comparing minutes 5 through 10. Obvious in hindsight; not obvious at 7am with a halted deploy.

Week 7, rollback drill. Deployed and rolled back three times on a Wednesday evening. Found that keep_releases was 3 and that the media symlink was being recreated on rollback in a way that briefly 404'd product images. Both fixed in an hour, both would have been discovered during an incident otherwise.

Where it landed, measured across the following quarter:

The number I care about is the last one. The freeze did not disappear, and I do not think it should have — on the highest-revenue days of the year there is a reasonable argument for not changing the database. But it shrank from five weeks of paralysis to four days of a specific, justified restriction, and that is a business outcome rather than an engineering one.

If you are building the pipeline that runs all this, the mechanics of gating and artifact promotion are covered in more detail in the piece on CI/CD pipelines for enterprise ecommerce; this article assumes you have somewhere to run these steps from.

17. Questions I Get Asked

"Do we need Kubernetes for zero-downtime deploys?"

No, and for a Magento monolith I would generally say no even if you are willing. Kubernetes gives you rolling updates, readiness gates and instant rollback of the pod spec for free, which is genuinely valuable. It does not give you schema compatibility, session compatibility, cache warming or CDN coordination, and those are four of the five hard problems. A team that moves a monolith to Kubernetes hoping to fix deploys usually ends up with the same deploy problems plus a cluster to run. If you are already containerised, use the tools. If you are not, symlink releases plus a load balancer gets you the same customer-facing outcome for a fraction of the operational surface.

"How long should the deregistration delay be?"

Longer than your p99.9 request duration, and measure it rather than guessing. On most storefronts p99.9 is dominated by checkout and payment callbacks and lands between 8 and 20 seconds. I set 30. Setting it to 300 "to be safe" means every deploy takes an extra five minutes per node, which on eight nodes is forty minutes and will get the whole practice abandoned.

"Can we skip expand/contract if we use blue/green?"

No. This is the most common misconception in the whole topic. Blue and green share one database. During cut-over both are serving traffic for at least a few seconds, and after cut-over you want the ability to flip back — which requires the old code to work against the new schema. Blue/green makes the code swap cleaner. It changes nothing about the data.

"What about deploying during business hours?"

That is the goal, and it is the honest test of whether any of this works. A team that only deploys at 2am has not achieved zero-downtime deployment; they have achieved low-traffic deployment, which is a different and much weaker property. Once you can deploy at 11am on a Tuesday without checking the order graph afterwards, you are done. Until then you are not, whatever the runbook says.

"Our third-party extensions break these rules. What do we do?"

Read the schema diff before you take the update, and when it contains a destructive change, decide consciously: hold the update, fork the module, or take the update in a scheduled maintenance slot with a database snapshot taken first. What you must not do is discover it during a rollback attempt. This is the strongest practical argument I have for keeping third-party module count low — every extension is a schema you do not control.

"Is a database snapshot before deploy enough of a safety net?"

It is a safety net for catastrophe, not for rollback. Restoring a snapshot loses every order placed since it was taken, so using it means you have accepted data loss. It belongs in your disaster plan and it does not belong in your deploy plan. If your rollback strategy is "restore the snapshot", you do not have a rollback strategy.

18. What I'd Do First

Assume you deploy today and it mostly works but nobody is comfortable. In this order.

One. Measure what actually happens during a deploy. Put a synthetic check on your add-to-cart flow at 10-second intervals, deploy, and read the result. Most teams have never looked, and the answer is either "nothing happens, we are fine" — which lets you stop worrying — or a specific failure you can now name. Do this before changing anything.

Two. Fix the realpath cache. If you are on PHP behind a symlinked document root, switch nginx to $realpath_root and drop the FPM reload. It is a config change and half an hour of testing, and on a multi-node fleet it eliminates the largest source of mixed-version weirdness.

Three. Audit your last twenty migrations for destructive changes. Not to fix them — they have already shipped — but to find out whether your rollback plan is real. If it is not, you need to know that before you need it.

Four. Make draining explicit. Deregister before you touch the node, poll for quiescence rather than sleeping, and put a drain flag in the health check. This is an afternoon.

Five. Rehearse a rollback in production at a quiet hour. It will find something. It always finds something, and the something is always cheaper to fix on a Wednesday evening than at 4pm on Black Friday.

Six. Only now, add a canary step. It is the highest-value control on the list and it is last because it is useless without the five above — a canary that halts on a cold-cache false positive, or that cannot roll back cleanly, is a mechanism for making deploys more frightening rather than less.

None of this is technically difficult. The realpath fix is four lines. The drain script is thirty. What makes it hard is that it is spread across six systems owned by four people, and the failure it prevents is invisible until it is expensive. That is why the code freeze exists, and that is what the work is actually for.