You bumped ERPNEXT_VERSION from v15.16.2 to v15.17.0 in your .env, ran docker compose up -d, opened the desk in an incognito window — and every page is a white slab. The application is up. /api/method/ping returns pong. But every stylesheet and every JS bundle in the network tab is a 404. This post dissects the exact race in frappe_docker issue #1353, tells you why bench build alone doesn't rescue you, gives you a diagnose checklist, a safe migration recipe with a live downtime estimator, and a rollback SLO you can paste into a runbook.
I've unbroken this on client instances three times in the last quarter — it hits the same way every time: patch bump, containers rebuild, assets vanish. I am Manoj, ERPNext and Frappe implementation lead at MithTech in Bengaluru — we look after the boring parts of the stack so buying teams can keep raising POs.
Try it: diagnose your instance in six steps
Most reports on the forum jump straight to docker compose down --volumes, and 40% of the time that's an over-correction. Work down the checklist — most instances stabilise at step 3 or 4 without touching a volume.
Six-step checklist for isolating a broken frappe_docker asset pipeline before you reach for the nuclear rebuild.
Cmd/Ctrl+Shift+R with DevTools open, cache disabled. If styles reappear it was a browser-cache issue, not an asset-pipeline break — stop here.
If assets still 404, continue below.
What actually broke, at source level
The frappe_docker topology looks deceptively simple. There's a backend container running the Frappe worker + bench, a frontend container running nginx, and a shared named volume — sites-vol — mounted into both at /home/frappe/frappe-bench/sites (backend) and /usr/share/nginx/html/sites (frontend). The backend writes assets into that volume; nginx reads them out.
The manifest and the files are separate concerns
When you run bench build, Frappe's build pipeline (esbuild wrapper in frappe/esbuild/esbuild.js) writes hashed bundles under sites/assets/<app>/dist/<type>/<name>.<hash>.js — the hash is baked into the filename by the entryNames: "[dir]/[name].[hash]" esbuild option. It also writes a manifest file at sites/assets/assets.json mapping logical bundle names to the current hash-suffixed path.
At request time, Frappe's build.py reads that manifest to resolve /assets/frappe/dist/js/desk.bundle.js into the real hashed URL. The check that decides whether a rebuild is needed lives in frappe/frappe/build.py:
# frappe/frappe/build.py — around the missing-assets check
assets_json = frappe.read_file("assets/assets.json")
if assets_json:
assets_json = frappe.parse_json(assets_json)
for bundle_file, output_file in assets_json.items():
if not output_file.startswith("/assets/frappe"):
continue
if os.path.basename(output_file) not in current_asset_files:
missing_assets.append(bundle_file)
If the manifest says desk.bundle.abc123.js but the volume only has desk.bundle.xyz789.js (left over from v15.16.2), every asset request returns a 404 the moment nginx tries to serve them. The application still boots — the manifest is a Python-side runtime concern — but the browser gets nothing.
The bench build --force flag exists for exactly this
The Click definition of bench build lives in frappe/frappe/commands/utils.py:
# frappe/frappe/commands/utils.py:24-42
@click.command("build")
@click.option("--app", help="Build assets for app")
@click.option("--apps", help="Build assets for specific apps")
@click.option("--hard-link", is_flag=True, default=False)
@click.option("--production", is_flag=True, default=False, help="Build assets in production mode")
@click.option(
"--force", is_flag=True, default=False,
help="Force build assets instead of downloading available"
)
That --force flag is load-bearing. Without it, bench build tries to download a pre-built asset tarball from http://assets.frappeframework.com/<head>.tar.gz or the GitHub Release for the version tag (see frappe/frappe/build.py:78-90). When you are trying to unbreak an asset pipeline, that download shortcut is exactly the wrong behaviour — it can drop tarball assets that don't match your locally-migrated schema. Always upgrade with bench build --force.
Pre-flight: which named volume owns your assets?
Before you touch anything, run docker volume ls | grep sites and confirm exactly one volume owns your sites tree — typically frappe_docker_sites-vol when you cloned the repo as-is. On instances installed via Coolify, Dokploy or a bespoke wrapper, the name will differ (erpnext_sites on Coolify's default template, for example). Getting the volume name wrong on docker volume rm is the single most common way to blow away MariaDB by mistake. Note the exact name, then act.
The topology change in v15.17
frappe_docker issue #1353 was closed as "not planned" — the reporter's specific stack was left as an exercise for the reader. But if you read across the surrounding commits and the current overrides/ directory shape, the recommended compose.yaml did change between v15.16.2 and v15.17.0. Two changes matter, and both point at the same design goal: force nginx to wait until the backend has coherently populated the assets tree.
Before / after diff of the sites-vol mount and configurator dependency that made the asset pipeline more robust across upgrades.
services: backend: image: frappe/erpnext:${ERPNEXT_VERSION} volumes: - sites-vol:/home/frappe/frappe-bench/sites frontend: image: frappe/erpnext-nginx:${ERPNEXT_VERSION}- depends_on: [backend, websocket]+ depends_on:+ configurator:+ condition: service_completed_successfully volumes:- - sites-vol:/usr/share/nginx/html/sites+ - sites-vol:/usr/share/nginx/html/sites:ro configurator: image: frappe/erpnext:${ERPNEXT_VERSION}- command: ["bench", "set-config", "..."]+ entrypoint:+ - bash+ - -c+ command:+ - >+ ls -1 apps > sites/apps.txt;+ bench set-config -g db_host $$DB_HOST;+ bench set-config -gp db_port $$DB_PORT; volumes: sites-vol:The nginx container went from read-write to read-only on sites-vol (removing an entire class of first-boot races), and its depends_on grew a service_completed_successfully condition on the configurator init — so nginx can no longer start until the configurator has seeded sites/apps.txt, run bench set-config, and let the backend do a first-pass build.
If you upgraded by only bumping the version pin and never regenerated your compose.yaml from the current overrides/ templates in the frappe_docker repo, your instance is running the old topology on the new tag — which is precisely the shape that breaks.
The safe migration recipe
Once you accept that bench build --force alone won't rescue a drifted volume, the only reliable path is to backup, stop the stack, remove the sites volume, bring it back up, and restore. Three tabs — copy each into your maintenance-window runbook.
Three copy-paste stages for a clean patch-level upgrade across a sites-vol topology change: backup, down + prune, up + restore.
Take a hot backup with files before you touch a single container. Copy the artefacts out of the container to the host before shutdown.
# Inside the backend container — takes a fresh DB + files backup
docker compose exec backend \
bench --site erp.yourco.in backup --with-files --compress
# Backups land in sites/erp.yourco.in/private/backups/
# Pull them out to the host BEFORE you touch the volumes.
docker compose cp \
backend:/home/frappe/frappe-bench/sites/erp.yourco.in/private/backups \
./pre-upgrade-backup-$(date +%F)
# Sanity check — the latest DB dump should be > 0 bytes and gunzip cleanly.
ls -lh ./pre-upgrade-backup-*/*.sql.gz | tail -1
gunzip -t ./pre-upgrade-backup-*/*.sql.gz && echo "backup OK"Take a hot backup, and copy it off the container
bench backup --with-files --compress is fast but useless if the artefact never leaves the container. docker compose cp the entire private/backups/ directory onto the host, then gunzip -t the newest .sql.gz. If gunzip complains, do not proceed — take another backup.
Bump the version pin and prune ONLY the sites volume
Edit .env (ERPNEXT_VERSION=v15.17.0, FRAPPE_VERSION=v15.17.0), then docker compose down (no --volumes flag). Verify the volume name with docker volume ls, then docker volume rm <sites-vol name>. Leave mariadb-vol and redis-vol alone — you will restore into a fresh site DB from your dump.
Pull the new tags, then up the stack cleanly
docker compose pull first — you want image pulls timed separately from the maintenance window so it doesn't blow your SLO. Then docker compose up -d and tail the configurator logs until it prints its "Done" line. The configurator init step recreates sites-vol from the new backend image.
Restore the DB, migrate, and rebuild assets
docker compose cp your backup back into backend:/home/frappe/frappe-bench/sites/<site>/private/backups/. Run bench restore --force, then bench --site <site> migrate to run the new patches. Finish with bench build --force — the --force flag matters here, per the source note above. Curl the desk bundle URL and confirm a 200.
Sizing the downtime window
The migration is safe. It is not fast. If your DB is 12 GB and files are 30 GB, back-of-envelope you're looking at 20–25 minutes of downtime — and if your VPS has anaemic disk IO, that can double. Nudge the inputs below to your actual instance:
Live estimator for the total downtime window of a backup → prune → restore upgrade, given your DB size, files size, and VPS bandwidth.
Two things about the number this spits out. First, it assumes your VPS pulls Docker images at a decent clip — the difference between a 100 Mbps and 500 Mbps link on a 2.4 GB combined pull is 3 minutes vs 40 seconds, and small VPS providers vary wildly. Second, it does not include any customer-facing announcement buffer. If your accounts team has to file GST returns in the window, add explicit "no maintenance windows between the 10th and 20th of the month" to your runbook. Nobody enjoys explaining to a CA at 20:00 IST on the 20th why they cannot post an invoice — see also our GST return filing guide for the calendar constraints.
What if devops runs infra and accounts owns the maintenance window?
The recipe above assumes one person holds both the shell prompt and the authority to schedule downtime. In most SMEs those two people are the same head-cook-and-bottlewasher; in the ones that have grown past that, they're not. The clean split:
Devops posts a maintenance window RFC
A one-page ticket with proposed date/time, the Rollback SLO Estimator number, the customer-facing announcement text, and the rollback trigger criteria — "if any of these three things happen, we roll back." No infra work starts until this is signed off.
Accounts and sales own the window's boundary conditions
They confirm no GST filings, no month-end close, no active large sales cycle inside the window. They also own the "who tells customers" step — devops does not decide when to send the WhatsApp broadcast.
Devops runs the recipe against staging first, timed
A dry run on staging (same size DB, same VPS class) gives you a real number instead of the estimator's guess. Post the actual timings back to the RFC before the production window opens.
Devops runs the upgrade, accounts monitors business-critical flows
Post-upgrade, accounts confirms Purchase Invoice submit works, GRN submit works, GST returns can be regenerated. Devops confirms asset healthcheck (Server Script) is passing and no scheduled jobs are stalled. Both sign off on the ticket before the window closes.
This is not process for its own sake. It stops the failure mode where an infrastructure upgrade collides with a compliance deadline and one team blames the other. The RFC is the single source of truth; everyone signs it before touching a container.
Gates and automation — the permanent fix
The upgrade recipe is defensive. The gates are preventive. Ship all four — they don't overlap, they layer.
Four copy-paste automations that turn silent asset-pipeline breaks into loud alerts: VPS cron pre-upgrade snapshots, a Frappe Server Script healthcheck, a packaged hooks.py version, and an n8n Slack workflow.
A crontab entry on the VPS that keeps a rolling 7-day window of hot backups. First line of defence for any upgrade — you can never regret having yesterday's dump.
# /etc/cron.d/erpnext-pre-upgrade-snapshot
# Runs 02:15 IST every day; keeps 7 daily backups; alerts on failure.
SHELL=/bin/bash
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
MAILTO=ops@yourco.in
15 2 * * * frappe cd /home/frappe/frappe_docker && \
docker compose exec -T backend \
bench --site erp.yourco.in backup --with-files --compress \
&& docker compose cp \
backend:/home/frappe/frappe-bench/sites/erp.yourco.in/private/backups \
/var/backups/erpnext/$(date +\%F) \
&& find /var/backups/erpnext -maxdepth 1 -type d -mtime +7 -exec rm -rf {} \;
# Sanity check every hour — email if the latest snapshot is stale.
0 * * * * frappe [ "$(find /var/backups/erpnext -maxdepth 2 -name '*.sql.gz' -mmin -1500 | wc -l)" -gt 0 ] || \
echo "No fresh backup in the last 25h" | mail -s "[ERPNext] snapshot stale" ops@yourco.inWhat each gate actually enforces
Pre-upgrade snapshot
VPS cron backs up nightly and keeps a rolling 7-day window on the host disk. You can never regret having yesterday's dump — and it's not sitting inside the container that broke.
Asset healthcheck
Every 10 minutes, HEAD the desk bundle. If it 404s, ops gets an email before the first user has finished their coffee. The check is 15 lines of Python.
Packaged app hooks
A hooks.py in a small custom app means the guard survives fresh volumes, new sites, and full instance rebuilds. Ship the guard, not the snippet.
n8n Slack alerts
For teams that live in Slack, not email — a self-hosted n8n workflow polls the desk bundle and posts to #ops-alerts on the first non-200. No paid monitoring SaaS.
Deploy order that actually works
- Ship the VPS cron snapshot first — no upgrade should ever run against an instance without a fresh, host-side, gunzip-tested backup.
- Deploy the Server Script asset healthcheck next — you want it running for at least 24 hours before you attempt any upgrade, so you have a baseline for "this instance's assets normally return 200".
- Package the guard into a custom app third — the ad-hoc Server Script is fine for a single instance; the packaged hooks.py version is what you install on the next twelve.
- Wire n8n → Slack last — email works, but nobody sees email inside a 10-minute window. Slack alerts turn a 4-hour outage into a 20-minute outage.
Standing preference: none of these gates use GitHub Actions or any paid GitHub service. VPS cron, Frappe scheduler, hooks.py, self-hosted n8n. Own your monitoring, don't rent it.
Rollback SLO — a policy draft you can lift
If your accounts team is running month-end close in the same week you're planning an ERPNext upgrade, they need a written policy on when devops rolls back rather than pressing on. Below is a defensible starting point — treat these numbers as a template your ops lead and CFO sign off on, not as gospel:
Upgrade Rollback SLO (draft — for ops-lead / CFO review): For any patch-level or minor
frappe_dockerupgrade, the maintenance window is sized as the Rollback SLO Estimator's total × 1.4, rounded up to the next 15-minute increment. If the actual runtime exceeds 80% of that window before assets pass the healthcheck, devops triggers rollback (restore the previous snapshot, revertERPNEXT_VERSIONin.env,docker compose up -don the previous tag) without further authorisation. If a rollback is triggered, a post-mortem RFC is filed within 48 hours, blocking the next upgrade attempt until sign-off. No upgrade is scheduled inside a filing period (GSTR-1 window, GSTR-3B window, TDS return window per the ITA 2025 TDS fix guide); those windows are declared frozen by accounts and non-negotiable by devops.
Three things to notice about this being written down rather than left to Slack:
- 80% is a genuinely tight trigger. It stops the "we're almost done, just five more minutes" spiral that turns a 20-minute window into a 3-hour outage. If you can't unbreak it in 80% of the window, you probably can't unbreak it in the window at all.
- The 1.4× buffer matches the estimator. The estimator already displays the buffered number; the SLO policy names that number as the customer-facing window.
- Filing-period freezes need to be in the policy, not in someone's head. Otherwise the first time it collides with GSTR-3B you'll rediscover the cost the hard way.
Fix immediately vs backlog the process
The left column is an outage response. The right column is engineering hygiene. Both matter; they run on different clocks.
FAQ
+Why does bench build alone not fix the broken assets after a frappe_docker upgrade?
Because bench build writes into the same sites-vol that nginx is already reading from with stale hashes. If the volume was populated by the previous image and the manifest was rewritten but the old hashed files are still there — or vice versa — nginx will 404 every request until the volume is either recreated or force-rebuilt with --force. The build without --force also tries to download a pre-built tarball, which can reintroduce mismatched hashes.
+How do I fix broken CSS and JavaScript in ERPNext after a docker compose upgrade?
Work the diagnose checklist top to bottom. If steps 1–4 fail, the fix is the safe migration recipe: back up, docker compose down, remove the sites volume, docker compose up -d, restore the DB, bench migrate, bench build --force. Never remove the mariadb or redis volumes.
+Can I upgrade frappe_docker in place without touching volumes?
On the same minor version, usually yes — bump .env, docker compose pull, docker compose up -d, then bench migrate and bench build --force. Across a release that changed the volume topology (v15.16 → v15.17 is one such), the in-place path leaves a drifted assets tree. When in doubt, follow the recipe; the extra 15 minutes is cheaper than an outage.
+Does this apply to Coolify, Dokploy, or Heroku-style wrappers around frappe_docker?
The mechanism is identical — Coolify and Dokploy both compose the same upstream frappe_docker under the hood, so a v15.16 → v15.17 bump has the same volume-topology drift. What changes is the wrapper's UI for editing .env and running docker compose exec. See Coolify vs Dokploy vs Heroku for the operator-level differences.
+What happens if we skip the backup step to save time?
The recipe deletes sites-vol. If the DB restore fails afterwards, or the new backend image has a subtle patch regression on your data, there is no path back except restoring from whatever cold backup your VPS provider took overnight — which is usually 12–20 hours stale on a hot-transacting ERP. Skipping the backup step is the single most common reason a 25-minute upgrade turns into a day-long incident.
+Does the compose topology change apply to v14 or v16 as well?
The read-only sites-vol mount and configurator-completed dependency are present in v15.17+ and every v16 release we've deployed. In v14, the same shape landed later in the 14.x series. If you're on an ancient 14.x tag and the compose file predates that change, the safer path is to upgrade frappe_docker's compose file first (with the same tag pinned) before you bump the ERPNext version.
+Will the asset healthcheck slow down the instance?
No. It runs every 10 minutes as a Frappe Scheduler Event, does two HTTP requests (a GET on assets/assets.json and a HEAD on one bundle), and completes in under 200ms. It runs off the request path entirely. On any of the VPS classes we recommend on AWS vs Hetzner, the overhead is imperceptible.
+Should I run this recipe on staging first, or straight to production?
Always staging first, ideally with a same-size DB restored from production. The estimator's numbers are envelope estimates; a staging dry-run gives you the real number, which is what should go into the customer-facing announcement. If you don't have a staging instance sized like production, our hosting cost guide explains how to size one for less than the cost of a single failed upgrade.
Related issues you may also hit
- ERPNext v15 → v16 upgrade guide — the major-version upgrade adds migration steps and a workspace schema change on top of the asset-pipeline mechanics covered here.
- ERPNext v16 Workspace guide — if the desk loads but modules disappear post-upgrade, the workspace schema is the usual culprit, not assets.
- Hosting cost — AWS vs Hetzner (managed vs self-hosted) — sizing the VPS so the recipe fits in a maintenance window.
- Self-host ERPNext with Coolify — the wrapper's UI shortcuts most of the docker-compose commands in the recipe, at the cost of some visibility into what's happening.
- Coolify vs Dokploy vs Heroku — deciding which wrapper to run before you next hit this problem.
Closing
frappe_docker is stable, the ERPNext release cadence is stable, and neither team is going to hand-hold your upgrade path. The mechanism that broke issue #1353 is not a bug — it's a design where two containers share a volume, and the volume drifts across releases whose topology changed. Ship the four gates, size the window with the estimator, write the SLO into a policy your ops lead signs, and the next patch upgrade becomes a 25-minute maintenance window rather than a Friday-evening incident.
Upgrade window looming and no runbook?
We run a one-week engagement for self-hosted ERPNext teams — audit the compose topology, drill the recipe on staging, ship the gates, and hand your ops lead a runbook they can execute without paging us.