Troubleshooting failed deploys: errors and fixes

When an ox deploy fails, the run's log names the failed step, its cause and the fix, then the last lines of output, and the previous release keeps serving. Find the message you see below: each entry gives the symptom, the cause and the fix. In the log the shape is always the same.

<step>: <cause>
fix: <fix>
--- last output ---
<the last lines the command or the app printed>

Open the run in the console, or run ox runs <project> and then ox logs <project> --run <id>. A failed ox deploy --wait exits with status 1.

Health check failed: the app did not answer within 120s

Symptom

The log shows waiting for … (up to 120s), then a step named health (<unit>) with one of these causes.

the app exited while starting (…)
… did not answer within 120s: …

Cause

The new release started but never answered. ox polls every 2 seconds for up to 120 seconds: an HTTP GET on [app] health that must return a 2xx or 3xx status, or a TCP connect when no health path is set. The usual reasons are a start command that does not listen on $PORT, a missing variable that makes the app crash, or a health path that returns 404.

When the app tried to write inside its own release, the cause says so: the app wrote to <path>, and a release's files are read-only while it runs.

Fix

Nothing changed for your visitors: the fix line ends with the previous release is still serving. Read the app's own log above the failure, then fix it and push.

• Bind to 127.0.0.1:$PORT, as in [app] start.
• Point [app] health at a path that answers without a login.
• For a read-only write, the fix line names the folder to add to [storage] keep, which links it to a writable folder that survives deploys.

Build failed: exit status 1, or set these before deploying

Symptom

The run stops at an install, build or migrate step, or before anything starts.

build 1/2: npm run build: exit status 1
fix: the command's own output is above; fix it in the repo and push

preflight: 1 problem(s); nothing on the host changed
fix: fix the listed problems and deploy again

Cause

A command from [build] exited non-zero, or preflight found a problem, such as set these before deploying: SECRET_KEY for a variable with no value. A failed build or migrate never touches the live release.

Fix

Run ox check in the repo first: it prints Ready to deploy., or a list of problems with the field each one is about. Then reproduce the failing command locally, since the log shows each one as it ran. Set missing variables on the dashboard or with ox vars set <project> KEY; Variables shows how.

Domain not working: waiting for DNS, points elsewhere, no certificate

Symptom

The domain's row on the project's Settings tab says waiting for DNS, points elsewhere or No certificate instead of HTTPS on.

Cause

The name does not resolve to the server yet, or it resolves somewhere else, or a Cloudflare proxy in front of it has no certificate for it. The row says which one, for example It resolves to 203.0.113.9, not this server. Change the record to: followed by the A record to create.

Fix

Create or change the record the row shows, at your DNS provider (how to point a domain). From a terminal, ox domains <project> lists each domain with its DNS state and prints Point each domain's A record at <ip>. The row turns green by itself once the record resolves.

For No certificate behind Cloudflare, check the certificate settings for the name in Cloudflare, or set its record to DNS only so the server makes its own certificate.

another operation is already running

Symptom

another operation is already running for shop; wait for it to finish, then retry

Cause

ox runs one thing at a time per project: a deploy, a rollback, a backup, a restore or a delete. A push, the daily backup or another person may have started one a moment before you.

Fix

Wait for it to end, then run your command again. ox runs shop lists what is running, and ox logs shop --run <id> follows it.

Disk full: a deploy needs at least 2 GB

Symptom

disk: only 1400 MB free on /srv/ox; a deploy needs at least 2 GB
fix: delete old projects or grow the disk

sudo ox doctor on the server says only 4% of the disk is free (…); deploys and backups fail soon once less than 5% is free.

Cause

The server's disk is close to full: releases, backups, logs and your app's own files share it. ox deletes older releases as the disk runs low, and keeps one project's backups under 20% of the disk, but it always keeps the newest release you can roll back to and the newest backup of each service.

Fix

Delete projects you no longer need, or resize the disk at your provider. Backups move off the server when you add an S3 bucket, so fewer stay on the disk. See how many backups ox keeps.

Not enough memory: killed by the out-of-memory killer

Symptom

Preflight refuses the deploy, or a build step dies:

capacity: the host would need 2304 MB but has 1987 MB: …
killed by the out-of-memory killer (the build needs more memory than the project or host allows)

Cause

The app, its services and its staging and preview copies add up to more memory than the server has, or one build step needs more than its limit.

Fix

The first fix line says it: drop a service or make it shared, lower a [limits] memory, or use a bigger host. Fewer previews at once also frees memory.

The app keeps crashing: systemd restarted it 5 times

Symptom

The deploy passed, but later an alert says:

web of shop on my-server keeps crashing: systemd restarted it 5 times in the last 10 minutes. The Logs tab shows why.

Cause

The process starts, then exits, and the server restarts it again and again. A missing variable, a lost database connection or a full disk are common reasons.

Fix

Open the Logs tab, pick Errors, and read the lines just before each restart, or run ox logs shop. See Logs. If the last deploy caused it, roll back while you fix it.

Server offline: the server has not connected yet

Symptom

A deploy or a command is refused with one of these:

server my-server has not connected yet
server my-server is offline since 12 minutes ago

Cause

The ox agent on the server is not talking to ox. The server may be off, out of network, or the agent stopped. Your sites keep running unless the server itself is down, but ox cannot deploy to it until it reconnects.

Fix

Check the server is on at your provider. Then, on the server:

sudo systemctl status ox-agent
sudo journalctl -u ox-agent -n 50

sudo systemctl restart ox-agent starts it again. If the server was rebuilt, add it again from the dashboard.

Install refused: this server can't take ox yet

Symptom

The Add server command stops before it changes anything:

this server can't take ox yet; nothing was changed:
  - port 80 is in use by nginx; ox's proxy (Caddy) needs it: stop that program, or use a fresh server
  - the server has 512 MB of RAM and ox needs at least 1 GB: resize it
  - the disk has 6.2 GB free on / and ox needs at least 10 GB: free space or resize the disk

Cause

ox needs ports 80 and 443 free for Caddy, at least 1 GB of memory and at least 10 GB of free disk.

Fix

Do what each line says, then run the Add server command again. sudo ox install --check runs only these checks. A fresh Ubuntu LTS server is the easy way.

repository access removed: GitHub gives ox no access

Symptom

repository access removed: GitHub gives ox no access to acme/shop (Not Found); connect GitHub again and pick the repository

Cause

The ox GitHub App was removed from the repository or the account on GitHub, or the repository was renamed or moved.

Fix

Connect GitHub again from the dashboard and pick the repository, then deploy.

not signed in; run ox login

Symptom

not signed in; run ox login
the plane refused the token (expired or revoked); sign in again with ox login

Cause

This computer has no token yet, or its token expired or was revoked in the dashboard. The command exits with status 3.

Fix

Run ox login and approve the code in the browser. A could not reach … message with status 4 is the network, not the sign-in. See exit codes.