Troubleshooting
For the organizer mid-event, ordered by the symptom you actually see — diagnosis, then fix. Setting up for the first time? That’s hosting.md. Verifying before the event? The checks in operations.md and the security checklist catch most of this page before it happens.
The two commands this page leans on:
docker compose logs -f sync # the poller's own account of itself
docker compose exec redis redis-cli HGETALL ctf:admin:settings # the live overrides
(redis-cli authenticates itself inside the container via REDISCLI_AUTH —
no password on the command line.)
On AWS (ECS) there is no docker compose and no redis-cli. The same
two views are:
aws logs tail /ecs/<name>/sync --follow # the poller's own account of itself
aws ecs describe-services --cluster <cluster> --services app srh scorer sync \
--query 'services[].{name:serviceName,running:runningCount,desired:desiredCount}'
and a Redis read goes through srh from an ECS Exec shell in the app container. The exact command, how to get the shell, and every other ECS recipe are in the AWS event-day runbook.
docker compose up refuses to start: “set REDIS_PASSWORD in .env”
Symptom. Compose exits immediately at interpolation with that message.
Diagnosis. Deliberate. Redis runs with requirepass, and compose reads
the variable with :? so a missing or empty value fails the bring-up
instead of quietly starting an unauthenticated Redis.
Fix. Add REDIS_PASSWORD=$(openssl rand -hex 24) to .env (a .env
from before the kit required it is the common case), or run
./setup/ctf-setup.sh secrets on a fresh checkout. Note the same
interpolation failure inside a script that pipes compose’s stderr to
/dev/null looks like an empty service list, not an error.
/admin 403s for everyone, including you
Symptom. Nobody — including you — can open /admin; every listed admin
gets the 403 wall.
Diagnosis. ADMIN_LOGINS is empty or unset in .env, or it is set but
the app was not restarted after you changed it. Since #386 part 4 the app no
longer bakes an admins list at build time — it reads the comma-separated
ADMIN_LOGINS env var at runtime, and an empty set (unset, blank, or every
entry failing the GitHub-login shape check) refuses every login, including
one that used to work. (The event’s name and the rest of its branding, the
fork org, and which Secure Development targets run are separate runtime
settings — the name and targets default to “OWASP CTF in a Box” / all six
either way, so a stock-looking name or a full target list is not a symptom of this
problem on its own.)
Fix. Set ADMIN_LOGINS in .env to a comma-separated list of GitHub
logins (case doesn’t matter — logins join case-insensitively everywhere in
this repo), then restart the app container — no rebuild needed, since this is
a runtime read, not a build arg:
docker compose --profile secdev --profile app up -d
(Quiz/classic/ai-only events: --profile app alone.) Compose must also be
passing ADMIN_LOGINS through to the app service’s environment — it is on
recent docker-compose.yml, but a customized override file that dropped it
would reproduce this exact symptom.
On AWS the list is the admin_logins Terraform variable, not .env.
Set it in terraform.tfvars and run terraform apply, which rolls out a new
app task definition. An empty value is refused at plan time.
Only the landing page loads (module pages redirect to /)
Symptom. Contestants open /challenges, /flags, /quiz, /ai or
/leaderboard and land back on /, which says Launching soon. or shows
a countdown; a script calling a submit or answer API gets
403 { "error": "not-launched" }. You, signed in as an admin, see every page
normally.
Diagnosis. The event is not launched. Since #464 an event is launched
only once its Scoring opens time (scoringStartsAt) is set and has
passed; until then every module page redirects a non-admin to the landing
page and every module API refuses. Admins pass the lock as a preview, which
is why it looks fine from your own session — check from a private window.
An empty field means not launched, not “no start bound”, and so does a
value the settings read cannot parse. The lock fails closed, so a Redis
error also shows this symptom — if /admin itself is failing, fix that
first (NOAUTH,
fetch failed).
Fix. In /admin → Event, set Scoring opens to now to launch
immediately, or leave the scheduled time and wait — the lock lifts on its
own at that instant, no restart. The Right now: readout under the
schedule says “scoring is closed (not launched — set a scoring start)” while
the field is empty, and “scoring is frozen (outside its window)” while the
start it holds is still in the future. A leftover
CHALLENGES_GATE_ENABLED/CHALLENGES_GATE_PASSWORD in .env does nothing
any more (the password gate was removed in v0.6) — delete them. See
Before launch.
A board that was on before the upgrade is gone
Symptom. After upgrading, Quiz, Jeopardy, or AI no longer appears in the nav or on the landing page, even though nothing about that module changed.
Diagnosis. Not a bug and nothing was deleted. Since #386, module
enablement is decided at runtime in /admin, not baked into the image —
and on a fresh deployment (or one whose stored settings have never named
this module) only Secure Development starts on, and only when the stack has
a scorer image; Quiz, Jeopardy and AI always start off.
Fix. /admin → Event tab → Modules, and switch it on. The board’s
data was never touched — solves, attempts and points come back exactly as
they were.
A scored PR isn’t landing on the leaderboard
Work down this list — each item is a different subsystem:
- Is the event frozen, outside the scoring window, or not launched?
Check the Event tab (or
HGETALL ctf:admin:settings—paused,scoringStartsAt/EndsAt). An emptyscoringStartsAtmeans the event has not launched and nothing scores — the usual cause right after an upgrade, since a blank start used to mean “always open” (#464). A score is judged at its PR comment’s time (ADR 67): one posted while frozen, before the launch or after the end never counts, not even after you unfreeze or set a start — the poller logs it asrefused (paused)orrefused (outside-window). A score timed before the last master reset never counts either, loggedrefused (pre-reset). The contestant has to push again once scoring is live. One exception: a freeze set straight on the settings hash (HSET … paused 1, no recordedpausedSince) has an unknown start, so the scorer holds the scores it reads during it and judges them once the freeze lifts. - Did the fork’s Action run and post the score comment? Open the PR: you
should see the
github-actions[bot]comment with the score table. No comment → the fork’s workflow didn’t run or failed; check the fork’s Actions tab, then./setup/ctf-setup.sh doctorfor the fork’s provisioning row (workflow present? version current? image grant observed?).upgradere-applies a stale workflow. - Is
syncactually polling?docker compose logs -f sync(on AWS,aws logs tail /ecs/<name>/sync --follow). Nosynccontainer at all means the stack came up without--profile secdev— correct for an event with noSCORE_IMAGE, wrong if you expected scoring. A container that exits non-zero withctf-sync: GITHUB_ORG is not setmeans the key is missing from.env; set it and bring the stack back up. A tick that polls nothing and records alastErrormeans the poller could not read the Secure Development target list fromctf:admin:settings— it waits rather than guessing, so fix Redis and the next tick catches up. A tick log withdroppedcounts names why a comment was refused (forged author, unknown target, malformed marker). - Is the score comment authored by
github-actions[bot]? Only that author is trusted — a comment posted any other way (including by you) is dropped by design. - Poll cadence is ~30 s — a score that lands a minute late in a busy tick is normal, not stuck.
The fork’s Action fails: “no matching manifest for linux/amd64”
Diagnosis. The scorer image was built on Apple Silicon without a platform pin — GitHub’s runners are amd64.
Fix. Rebuild pinned and push:
docker buildx build --platform linux/amd64 --provenance=false --sbom=false -t <SCORE_IMAGE> --push scorer/
(the wizard’s own build step already pins this).
Services log NOAUTH Authentication required
Symptom. srh (or anything behind it) errors with NOAUTH; reads and
writes fail.
Diagnosis. The password the running containers carry doesn’t match the
Redis they’re talking to — typically .env was edited after the stack came
up (compose does not re-read .env into running containers).
Fix. docker compose up -d again (recreates with current env). On AWS,
srh reads its connection string from the SRH_CONNECTION_STRING SSM
parameter when a task starts, so a task started before that parameter
changed still carries the old one. Start fresh srh tasks with
aws ecs update-service --cluster <cluster> --service srh --force-new-deployment.
Note an
unauthenticated PING answering NOAUTH is the correct state —
scripts/smoke.sh asserts it — the bug is only when the kit’s own services
hit it.
sync is crash-looping (or ingestion died mid-event)
Symptom. docker compose ps shows sync restarting; every tick throws.
Diagnosis. The poller’s cursor lives in Redis at ctf:sync:state (a
JSON string). Older builds kept it in /state/state.json on the
sync-state volume; the first boot of a current build moves that file’s
contents into Redis and renames the file to state.json.migrated. Current
sync validates and repairs damaged state field by field (each repair
is logged — look for repair lines before assuming worse). Historically a
bare {} — valid JSON, unusable shape — crash-looped the poller for a whole
event, which is exactly why the repair exists.
If the logs instead repeat cannot load poll state from Redis
(ctf:sync:state) … not polling until it is readable, sync is not crashing:
it is holding on purpose, because starting from an empty cursor would
re-ingest every score comment. Fix srh/Redis (the fetch failed and
NOAUTH entries in this runbook) and it resumes on its own at the next
retry. The same hold applies on the first boot after an upgrade when the old
state.json exists but cannot be read (cannot read the legacy state file
…): fix its permissions, or delete it to start from an empty cursor on
purpose. It also holds, after copying the file into Redis, if it cannot move
the file aside (could not move … aside), because a leftover file would
re-seed a wiped Redis with a stale cursor: fix the directory’s permissions,
or delete the file — its contents are already in Redis.
Fix. Read the first error line of docker compose logs sync. If state
is beyond repair: docker compose exec redis redis-cli DEL ctf:sync:state,
then restart sync. Losing the cursor does not double-count (the scorer’s
writes are idempotent on replay) — but the re-read brings back any Secure
Development solves a per-contestant reset removed, and resets /admin’s
ingested/dropped counters to 0.
On AWS there is no sync-state volume and none is needed: the cursor is
in ctf:sync:state on ElastiCache, so a new sync task resumes from it, and
holds while srh is unreadable. Read the error
with aws logs tail /ecs/<name>/sync --since 30m, and why the task stopped
with describe-tasks (see the
AWS runbook). A GITHUB_ORG is not set or a
missing App id means a Terraform variable is empty: set github_org and
github_app_id, then terraform apply. Only deleting ctf:sync:state
makes sync re-read every score comment, which undoes a per-contestant Secure
Development reset: see the sync drill in the
AWS runbook.
The monitor says /health/deep is 503 (but the site loads fine)
That combination is the check doing its job. Every read in this kit fails
open — a Redis blip must not stop contestants playing — so a dead dependency
leaves the site rendering while nothing scores. /health/deep is the one
place that failure is visible from outside. Read the body:
"redis": "down"— the app cannot reach Redis through srh. Nothing is scoring in any module and admin settings reads are serving defaults. On Fly:fly ssh console --app <app> -C "redis-cli PING", thenfly logsforsrhandredis; aNOAUTHthere is the password mismatch described above. On compose:docker compose psand thesrh/redislogs. On AWS:aws logs tail /ecs/<name>/srh, anddescribe-servicesfor whether the srh tasks are running at all. The module runs two, so the stack reads as down only when both are. See the srh and failover drills in the AWS runbook."scorer": "down"— Redis is fine but the scorer’s/healthzdid not answer. Quiz, Jeopardy and AI keep scoring; Secure Development scores stop landing and the board shows stale SD totals. Restart the scorer container (on AWS,aws ecs update-service --cluster <cluster> --service scorer --force-new-deployment, after reading itsstoppedReason);LEADERBOARD_API_URLunset on a box withSCORE_IMAGEset also reports this, since a scorer nothing can reach is as good as down."sync.ageSec"growing while the check is 200 — not a failure of the check (the poller is reported, never failed on), but it means no poll has completed since that many seconds ago. Scored PR comments are accumulating on GitHub; see “syncis crash-looping” above. If the machine is idle-suspending (FLY_AUTO_STOPnotoff), that is the cause.
The body never says why — no host, no error text; that is deliberate for a
public URL. The reason is in the server log: fly logs --app <app>,
docker compose logs app, or on AWS aws logs tail /ecs/<name>/app, lines
tagged [health/deep].
What a dead container actually does on Fly (measured, not assumed). The
rendered compose file keeps each service’s restart: line from
docker-compose.yml, and deploy.sh sets no restart policy of its own, so
the machine runs at Fly’s default on-failure, ten retries. Killing PID 1 in
the app container on the live box (2026-09-16, `fly ssh console –machine