ECC DPS · Engineering Docs

Appearance

Troubleshooting ​

Every entry follows the same structure: Symptoms → Likely causes → Checks → Resolution → Escalation.

API unavailable ​

Symptoms: dashboard shows "Could not reach the DPS API"; curl https://api.eccdps.org/ fails or hangs; Render health checks fail.

Likely causes: Render service down or crashed (bad deploy, OOM, boot-time config failure); DNS/CDN issue on the api.eccdps.org record; deploy in progress; Cloudflare/CDN issue.

Checks:

  1. curl -i https://api.eccdps.org/ — expect DPS API Online.
  2. Render dashboard: service status, deploy history, memory, restart count.
  3. Render logs for boot-time exceptions (the app fails fast when production config is missing).
  4. Cloudflare: DNS record for api.eccdps.org and proxying status.
  5. curl the direct Render origin (it should still answer /).

Resolution: redeploy or roll back the previous deploy; fix the missing environment variable and redeploy; adjust memory/instance size if OOMing; fix DNS if the record drifted.

Escalation: if the service is up but data routes fail (503/500), move to Database connection failure. If Cloudflare/Render itself is degraded, contact the platform.

Dashboard cannot authenticate ​

Symptoms: login redirects back to /login with an error code; /auth/me returns 401; "not a DPS agent" or "agent not active" errors.

Likely causes: agent record missing or not active/onboarding; Discord OAuth misconfiguration (client id/secret/redirect URI); environment mismatch (demo cookie on production ingress); session secret rotation invalidating sessions; Discord outage.

Checks:

  1. The error code on /login?error=… — it maps to the failure step (not_a_dps_agent, agent_not_active, oauth_state_mismatch, discord_redirect_uri_mismatch, …).
  2. The agent's agents row (status, discord_id) via an admin/read path.
  3. Render env: DISCORD_CLIENT_ID/SECRET/REDIRECT_URI, OAUTH_SESSION_SECRET, DASHBOARD_ORIGIN.
  4. API logs for the OAuth failure reason.
  5. Discord status for an ongoing outage.

Resolution: fix the agent record (admin path, audited); correct OAuth config and redeploy; wait out Discord outages (the API's retry budget protects itself).

Escalation: if a class of users fails after a secret rotation, expect one-time re-authentication (environment-bound sessions invalidate old cookies); if logins are being abused, involve incident response.

Database connection failure ​

Symptoms: data routes return 500 with "Database error: …"; health's database section is error; all authenticated reads fail while / stays up.

Likely causes: Supabase project paused/outage; service-role key rotated or wrong; Supabase URL changed; network egress from Render blocked.

Checks:

  1. GET /admin/health database section (if admin access is available) or the API logs.
  2. Supabase dashboard: project status, API settings (URL, service key), usage/limits.
  3. Compare Render env (SUPABASE_URL/SUPABASE_KEY) with the dashboard values.
  4. Supabase status page.

Resolution: restore Supabase service; update the env vars and redeploy; raise project limits if exhausted.

Escalation: if a key was rotated/leaked, rotate again deliberately and audit access; if data appears affected, do not run recovery steps without a backup — follow Database recovery.

Migration failure ​

Symptoms: apply_rpc_migrations.py prints an error for a file; admin Schema tab shows a missing/partial entry; new endpoints 500 with "function does not exist" / "column does not exist".

Likely causes: file applied out of order; 033 re-applied after 034; connection string pointing at the wrong project; SQL error in the file; partially committed multi-statement file.

Checks:

  1. schema_migrations rows vs migrations/ filenames (checksums).
  2. Which file failed and its first error line.
  3. Whether earlier statements in the same file committed (pg8000 run does not wrap multi-statement input in one transaction unless the file does).
  4. Confirm the target project (production vs demo) before anything else.

Resolution: fix the SQL error and re-apply the file; if 033 was re-applied after 034, re-apply 034 (it drops the superseded overloads); apply missing files in order — schema before code.

Escalation: if the schema and schema_migrations have drifted beyond a single file, stop and reconstruct the applied set from backups/verification before proceeding.

Realtime connection failure ​

Symptoms: training UI shows disconnected/reconnecting; channel joins fail with CHANNEL_ERROR; events don't arrive live.

Likely causes: SUPABASE_ANON_KEY/SUPABASE_JWT_SECRET unset or wrong on the API; RLS policies on realtime.messages missing/regressed (migrations 026/027); JWT expiry handling (tokens refresh before expiry — a stuck clock breaks it); Supabase Realtime degraded.

Checks:

  1. /training/realtime-config returns URL + anon key; /training/realtime-token mints a token (decode claims: role/sub/exp).
  2. Supabase dashboard: Realtime enabled, private channels configured.
  3. pg_policies on realtime.messages and training_session_members match migrations 026/027.
  4. Browser console for channel errors; API logs for broadcast failures.
  5. Supabase status.

Resolution: set the env vars and redeploy; re-apply 026/027 (idempotent); restart the Realtime service if Supabase guidance says so. Remember: Realtime is best-effort — session state, notes, evidence, and grading all still work through the API; the UI shows the honest disconnected state.

Escalation: persistent CHANNEL_ERROR for private channels with correct config is a platform-level Realtime issue — engage Supabase support with the migration verification notes.

Evidence / media not loading ​

Symptoms: evidence images broken; lightbox empty; profile avatars missing; links open nothing.

Likely causes: external host down/blocked (Discord CDN, imgur); stored URL scheme rejected by client sanitization (legacy javascript:/data: rows render fallbacks by design); evidence row missing its report (should not happen — FK cascade); API error on GET /evidence.

Checks:

  1. Network tab: the media request URL and status.
  2. Open the stored URL directly; check the host.
  3. Check the raw evidence.url value (via an admin read) for the scheme.
  4. GET /evidence response for the item.

Resolution: external host recovery for outages; re-add/replace bad rows via the normal flow (or Director-only correction); nothing to fix when sanitization correctly blocks an unsafe URL — that is the security control working.

Escalation: if the API itself fails to serve evidence metadata, treat as an API issue (see API unavailable).

Discord integration failure ​

Symptoms: reports not created; queue actions never complete; reporter DMs not sent; punishments not delivered.

Likely causes: bot (BotGhost) offline; API key mismatch/rotated; queue rows stuck processing (bot crashed mid-action) or terminal failed (retry cap); Discord API outage/rate limits.

Checks:

  1. Queue views: GET /actions/pending/*, GET /admin/queue, GET /discipline/pending?status=failed — statuses, attempts, last_error.
  2. Bot-side logs for pickup errors (401 = key mismatch).
  3. API logs for completion calls and last_error values.
  4. Discord status.

Resolution: restart the bot (stale sweeps self-heal processing rows on the next pickup); requeue failed actions (POST /actions/<id>/requeue, /discipline/<id>/requeue, or the admin queue controls); fix the API key; wait out Discord incidents (the discipline queue supports paced retries via retry_after_seconds).

Escalation: if deliveries repeatedly fail with a specific Discord error, that error is visible in last_error — escalate with it. If the API key may have leaked, rotate it and audit recent bot-path activity.

Permission denied ​

Symptoms: 403 "Insufficient clearance" / "Supervisor reports require…" / "Not an admin user" / "Director access required" on actions that worked before.

Likely causes: clearance or rank changed server-side (demotion takes effect immediately); admin allowlist membership removed; assignment changed (only the assigned agent may mutate a case); supervisor-flag on the report.

Checks:

  1. /auth/me — current permissions, can_handle_supervisor, admin flags (server-computed from the live row).
  2. The target report's assigned_agent / is_supervisor.
  3. admin_users.active for the agent.
  4. The audit trail for recent agent updates (someone may have changed the record — correctly).

Resolution: deliberate access changes are working as designed; the UI re-fetches after 403s. If the change was a mistake, the Director/admin path fixes the record (audited, with a reason).

Escalation: a permission escalation (someone gained access they should not have) is an incident — contain via the audited admin path, rotate if credentials were involved.

Investigation claim conflict ​

Symptoms: "This case has already been claimed" or "This action can no longer be performed — the case status changed" (409) when clicking claim/validate.

Likely causes: another agent claimed or actioned the case first; the report was already in the target state; a contact thread is already open for contact_reporter.

Checks:

  1. The report's current status/assigned_agent (refresh the docket).
  2. The timeline for the competing action and its actor.
  3. The audit log (/audit) for the same window.

Resolution: this is the race guard working correctly — exactly one concurrent action wins. The losing agent refreshes and sees the new state. If the case is stuck (e.g. assigned to an absent agent), reassignment requires Head Investigator rank or above (reassign_docket).

Escalation: if a stuck action queue row accompanies the conflict, use the queue recovery paths; if the assigned agent is unavailable and no supervisor can act, escalate to leadership for reassignment.