Operations Overview

This section covers the day-to-day operational surface of the platform: backups, migrations, deployment, rollback, environment variables, health checks, logs, incident response, and database recovery.

Operational surface map

Concern	Where it lives	How to use
Backups	Supabase Dashboard → Database → Backups (PITR)	Take/verify before any migration; restore in incidents
Migrations	dps-code-api/migrations/ + apply_rpc_migrations.py	Apply in order, schema before code — see Migrations
Deployment	Render (API, auto-deploy from main), Cloudflare Pages (frontend), Cloudflare Workers	See Deployment
Environment variables	Render service env; Pages build env; Worker secrets	See Environment variables reference
Health checks	GET / (liveness), GET /admin/health (deep)	Probe + admin panel
Logs	Render logs; API app.logger; Worker logs	Grep for route/error markers
Incident history	admin_health_history + /admin/health	Sparse log of overall-status transitions
Queue health	GET /admin/queue, /actions/pending/*	Watch for stuck/failed actions

Backups

• Supabase manages backups/PITR at the platform level. Before any migration or schema-affecting operation, confirm a recent backup exists (or take one).
• The demo project is disposable by design: demo_seed.py can re-seed it completely (--reset --yes-demo, or POST /demo/reset with a demo session).

Migrations

• Apply in strict filename order (000–034), one file at a time, schema before code.
• Each file is recorded in schema_migrations (file, applied_at, checksum) and shown in the admin Schema tab.
• Never edit an applied migration; write corrective migrations.
• See Migrations and Database deployment.

Deployment procedures

• API: Render auto-deploys from main; verify boot (fail-fast config checks), liveness, and /auth/me.
• Frontend: npm run build then npx wrangler pages deploy dist --project-name=dashboard; verify both hostnames.
• Order for schema changes: backup → migrations → backend → frontend → smoke tests.
• Rollback strategy: redeploy the previous Render/Pages deployment; safe because migrations are backward compatible and the frontend consumes stable API contracts. Data-affecting rollbacks are corrective migrations or restores — never edits to applied migrations.

Health checks

• GET / — liveness; exempt from environment resolution (Render probe).
• GET /admin/health — deep health: overall + 9 sections (database, api, actions, queue, agents, evidence, auth, audit, events), each ok/warn/error, plus incident history recorded on overall-status changes.
• Queue metrics in the health payload (pending/processing/stale counts) indicate bot health: growing processing counts without completion means the bot is down or the queue is wedged.

Logs

• Render captures API stdout/stderr: Flask request logs, app.logger lines ([discord_profiles], [training], [admin_health], OAuth exchange failures, queue sweep warnings, Realtime broadcast failures).
• Cloudflare captures Worker logs (OAuth proxy, ingress worker).
• What to watch: repeated OAuth retries (Discord outage), Realtime broadcast failures (Supabase Realtime degraded), stale-action sweep warnings (bot polling stopped), admin_health history transitions.

Troubleshooting

The Troubleshooting page is symptom-driven: API unavailable, authentication failures, database connection failures, migration failures, Realtime disconnects, evidence not loading, Discord integration failure, permission denials, and claim conflicts. Every entry follows: symptoms → likely causes → checks → resolution → escalation.

Incident response

1. Assess — confirm scope (single user, all agents, all requests?) via liveness, health, and logs.
2. Contain — for suspected abuse: revoke/rotate the affected credential, suspend the agent (agents.status change via admin path — itself audited), or block the ingress at Cloudflare. Never delete evidence.
3. Diagnose — use the audit/event streams and admin_audit_log (they are append-only for exactly this reason).
4. Recover — re-apply/requeue failed queue work (/admin/queue controls), restore from backup for data loss, redeploy for bad code.
5. Record — the health history and audit trails capture transitions; document the incident in the Changelog and update this documentation if procedures change.

Database recovery

• Data corruption / loss: restore the Supabase backup or PITR point, then re-apply migrations after the restore point (verify schema_migrations matches).
• Schema breakage: apply the corrective migration; never hand-edit.
• Queue recovery: stale sweeps run on every pickup (10-minute timeout, 3-attempt cap); failed actions are visible in the queue lists and can be requeued/cancelled from the admin panel or the recovery endpoints.
• Realtime degradation: broadcast failures are logged and non-fatal; clients reconcile via authoritative GETs. Verify Realtime settings (SUPABASE_ANON_KEY/SUPABASE_JWT_SECRET) when channels fail.

Common failure modes

Mode	First sign	Response
Bot offline	queue processing rows grow stale	Restart bot; stale sweep self-heals on next pickup; requeue failed
Supabase outage	/ ok, data routes 500	Check Supabase status; API fails closed on env resolution only for unknown hosts
Discord OAuth outage	logins fail with retry logs	Bounded retries + capped sleep protect the API; retry after Discord recovers
Realtime outage	training shows disconnected	State still correct via REST; check anon key/JWT secret config
Credential leak	anomalous audit entries	Rotate the credential; audit the trail; restrict ingress
Migration mismatch	admin Schema tab vs migrations/	Compare checksums; apply missing files in order (with backup)

Related reading

• Troubleshooting
• Deployment
• Events & audit