Runbooks

Each runbook names a symptom, the commands to run and how to confirm the result. The reasons live in the linked decisions; this page lists only the steps. Run database commands against the environment you mean, and never against a database you need without a backup.

Database restored or moved to another host

Symptom. After a pg_restore or a provider move, tenant requests see no rows, fail with permission errors, or db:apply-pending reports RLS problems. A restore with --no-owner --no-acl loses grants, and loses policies when the app_tenant role is missing.

Create the role if the host cannot
If the target connection role cannot create roles, have an administrator create app_tenant and grant it once:
```
CREATE ROLE app_tenant NOLOGIN NOINHERIT NOBYPASSRLS;
GRANT app_tenant TO <connection role> WITH INHERIT FALSE, SET TRUE;
```
Restore and apply
Restore the dump, then run the pending migrations against the target. The command ends with the RLS verification and repairs its idempotent part:
```
pnpm --filter @template-cloudflare/db db:apply-pending
```

Verify and repair

Check, and repair what is left if needed:

pnpm --filter @template-cloudflare/db db:verify-rls
pnpm --filter @template-cloudflare/db db:verify-rls --repair

Verify. db:verify-rls exits 0, and select pg_has_role(current_user, 'app_tenant', 'SET') is true on the runtime connection. If the runtime role differs from the migration role, add it to app.session_roles; db:verify-rls says so. Reason: ADR-0009.

Tenant statements fail after a database or driver change

Symptom. Tenant queries error or time out after a PostgreSQL, Hyperdrive or pg change, while system paths (Better Auth, cron) work.

Set the kill switch
Turn off statement pipelining without a code change. Set the Worker variable or secret DB_TENANT_PIPELINE=0. Tenant statements then use explicit BEGIN, settings, statement and COMMIT.
Retry
Redeploy or wait for the variable to apply, and retry the failing request.
Prove both modes locally
Reproduce locally with both modes before turning pipelining back on:
```
pnpm --filter @template-cloudflare/db test:rls-matrix
```

Verify. The failing request succeeds with the switch set. test:rls-matrix passes in both modes before you remove the variable.

Hyperdrive query caching is on

Symptom. The production deploy stops at the Hyperdrive check, or a Hyperdrive was created without pnpm setup. With caching on, reads are keyed by query text and parameters and can cross the tenant boundary.

node packages/project-setup-cli/src/hyperdrive.mjs check
pnpm setup --hyperdrive-id <id>
pnpm doctor --connect

Verify. The check exits 0 and pnpm doctor --connect reports caching disabled. Reason: ADR-0009.

Prove tenant isolation on a deploy

When. Before a production deploy that touches tenancy, the database driver or Hyperdrive.

node scripts/rls-preview-proof.mjs <previewUrl>

Run it against the pull request's preview: the tenancy.probe procedure exists only where TENANCY_PROBE=1, which is set for previews and refused in the production configuration. When sign-up needs email verification, pass two verified accounts with RLS_PROOF_USER_A=email:password and RLS_PROOF_USER_B=email:password.

Verify. Exit 0 means no leak. Exit 1 is a leak or a broken guarantee, so do not deploy. Exit 2 means setup failed (for example, verification required).

Take production offline

Symptom. Production must stop serving traffic now.

Run the Emergency shutdown (wrangler worker) workflow with action=shutdown and confirm=yes. Run it again with action=restore to bring it back.

The custom domain stays online

The workflow toggles the Worker's workers.dev subdomain only. Production is served from a custom domain (routes in apps/web/wrangler.jsonc), which the workflow does not block. To stop traffic on the custom domain, also disable the route or add a WAF rule that blocks it in the Cloudflare dashboard, and remove it afterwards.

Verify. Request the custom domain and the workers.dev URL. Both must refuse traffic during the shutdown, and both must serve again after the restore.

Roll back a Worker deploy

Follow Deployment and rollback. A Worker rollback does not reverse migrations, restore deleted records or undo external deliveries.

After an incident

Write a short note in the issue or pull request that closed it, with three headings:

What is back: the service, the data and the time it returned.
What is lost: rows, deliveries or files that could not be recovered, with evidence.
Verdict: recovered, partly recovered or not recovered. Add follow-up issues.

Create the role if the host cannot

Restore and apply

Verify and repair

Set the kill switch

Retry

Prove both modes locally