Local incident runbooks
Use scoped cases, existing domain services and verified outcomes. These runbooks do not authorize live outreach, charges or production cleanup.
Worker outage or delayed job
Detect
Open Operations → Overview. Check the worker heartbeat age and grouped queue alert; open Resources and select the exact workspace/import/export/review UUID. A missing heartbeat is unavailable telemetry, not a healthy zero.
Recover
Record the request/resource IDs in a support case. Inspect `python3 platform-foundation/dev.py status`. Restore only the local product worker with `python3 platform-foundation/dev.py worker-start`. For an eligible retry_wait workspace/review job, enter a reason and schedule its existing retry. Expired running leases are recovered by the owning worker; do not reset attempts or lease tokens manually.
Verify
Refresh the case and original user workflow. Verify succeeded/ready on the same job (or explicitly follow the owner-facing retry contract), a single valid output and recovered alert. If unsuccessful, leave the case investigating. Stop new relevant admissions when needed; do not revoke stored answers or completed results.
Payment mismatch or uncertainty
Detect
Use the payment resource kind. Observe original order ID, state, checks and error category. Never paste card details or full provider payloads into notes.
Recover
A scoped billing grant may schedule reconciliation only when the original order account matches a ready Razorpay test adapter and no lease is active. The existing billing worker fetches provider facts and preserves the buyer/order/payment binding. Missing credentials means the action is unavailable. Never make a new order/refund key to escape uncertainty and never edit a grant directly.
Verify
Inspect the original order and entitlement history. Confirm at most one purchased grant and no additional provider order. Held/disputed outcomes remain held; there is no release-hold shortcut. Live credentials, prices/refund policy and real sandbox acceptance are earlier open gates.
Notification backlog
Detect
Inspect local incident inbox and notification metadata. Classify queued/retry_wait, failed/uncertain, suppression and provider acceptance separately.
Recover
For known preacceptance retry_wait failure, use a case and requeue the existing notification. Existing attempt budget remains at three. Verify the notifications pause control and current recipient preference. A paused queue waits rather than being converted into an external send. Uncertain delivery has no blind requeue action; use the selected provider reconciliation contract when a real adapter is introduced.
Verify
Inspect the original notification/receipt and owner inbox. Confirm one logical notice, correct recipient, recorded local delivery and a current authorized link. Case updates themselves send no message. Leave inaccessible/expired/deleted targets without an active result link.
Storage/render/media failure and deletion backlog
Detect
Inspect Resources → Attachment cleanup and Overview deletion alerts. First distinguish storage failure from active content that has not been deleted. Only tombstoned resources with an existing cleanup record are eligible.
Recover
Restore the local storage dependency. Open the scoped cleanup case, record the reason and choose Retry deleted attachment cleanup. The domain helper removes only registered attachment objects; failure leaves pending with an incremented attempt and completion never reactivates the resource. Render failures use the owner-facing PDF retry contract; no direct final-asset publication is permitted.
Verify
Verify cleanup complete and actual object absence using the scoped synthetic drill or local storage tooling. Confirm authenticated download and late upload are denied. Speech/video keep their existing consent/expiry cleanup; preserve the independent video-deletions journal. Retained database document rows are not covered by attachment erasure.
Unauthorized access or configuration event
Detect
Inspect scoped activity for actor, reason, target, request ID and configuration version. Preserve the immutable event history.
Recover
The local database owner revokes the exact grant using `python3 platform-foundation/ops-admin.py revoke --grant GRANT_UUID --reason REASON`. A maintainer can pause new exports, starts, media admissions, checkout or local delivery in the assigned tenant. To reverse a configuration change, read the latest version and record a new reasoned change. Existing jobs drain under their original policy.
Verify
Verify the open operator session receives 403 on the next request and console data clears. Test the current control against a new admission; confirm existing work and audit remain. No raw-content inspection, impersonation or arbitrary provider/model switch is available.
Isolated database restore
Detect
Stop test/browser writers. Run `python3 database/db.py restore-check`; it refuses to overwrite a pre-existing restore database and guards the source test identity.
Recover
The helper creates an isolated database, revokes PUBLIC connection, restores and compares hashes for all included domains. It then revokes restored sessions/grants, pauses six controls and verifies the application cannot connect. No workers or outbound secrets are attached. The helper drops only the disposable database it created.
Verify
Review docs/phase-16/restore.txt. This verifies fidelity/quarantine, not live reactivation. Before a future real reopen, reconcile all post-snapshot writes and revocations, object files, independent video journal, provider receipts and financial facts; select fresh operators and meet accepted RTO/RPO. Escalate unresolved mismatches rather than starting consumers.