Kill Switch Runbook for AI Embeds
Even well-behaved assistants need emergency brakes. A kill switch runbook keeps incidents controlled and auditable.
Kill switch tiers
- Per-embed: Disable a specific placement if it misbehaves (e.g., marketing page only).
- Per-tenant: Pause all embeds for a customer (billing, security, policy).
- Global: Turn off every embed if the LLM provider or data pipeline is compromised.
Runbook steps
- Identify trigger: security incident, hallucination spike, billing issue, or regulator request.
- Decide scope: embed, tenant, or global; document reason.
- Execute: flip flag in admin-ops UI or API; log actor, timestamp, scope.
- Notify: send Google Chat alert, email tenant admins, and update status page if global.
- Mitigate: fix root cause (crawl, prompt, provider).
- Restore: re-enable embeds, notify stakeholders, and annotate analytics timeline.
UX considerations
- Show a friendly message in the widget: “Assistant temporarily disabled. Please contact support.”
- Provide admin dashboard banners explaining the status.
- For global incidents, add a public status page entry.
Testing
- Exercise kill switches monthly in staging and quarterly in production with low-risk tenants.
- Record results in the audit log (who tested, outcome, follow up).
- Ensure analytics continue logging “disabled” events for postmortems.
CrawlBot controls
CrawlBot exposes kill switches in admin-ops, logs every change, and blocks new chats while allowing existing conversations to wrap up gracefully. Adopt a similar runbook to keep AI incidents contained.***