Weekly AI Assistant Health Checklist

AI assistants need predictable maintenance. This weekly routine keeps crawl freshness, retrieval accuracy, billing, and access controls in check.

1. Crawl and freshness

• Check the latest sitemap runs per tenant; verify run durations, page counts, and error logs.
• Ensure stale_page_ratio <5 percent; if not, schedule catch-up crawls and notify the customer.
• Confirm IndexNow queues are empty and headless renderer jobs completed within budget.

2. Retrieval and fallback

• Review fallback_reason dashboards (low_score, no_context, provider_error, timeout, context_overflow).
• Compare containment week over week; flag embeds below target.
• Sample transcripts where fallback_reason spikes to confirm refusal copy still meets brand tone.

3. Billing and quotas

• List tenants in grace_state; ensure finance has reached out.
• Confirm usage counters reset where expected; accuracy guardrails should never disable paying tenants by accident.
• Schedule kill switch tests quarterly to ensure disablement flows still work.

4. Security and access

• Audit SSO login logs for anomalies; verify SCIM sync or nightly disablement ran.
• Check impersonation logs inside admin-ops tools; ensure approvals are captured.
• Validate kill switch statuses and environment toggles (staging/prod) remain correct.

5. Feedback triage

• Filter negative feedback and “unanswered” reports; assign to content or prompt owners.
• Reply to demo submissions that left high-value questions unanswered.
• Export flagged transcripts for training or documentation updates.

Automation tips

• Use scheduled Looker/Metabase dashboards to email this checklist Monday mornings.
• Pipe high-severity alerts (billing grace, stale crawl) into Google Chat so they are already acknowledged before the weekly review.
• Track checklist completion in a lightweight ticket or Notion doc to maintain accountability.

CrawlBot tooling

CrawlBot’s admin dashboard centralizes crawl manifests, fallback charts, billing states, SSO logs, and feedback queues. Mirror this structure even if you build in-house; weekly discipline keeps AI assistants trustworthy.***