HCAA Reporting · planned and built Oct 30, 2026 (migration 408). The list of every automatic fix is on Admin → Alerts that fix themselves. Figures below are from live Supabase on Oct 30 (last 30 days).
Most of the 59 emails came from test runs (now filtered by migration 407) and 17 "still broken" repeats from SendJim, which you canceled on Oct 20.
| Alert | Watches | Tells you | Fixes itself today? | Plan |
|---|---|---|---|---|
| Edge-call alarm (every 10 min) | Scheduled jobs and buttons that fail, plus 10 "must have run by" checks (Twist posts, AI Voice email…) | Email; repeats every 4 h; all-clear | Partly Retries outages and timeouts at 1/3/10 min, re-runs missed jobs, ignores "reply lost, data OK" | Layer 1 + Layer 2 |
| Sync health (6:30 AM daily) | 32 data sources: failed, stale or bad rows | Email when amber or red | No | Re-run stale syncs at 6:00 before the email; Layer 2 if still red |
| Jobber schedule watchdog (mornings) | Visit data older than 12 h | Nothing (silent) | Yes Re-runs the schedule sync | Log its fixes in the new list |
| Security monitor (daily) | CM Locator database permissions, users, MFA | Email on any finding | No | Layer 2 diagnoses and reports only. Security changes always wait for you |
| QuickBooks token lock | One token refresh at a time | Nothing | Yes 203 refreshes, 0 races in 30 days | No change |
| Helcim ACH void / returned (hourly) | Customer payments voided or returned | Email per payment | Business notice | No change (not a system failure) |
| Field app feedback | Craftsman feedback | Email to you and Max | Business notice | No change |
Also business notices, unchanged: compliance reminders, booked-without-assessment, AI Voice daily, bank charges, EMC "ready to post". The Outlook email syncs stay as Cowork tasks, as already settled.
| Gap | Last 30 days | Proposed |
|---|---|---|
| Jobber webhook token refresh fails, and nothing alerts on it | 75 errors (68 × 503, 7 × 401) | Retry 503s in code; add to sync health; 401 → Layer 2 |
| Network errors and "Throttled" from Jobber are classed as our own errors, so they are never retried | 3 | Treat as temporary and retry with backoff |
| SendJim nightly fetch still scheduled after the Oct 20 cancel | 17 repeat emails | Turn off its schedule and "must have run" check your OK |
| eLocal and Project Quote GitHub workflows still scheduled (retired Sep 3) | not checked | Turn off their schedules your OK |
| Security monitor isn't in sync health, so if it stops running nobody hears | — | Add it |
Leftover Oct 3 test row (zz-retry-test) | — | Delete |
How it starts, fully in the cloud: one new scheduled task, "HCAA alert fixer", with no schedule of its own. It has an API trigger: Anthropic's routine API (POST /v1/claude_code/routines/{id}/fire) starts a session and passes the alert text. A small Supabase function calls it, with the trigger's token kept in the Supabase vault. This uses your Claude plan, not API credits.
When it fires: a scheduled job that is still failing after Layer 1 has finished, or one that failed in a way code shouldn't guess at (a vendor page changed, 401 / login, bad data, a code error). Test and one-off calls never fire it.
What the session may do (your Oct 30 rules):
Last 30 days, this would have started about 3–8 sessions: SendJim export failing (session expired), Valpak returning a web page instead of data, the EMC split not adding up, and the Jobber webhook 401s. This is an estimate from those failure counts.
| Rule | Setting |
|---|---|
| Same failure (job + error type) | At most 1 session per 24 h, 2 in total; then "Needs you" |
| Sessions per day, all failures | 5 confirm |
| Start request | Logged before sending and never re-sent. The API has no duplicate protection, and every resend starts another session. |
| Off switch | One setting turns Layer 2 off; Layer 1 keeps running |
| Anthropic's own limits | 30 starts/hour per task, 100/hour per account. We stay far below. |
The fix list on the Reporting site: Admin → Sync gains Automatic fixes, showing when, which alert, Layer 1 or 2, what was done, the result, a review link, and whether it's waiting for you.
cron:, dispatch: and button: callers and expected jobs alarm.| Question | Decision |
|---|---|
| Edge function code fixes | Apply, then tell. Wait for Tim only when payroll, QuickBooks or customers are involved. |
| Claude sessions per day | 5 |
| Trigger token | Claude creates it on claude.ai through Chrome and saves it straight into the Supabase vault. |
| Cleanup | SendJim fetch and its check turned off (canceled Oct 20); Oct 3 test row deleted. eLocal and Project Quote workflows left as they are. |
| Sync Status table | Full browser width, so it doesn't need a side scroll. |
Files: migrations/408_self_healing_alerts.sql (rollback: ROLLBACK_408_self_healing_alerts.sql), supabase/functions/edge-call-alarm, sync-health-alert, auto-fix-notify, jobber-request-webhook. Original plan: claude_reports/Alerts_Self_Healing_Plan_2026-10-30.html.