Part 7 of 12 · 1 min read
How to build it so it holds up
- Put timeouts on every call and a deadline on every run; keep inner timeouts shorter than outer ones.
- Retry only transient errors, at one layer, with backoff, jitter and a budget; honour
Retry-After; only retry idempotent or keyed operations. - Use a circuit breaker per dependency (per provider, per tool) with a probe; alert when it opens.
- Define a fallback chain and test it: secondary provider or model, cached answer, degraded tools, human handoff; log which path served each request.
- Check fallback permissions (data residency, regulation, quality floor) before using them (Level 10 module 10).
- Rate-limit on the client with a shared token bucket; queue background work, shed interactive work.
- Isolate with bulkheads: per-tool and per-tenant pools or concurrency limits (module 7).
- Degrade honestly: tell users when an answer is reduced; do not silently return stale or weaker output for critical tasks.
- Run failure drills: inject outages, slowness and 429s in staging and watch the dashboards (module 11).
- Alert on symptoms (error rate, p95, breaker state, fallback rate) not just causes.