LLM-Aware Resilience

Overview

Status-code retries treat every 5xx the same and ignore the LLM-specific failure modes a provider signals in the response: a context-window overflow, a content-policy refusal, a rate limit. LLM-aware resilience classifies each upstream failure into a typed cause and lets you set retry counts per error class, so a transient failure is retried while a request that would only fail again is sent to a fallback instead. It is opt-in; without a retry_policy the default status-code retry set is unchanged.

Failure classification

Each failure is classified from the status and, for the body-refined classes, the response body: timeout (408/504), rate_limit (429), server_error (5xx), context_window_exceeded and content_policy (from the message, even on a 200), auth (401/403), and bad_request. Timeouts, rate limits, and server errors are retryable by default; the rest are not. Each cause maps to its own fallback list, separating the context-window and content-policy lists from the general one.

Configuration

action:
  type: ai_proxy
  routing: fallback_chain
  resilience:
    retry_policy:
      rate_limit: 3      # retry a 429 up to 3 times
      server_error: 2    # retry a 5xx up to 2
      content_policy: 0  # never retry a refusal in place
      bad_request: 0

During failover the loop retries on the default status set or when the classified cause clears the policy. A class with an explicit count caps its retries; a class with no entry uses its default retryability. The overall max_attempts still bounds the total.