Overview
The latency- and cost-aware strategies decide from live signals or static catalog price. None consume the realized outcome of a request: whether it succeeded, was refused, what it cost, and how long it took. The outcome_aware strategy closes that loop. Every completed call feeds a per-provider rolling estimate, and selection scores candidates by realized cost-per-success, demoting a provider whose refusal or error rate is rising.
Configuration
action:
type: ai_proxy
routing: outcome_aware
providers:
- name: openai-primary
provider_type: openai
api_key: ${OPENAI_API_KEY}
default_model: gpt-4o-mini
models: [gpt-4o-mini]
- name: openai-secondary
provider_type: openai
api_key: ${OPENAI_API_KEY}
default_model: gpt-4o-mini
models: [gpt-4o-mini]How it scores
For each provider the store keeps an EWMA of realized cost, success rate, refusal rate, and latency. The score is (ewma_cost / success_rate) * (1 + refusal_rate); lower is better, and a provider that never succeeds scores infinity. Selection routes to the lowest-scoring eligible provider.
Warm-up
While any candidate has fewer than a handful of samples, the strategy round-robins so every provider earns an estimate first. A fresh deployment therefore behaves exactly like round-robin until it has data, which makes the strategy safe to enable with no other change. A provider that starts refusing sees its score rise and traffic shift to a healthier alternative within a bounded window.