Ship AI features
without duct tape.
Your LLM features are in production. The duct tape holding them there is not. Replace a router, a guardrails vendor, a cache, and a spend spreadsheet with one runtime that does all four, better.
# AI gateway: routing, fallbacks, guardrails, budget
origins:
"ai.example.com":
action:
type: ai_proxy
routing: fallback_chain
providers:
- name: anthropic
api_key: ${ANTHROPIC_API_KEY}
priority: 1
default_model: claude-3-5-sonnet-latest
- name: openai
api_key: ${OPENAI_API_KEY}
priority: 2
default_model: gpt-4o
guardrails:
input:
- {type: pii, action: redact}
- {type: injection, action: block}
budget:
on_exceed: downgrade
limits:
- {scope: workspace, max_tokens: 1000000, period: daily}The AI stack is held together with duct tape.
Every team shipping a real AI product ends up with the same five vendors. A router to juggle providers. A guardrails service for PII. A cache to cut spend. A WAF in front. A bill tracker that runs at the end of the month.
Five products. Five control planes. Five places to look when something breaks. SBproxy collapses the stack into one runtime, one configuration, one trace.
When you put SBproxy in front of your stack.
The work below moves out of the engineering backlog and into a config change.
- Replace a router, a guardrails vendor, a cache, and a spend tracker. With one runtime.
- Cap AI spend per team, customer, or virtual key. Automatically.
- Redact PII and stop prompt injection on every call. In the same runtime.
- See who called what, at what cost, before finance asks.
What’s shipping in the data plane.
For engineers in the buying committee. The capabilities behind the outcomes above.
- 01Switch providers without rewriting codeRoute across OpenAI, Anthropic, Google, Bedrock, and the rest. Fallback when a provider stumbles. Pick the cheapest model that still works.
- 02Guardrails on every callPII redaction, secret scanning, prompt injection defense. Runs in the stream, before the response reaches the user.
- 03Cut tokens without losing freshnessSemantic cache answers the same question once. Tunable similarity, per-route TTLs.
- 04Set the spend rules. We enforce them.Cap budgets per team, customer, or key. The gateway downgrades or stops the call when the cap hits. No spreadsheets.
- 05Stream nativelySSE, WebSocket, and gRPC with backpressure. Guardrails run on streamed deltas. Cancellation works the way users expect.
- 06Know what each request costEvery call tagged, counted, and costed. Stream to OpenTelemetry or your warehouse. Answer cost questions before they get asked.
- You ship LLM features and your runtime budget just became a line item.
- You are running a router in front of a WAF in front of a guardrails vendor.
- You need answers to who called what, at what cost, before the quarter closes.
Want to talk
before you install?
The gateway is on GitHub under Apache 2.0 and you can install it right now, so nothing here is gated behind a sales call. Use this form for deployment questions, migration scoping, or anything you would rather not file as a public issue.