Is SBproxy a reverse proxy, an AI gateway, or a model server?

All three, from one binary. It terminates TLS and handles HTTP/1.1, HTTP/2, WebSocket, and gRPC like nginx or Envoy. It routes LLM traffic to 200+ hosted models across 72 providers with guardrails, budgets, and MCP federation. And it serves open-weight models on your own GPUs by supervising vLLM or llama.cpp underneath. One config file drives all of it, and the AI features sit on a runtime that was already a production-grade proxy.

SBproxy is the open-source enterprise gateway for API, MCP, and AI model traffic. Outbound, it routes to 200+ hosted models across 72 providers through one OpenAI-compatible endpoint, picking models by prompt difficulty and rerouting on live cost-per-success, and serves open-weight models on your own GPUs. Inbound, it gates the crawlers, signed agents, and MCP clients that hit your APIs, with controls like Pay Per Crawl and Web Bot Auth. Both directions share one policy of virtual keys, budgets, and guardrails. Underneath sits a production-grade reverse proxy running auth, TLS, WAF, rate limiting, transforms, and caching. Single static binary, Apache License 2.0.

Can SBproxy serve models on my own GPUs?

Yes, on any GPU. A provider whose entire body is a serve: block boots to a first token: SBproxy resolves the weights, fits an engine and quantization to your card (bigger cards take FP8, smaller ones fall back to an int4 GGUF), runs vLLM or llama.cpp as a supervised subprocess, and routes to it. Run sbproxy doctor first and it reports what the host can serve. Every governance plane, meaning virtual keys, budgets, guardrails, and the usage ledger, applies to the local model unchanged, and a hosted provider placed after the local lane catches overflow under policy.

How is SBproxy different from vLLM or Ollama?

vLLM and Ollama serve a model. SBproxy stands up the environment around it: the governed OpenAI-compatible endpoint, virtual keys, budgets, guardrails on prompt and response, failover to hosted providers, and a verifiable usage ledger. It runs vLLM or llama.cpp underneath as supervised engines, so you keep the engine and gain the gateway.

How is SBproxy different from Cloudflare Pay Per Crawl?

Cloudflare runs at its own edge. SBproxy runs in your VPC, region, Kubernetes cluster, or air-gapped environment. Pricing is tier-aware, per route, per shape, and per agent rather than domain-wide. Inbound monetization, agent identity, content negotiation, and outbound AI governance live on one policy graph instead of separate Cloudflare products. Settlement supports both x402 and Stripe MPP, and you can act as your own Merchant of Record.

How is SBproxy different from other AI gateways like LiteLLM, Helicone, Portkey, or Kong AI Gateway?

Most AI gateways stop at routing calls to hosted providers. SBproxy also serves the weights: the same binary runs open models on your GPUs and applies the same keys, budgets, and guardrails to local and hosted traffic alike. It is also a real reverse proxy underneath, so auth, WAF, and rate limiting ride along instead of needing a second gateway. Helicone is excellent for LLM observability. LiteLLM is a Python library that grew a server, and sbproxy config import-litellm converts its config in an afternoon. Kong is a traditional API gateway with AI bolt-ons. None of them serve the model.

Where does SBproxy sit in my stack?

At the network hop between your apps or agents and the models they call, and between inbound AI traffic and your APIs. SDK gateways live inside one app's code, hosted edges govern from someone else's network, and agent platforms ask you to build on them; SBproxy is standalone infrastructure that deploys like nginx, works under any agent framework or platform, and fronts any hosted provider or your own GPUs. It terminates TLS with certificates it issues and renews from Let's Encrypt, so the governed endpoint can face the public internet behind virtual keys you mint and revoke at runtime.

How does SBproxy handle API keys and state across a fleet of instances?

SBproxy ships a dynamic key plane. Mint, rotate, and revoke virtual keys at runtime through the admin API with no reload, and a revoke takes effect on the next request. Keys are hashed at rest with HMAC-SHA256 and a server pepper, and upstream provider credentials are sealed with AES-256-GCM. Across a fleet of replicas, a built-in mesh keeps the keys, budgets, and per-key rate counters coherent using SWIM gossip and CRDT counters, so the cluster coordinates itself without an external Redis. LiteLLM and most AI gateways need a separate Redis to share that state across instances.

Where does SBproxy deploy?

Single binary on a server, Kubernetes operator on a fleet, or air-gapped install on a regulated network. Same policy graph in every shape. No phone-home. No SaaS edge. Bring your own Stripe, your own SSO, your own observability, and your own audit sink. Suitable for EU, India, Saudi Arabia, US public sector, defense, healthcare, and finance workloads with sovereignty requirements.

Is SBproxy open source?

SBproxy is open source under the Apache License 2.0. Free for any use, including production and commercial, with no field-of-use restriction. Read every line on GitHub. Verify every binary.

What is Pay Per Crawl and how does SBproxy support it?

Pay Per Crawl is the pattern of charging AI crawlers for access to content using HTTP 402 Payment Required. SBproxy ships the ai_crawl_control policy: AI crawlers that arrive without a valid payment token receive a 402 response with a JSON challenge. The crawler retries with the issued token and the proxy validates it through a pluggable ledger. Settlement supports x402 stablecoin and Stripe MPP. Tokens can be priced per route, per shape, and per agent.

What is Web Bot Auth and how does SBproxy verify AI agents?

Web Bot Auth is the IETF pattern for verifying signed AI crawlers and agents. SBproxy implements it through the bot_auth provider. Agents sign each request with an Ed25519 key under RFC 9421 HTTP Message Signatures and advertise their key id in the Signature-Input header. The gateway looks up the matching public key in its directory and verifies the signature. Agents that pass come through. Everything else gets 401.

Does SBproxy support MCP?

Yes. SBproxy ships an MCP gateway that federates internal tools under one identity model. Authentication supports OAuth2 with DPoP, PKCE, and DCR. Federated servers can be wrapped from any OpenAPI service, with per-server prefix and RBAC. The same policy graph that governs API and AI traffic also governs MCP tool invocations.

What does SBproxy replace?

A hosted model router (OpenRouter), an LLM proxy (LiteLLM, Portkey, Helicone), the hand-rolled shim in front of vLLM or Ollama, a traditional API gateway (Kong, nginx, Envoy), a guardrails service, a bot management product, and a payment rail for crawlers (TollBit, Skyfire). One runtime instead of six vendors stitched together, one policy plane instead of six control planes, one audit log instead of six disconnected feeds.

How many AI providers does SBproxy support?

SBproxy reaches 200+ models behind one OpenAI-compatible API through 72 native providers, including OpenAI, Anthropic, Google Gemini, AWS Bedrock, Azure OpenAI, Azure AI Foundry, Groq, Mistral, Cohere, Together AI, Fireworks, xAI Grok, OpenRouter, Cerebras, NVIDIA NIM, Databricks, Snowflake Cortex, IBM watsonx, and Oracle GenAI, plus the models it serves itself on your GPUs via vLLM or llama.cpp and external self-hosted runtimes like Ollama and TGI. Routing supports fallback chains, weighted, round-robin, latency-based, cost-optimized, and outcome-aware strategies.

Is SBproxy safe to run in front of production traffic?

SBproxy ships as a single static Rust binary. Releases are cut from tagged commits in a public repository on GitHub. Dependencies are pinned and checksummed in Cargo.lock. The runtime does not fetch code from a public package registry when it starts. The deployment shape matches nginx, HAProxy, and Envoy. Hot reload is supported without dropping connections.

Your AI gateway should be a real proxy

Your AI gateway is probably a Python SDK. That works on a laptop. It does not survive contact with real traffic.

This is not a dig at the people who built those libraries. They solved a real problem: developers wanted one way to talk to every model, and they wanted it yesterday. A Python wrapper around OpenAI, Anthropic, Bedrock, and friends was the fastest way to do that. Ship the library, done.

Then the library grew a server. Then the server grew a UI. Then the UI grew a billing plan. Somewhere along the way, the thing stopped being an SDK and started calling itself a "gateway." Nothing underneath changed. It is still a Python process wrapping HTTP clients.

That is fine for weekend projects. It is not fine for the gateway that sits in front of your production traffic.

What a gateway actually has to do

A gateway is the thing every request hits first. That means:

Terminate TLS, including ALPN negotiation for HTTP/2.
Keep connection pools warm to upstreams, for tens of thousands of concurrent streams.
Survive slow clients, slow upstreams, half-open TCP, broken WebSockets, and HTTP smuggling attempts.
Enforce auth before a single byte of the body reaches your application code.
Apply rate limits that understand request shape, not just request count.
Apply WAF rules that understand request shape, not just URL suffixes.
Transform headers, bodies, and streams without buffering everything into memory.
Emit structured logs, metrics, and traces that look the same as the rest of your stack.
Restart cleanly, reload config without dropping connections, and run in every deployment shape you already own.

None of this is specific to AI. All of it is what people spent twenty years teaching nginx, HAProxy, and Envoy to do. It is the reason those projects exist.

The AI-SDK-that-grew-a-server crowd has to rebuild every one of those pieces from scratch, in Python, usually while also shipping features. Some of them do a reasonable job. None of them do it the way a production operator expects.

What AI traffic actually is

AI traffic is HTTP traffic. A POST to /v1/chat/completions is a POST. The body is JSON. The response is a stream of server-sent events or chunked JSON. The auth is a bearer token. The upstream is at api.openai.com or a model endpoint inside your VPC.

The things that make it feel different are surface features, not architectural ones:

Responses stream for longer. Tens of seconds, sometimes minutes. A proxy that handles WebSockets handles this already.
Costs are measured in tokens, not requests. A proxy that can count bytes can count tokens.
You want provider fallbacks when one is down. A proxy that does health-checked upstream pools does this already.
You want to cache. A proxy that caches GETs can be taught to cache prompt-keyed responses.
You want prompt-level policy. That is a request transform and a response transform.

The honest read is that everything new about AI gateways fits inside the extension points a real proxy already has. You do not need a different runtime. You need the same runtime, with a handful of features added.

Why this matters in practice

The teams we talk to all hit the same wall. They stand up an AI gateway for their first use case. It works. Then one of two things happens:

The first is they put it in front of traffic that is not AI. Maybe an internal API that also wants auth and rate limits. The AI gateway cannot do that well, because it was never a gateway. So they keep the existing API gateway and put the AI gateway behind it. Now they run two gateways.

The second is that the AI use case grows. Streaming to a browser. WebSocket tunnels for an agent. MCP traffic. A quiet request to add a WAF rule because someone pasted a jailbreak prompt into support. The AI gateway starts to wobble. The ops team finds themselves patching a Python service to do what their existing reverse proxy does out of the box.

Both paths end in the same place. Two control planes. Two sets of configs. Two sets of dashboards. Two on-call runbooks. All of it to solve a problem that should have been one box.

What SBproxy is

SBproxy is a single static binary. A reverse proxy that happens to speak LLM, MCP, and agent traffic.

It started life as a production reverse proxy. TLS, HTTP/1.1, HTTP/2, WebSockets. Auth, WAF, rate limits, IP filters, transforms, caching, multi-tenant routing. The kind of proxy you would put in front of any HTTP service.

Then we taught it AI. Provider fallbacks across 200-plus models. Token accounting. Semantic caching. Budget limits. Prompt-level policy as a first-class rule, not a middleware afterthought. MCP federation, so one proxy fronts N MCP servers.

Same runtime. Same YAML. One config file describes an auth policy in front of your internal API, a rate limit in front of your checkout endpoint, and a provider fallback chain in front of your chat completions route. One process reads that file. One log stream reports what happened.

This is not a clever architecture. It is the obvious one. The only reason nobody did it first is that the AI gateway market started inside application code, and the API gateway market started inside infrastructure, and nobody had standing to merge them.

What to do with this

If you are running a Python AI gateway behind an API gateway, you are running two things to get one job done. That is a stitch, not a stack. Rip one of them out.

If you are picking one now, pick the one that was built to sit in front of traffic. The AI parts are easy to add to a proxy. The proxy parts are very hard to add to a Python SDK.

If you want to see what it looks like:

curl -fsSL https://sbproxy.dev/install.sh | sh

Point it at a YAML with an origin. Send it traffic. Keep the parts that work.

We did not set out to build an AI gateway. We set out to stop running two of them.