Run open-weight models
on your own GPUs.
Your prompts and weights should not have to leave your network to get a completion. SBproxy resolves the model, fits the engine and quantization to your hardware, and serves an OpenAI-compatible endpoint behind the same keys, budgets, and guardrails as everything else.
# One serve block, from a bare GPU to a governed endpoint
origins:
"ai.internal":
action:
type: ai_proxy
routing:
strategy: fallback_chain
providers:
- name: local # your GPU serves this
priority: 1
serve:
models:
- model: qwen3-14b
- name: openai # overflow, under policy
priority: 2
guardrails:
input:
- { type: injection, action: block }
budget:
limits:
- { scope: workspace, max_tokens: 1_000_000, period: daily }The problem we hear.
You have GPUs, or cloud credits for a few, and models you would rather not send to a hosted API. Standing up vLLM by hand means matching the engine to the card, picking a quantization that fits VRAM, wiring an OpenAI-compatible shim, and then bolting on keys and budgets after the fact.
You want one command from a bare box to a served model, with the governance already attached, and a fallback to a hosted provider for when the GPU runs hot.
When you put SBproxy in front of your stack.
The work below moves out of the engineering backlog and into a config change.
- Go from a bare GPU to a first token in minutes, not an afternoon of engine tuning.
- Keep weights, prompts, and embeddings on your own hardware. Nothing phones home.
- Point a coding assistant at a local model with a base-URL change.
- Spill overflow to a hosted provider under policy when the GPU is saturated.
What’s shipping in the data plane.
For engineers in the buying committee. The capabilities behind the outcomes above.
- 01Weights to first tokenA provider whose entire body is a serve block boots to a first token. SBproxy resolves the weights, picks the engine and quant, and supervises vLLM or llama.cpp underneath.
- 02Fit to your hardwareEngine and quantization are fitted to whatever card you have. Bigger cards take FP8, smaller ones fall back to an int4 GGUF. sbproxy doctor reports what a host can serve before anything downloads.
- 03Governed like everything elseVirtual keys, budgets, and guardrails apply to the local model unchanged. The same policy covers your GPU and your hosted providers.
- 04Local first, cloud spilloverPut a hosted provider after the local lane and overflow spills to the cloud under policy. The usage ledger reports what the GPU saved you.
- 05Air-gapped and sovereignWeights, embeddings, and guardrail classifiers all run on-box. Suitable for networks where nothing may leave the perimeter.
- You have GPUs, or cloud credits, and models you would rather not send to a hosted API.
- You want a coding assistant or app running against hardware you control.
- You operate in a network where weights and prompts cannot leave the perimeter.
Want to talk
before you install?
The gateway is on GitHub under Apache 2.0 and you can install it right now, so nothing here is gated behind a sales call. Use this form for deployment questions, migration scoping, or anything you would rather not file as a public issue.