Local Inference & Semantic Cache

On-box embeddings and prompt-injection classification with no per-call cost and no prompt egress

Overview

SBproxy can run two AI-gateway features on local models instead of paid APIs: the embedding semantic cache, which vectorizes prompts to serve near-duplicate requests from cache, and prompt-injection v2, which classifies prompts for injection attempts. Both run on a self-contained inference engine with no Python and no native ONNX Runtime to install.

What makes it different

  • No per-call cost. Embedding and classification happen on-box, not against a metered API. A semantic cache hit avoids the upstream completion entirely.
  • No prompt egress. The prompt is vectorized locally and never leaves your network. This is the difference between a semantic cache you can run in a regulated VPC and one you cannot.
  • Air-gap ready. Models are operator-supplied files you download once and pin by SHA-256. Nothing phones home.
  • Two deployment modes. Run the model in a co-located sidecar (recommended, so a bad model can only restart the sidecar) or in-process for a true single binary.
  • It pays for itself, visibly. The cache reports tokens and dollars saved against the same cost table used for spend, so a dashboard shows spend and savings side by side and they reconcile.

Models

The default embedding model is all-MiniLM-L6-v2 (384-dimensional, Apache-2.0, ~90 MB). The default classifier is protectai/deberta-v3-base-prompt-injection-v2 (Apache-2.0, ~70 MB int8). Both are small enough to bake into a container image or copy onto an air-gapped host. Download them once:

mkdir -p /var/lib/sbproxy/models/minilm

curl -fSL -o /var/lib/sbproxy/models/minilm/model.onnx \
  https://huggingface.co/sentence-transformers/all-MiniLM-L6-v2/resolve/main/onnx/model.onnx
curl -fSL -o /var/lib/sbproxy/models/minilm/tokenizer.json \
  https://huggingface.co/sentence-transformers/all-MiniLM-L6-v2/resolve/main/tokenizer.json

Run the sidecar

The sidecar serves both embed and classify over gRPC (TCP or a Unix domain socket). The proxy connects lazily, so the sidecar does not have to be up before the proxy starts:

sbproxy-classifier-sidecar \
  --listen 127.0.0.1:9440 \
  --embed-model all-MiniLM-L6-v2=/var/lib/sbproxy/models/minilm/model.onnx:/var/lib/sbproxy/models/minilm/tokenizer.json

For a co-located deployment, use --listen-uds /run/sbproxy/classifier.sock to skip the loopback TCP round trip.

Local semantic cache

Point the semantic cache at the sidecar with source: sidecar. On a miss the proxy vectorizes the prompt, scans the cache, and replays the closest cached response when cosine similarity meets threshold. If the sidecar is unreachable the lookup is treated as a miss and the request proceeds uncached, so the cache never wedges a request.

origins:
  ai.example.com:
    action:
      type: ai_proxy
      providers:
        - name: openai
          api_key: ${OPENAI_API_KEY}
          models: [gpt-4o]
      semantic_cache:
        enabled: true
        threshold: 0.85          # cosine similarity for a near-duplicate hit
        ttl_secs: 3600
        max_entries: 1024
        source: sidecar
        sidecar:
          endpoint: http://127.0.0.1:9440
          model: all-MiniLM-L6-v2
          timeout_ms: 500

The semantic cache is configured on each AI origin under action.semantic_cache. The default source is provider, which calls an AI provider's /v1/embeddings API. Existing configs are unchanged.

First-class prompt-injection classification

Select the sidecar detector in the prompt_injection_v2 policy. The default detector is a zero-dependency regex pass; choosing sidecar runs the classifier:

origins:
  ai.example.com:
    action:
      type: ai_proxy
      providers:
        - name: openai
          api_key: ${OPENAI_API_KEY}
          models: [gpt-4o]
    policies:
      - type: prompt_injection_v2
        threshold: 0.8
        action: block
        detector: sidecar
        detector_config:
          endpoint: http://127.0.0.1:9440
          model: prompt-injection
          injection_label: INJECTION
          timeout_ms: 250
          fail_closed: false       # a sidecar outage degrades to "clean" (allow)

In-process, single binary

For a single binary, run either feature in-process. This loads a model into the proxy's address space, so it is gated behind explicit config and a max_model_bytes guard. Prefer the sidecar for isolation.

origins:
  ai.example.com:
    action:
      type: ai_proxy
      providers:
        - name: openai
          api_key: ${OPENAI_API_KEY}
          models: [gpt-4o]
      semantic_cache:
        enabled: true
        threshold: 0.85
        source: inprocess
        inprocess:
          model: all-MiniLM-L6-v2
          model_path: /var/lib/sbproxy/models/minilm/model.onnx
          tokenizer_path: /var/lib/sbproxy/models/minilm/tokenizer.json
          max_model_bytes: 209715200   # 200 MB guard

Remote OpenAI-compatible embeddings

To vectorize through an OpenAI-compatible /v1/embeddings endpoint that is not one of the origin's chat providers, set source: openai. Point it at another sbproxy that fronts an embedding model, at OpenRouter, or at a hosted provider, each with its own URL and key. This source is not on-box, but it keeps the embedding endpoint decoupled from the chat providers.

origins:
  ai.example.com:
    action:
      type: ai_proxy
      providers:
        - name: openai
          api_key: ${OPENAI_API_KEY}
          models: [gpt-4o]
      semantic_cache:
        enabled: true
        threshold: 0.85
        source: openai
        openai:
          base_url: https://openrouter.ai/api/v1   # or http://sbproxy.internal/v1
          api_key: ${EMBEDDING_API_KEY}
          model: text-embedding-3-small
          timeout_ms: 2000

Auth defaults to Authorization: Bearer. For endpoints that expect a different header (Azure api-key, an x-api-key gateway), set auth_header and clear auth_prefix; endpoints that need extra headers, such as OpenRouter's HTTP-Referer and X-Title, take a headers list. On any embedding error the lookup degrades to an uncached upstream call.

Metrics

Local inference and the semantic cache emit metrics attributed per tenant:

  • sbproxy_semantic_cache_results_total{tenant,origin,source,result} - hit / miss / error rate by embedding source.
  • sbproxy_inference_requests_total{kind,backend,model,result} - embed and classify call counts.
  • sbproxy_inference_duration_seconds{kind,backend,model} - embed and classify latency.
  • sbproxy_ai_tokens_saved_total{tenant,origin,model,kind} - tokens a cache hit avoided.
  • sbproxy_ai_cost_saved_micros_total{tenant,origin,model} - micro-USD a cache hit avoided.

If near-duplicates score just under your threshold, watch sbproxy_ai_semantic_cache_similarity_bucket and tune threshold from the observed distribution.

← PreviousCachingNext →Cedar Authorization