Caching

Overview

SBproxy includes a response cache that sits in front of the action handler. Cache hits skip action execution entirely. The cache is backed by a pluggable key-value store, so the same configuration shape works for in-memory, on-disk, and distributed deployments. An optional cold tier lets you keep large or infrequently-accessed entries on slower, cheaper storage without evicting them from the working set.

Capabilities

• Response cache - Full response caching with stale-while-revalidate, conditional revalidation, and tag-based invalidation
• Pluggable backends - Choose memory, file, or memcached/redis for the response cache without changing your origin configs
• Cold tier (secondary cache) - Optional second-tier store for entries evicted from the hot cache. Hits in the cold tier are promoted back, misses fall through to the action
• Tag-based invalidation - Tag entries on write and invalidate by tag, key, or pattern
• Stale-while-revalidate - Serve stale content immediately while refreshing in the background
• Conditional caching - Per-method, per-status-code, and per-rule cache eligibility

Example

Enable response caching with stale-while-revalidate:

proxy:
  http_bind_port: 8080

origins:
  "cdn.example.com":
    action:
      type: proxy
      url: https://test.sbproxy.dev
    response_cache:
      enabled: true
      ttl: 5m
      stale_while_revalidate: 30s
      methods:
        - GET
      status_codes:
        - 200
      tags:
        - api-data

Test it:

curl -v -H "Host: cdn.example.com" http://localhost:8080/ 2>&1 | grep -i "x-cache|content-type|x-sb"

Cold tier

The cold tier is an optional secondary cache that backs the hot cache. When an entry is evicted from the hot cache (because of size limits or LRU pressure), it lands in the cold tier instead of being thrown away. Subsequent requests check the hot cache first; on a miss, they look in the cold tier and promote the entry back. This is useful for long-tail content that does not earn space in the hot working set but is still cheaper to serve from the cold tier than to refetch from the upstream.

response_cache:
  enabled: true
  ttl: 1h
  cold_tier:
    enabled: true
    backend: file
    path: /var/cache/sbproxy
    max_size: 10GB

Use the cold tier for large media, infrequently-accessed API responses, or when your upstream charges per request. Disable it for low-latency endpoints where the extra lookup cost is not worth the hit rate gain.

Semantic cache for AI traffic

The response cache above keys on the exact request. For LLM traffic, the AI gateway also offers a semantic cache that serves near-duplicate prompts: it vectorizes the prompt and replays the closest cached completion when cosine similarity clears a threshold. Prompts can be vectorized by a provider's embeddings API or, to keep them on your network and skip the per-call cost, by a local embedder running in a sidecar or in-process. See Local Inference & Semantic Cache.