Content for agents

Why content negotiation matters for agents

AI crawlers and agents pay tokens for HTML they did not need. A typical article wrapped in navigation, ads, and footer markup costs 5x to 10x the tokens of the same article in Markdown. Content negotiation at the gateway lets you serve the cheaper shape on demand without rewriting the upstream.

HTML to Markdown on demand

Add the html_to_markdown transform with an on_accept trigger. When a client sends Accept: text/markdown, the gateway converts the upstream HTML response to Markdown before it leaves the perimeter.

origins:
  "blog.example.com":
    action: { type: proxy, url: https://upstream.example }
    transforms:
      - type: html_to_markdown
        on_accept: "text/markdown"
        strip_navigation: true
        preserve_code_blocks: true

The transform runs after authentication and policy, so the agent identity check and Pay Per Crawl ledger see the original request. The conversion is streaming-safe and respects upstream Cache-Control directives.

JSON projection and schema validation

For data APIs, project the upstream payload to the fields a partner contract requires, and validate inbound bodies against an OpenAPI schema:

origins:
  "api.example.com":
    transforms:
      - type: json_projection
        keep:
          - id
          - title
          - published_at
          - author.name
    policies:
      - type: openapi_validation
        spec: /etc/sbproxy/openapi.yaml
        on_violation: reject_4xx

Live OpenAPI emission

The gateway publishes an OpenAPI 3.0 document derived from its compiled config. Buyers consume the emitted spec with Postman, Stainless, ReadMe, or any standard tooling.

# Per-host (public, opt-in)
curl -s -H 'Host: api.example.com' \
  http://127.0.0.1:8080/.well-known/openapi.json

# Admin endpoint (all hosts, basic auth)
curl -s -u admin:changeme http://127.0.0.1:9090/api/openapi.json | jq

What gets emitted:

• Origin hostname becomes servers[].url
• Forward rule matchers become paths keys
• Auth config becomes securitySchemes and security
• Response cache statuses become responses keys
• CORS becomes the x-sbproxy-cors extension

Coverage is bounded by what the gateway config knows. Upstream request and response bodies are not described here unless declared explicitly.

Content-Signal headers

Publish a real licensing posture that crawlers can read before they hit the request:

response_modifiers:
  - type: add_headers
    headers:
      Content-Signal: "ai-train=deny, search=allow, ai-input=charge"
      Link: '<https://example.com/llms.txt>; rel="ai-policy"'

See also

• Pay Per Crawl
• Response transforms
• Forward rules