StreamGuard

Overview

StreamGuard is a Go reverse proxy for streaming LLM workloads with one promise: users never see a broken stream, and you never pay for a failed generation. It detects mid-stream upstream failures, fails over deterministically across configured providers (e.g. OpenAI → Anthropic), and exposes every recovery decision through an explicit SSE wire protocol clients can render safely.

Mid-stream failover

Detects dead_socket, silent_hang, and malformed upstream failures mid-stream, replays the full original request against the next provider, and preserves stream continuity — emitting gateway_failover and gateway_regenerating events so clients can render failover and truncation states without corruption.

Billing integrity

Separates tokens_delivered from tokens_billed so failed attempts stay visible without being billed, backed by a reconciliation job that applies drift and a calibration layer that collects samples and percentiles from a local tokenizer registry.

Admission control

Enforces per-key budgets with atomic token reservation, allowlists, and admission-time rate limiting on the live path, with per-provider circuit breakers guarding each upstream.

Deterministic testing

A mock-upstream harness produces deterministic SSE, with split-frame validation and build-tag-gated chaos execution — plus CI and a Go Report Card gate.

Built with

• Go
• SSE
• Reverse Proxy
• OpenAI
• Anthropic