When do I need an LLM gateway versus calling model provider APIs directly?
TL;DR
Call provider SDKs directly in single-model prototypes with one API key. Introduce an LLM gateway as soon as you need automatic provider failover, per-tenant spend caps, unified OpenTelemetry traces, or multi-model routing.
Direct provider SDK calls keep early prototypes simple. Once an application serves real users or runs background agent loops, direct API calls scatter retry logic, billing telemetry, and key rotation across every service.
What an LLM gateway centralizes
An LLM gateway sits between your application (or coding harness) and upstream providers such as Anthropic, Google Gemini, OpenAI, and local vLLM endpoints:
- 1.Resilient failover — Automatically retries 429/529 provider rate limits against a warm fallback region or secondary model.
- 2.Hard budget enforcement — Enforces daily and per-workflow USD caps before runaway agent loops exhaust monthly credits.
- 3.Unified observability — Emits standardized OpenTelemetry GenAI spans (latency, prompt/completion tokens, cache hits, cost) to Langfuse, Grafana, or Datadog.
- 4.Credential isolation — Issues scoped virtual keys to developers and agents while keeping master provider keys in one vault.
Practical threshold
Stay on direct SDK calls (or Vercel AI SDK provider packages) when running one model under $100/month. Move behind LiteLLM, Cloudflare AI Gateway, Kong AI Gateway, or Portkey when adding a second model family, onboarding multiple engineers, or running unattended agent schedules. See the full LLM gateway architecture guide.