Standardize API models under a single interface, configure automatic fallbacks, and save up to 80%+ on API costs using vector similarity-based prompt caching.
When you run AI agentic workflows in production, calling provider endpoints directly is a design anti-pattern. If Anthropic experiences rate limits, or OpenAI encounters latency spikes, your agent loops crash. By wrapping your model calls in a combined LiteLLM Proxy and Portkey Gateway, you get bulletproof failovers, unified interfaces, and massive cost savings through semantic caching.
A robust production AI gateway uses LiteLLM to unify upstream providers and Portkey to control cache policy. Group your API endpoints into virtual models with failover groups in a LiteLLM config.yaml. Then, route outbound requests through Portkey to check a semantic (vector similarity-based) cache at 0.95+ cosine threshold. Identical or closely rephrased queries return instantly, saving up to 80%+ on your total token bills.
Hardcoding LLM calls directly to provider APIs (like calling Anthropic's SDK directly) creates single points of failure. As agentic applications scale, three main bottlenecks emerge that require a dedicated proxy layer:
If your autonomous code agent loops hit Anthropic's Tier-1 limits (or OpenAI's requests/min cap), the script throws a 429 error and dies. Resolving this requires immediate routing to fallback models.
Agents often run repetitive check loops, executing the exact same prompts (or minor variations of them) against codebases or databases. Standard exact-string caching misses slight rephrasings, wasting thousands of dollars.
If OpenAI experiences global API degradation, switching to Claude requires changing codebase logic, redeploying code, and refactoring API response parsing. An abstraction layer lets you redirect traffic instantly.
The production gateway splits responsibilities between two layers: LiteLLM Proxy (acting as the provider aggregator) and Portkey AI (acting as the control plane). Here is the flow of an inbound prompt request:
1. Client Call: The client sends a request to the Portkey Endpoint containing a unified OpenAI-compatible payload.
2. Semantic Cache Check: Portkey generates an embedding of the query and checks if a semantically similar prompt exists in the cache database. If a match is found (> 0.95 cosine similarity), the cached response returns in <150ms.
3. LiteLLM Routing: On a cache miss, the request goes to the LiteLLM Proxy. LiteLLM routes the request to the active primary model (e.g., Claude 3.5 Sonnet).
4. Automatic Failover: If Claude returns a timeout or rate-limit error, LiteLLM redirects the request to Gemini 1.5 Pro or GPT-4o seamlessly without the client noticing.
LiteLLM takes model configurations and maps them to a single virtual group. Here is how to configure a multi-provider model group inside config.yaml.
# Define model list mapping and parameters
model_list:
- model_name: llm-agent-core
litellm_params:
model: claude-3-5-sonnet-20241022
api_key: os.environ/ANTHROPIC_API_KEY
rpm: 1000
timeout: 10
- model_name: llm-agent-core
litellm_params:
model: gemini/gemini-1.5-pro
api_key: os.environ/GEMINI_API_KEY
rpm: 2000
timeout: 10
- model_name: llm-agent-core
litellm_params:
model: gpt-4o
api_key: os.environ/OPENAI_API_KEY
rpm: 5000
timeout: 8
router_settings:
routing_strategy: failover
allowed_fails: 2
cooldown_time: 30
Running this configuration locally or in Docker creates an endpoint mapping all requests from virtual model `llm-agent-core` to the active backup models on rate-limit triggers.
docker run -d -p 4000:4000 \ -v $(pwd)/config.yaml:/app/config.yaml \ -e ANTHROPIC_API_KEY="your-key" \ -e GEMINI_API_KEY="your-key" \ -e OPENAI_API_KEY="your-key" \ ghcr.io/berriai/litellm:main --config /app/config.yaml
Portkey sits in front of LiteLLM, intercepting queries and querying the local semantic database using embeddings. Standard caches require exact string matches; semantic caching relies on similarity.
| Prompt Variant | Exact Cache | Semantic Cache (Cosine Similarity > 0.95) |
|---|---|---|
| "Write a Python script to sort a list." | MISS (Saves initial response) | MISS (Saves initial response) |
| "Write a Python script to sort a list." | HIT (Returns saved response) | HIT (Returns saved response) |
| "Write Python code that sorts a list." | MISS (Treated as completely new) | HIT (Recognized similarity, returns cached result) |
| "Help me write a list sorting script in Python." | MISS (Treated as completely new) | HIT (Recognized similarity, returns cached result) |
Create a JSON config containing the routing and semantic cache settings for Portkey:
{
"strategy": {
"mode": "fallback"
},
"cache": {
"mode": "semantic",
"max_age": 86400,
"similarity_threshold": 0.96,
"vector_embeddings": {
"provider": "openai",
"model": "text-embedding-3-small"
}
},
"retry": {
"attempts": 3,
"backoff": "exponential"
}
}
With LiteLLM running on `localhost:4000` and Portkey routing the calls, we initialize the unified SDK and run semantic cache hit assertions.
import os
import time
from portkey_ai import Portkey
# Initialize client targeting LiteLLM's local proxy endpoint
client = Portkey(
api_key=os.environ.get("PORTKEY_API_KEY"),
base_url="http://localhost:4000",
config={
"cache": {
"mode": "semantic",
"max_age": 86400,
"similarity_threshold": 0.95
}
}
)
def query_gateway(prompt: str):
start_time = time.time()
response = client.chat.completions.create(
model="llm-agent-core",
messages=[
{"role": "system", "content": "You are a code optimization engine."},
{"role": "user", "content": prompt}
]
)
duration = time.time() - start_time
cache_status = response.headers.get("x-portkey-cache", "MISS")
print(f"Prompt: '{prompt}'")
print(f"Latency: {duration:.2f}s | Cache Status: {cache_status}")
print(f"Model Used: {response.model}\n")
# Run 1: Cold start (Cache Miss)
query_gateway("Write a quicksort implementation in Python.")
# Run 2: Semantic Variant (Cache Hit!)
query_gateway("Can you write a Python quicksort script?")
Observability is built-in. By monitoring traces, you can identify failing models, token costs, and exact cache hit rates across your architecture.
| Metric | Without Gateway Stack | With Gateway Stack |
|---|---|---|
| Downtime Resiliency | Low (Single API failure crashes script) | High (Auto-failover to backup providers) |
| Average Latency | ~1200ms - 2500ms | ~120ms (On semantic cache hits) |
| Total Token Bills | 100% full cost | Saved 80%+ (Repetitive tasks) |
| Multi-provider Code integration | Custom code wrappers per SDK | Unified OpenAI-compatible SDK |
| Action | Configuration / Command |
|---|---|
| Unify Models | LiteLLM config.yaml mapping parameters |
| Enable Semantic Caching | Set "mode": "semantic" in Portkey Config JSON |
| Similarity Threshold | Configure "similarity_threshold": 0.95 (Cosine Similarity) |
| Run Proxy (Local) | litellm --config config.yaml |
| Verify Failover logs | Run LiteLLM CLI with --debug flag |
| Cache Validation header | Inspect response header x-portkey-cache |
| Symptom | Potential Cause | Solution |
|---|---|---|
| Cache is always a MISS | Low similarity score, or different system prompts | Set similarity threshold down to 0.94, or verify system prompts match exactly. |
| LiteLLM returns 500 error | Missing environment API keys | Ensure environment variables are exported (e.g., export ANTHROPIC_API_KEY="key"). |
| Failover takes too long | High individual model timeout settings | Reduce individual model timeout parameters in config.yaml to 5-8 seconds. |
Yes. If the context of your data changes frequently, set a lower cache TTL (e.g., "max_age": 3600 for 1 hour) or use conditional tags to bypass cache on writes.
If you require local compliance, Portkey's gateway proxy can be self-hosted on your own AWS/GCP clusters, storing embeddings and cached completions entirely within your private subnet.
No. LiteLLM adds <5ms proxy overhead. Portkey's cache checks run concurrent to API calls on miss, meaning zero added penalty on miss, and huge speed gains on hit.
Yes. You can route embedding generations to a local HuggingFace/Ollama server by changing the vector_embeddings provider setting in the JSON config.