Throttle

shipping Throttle today. spent the last few months building this because teams are getting destroyed by LLM inference costs. the problem is real: you're paying for routes that don't need your best model, you're recomputing the same queries, your batching sucks. we cut our own spend from 8k to 2.4k just by being smart about it. open sourced it because if you're building with Claude or OpenAI at scale, you probably need this. grab it on github, test it on your actual traffic. would love to hear what breaks

Routing to cheaper models "when quality doesn't matter" is the hard part. How does the proxy judge task complexity: a heuristic on the prompt, or a classifier? Can I pin certain calls, like tool use or structured output, to a specific model so they never get downgraded?

The drop-in proxy approach is appealing since there's no code change needed to start trimming Claude/OpenAI spend. I'd be curious how it decides when a request is simple enough to route to a cheaper model.

Smart model routing that sends easy requests to cheaper models while caching repeated queries is a practical way to cut LLM bills. I like that it works as a drop-in proxy for Claude and OpenAI, so there's no code change needed.

The routing piece is the interesting part. How does Throttle decide a request is "low stakes" enough for a cheaper model? Is it a classifier on the prompt, rules per endpoint, or something you configure? I'd also like to see per-route cost and quality numbers in a dashboard before trusting it in production.

Going from $8k to $2.4k a month on your own spend is a pretty convincing proof point. How does the router decide when a request is simple enough for a cheaper model, and have you seen cases where it downgraded something it shouldn't have? Also curious whether the cache handles near-duplicate prompts or only exact matches.

How does the proxy handle fallbacks if the cheaper model returns a weak answer? Curious whether quality checks are configurable per request.

Nice idea. Routing easy requests to cheaper models and caching repeats without any code changes is the part most people would actually use. Does the router let you set which requests count as easy, or is it automatic?

The routing idea of sending low-stakes requests to cheaper models while caching repeats makes sense. How do you decide task complexity without adding noticeable latency, and can teams set a quality floor per endpoint so critical prompts never get downgraded?

Routing simple requests to cheaper models while caching repeated queries is a practical way to cut spend, and the drop-in proxy setup means teams don't have to touch app code. I'm curious how you decide when a task is 'simple enough' to downgrade, and whether that threshold can be tuned per endpoint. Seems especially useful for startups with high-volume Claude/OpenAI usage.

The drop-in proxy approach with no code changes is what makes this easy to try. How do you decide when a request is simple enough to route to a cheaper model, and can teams override that per endpoint if quality dips?

Dropping in as a proxy between the app and the Claude/OpenAI APIs with no code changes is a very low-friction way to adopt this, and combining model routing with query caching targets a real cost problem. How do you decide when a request is simple enough for a cheaper model, and can teams set their own routing rules?

Caching repeated queries is usually where the big savings hide. How do you key the cache: exact prompt match, or similar prompts too? And since every prompt passes through the proxy, can it run self-hosted so prompts with user data stay on our own servers?

Caching repeated queries alongside model routing is a sensible pairing, since those two cut cost without touching application code. How do you decide when a cached answer is stale, for example if a system prompt or the underlying model version changes? A per-route cache expiry setting would be a useful addition.

Routing simple requests to cheaper models automatically while caching repeated queries is a practical way to cut LLM spend. Being a drop-in proxy with no code changes makes it very easy to try on an existing Claude or OpenAI app.

A drop-in proxy is the right shape for this. Nobody wants to rewrite their LLM calls just to save money. How do you handle streaming responses and tool calls? Do they pass through untouched, or does the 70% come from routing some requests to smaller models?

Is there a dry-run or shadow mode that logs which requests would have been routed to a cheaper model, without actually switching them? That would let teams check quality on their own traffic before trusting the 70% savings figure.