Model routing

How provider/model strings resolve, what the registry guarantees, and why unknown names are errors.

Every request names a model as provider/model. That string is the whole routing decision: it picks the provider, its base URL, the environment variable holding its key, and the name sent upstream.

{ "model": "openai/gpt-5-nano" }

Provider

openai — a host, its base URL, and one API-key variable.

Model

gpt-5-nano — a routable name under that provider.

Upstream name

What actually goes on the wire, which may differ from warpllm’s alias.

The prefix is required

warpllm matches the whole string against its registry. A bare gpt-5-nano does not resolve, and neither does a name the registry has never heard of.

An unlisted model is an error, not a guess. warpllm will not forward an unknown name to some upstream default and let you discover the routing on your bill.

client.chat_completions({"model": "gpt-5-nano", ...})
# warpllm.BadRequestError: invalid model string 'gpt-5-nano':
# no registered model spec

There is no wildcard either. A registry entry of openai/* would register a model literally named *.

Switching providers

Because the request shape does not change, moving between providers is a string edit plus the key that provider needs.

Python

# OpenAI — needs OPENAI_API_KEY
client.chat_completions({"model": "openai/gpt-5-nano", "messages": messages})

# DeepSeek — needs DEEPSEEK_API_KEY
client.chat_completions({"model": "deepseek/deepseek-v4-flash", "messages": messages})

# Anthropic via OpenRouter — needs OPENROUTER_API_KEY
client.chat_completions({"model": "openrouter/anthropic/claude-sonnet-4", "messages": messages})

A model name may contain further slashes after the provider prefix — openrouter/anthropic/claude-sonnet-4 is the OpenRouter provider serving the anthropic/claude-sonnet-4 slug.

Claude over Anthropic’s own protocol

anthropic/ models go to api.anthropic.com directly, and they are the first on the roster reached over a protocol that is not chat completions. Nothing changes in your code: you send the same request shape and read the same reply shape, and warpllm translates both ways.

Python

# Claude direct — needs ANTHROPIC_API_KEY
client.chat_completions({"model": "anthropic/claude-opus-5", "messages": messages})

# The same model through OpenRouter — needs OPENROUTER_API_KEY
client.chat_completions({"model": "openrouter/anthropic/claude-sonnet-4", "messages": messages})

Both routes work. Direct is one hop fewer and bills through Anthropic; the OpenRouter entry stays registered and unchanged.

Two differences from a chat-completions provider are worth knowing:

max_tokens is always sent

Anthropic requires it and chat completions does not, so warpllm falls back to the model’s published output ceiling when you name none. Your own value always wins. A model whose ceiling is undocumented and a request naming no max_tokens is refused rather than given an invented default — truncating a reply at a length nobody asked for looks like the model stopping early.

temperature, top_p and top_k are rejected by newer models

Anthropic deprecated all three on Claude Opus 4.7 and later: a non-default value returns a 400. warpllm forwards what you write rather than dropping it, because dropping a parameter would change your request instead of translating it — so the refusal is Anthropic’s, with Anthropic’s message. Omit them for those models and steer with the prompt.

Controls Anthropic cannot express are refused, not dropped

Everywhere else warpllm forwards what you write and lets the provider judge it. On a translated route the provider never receives the field, so it cannot judge it — and a silent drop would answer a different request than the one you sent. n: 2 would come back as a single choice, which looks exactly like a model that ignored you.

So these are refused with an InvalidInput error naming the field:

Field	Refused when
n	greater than 1
frequency_penalty, presence_penalty	non-zero
logprobs	true
top_logprobs	greater than 0
logit_bias	non-empty
seed	any value

The value your SDK fills in when you asked for nothing — n: 1, zero penalties, logprobs: false, an empty bias map — is not refused, so an ordinary request carrying them still routes.

top_k and parallel_tool_calls are NOT on this list: Anthropic has an equivalent for each, so warpllm translates them instead of refusing them.

Extended thinking cannot be combined with tool results

Anthropic requires the assistant turn holding a tool call to open with its own signed thinking block, and a chat-completions reply has nowhere to carry one — so the request that sends your tool results back cannot be built. warpllm refuses it with a NotImplemented error rather than letting Anthropic answer it with a 400.

A single-turn tool call with thinking works, and so does thinking-without-tools; it is sending the results back that stops. Turn thinking off for a tool loop, or run the loop on a chat-completions provider. Tracked in #101.

The thinking itself does reach you, streamed or whole, under reasoning_content — the same field DeepSeek and vLLM use. Its signature does not, because chat completions has no field for one, and that missing signature is exactly what the tool-result turn would need.

What the registry records

The registry lives at crates/warpllm/src/registry/specs.yaml and is compiled into the binary. Each model entry carries:

supported_apis · list · required

Every surface the model serves. A model serves exactly what it lists — one it does not list is refused before a request goes out.

model · string

The name sent upstream. Defaults to the key’s last segment; set explicitly when warpllm’s alias differs from the provider’s own name.

capabilities.max_input_tokens · integer

The provider’s published context window, recorded verbatim.

capabilities.max_output_tokens · integer

The provider’s published output ceiling, where one exists.

capabilities.max_concurrent_requests · integer

A documented per-model concurrency ceiling, where the provider publishes one.

deprecation_date · string

YYYY-MM-DD, the day the provider stops serving the model, when one has been announced. Nothing acts on it — an entry keeps resolving past its date.

Every optional field is genuinely optional, and an absent limit means the provider does not publish one — not that the limit is unlimited. warpllm omits a number rather than guessing it.

Reading capabilities at runtime

Rust callers can look a spec up directly, which is useful for sizing prompts against the routed model’s window.

use warpllm::fetch_model;

let (provider, spec) = fetch_model("openai/gpt-5-nano")?;

println!("{} sends this upstream as {}", provider.name(), spec.model());
if let Some(window) = spec.capabilities().max_input_tokens() {
    println!("context window: {window}");
}

fetch_model returns the provider spec and the model spec as a pair, or Error::InvalidModel if the string does not resolve.

Surfaces

A surface name is <protocol>_<endpoint>, which is how the registry records the wire format a model speaks.

Surface	Meaning	Status
openai_compat_chat_completions	One request, one whole reply	Implemented
openai_compat_chat_completions_stream	The same, as server-sent events	Implemented
openai_compat_responses	OpenAI’s Responses API	Not implemented

Spreading load, retrying, and failing over

A request naming one model goes to exactly that model, and a failure comes back to you to decide about. There is no retry loop yet, so the error taxonomy is what lets you write your own policy — see Errors.

Failover is the other shape: name an ordered list in models instead of model, and warpllm tries each in turn until one answers.

client.chat_completions({
    "models": ["anthropic/claude-sonnet-5-5", "openai/gpt-6-sol"],
    "messages": [{"role": "user", "content": "Hello"}],
})

• Up to 8 candidates, each one validated before anything is sent. Duplicates are dropped, and an empty list is an error. model and models are mutually exclusive.
• A failure scoped to one provider moves on: network, rate limit, overload, a 5xx, model not found, and a rejected or exhausted key — a dead key on one provider should not block a healthy one on another. A failure about the request itself, such as InvalidInput, stops the chain. So does Decode: the provider already answered 200 and billed, and retrying would buy a second completion.
• The whole chain shares one deadline, the client’s timeout. Running out is DeadlineExceeded (504); every candidate failing is CandidatesExhausted (503), whose body lists each attempt and its error.
• A stream fails over until its first chunk arrives. After that it is committed, and a failure is the stream’s to report.
• BalancedClient refuses a request carrying models, rather than balancing over one list and ignoring the other.

Load balancing shipped in 0.5.0. BalancedClient spreads requests across a weighted set of provider/model pairs using smooth weighted round-robin, rewriting the request’s model to the candidate it picked — so the name you pass acts as a group, and each candidate is a concrete route within it.

use warpllm::{BalancedClient, Client, ClientConfig};

let client = Client::new(ClientConfig::default())?;
let balanced = BalancedClient::new(&client, &[
    ("openai/gpt-5.6", 3),
    ("deepseek/deepseek-v4-pro", 1),
])?;

The candidate set is fixed at construction, and each selected candidate still goes through the same admission checks a direct call does. It is Rust-only for now — neither binding exposes it, and the gateway has no flag for it.

Load balancing is tracked in issue #28.

Next

Model roster

Every routable model and its limits.

Add a model

The registry edit, and the rules its tests enforce.