Streaming

Consume a reply as it is generated, and tell a complete stream from a truncated one.

Set stream: true and you get chunks instead of one reply. Every model on the roster that serves chat completions also serves them streamed.

Consuming a stream

Python

from warpllm import WarpLLM

client = WarpLLM()

stream = client.chat_completions({
    "model": "openai/gpt-5-nano",
    "messages": [{"role": "user", "content": "Count to five."}],
    "stream": True,
})

for chunk in stream:
    print(chunk["choices"][0]["delta"].get("content", ""), end="", flush=True)

How each language decides

Python

chat_completions dispatches on the value of stream, matching the official OpenAI SDK’s one-method shape. Runtime behaviour is exact.

Static typing is best-effort: a dict literal carrying keys warpllm does not model matches no TypedDict, so a checker may infer the non-streaming return type. Where the annotation has to be right, call chat_completions_stream(request) — same behaviour, precisely typed, and it sets stream for you.

stream = client.chat_completions_stream({
    "model": "openai/gpt-5-nano",
    "messages": messages,
})

Node

chatCompletions has three overloads. A literal stream: true returns ChatCompletionStream; its absence, false, or null returns a whole completion; anything else — a boolean variable, a value threaded through a helper — is genuinely either and types as the union.

Narrowing a union is a nuisance; being handed a completion that is a stream at runtime is a crash on .choices[0].message. Runtime dispatch is on the value, so behaviour is right even where the overload cannot resolve.

Rust

Streaming is its own method. chat_completions with stream: Some(true) returns Error::InvalidInput telling you to call chat_completions_stream — a whole-reply return type cannot carry chunks, so the entrypoint says where to go rather than quietly serving the wrong shape.

chat_completions_stream sets stream on your behalf; you do not have to.

The chunk shape

Each chunk is a CreateChatCompletionStreamResponse. The difference from a whole reply is choices[].delta in place of choices[].message.

{
  "id": "chatcmpl-...",
  "object": "chat.completion.chunk",
  "created": 1770000000,
  "model": "openai/gpt-5-nano",
  "choices": [
    {
      "index": 0,
      "delta": { "content": "One" },
      "finish_reason": null
    }
  ]
}

choices[].delta.content · string

The text fragment. Absent on chunks that carry only a role or tool-call fragment.

choices[].delta.role · string

Usually present on the first chunk only.

choices[].delta.tool_calls · array

Tool-call fragments, each with an index to reassemble by. See Tool calling.

choices[].finish_reason · string | null

null until the final content chunk.

model on every chunk is your provider/model routing string, not the upstream name.

Token usage on a stream

Ask for it explicitly; providers do not send it by default.

{
  "model": "openai/gpt-5-nano",
  "messages": [{ "role": "user", "content": "Count to five." }],
  "stream": true,
  "stream_options": { "include_usage": true }
}

A final chunk then carries usage with the same fields a whole reply has.

Knowing the reply is complete

This is the part worth getting right.

1. The stream ending cleanly means the reply is whole

   Python raises StopIteration, Node’s for await finishes, Rust’s next() returns None. In every case: the model finished.

2. An upstream that stopped early raises instead

   StreamTruncated arrives after every chunk that did make it, so a truncated answer can never be mistaken for a finished one. Without this, both endings would look identical.

3. An error item is terminal

   Whatever produced it also ended the stream. There is nothing left to read, and warpllm sends no completion sentinel after a failure.

from warpllm import APIError

buffer = []
try:
    for chunk in stream:
        buffer.append(chunk["choices"][0]["delta"].get("content", ""))
except APIError as e:
    # Terminal. `buffer` holds a partial answer — discard or salvage it,
    # but do not treat it as complete.
    log.warning("stream failed after %d chunks: %s", len(buffer), e.message)

Set stream_read_timeout above the slowest time-to-first-token you expect. The total request timeout cannot tell a slow stream from a wedged one.

Through the gateway

The gateway serves streams as server-sent events, the shape an OpenAI SDK expects, terminated by data: [DONE].

A failure that arrives before the response commits is an ordinary error response with its real status and headers. A failure after the 200 is committed travels as the last event, carrying the same OpenAI envelope — and no [DONE] follows it, because the sentinel means the model finished.

Next

Tool calling

Reassembling streamed tool-call fragments.

Timeouts

The gap-based deadline that streams actually need.