Guides
Streaming
Stop long reasoning streams from dropping
A model that thinks quietly for a minute can look like a dead connection to a CLI socket timeout or an idle-reaping proxy. The stream is cut before the answer arrives.
Keepalives in each wire's own idiom
While the upstream is silent, OpenLLM sends the keepalive each client expects: a real ping event on the Anthropic wire, and an SSE comment on the OpenAI wire.
Never a half-written frame
On byte-for-byte passthrough streams, keepalives are inserted only between whole events, so a partially sent frame is never split.
Questions
Something else? Contact us