Streaming

With "stream": true in the request, all three chat endpoints (Chat Completions, Responses and Anthropic Messages) return content piece by piece over SSE (Server-Sent Events). The three protocols differ in event format, end marker and where usage appears, so a client must parse according to the protocol it uses and rely on that protocol's end marker to decide whether the output is complete. This chapter covers protocol formats and client implementation; for troubleshooting interrupted streams and timeouts, see Timeouts and interrupted streams.

Enabling streaming

• If a request is rejected before output starts (authentication failure, insufficient balance, unavailable model and so on), the response is an ordinary HTTP error, not an event stream: requests rejected by the gateway get a JSON error body (in Anthropic error format for /v1/messages), and 500 / 504 are HTML pages. Check the status code and Content-Type before parsing the event stream.
• Once streaming output has started, the gateway no longer switches channels; for what happens when the upstream fails, see "Mid-stream failures" below.

SSE basics

An event consists of several lines and ends with a blank line:

event: <event name>
data: <JSON or text>

• A data: line carries the event content; when an event has several data: lines, join them with newlines. Chat Completions has only data: lines; Responses and Anthropic Messages also have an event: line that gives the event name.
• The space after the colon is optional: data: {...} and data:{...} are both valid under the SSE specification, and both occur in real streams, so handle both if you write your own parser.
• There may also be other field lines such as id:; for example, every event in a Responses stream is preceded by an id: line. Ignore them if you do not need them.
• Lines starting with : are comments, which some upstreams use to keep the connection alive; ignore them when parsing.
• One chunk of data on the network may contain only half an event, or several events. Accumulate data in a buffer and process one complete event each time you reach a blank line.

Chat Completions

Each event is a single data: <chat.completion.chunk> line, and the stream ends with data: [DONE]:

data: {"id":"chatcmpl-9f1c2a7e4b","object":"chat.completion.chunk","created":1791950400,"model":"MODEL_ID","choices":[{"index":0,"delta":{"role":"assistant","content":""},"finish_reason":null}]}

data: {"id":"chatcmpl-9f1c2a7e4b","object":"chat.completion.chunk","created":1791950400,"model":"MODEL_ID","choices":[{"index":0,"delta":{"content":"Autumn wind rises,"},"finish_reason":null}]}

data: {"id":"chatcmpl-9f1c2a7e4b","object":"chat.completion.chunk","created":1791950400,"model":"MODEL_ID","choices":[{"index":0,"delta":{"content":" leaves turn gold."},"finish_reason":null}]}

data: {"id":"chatcmpl-9f1c2a7e4b","object":"chat.completion.chunk","created":1791950400,"model":"MODEL_ID","choices":[{"index":0,"delta":{},"finish_reason":"stop"}]}

data: {"id":"chatcmpl-9f1c2a7e4b","object":"chat.completion.chunk","created":1791950400,"model":"MODEL_ID","choices":[],"usage":{"prompt_tokens":18,"completion_tokens":12,"total_tokens":30,"prompt_tokens_details":{"cached_tokens":0}}}

data: [DONE]

• The gateway always sets stream_options to {"include_usage": true}, so there is always a chunk with usage, and its choices is an empty array. Check that the array is not empty before reading choices[0].
• When an Anthropic-protocol channel serves the request, the gateway generates the chunks by conversion: the chunk with usage has only prompt_tokens, completion_tokens and total_tokens, and prompt_tokens excludes cache hits. Check cache counts in the request logs.
• For what the fields mean and how they map to the four token types in logs, see Chat Completions.

Responses

Every event has an event name; type in data matches the event name, and sequence_number increases. Each event is also preceded by an id: line, which the fragment below omits:

event: response.created
data: {"type":"response.created","sequence_number":0,"response":{"id":"resp_68f1a2b3","object":"response","created_at":1791950400,"status":"queued","model":"MODEL_ID","output":[],"usage":null}}

event: response.output_item.added
data: {"type":"response.output_item.added","sequence_number":2,"output_index":0,"item":{"id":"msg_68f1a2c4","type":"message","status":"in_progress","role":"assistant","content":[]}}

event: response.output_text.delta
data: {"type":"response.output_text.delta","sequence_number":4,"item_id":"msg_68f1a2c4","output_index":0,"content_index":0,"delta":"Autumn wind rises,"}

event: response.output_text.delta
data: {"type":"response.output_text.delta","sequence_number":5,"item_id":"msg_68f1a2c4","output_index":0,"content_index":0,"delta":" leaves turn gold."}

event: response.output_text.done
data: {"type":"response.output_text.done","sequence_number":6,"item_id":"msg_68f1a2c4","output_index":0,"content_index":0,"text":"Autumn wind rises, leaves turn gold."}

event: response.completed
data: {"type":"response.completed","sequence_number":9,"response":{"id":"resp_68f1a2b3","object":"response","status":"completed","model":"MODEL_ID","output":[{"id":"msg_68f1a2c4","type":"message","status":"completed","role":"assistant","content":[{"type":"output_text","text":"Autumn wind rises, leaves turn gold.","annotations":[]}]}],"usage":{"input_tokens":18,"input_tokens_details":{"cached_tokens":0},"output_tokens":12,"output_tokens_details":{"reasoning_tokens":0},"total_tokens":30}}}

• The gateway forwards the Responses events returned by the channel unchanged and does not add stream_options; there is usually no data: [DONE] either.
• Usage is in response.usage of response.completed: input_tokens includes cache hits, and the number of hits is in input_tokens_details.cached_tokens.

Anthropic Messages

event: message_start
data: {"type":"message_start","message":{"id":"msg_01XFDUDYJgAACzvnptvVoYEL","type":"message","role":"assistant","model":"MODEL_ID","content":[],"usage":{"input_tokens":18,"output_tokens":0}}}

event: content_block_start
data: {"type":"content_block_start","index":0,"content_block":{"type":"text","text":""}}

event: content_block_delta
data: {"type":"content_block_delta","index":0,"delta":{"type":"text_delta","text":"Autumn wind rises,"}}

event: content_block_delta
data: {"type":"content_block_delta","index":0,"delta":{"type":"text_delta","text":" leaves turn gold."}}

event: content_block_stop
data: {"type":"content_block_stop","index":0}

event: message_delta
data: {"type":"message_delta","delta":{"stop_reason":"end_turn"},"usage":{"input_tokens":18,"output_tokens":12}}

event: message_stop
data: {"type":"message_stop"}

• Rely on usage in message_delta. When an OpenAI-compatible channel serves the request, the gateway generates the events by conversion: input_tokens in message_start is 0, and the final counts appear only in message_delta. When there are cache reads or writes, message_delta.usage also carries cache_read_input_tokens and cache_creation_input_tokens respectively.
• input_tokens excludes cache reads and cache writes; for how to calculate costs, see Estimating costs and cache pricing.

End markers and usage at a glance

Treat every stream that closes without a normal end marker as failed.

Mid-stream failures

If the upstream breaks off or times out after streaming output has started, the gateway does not switch channels; it appends an error event to the end of the stream and closes the connection:

data: {"error": {"message": "upstream timed out after the response had started", "type": "api_error", "code": "stream_interrupted"}}

event: error
data: {"type": "error", "error": {"type": "api_error", "message": "upstream timed out after the response had started"}}

• The first form is used for Chat Completions and the second for Anthropic Messages. Responses gets a response.failed event instead, with response.error.code server_error and a sequence_number that continues from the events already sent:

event: response.failed
data: {"type": "response.failed", "sequence_number": 42, "response": {"id": "resp_...", "object": "response", "status": "failed", "error": {"code": "server_error", "message": "upstream timed out after the response had started"}}}

• On a timeout, message is upstream timed out after the response had started; on a connection or read failure, it is upstream failed after the response had started.
• After an error event, no data: [DONE], message_stop or response.completed follows. Error events that the upstream itself sends in the stream (such as Anthropic's overloaded_error) are passed on to you in the protocol you called.
• The official OpenAI and Anthropic SDKs raise an exception when they receive an error event in their format. In the logs these requests have status 200; for causes and retry advice, see Timeouts and interrupted streams.

Client implementation notes

• Parse complete events: accumulate data and process it when you reach a blank line; join multiple data: lines; ignore comment lines and events you do not recognize; data: [DONE] is not JSON, so check for it before parsing.
• Use the end marker to judge completeness: treat the result as complete output only after receiving the end marker; treat an error event or an early connection close as a failure, and deduplicate before retrying (for example, tool operations that have already run).
• Timeouts: while waiting for the upstream, the gateway waits at most 6 minutes between two pieces of data. Set the client's read timeout longer than 6 minutes, and prefer an idle timeout between two reads over a total timeout for the whole request.
• No buffering in proxies: when a reverse proxy such as Nginx sits between your application and LYRouter, turn off response buffering (in Nginx, set proxy_buffering off) and set the proxy's read timeout longer than 6 minutes; otherwise content arrives in one large block, or the proxy disconnects early.
• Command-line testing: add -N to curl to turn off output buffering; only then do you see events arrive one by one.

Below is a way to parse a Chat Completions event stream by hand without an SDK, which you can use as a reference for implementations in other languages:

curl -N https://api.lyrouter.com/v1/chat/completions \
  -H "Authorization: Bearer $LYROUTER_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"model":"MODEL_ID","stream":true,"messages":[{"role":"user","content":"Write a two-line poem about autumn"}]}'

import json
import os
import requests

resp = requests.post(
    "https://api.lyrouter.com/v1/chat/completions",
    headers={"Authorization": f"Bearer {os.environ['LYROUTER_API_KEY']}"},
    json={
        "model": "MODEL_ID",
        "stream": True,
        "messages": [{"role": "user", "content": "Write a two-line poem about autumn"}],
    },
    stream=True,
    timeout=(10, 400),  # 10 s connect timeout; wait at most 400 s between two reads
)
resp.raise_for_status()
resp.encoding = "utf-8"

done = False
data_lines = []
for line in resp.iter_lines(decode_unicode=True):
    if line.startswith(":"):
        continue  # comment line
    if line.startswith("data:"):
        data_lines.append(line[5:].lstrip())
        continue
    if line != "" or not data_lines:
        continue  # other fields such as event:, or extra blank lines
    data = "\n".join(data_lines)  # blank line: one complete event
    data_lines = []
    if data == "[DONE]":
        done = True
        break
    event = json.loads(data)
    if "error" in event:
        raise RuntimeError(event["error"].get("message"))
    for choice in event.get("choices", []):
        print(choice.get("delta", {}).get("content") or "", end="", flush=True)
    if event.get("usage"):
        print("\n", event["usage"])

if not done:
    raise RuntimeError("Stream ended before the end marker; output is incomplete")

const resp = await fetch("https://api.lyrouter.com/v1/chat/completions", {
  method: "POST",
  headers: {
    Authorization: `Bearer ${process.env.LYROUTER_API_KEY}`,
    "Content-Type": "application/json",
  },
  body: JSON.stringify({
    model: "MODEL_ID",
    stream: true,
    messages: [{ role: "user", content: "Write a two-line poem about autumn" }],
  }),
});
if (!resp.ok) throw new Error(`HTTP ${resp.status}: ${await resp.text()}`);

const decoder = new TextDecoder();
let buffer = "";
let done = false;
for await (const bytes of resp.body) {
  buffer = (buffer + decoder.decode(bytes, { stream: true })).replace(/\r\n/g, "\n");
  let end;
  while (!done && (end = buffer.indexOf("\n\n")) !== -1) {
    const block = buffer.slice(0, end); // one complete event
    buffer = buffer.slice(end + 2);
    const data = block
      .split("\n")
      .filter((line) => line.startsWith("data:"))
      .map((line) => line.slice(5).trimStart())
      .join("\n");
    if (!data) continue; // comment or other fields
    if (data === "[DONE]") {
      done = true;
      break;
    }
    const event = JSON.parse(data);
    if (event.error) throw new Error(event.error.message);
    const text = event.choices?.[0]?.delta?.content;
    if (text) process.stdout.write(text);
    if (event.usage) console.log("\n", event.usage);
  }
  if (done) break;
}
if (!done) throw new Error("Stream ended before the end marker; output is incomplete");

With an official SDK, the SDK already handles event parsing and the end marker; for usage, see the streaming examples in Chat Completions. For streaming with the Responses and Anthropic SDKs, see Responses and Anthropic Messages.

FAQ

Why is choices an empty array in the last chunk?

Answer: The gateway always turns on include_usage, and the upstream sends a separate chunk that carries only usage before the end; its choices is empty. Code that reads choices[0] directly fails, so check that the array is not empty first. This chunk cannot be turned off.

Why did I get a JSON error instead of an event stream with stream: true?

Answer: The request was rejected before output started, for example because of an authentication failure, insufficient balance or an unavailable model, and the gateway returned the status code and error body as an ordinary response. Handle it by status code; see Errors and request identifiers.

Why does streaming output appear all at once after a long wait?

Answer: Usually a proxy in between is buffering the response. Compare by requesting LYRouter directly with curl -N; if the direct request works and the proxied one does not, turn off response buffering in the proxy. If the first piece of content takes a long time even on a direct request, check "TTFT" in the request logs; see Timeouts and interrupted streams.

Can I use the content I received if the end marker never arrived?

Answer: Not as a complete result. The gateway sends an error event when a stream fails midway, but when the upstream disconnects directly there may be no error event at all. Treat the call as failed, retry as needed, and deduplicate repeated content and tools that have already run.