# Streaming

Source: https://platform.lyrouter.com/ai_gateway/docs/streaming/en
Last updated: 2026-10-22

With `"stream": true` in the request, all three chat endpoints (Chat Completions, Responses and Anthropic Messages) return content piece by piece over SSE (Server-Sent Events). The three protocols differ in event format, end marker and where `usage` appears, so a client must parse according to the protocol it uses and rely on that protocol's end marker to decide whether the output is complete. This chapter covers protocol formats and client implementation; for troubleshooting interrupted streams and timeouts, see [Timeouts and interrupted streams](https://platform.lyrouter.com/ai_gateway/docs/stream-errors/en).

## Enabling streaming

| Endpoint | How to enable | Response `Content-Type` |
| --- | --- | --- |
| [Chat Completions](https://platform.lyrouter.com/ai_gateway/docs/chat-completions/en) (`/v1/chat/completions`) | `"stream": true` | `text/event-stream` |
| [Responses](https://platform.lyrouter.com/ai_gateway/docs/responses-api/en) (`/v1/responses`) | `"stream": true` | `text/event-stream` |
| [Anthropic Messages](https://platform.lyrouter.com/ai_gateway/docs/anthropic-messages/en) (`/v1/messages`) | `"stream": true` | `text/event-stream` |

- If a request is rejected before output starts (authentication failure, insufficient balance, unavailable model and so on), the response is an ordinary HTTP error, not an event stream: requests rejected by the gateway get a JSON error body (in Anthropic error format for `/v1/messages`), and 500 / 504 are HTML pages. Check the status code and `Content-Type` before parsing the event stream.
- Once streaming output has started, the gateway no longer switches channels; for what happens when the upstream fails, see "Mid-stream failures" below.

## SSE basics

An event consists of several lines and ends with a blank line:

```text
event: <event name>
data: <JSON or text>

```

- A `data:` line carries the event content; when an event has several `data:` lines, join them with newlines. Chat Completions has only `data:` lines; Responses and Anthropic Messages also have an `event:` line that gives the event name.
- The space after the colon is optional: `data: {...}` and `data:{...}` are both valid under the SSE specification, and both occur in real streams, so handle both if you write your own parser.
- There may also be other field lines such as `id:`; for example, every event in a Responses stream is preceded by an `id:` line. Ignore them if you do not need them.
- Lines starting with `:` are comments, which some upstreams use to keep the connection alive; ignore them when parsing.
- One chunk of data on the network may contain only half an event, or several events. Accumulate data in a buffer and process one complete event each time you reach a blank line.

## Chat Completions

Each event is a single `data: <chat.completion.chunk>` line, and the stream ends with `data: [DONE]`:

```text
data: {"id":"chatcmpl-9f1c2a7e4b","object":"chat.completion.chunk","created":1791950400,"model":"MODEL_ID","choices":[{"index":0,"delta":{"role":"assistant","content":""},"finish_reason":null}]}

data: {"id":"chatcmpl-9f1c2a7e4b","object":"chat.completion.chunk","created":1791950400,"model":"MODEL_ID","choices":[{"index":0,"delta":{"content":"Autumn wind rises,"},"finish_reason":null}]}

data: {"id":"chatcmpl-9f1c2a7e4b","object":"chat.completion.chunk","created":1791950400,"model":"MODEL_ID","choices":[{"index":0,"delta":{"content":" leaves turn gold."},"finish_reason":null}]}

data: {"id":"chatcmpl-9f1c2a7e4b","object":"chat.completion.chunk","created":1791950400,"model":"MODEL_ID","choices":[{"index":0,"delta":{},"finish_reason":"stop"}]}

data: {"id":"chatcmpl-9f1c2a7e4b","object":"chat.completion.chunk","created":1791950400,"model":"MODEL_ID","choices":[],"usage":{"prompt_tokens":18,"completion_tokens":12,"total_tokens":30,"prompt_tokens_details":{"cached_tokens":0}}}

data: [DONE]
```

| Content | Location |
| --- | --- |
| Answer text deltas | `choices[0].delta.content` |
| Thinking deltas | `choices[0].delta.reasoning_content` (`delta.reasoning` on a few upstreams) |
| Tool call deltas | `choices[0].delta.tool_calls`, joined by `index`; see [Tool calls](https://platform.lyrouter.com/ai_gateway/docs/tool-call/en) |
| Finish reason | `choices[0].finish_reason` in the last few chunks |
| Usage | `usage` in the chunk whose `choices` is an empty array |
| End marker | `data: [DONE]` |

- The gateway always sets `stream_options` to `{"include_usage": true}`, so there is always a chunk with `usage`, and its `choices` is an empty array. Check that the array is not empty before reading `choices[0]`.
- When an Anthropic-protocol channel serves the request, the gateway generates the chunks by conversion: the chunk with `usage` has only `prompt_tokens`, `completion_tokens` and `total_tokens`, and `prompt_tokens` excludes cache hits. Check cache counts in the [request logs](https://platform.lyrouter.com/ai_gateway/docs/logs/en).
- For what the fields mean and how they map to the four token types in logs, see [Chat Completions](https://platform.lyrouter.com/ai_gateway/docs/chat-completions/en).

## Responses

Every event has an event name; `type` in `data` matches the event name, and `sequence_number` increases. Each event is also preceded by an `id:` line, which the fragment below omits:

```text
event: response.created
data: {"type":"response.created","sequence_number":0,"response":{"id":"resp_68f1a2b3","object":"response","created_at":1791950400,"status":"queued","model":"MODEL_ID","output":[],"usage":null}}

event: response.output_item.added
data: {"type":"response.output_item.added","sequence_number":2,"output_index":0,"item":{"id":"msg_68f1a2c4","type":"message","status":"in_progress","role":"assistant","content":[]}}

event: response.output_text.delta
data: {"type":"response.output_text.delta","sequence_number":4,"item_id":"msg_68f1a2c4","output_index":0,"content_index":0,"delta":"Autumn wind rises,"}

event: response.output_text.delta
data: {"type":"response.output_text.delta","sequence_number":5,"item_id":"msg_68f1a2c4","output_index":0,"content_index":0,"delta":" leaves turn gold."}

event: response.output_text.done
data: {"type":"response.output_text.done","sequence_number":6,"item_id":"msg_68f1a2c4","output_index":0,"content_index":0,"text":"Autumn wind rises, leaves turn gold."}

event: response.completed
data: {"type":"response.completed","sequence_number":9,"response":{"id":"resp_68f1a2b3","object":"response","status":"completed","model":"MODEL_ID","output":[{"id":"msg_68f1a2c4","type":"message","status":"completed","role":"assistant","content":[{"type":"output_text","text":"Autumn wind rises, leaves turn gold.","annotations":[]}]}],"usage":{"input_tokens":18,"input_tokens_details":{"cached_tokens":0},"output_tokens":12,"output_tokens_details":{"reasoning_tokens":0},"total_tokens":30}}}
```

| Event | Meaning |
| --- | --- |
| `response.created`, `response.in_progress` | The response has been created / is being generated. The `status` in `response.created` may be `queued` |
| `response.output_item.added` / `.done` | An output item starts / ends; its type is `message`, `reasoning`, `function_call` and so on |
| `response.content_part.added` / `.done` | A content part within a message starts / ends |
| `response.output_text.delta` / `.done` | Text delta (`delta`) / the full text of that content part (`text`) |
| `response.reasoning_text.delta` / `.done` | Reasoning text delta / full reasoning text, returned by reasoning models |
| `response.function_call_arguments.delta` / `.done` | Function call argument delta / the full arguments; see [Tool calls](https://platform.lyrouter.com/ai_gateway/docs/tool-call/en) |
| `response.completed` | Normal end; `response.usage` is the usage for this call |
| `response.incomplete` | Ended before generation was complete, for example on reaching `max_output_tokens` |
| `response.failed`, `error` | Generation failed / the upstream reported an error |

- The gateway forwards the Responses events returned by the channel unchanged and does not add `stream_options`; there is usually no `data: [DONE]` either.
- Usage is in `response.usage` of `response.completed`: `input_tokens` includes cache hits, and the number of hits is in `input_tokens_details.cached_tokens`.

## Anthropic Messages

```text
event: message_start
data: {"type":"message_start","message":{"id":"msg_01XFDUDYJgAACzvnptvVoYEL","type":"message","role":"assistant","model":"MODEL_ID","content":[],"usage":{"input_tokens":18,"output_tokens":0}}}

event: content_block_start
data: {"type":"content_block_start","index":0,"content_block":{"type":"text","text":""}}

event: content_block_delta
data: {"type":"content_block_delta","index":0,"delta":{"type":"text_delta","text":"Autumn wind rises,"}}

event: content_block_delta
data: {"type":"content_block_delta","index":0,"delta":{"type":"text_delta","text":" leaves turn gold."}}

event: content_block_stop
data: {"type":"content_block_stop","index":0}

event: message_delta
data: {"type":"message_delta","delta":{"stop_reason":"end_turn"},"usage":{"input_tokens":18,"output_tokens":12}}

event: message_stop
data: {"type":"message_stop"}
```

| Event | Meaning |
| --- | --- |
| `message_start` | The message starts; `message.usage` contains the input token count |
| `content_block_start` | A content block starts; its type is `text`, `thinking` or `tool_use` |
| `content_block_delta` | Deltas: `text_delta` (answer text), `thinking_delta` (thinking), `signature_delta` (thinking block signature), `input_json_delta` (tool arguments) |
| `content_block_stop` | The content block ends |
| `message_delta` | `delta.stop_reason` (`end_turn`, `max_tokens`, `tool_use` and so on) and the final `usage` |
| `message_stop` | Normal end |
| `ping` | Keeps the connection alive; ignore it |
| `error` | Error |

- Rely on `usage` in `message_delta`. When an OpenAI-compatible channel serves the request, the gateway generates the events by conversion: `input_tokens` in `message_start` is 0, and the final counts appear only in `message_delta`. When there are cache reads or writes, `message_delta.usage` also carries `cache_read_input_tokens` and `cache_creation_input_tokens` respectively.
- `input_tokens` excludes cache reads and cache writes; for how to calculate costs, see [Estimating costs and cache pricing](https://platform.lyrouter.com/ai_gateway/docs/estimate/en).

## End markers and usage at a glance

| Endpoint | Normal end | Where usage is | Error event for a mid-stream failure |
| --- | --- | --- | --- |
| Chat Completions | `data: [DONE]` | The chunk whose `choices` is an empty array | `data: {"error": {...}}` |
| Responses | `response.completed` event | `response.usage` in `response.completed` | `response.failed` event |
| Anthropic Messages | `message_stop` event | `usage` in `message_delta` | `event: error` |

Treat every stream that closes without a normal end marker as failed.

## Mid-stream failures

If the upstream breaks off or times out after streaming output has started, the gateway does not switch channels; it appends an error event to the end of the stream and closes the connection:

```text
data: {"error": {"message": "upstream timed out after the response had started", "type": "api_error", "code": "stream_interrupted"}}
```

```text
event: error
data: {"type": "error", "error": {"type": "api_error", "message": "upstream timed out after the response had started"}}
```

- The first form is used for Chat Completions and the second for Anthropic Messages. Responses gets a `response.failed` event instead, with `response.error.code` `server_error` and a `sequence_number` that continues from the events already sent:

```text
event: response.failed
data: {"type": "response.failed", "sequence_number": 42, "response": {"id": "resp_...", "object": "response", "status": "failed", "error": {"code": "server_error", "message": "upstream timed out after the response had started"}}}
```

- On a timeout, `message` is `upstream timed out after the response had started`; on a connection or read failure, it is `upstream failed after the response had started`.
- After an error event, no `data: [DONE]`, `message_stop` or `response.completed` follows. Error events that the upstream itself sends in the stream (such as Anthropic's `overloaded_error`) are passed on to you in the protocol you called.
- The official OpenAI and Anthropic SDKs raise an exception when they receive an error event in their format. In the logs these requests have status 200; for causes and retry advice, see [Timeouts and interrupted streams](https://platform.lyrouter.com/ai_gateway/docs/stream-errors/en).

## Client implementation notes

- **Parse complete events**: accumulate data and process it when you reach a blank line; join multiple `data:` lines; ignore comment lines and events you do not recognize; `data: [DONE]` is not JSON, so check for it before parsing.
- **Use the end marker to judge completeness**: treat the result as complete output only after receiving the end marker; treat an error event or an early connection close as a failure, and deduplicate before retrying (for example, tool operations that have already run).
- **Timeouts**: while waiting for the upstream, the gateway waits at most 6 minutes between two pieces of data. Set the client's read timeout longer than 6 minutes, and prefer an idle timeout between two reads over a total timeout for the whole request.
- **No buffering in proxies**: when a reverse proxy such as Nginx sits between your application and LYRouter, turn off response buffering (in Nginx, set `proxy_buffering off`) and set the proxy's read timeout longer than 6 minutes; otherwise content arrives in one large block, or the proxy disconnects early.
- **Command-line testing**: add `-N` to `curl` to turn off output buffering; only then do you see events arrive one by one.

Below is a way to parse a Chat Completions event stream by hand without an SDK, which you can use as a reference for implementations in other languages:

**Streaming request examples**

cURL:
```bash
curl -N https://api.lyrouter.com/v1/chat/completions \
  -H "Authorization: Bearer $LYROUTER_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"model":"MODEL_ID","stream":true,"messages":[{"role":"user","content":"Write a two-line poem about autumn"}]}'
```

Python:
```python
import json
import os
import requests

resp = requests.post(
    "https://api.lyrouter.com/v1/chat/completions",
    headers={"Authorization": f"Bearer {os.environ['LYROUTER_API_KEY']}"},
    json={
        "model": "MODEL_ID",
        "stream": True,
        "messages": [{"role": "user", "content": "Write a two-line poem about autumn"}],
    },
    stream=True,
    timeout=(10, 400),  # 10 s connect timeout; wait at most 400 s between two reads
)
resp.raise_for_status()
resp.encoding = "utf-8"

done = False
data_lines = []
for line in resp.iter_lines(decode_unicode=True):
    if line.startswith(":"):
        continue  # comment line
    if line.startswith("data:"):
        data_lines.append(line[5:].lstrip())
        continue
    if line != "" or not data_lines:
        continue  # other fields such as event:, or extra blank lines
    data = "\n".join(data_lines)  # blank line: one complete event
    data_lines = []
    if data == "[DONE]":
        done = True
        break
    event = json.loads(data)
    if "error" in event:
        raise RuntimeError(event["error"].get("message"))
    for choice in event.get("choices", []):
        print(choice.get("delta", {}).get("content") or "", end="", flush=True)
    if event.get("usage"):
        print("\n", event["usage"])

if not done:
    raise RuntimeError("Stream ended before the end marker; output is incomplete")
```

Node.js:
```js
const resp = await fetch("https://api.lyrouter.com/v1/chat/completions", {
  method: "POST",
  headers: {
    Authorization: `Bearer ${process.env.LYROUTER_API_KEY}`,
    "Content-Type": "application/json",
  },
  body: JSON.stringify({
    model: "MODEL_ID",
    stream: true,
    messages: [{ role: "user", content: "Write a two-line poem about autumn" }],
  }),
});
if (!resp.ok) throw new Error(`HTTP ${resp.status}: ${await resp.text()}`);

const decoder = new TextDecoder();
let buffer = "";
let done = false;
for await (const bytes of resp.body) {
  buffer = (buffer + decoder.decode(bytes, { stream: true })).replace(/\r\n/g, "\n");
  let end;
  while (!done && (end = buffer.indexOf("\n\n")) !== -1) {
    const block = buffer.slice(0, end); // one complete event
    buffer = buffer.slice(end + 2);
    const data = block
      .split("\n")
      .filter((line) => line.startsWith("data:"))
      .map((line) => line.slice(5).trimStart())
      .join("\n");
    if (!data) continue; // comment or other fields
    if (data === "[DONE]") {
      done = true;
      break;
    }
    const event = JSON.parse(data);
    if (event.error) throw new Error(event.error.message);
    const text = event.choices?.[0]?.delta?.content;
    if (text) process.stdout.write(text);
    if (event.usage) console.log("\n", event.usage);
  }
  if (done) break;
}
if (!done) throw new Error("Stream ended before the end marker; output is incomplete");
```

With an official SDK, the SDK already handles event parsing and the end marker; for usage, see the streaming examples in [Chat Completions](https://platform.lyrouter.com/ai_gateway/docs/chat-completions/en). For streaming with the Responses and Anthropic SDKs, see [Responses](https://platform.lyrouter.com/ai_gateway/docs/responses-api/en) and [Anthropic Messages](https://platform.lyrouter.com/ai_gateway/docs/anthropic-messages/en).

## FAQ

### Why is `choices` an empty array in the last chunk?

**Answer:** The gateway always turns on `include_usage`, and the upstream sends a separate chunk that carries only `usage` before the end; its `choices` is empty. Code that reads `choices[0]` directly fails, so check that the array is not empty first. This chunk cannot be turned off.

### Why did I get a JSON error instead of an event stream with `stream: true`?

**Answer:** The request was rejected before output started, for example because of an authentication failure, insufficient balance or an unavailable model, and the gateway returned the status code and error body as an ordinary response. Handle it by status code; see [Errors and request identifiers](https://platform.lyrouter.com/ai_gateway/docs/errors/en).

### Why does streaming output appear all at once after a long wait?

**Answer:** Usually a proxy in between is buffering the response. Compare by requesting LYRouter directly with `curl -N`; if the direct request works and the proxied one does not, turn off response buffering in the proxy. If the first piece of content takes a long time even on a direct request, check "TTFT" in the [request logs](https://platform.lyrouter.com/ai_gateway/docs/logs/en); see [Timeouts and interrupted streams](https://platform.lyrouter.com/ai_gateway/docs/stream-errors/en).

### Can I use the content I received if the end marker never arrived?

**Answer:** Not as a complete result. The gateway sends an error event when a stream fails midway, but when the upstream disconnects directly there may be no error event at all. Treat the call as failed, retry as needed, and deduplicate repeated content and tools that have already run.