Streaming

Stream Chat Completions over server-sent events, read chunk and tool-call deltas, request usage data, and handle incomplete streams and fallback behavior.

What is streaming in OpenGateway?#

Streaming returns Chat Completions chunks as server-sent events (SSE) while the model is generating a response. The gateway maps supported providers to the Chat Completions chunk shape so a compatible client can process one stream format.

Enable streaming#

Set stream: true on a POST /v1/chat/completions request. The model must list chat_completions in its endpoints field in GET /v1/models.

Bash

curl https://apis.opengateway.ai/v1/chat/completions \
  -H "Authorization: Bearer $OPENGATEWAY_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "deepseek/deepseek-v4.1-flash-ultrafast",
    "messages": [{"role": "user", "content": "Count to five slowly."}],
    "stream": true
  }'

Read the stream#

Each data: line contains a JSON chunk. Text arrives in choices[].delta.content; the final choice chunk includes a finish_reason. The stream ends with data: [DONE].

data: {"id":"chatcmpl_example","object":"chat.completion.chunk","choices":[{"index":0,"delta":{"content":"One"},"finish_reason":null}]}

data: {"id":"chatcmpl_example","object":"chat.completion.chunk","choices":[{"index":0,"delta":{},"finish_reason":"stop"}]}

data: [DONE]

To include token usage, set stream_options.include_usage to true. Otherwise the gateway omits the usage-only final chunk.

OpenAI SDK for Python#

Python

from openai import OpenAI

client = OpenAI(
    api_key="YOUR_OPENGATEWAY_API_KEY",
    base_url="https://apis.opengateway.ai/v1",
)

stream = client.chat.completions.create(
    model="deepseek/deepseek-v4.1-flash-ultrafast",
    messages=[{"role": "user", "content": "Count to five slowly."}],
    stream=True,
    stream_options={"include_usage": True},
)

for chunk in stream:
    if chunk.choices:
        text = chunk.choices[0].delta.content or ""
        print(text, end="", flush=True)

Tool-call chunks#

When a model calls a function, chunks contain choices[].delta.tool_calls. Function names and JSON arguments can arrive across multiple chunks; accumulate each call by its index before parsing the complete arguments. See tool calling for the request and response cycle.

Fallbacks and incomplete streams#

OpenGateway can try another configured route before it has sent the first stream frame. After a frame has been sent, it does not switch providers mid-response. If a stream ends unexpectedly, treat the response as incomplete rather than assuming the model finished successfully.