Tool calls

Tool calling (function calling) lets the model request a call to a function you define, with structured arguments: your application declares tools in the request, the model returns the function name and arguments to call, your application runs the function itself and sends the result back, and the model gives the final answer. LYRouter only forwards requests and converts between protocols; it never runs any of your business operations. Chat Completions, Responses and Anthropic Messages structure tools differently, and the gateway converts automatically between Chat Completions and the Anthropic protocol. This chapter uses Chat Completions for the complete examples and explains what happens during conversion.

How it works

1. Define functions in the request's tools: name, description of purpose, and the JSON Schema of the parameters.
2. When the model decides to call a tool, the response's message.tool_calls gives each call's id, function name and arguments (a JSON string), and finish_reason is usually tool_calls.
3. Your application parses the arguments, validates them and runs the function.
4. Append the assistant message returned by the model to messages unchanged, then append one role: tool message for each call, with tool_call_id set to that call's id.
5. Send the request again. The model answers based on the tool results, or calls more tools, until the response no longer contains tool_calls.

To decide whether to run tools, check whether the response contains tool_calls (tool_use blocks in Anthropic Messages) rather than relying only on finish_reason (stop_reason): for example, when tool_choice specifies a function, Alibaba Cloud Model Studio may return tool_calls with finish_reason stop; and through Anthropic Messages, specifying a tool with {"type": "tool"} currently returns a tool_use block but with stop_reason end_turn (any and auto behave normally). The example code below checks message.tool_calls.

Defining tools

{
  "type": "function",
  "function": {
    "name": "get_weather",
    "description": "Get the current weather for a city. Call this when the user asks about weather or temperature.",
    "parameters": {
      "type": "object",
      "properties": {
        "city": { "type": "string", "description": "City name, for example Beijing" },
        "unit": { "type": "string", "enum": ["celsius", "fahrenheit"] }
      },
      "required": ["city"]
    }
  }
}

The model must support tool calling; check this in the model's details in the model marketplace.

A complete two-turn example

Turn 1: send the question and tool definition

curl https://api.lyrouter.com/v1/chat/completions \
  -H "Authorization: Bearer $LYROUTER_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "MODEL_ID",
    "messages": [
      {"role": "user", "content": "What is the weather like in Beijing right now?"}
    ],
    "tools": [{
      "type": "function",
      "function": {
        "name": "get_weather",
        "description": "Get the current weather for a city.",
        "parameters": {
          "type": "object",
          "properties": {"city": {"type": "string"}},
          "required": ["city"]
        }
      }
    }]
  }'

The model decides to call the tool and returns:

{
  "id": "chatcmpl-5d1e8b20a3",
  "object": "chat.completion",
  "created": 1791950400,
  "model": "MODEL_ID",
  "choices": [
    {
      "index": 0,
      "message": {
        "role": "assistant",
        "content": null,
        "tool_calls": [
          {
            "id": "call_8f2a1c",
            "type": "function",
            "function": { "name": "get_weather", "arguments": "{\"city\":\"Beijing\"}" }
          }
        ]
      },
      "finish_reason": "tool_calls"
    }
  ],
  "usage": { "prompt_tokens": 96, "completion_tokens": 18, "total_tokens": 114 }
}

arguments is a JSON string that you need to parse yourself. content may be null or an empty string, or it may contain some explanatory text.

Turn 2: send back the tool result

Put the previous turn's assistant message back unchanged, followed by a role: tool message. Keep tools the same as in turn 1:

curl https://api.lyrouter.com/v1/chat/completions \
  -H "Authorization: Bearer $LYROUTER_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "MODEL_ID",
    "messages": [
      {"role": "user", "content": "What is the weather like in Beijing right now?"},
      {"role": "assistant", "content": null, "tool_calls": [
        {"id": "call_8f2a1c", "type": "function",
         "function": {"name": "get_weather", "arguments": "{\"city\":\"Beijing\"}"}}
      ]},
      {"role": "tool", "tool_call_id": "call_8f2a1c",
       "content": "{\"city\":\"Beijing\",\"condition\":\"Sunny\",\"temperature_c\":22}"}
    ],
    "tools": [{
      "type": "function",
      "function": {
        "name": "get_weather",
        "description": "Get the current weather for a city.",
        "parameters": {
          "type": "object",
          "properties": {"city": {"type": "string"}},
          "required": ["city"]
        }
      }
    }]
  }'

The model answers based on the tool result:

{
  "id": "chatcmpl-5d1e8b20b7",
  "object": "chat.completion",
  "created": 1791950403,
  "model": "MODEL_ID",
  "choices": [
    {
      "index": 0,
      "message": { "role": "assistant", "content": "It is sunny in Beijing right now, at 22°C." },
      "finish_reason": "stop"
    }
  ],
  "usage": { "prompt_tokens": 142, "completion_tokens": 14, "total_tokens": 156 }
}

When several tool_calls are returned at once, append one role: tool message for each call after the assistant message, then send the next request.

A minimal runnable loop

The program below turns the two turns above into a loop: it runs the tools the model requests until the model gives a final answer, for at most 5 turns. To install the SDK and set environment variables, see Your first model call.

import json
import os
from openai import OpenAI

client = OpenAI(
    api_key=os.environ["LYROUTER_API_KEY"],
    base_url="https://api.lyrouter.com/v1",
)

tools = [
    {
        "type": "function",
        "function": {
            "name": "get_weather",
            "description": "Get the current weather for a city.",
            "parameters": {
                "type": "object",
                "properties": {"city": {"type": "string"}},
                "required": ["city"],
            },
        },
    }
]


def get_weather(city):
    # Replace with a real weather lookup
    return {"city": city, "condition": "Sunny", "temperature_c": 22}


TOOL_FUNCTIONS = {"get_weather": get_weather}

messages = [{"role": "user", "content": "What is the weather like in Beijing right now?"}]

for _ in range(5):  # Limit the number of turns
    response = client.chat.completions.create(
        model="MODEL_ID", messages=messages, tools=tools
    )
    message = response.choices[0].message
    if not message.tool_calls:
        print(message.content)
        break

    # Append the assistant message unchanged (including tool_calls)
    messages.append(message.model_dump(exclude_none=True))
    for call in message.tool_calls:
        func = TOOL_FUNCTIONS.get(call.function.name)
        try:
            args = json.loads(call.function.arguments or "{}")
            result = func(**args) if func else {"error": "unknown tool"}
        except Exception as exc:  # Invalid arguments or failed execution: tell the model about the error
            result = {"error": str(exc)}
        messages.append(
            {
                "role": "tool",
                "tool_call_id": call.id,
                "content": json.dumps(result, ensure_ascii=False),
            }
        )

import OpenAI from "openai";

const client = new OpenAI({
  apiKey: process.env.LYROUTER_API_KEY,
  baseURL: "https://api.lyrouter.com/v1",
});

const tools = [
  {
    type: "function",
    function: {
      name: "get_weather",
      description: "Get the current weather for a city.",
      parameters: {
        type: "object",
        properties: { city: { type: "string" } },
        required: ["city"],
      },
    },
  },
];

function getWeather({ city }) {
  // Replace with a real weather lookup
  return { city, condition: "Sunny", temperature_c: 22 };
}

const toolFunctions = { get_weather: getWeather };
const messages = [{ role: "user", content: "What is the weather like in Beijing right now?" }];

for (let turn = 0; turn < 5; turn++) {
  const response = await client.chat.completions.create({
    model: "MODEL_ID",
    messages,
    tools,
  });
  const message = response.choices[0].message;
  if (!message.tool_calls?.length) {
    console.log(message.content);
    break;
  }

  messages.push(message); // Append the assistant message unchanged (including tool_calls)
  for (const call of message.tool_calls) {
    const fn = toolFunctions[call.function.name];
    let result;
    try {
      const args = JSON.parse(call.function.arguments || "{}");
      result = fn ? fn(args) : { error: "unknown tool" };
    } catch (err) {
      result = { error: String(err) }; // Invalid arguments or failed execution: tell the model about the error
    }
    messages.push({
      role: "tool",
      tool_call_id: call.id,
      content: JSON.stringify(result),
    });
  }
}

tool_choice and parallel calls

• parallel_tool_calls: false asks for at most one call at a time. It is passed through to OpenAI-compatible channels, and whether it takes effect depends on the model; it is removed for Anthropic-protocol channels, where the model may still return several calls at once. Always handle tool_calls as an array in your application.
• When a call is forced ("required" or a specified function) and the request has no enable_thinking / thinking_budget, Alibaba Cloud Model Studio channels automatically get enable_thinking: false, because these upstreams do not accept forced tool calls in thinking mode.

Assembling tool arguments when streaming

In a streaming request, a tool call arrives across several chunks: the first fragment carries the id, type and function name, and later fragments carry only a piece of arguments.

data: {"id":"chatcmpl-5d1e8b20a3","object":"chat.completion.chunk","created":1791950400,"model":"MODEL_ID","choices":[{"index":0,"delta":{"role":"assistant","content":null,"tool_calls":[{"index":0,"id":"call_8f2a1c","type":"function","function":{"name":"get_weather","arguments":""}}]},"finish_reason":null}]}

data: {"id":"chatcmpl-5d1e8b20a3","object":"chat.completion.chunk","created":1791950400,"model":"MODEL_ID","choices":[{"index":0,"delta":{"tool_calls":[{"index":0,"function":{"arguments":"{\"city\":"}}]},"finish_reason":null}]}

data: {"id":"chatcmpl-5d1e8b20a3","object":"chat.completion.chunk","created":1791950400,"model":"MODEL_ID","choices":[{"index":0,"delta":{"tool_calls":[{"index":0,"function":{"arguments":"\"Beijing\"}"}}]},"finish_reason":null}]}

data: {"id":"chatcmpl-5d1e8b20a3","object":"chat.completion.chunk","created":1791950400,"model":"MODEL_ID","choices":[{"index":0,"delta":{},"finish_reason":"tool_calls"}]}

data: {"id":"chatcmpl-5d1e8b20a3","object":"chat.completion.chunk","created":1791950400,"model":"MODEL_ID","choices":[],"usage":{"prompt_tokens":96,"completion_tokens":18,"total_tokens":114}}

data: [DONE]

• Group fragments by tool_calls[].index, and join the arguments that share an index in order of arrival to get the complete string.
• Take id and the function name from the first non-empty value. Some upstreams repeat id and name as empty strings in later fragments; do not overwrite earlier values with empty ones.
• Parse the arguments JSON only after you receive the chunk with finish_reason or the stream ends; the fragments in between are not valid JSON. finish_reason is not necessarily "tool_calls"; see "How it works" above.
• In Anthropic Messages the equivalents are content_block_start (a tool_use block with id and name) and several input_json_delta events; join partial_json by the block's index. In Responses they are response.output_item.added (a function_call item) and response.function_call_arguments.delta, with response.function_call_arguments.done giving the complete arguments. For event formats, see Streaming.

Structure across the three protocols

A call and its returned result in Responses:

{"type": "function_call", "id": "fc_68f1b0a1", "call_id": "call_8f2a1c", "name": "get_weather", "arguments": "{\"city\":\"Beijing\"}", "status": "completed"}
{"type": "function_call_output", "call_id": "call_8f2a1c", "output": "{\"condition\":\"Sunny\",\"temperature_c\":22}"}

A call and its returned result in Anthropic Messages:

{"role": "assistant", "content": [{"type": "tool_use", "id": "toolu_01A09q90qw90lq917835lq9", "name": "get_weather", "input": {"city": "Beijing"}}]}
{"role": "user", "content": [{"type": "tool_result", "tool_use_id": "toolu_01A09q90qw90lq917835lq9", "content": "{\"condition\":\"Sunny\",\"temperature_c\":22}"}]}

The gateway forwards Responses requests unchanged and does not convert their tool structure. When you continue with previous_response_id, the gateway uses it only to identify the session, and the continuation request is not necessarily sent to the upstream that created the previous response; see Responses.

Cross-protocol conversion

When a Chat Completions request is served by an Anthropic-protocol channel, or an Anthropic Messages request is served by an OpenAI-compatible channel, the gateway converts the tool-related structures:

Before sending a request, the gateway also applies the following:

• Fixed JSON key order: the gateway reorders the keys of every JSON object in the request body sent upstream into a fixed order (array order is unchanged), and does the same for the JSON inside the tool_calls[].function.arguments strings in history messages and the arguments strings of function_call items in a Responses input. When the client writes the same argument object with a different key order on each turn, the upstream still sees the same bytes, so the prompt cache is not invalidated by it. Arguments that are not valid JSON objects, or that contain 15 or more consecutive digits, are left as they are. This affects only the request sent upstream, not the content returned to you.
• Number precision: when the gateway parses and re-encodes JSON, numbers keep at most 14 significant digits. The input objects of Anthropic Messages, and tool arguments during cross-protocol conversion, are all parsed, so long integers such as 19-digit order numbers or Snowflake IDs become scientific notation; define them as string in the parameter schema.
• Thinking content: before sending to OpenAI-compatible channels, the gateway removes reasoning_content from history assistant messages. The DeepSeek family is the exception: its thinking mode requires the current turn's reasoning to be passed back, so non-empty reasoning_content on assistant messages after the last user message is kept; if DeepSeek still returns a 400 because it is missing, the gateway fills in an empty value and retries once on the same channel. When an Anthropic-protocol channel with thinking on serves the request, the message of a non-streaming response carries reasoning_blocks (a LYRouter extension field); put the assistant message back unchanged so the gateway can hand the signed thinking blocks back to the upstream.

Notes

• Send tool results and extra instructions separately: when calling through Anthropic Messages, put the tool_result block and any additional text for the model in two separate messages rather than the same user message, so that both are kept intact when converted for an OpenAI-compatible channel.
• Report failures too: when a tool fails, write the error into the result content (for example {"error": "City not found"}) so the model can adjust; the is_error flag of tool_result is not kept when converting for an OpenAI-compatible channel.
• Validate before running: arguments generated by the model may be incomplete or invalid. Validate them against the schema before running the function, and check that the caller is allowed to perform the operation. For tools with side effects (placing orders, sending email, deleting data), ask the user to confirm first, and carry an idempotency key with each operation so that a retry after an interrupted stream does not run it twice.
• Limit the number of turns: set a maximum number of turns and an overall timeout for the tool loop, so the model cannot keep calling tools indefinitely.
• Keep tool definitions stable: within one session, do not add, remove, reorder or modify tools; otherwise no later prompt cache can hit. See Reducing costs and improving cache reuse.
• Logs keep tool calls: tool calls returned by the model (function names and arguments) are stored even when content logging is off, so do not put passwords or other sensitive information in arguments; see Logs and data processing.

FAQ

Why did the model answer directly without calling a tool?

Answer: First confirm that the selected model supports tool calling, then check whether the tool's description says when it should be called. To make sure a tool is called, set tool_choice to "required" or to a specific function.

Why does the upstream return 400 when I force a tool call?

Answer: Some models do not accept forced tool calls in thinking mode; for example, Alibaba Cloud Model Studio returns The tool_choice parameter does not support being set to required or object in thinking mode. When you do not set thinking parameters, the gateway turns off thinking automatically for Alibaba Cloud Model Studio channels; if your request sets enable_thinking: true or thinking_budget itself, remove it, or change tool_choice to "auto".

Why does the turn 2 request fail with an error about a missing tool call ID or wrong message order?

Answer: Check three things: whether the assistant message was put back unchanged with its tool_calls; whether every call has a role: tool message whose tool_call_id matches the call's id; and whether the role: tool messages directly follow their assistant message. The error text comes from the upstream; view it under "Upstream response" in the "Raw packets" tab of the request log details.

How do I see tool calls in the logs?

Answer: Open the "Conversation" tab of the request log details. "Tool calls" lists the tools and tool_choice defined in the request, and the function name and arguments of each call the model returned. To confirm which provider the request was routed to and whether protocol conversion happened, check "Provider" and "Upstream request" under "Raw packets".