Chat completions

Impala serves an OpenAI-compatible /chat/completions endpoint. The change is the base URL, the key, and the model name — see Quickstart for those three values.

The request

model and messages are the only required fields.

resp = client.chat.completions.create(
    model="<MODEL>",
    messages=[{"role": "user", "content": "Summarize this ticket in one line."}],
)
print(resp.choices[0].message.content)

messages is the standard OpenAI chat format. A system message sets behavior, user carries the prompt, and assistant carries prior model turns when you’re replaying a conversation:

messages=[
    {"role": "system", "content": "You are a concise assistant."},
    {"role": "user", "content": "What changed in this diff?"},
    {"role": "assistant", "content": "Two functions were renamed."},
    {"role": "user", "content": "Which ones?"},
]

base_url routes your requests to Impala. Pass it exactly as provided — endpoints differ between accounts. model names the model your endpoint is provisioned for.

Timeouts and retries

Two settings matter more here than at an interactive provider. Raise your timeouts. A call takes seconds, and minutes under load. SDK defaults are tuned for interactive providers and will abort a request that would have succeeded. Reduce automatic retries. The SDK can’t tell a slow request from a failed one, so its default retries fire on timeouts for requests that are still running — adding load without shortening your wait. Lower the count and raise the timeout instead. Set it to 0 if your own harness handles retries.

client = OpenAI(
    base_url="<BASE_URL>",
    api_key="<API_KEY>",
    timeout=600.0,
    max_retries=1,
)

For an agent, total time is per-step time multiplied by the number of steps, so turn count is usually a bigger lever than any individual call. See Run async and open source.

Keep the key out of your source

The OpenAI SDK reads these environment variables automatically:

export OPENAI_BASE_URL="<BASE_URL>"
export OPENAI_API_KEY="<API_KEY>"

Which lets your code drop the explicit arguments:

from openai import OpenAI

client = OpenAI()  # reads OPENAI_BASE_URL and OPENAI_API_KEY

resp = client.chat.completions.create(
    model="<MODEL>",
    messages=[{"role": "user", "content": "Ping"}],
)

Don’t commit your key or share it outside your team. You can rotate it under API Keys on the platform.

Other parameters

The endpoint accepts the standard OpenAI chat completion fields. Sampling parameters in the request body, such as temperature, top_p, top_k, min_p and max_tokens, are passed through to the inference engine as sent. In a batch, put them in the body of each JSONL line.

Reasoning

Some models are provisioned as two jobs, one with reasoning and one without. Others, such as Qwen3.8, serve both from one job, with thinking on by default. For those, control it per request through chat_template_kwargs. To turn thinking off:

{"chat_template_kwargs": {"enable_thinking": false}}

To set how much the model reasons, pass reasoning_effort. Supported values are xhigh (the default), medium and low. Any other value, including high, returns an error.

{"chat_template_kwargs": {"reasoning_effort": "medium"}}

With the OpenAI SDK, pass chat_template_kwargs through extra_body. In a batch, add it to the body of each JSONL line.

Run a batch

File in, results out, at the lowest cost per token.

Anthropic SDKs and Claude Code

The same endpoint in Anthropic format.

Introduction

Serverless inference

Batch inference

Agent orchestration

Optimization

Bring your own model

Bring your own cloud (BYOC)

Integrations

Reference

API reference

The request

Timeouts and retries

Keep the key out of your source

Other parameters

Reasoning

Next

Run a batch

Anthropic SDKs and Claude Code

​The request

​Timeouts and retries

​Keep the key out of your source

​Other parameters

​Reasoning

​Next

Run a batch

Anthropic SDKs and Claude Code

The request

Timeouts and retries

Keep the key out of your source

Other parameters

Reasoning

Next