Skip to content
← Changelog
2.2.135
Fix

Fail over from an unreachable provider in seconds

A provider that never completed the connection used to hold your request for the attempt's whole wait; now the request moves to the next provider in seconds, and a stream that sets no timeout, to a model with one provider, is no longer cut at a fixed wait.

We changed how a request waits on a provider that does not answer. When the first provider never completes the connection, the request now moves to the next provider in seconds, where it used to wait out the attempt's whole wait. And a streamed request that sets no timeout of its own, to a model that has only one provider, is no longer cut at a fixed wait: it waits for that provider up to the request's deadline, so a slow start is served instead of failed.

What you see when a provider stalls

What the first provider doesWhat you see
Never completes the connection, because the host does not answerThe next provider serves the request. The wait is a short cap on connecting, not the transport's default.
Accepts the connection but never completes the TLS handshakeThe same: the next provider serves it.
Accepts a streamed request and never sends the start of the answer, and another provider can serve the modelThe first provider is cut at its own first wait, which is shorter than the request's deadline, and the next provider streams the answer. On /v1/messages, an X-Timeout above that wait holds the first provider for what you set.
Answers with an error statusUnchanged: the request is failed over as before.
Sends the start of a stream in time, then streams for a long whileNot cut by any of these waits, which end when the stream starts.
Is the model's only provider and is slow to start a stream, and the request sets no timeout of its ownThe request waits for it up to the request's deadline, not a fixed wait. If it starts in time, you get the stream.
Terminal: a streamed chat completion whose first provider stays silent, answered by the next provider, then the 408 a model with one silent provider answers at the request's deadline

On a stream, the providers after the first also get a shorter wait, so several silent providers in a row cannot spend the whole request. A request that is not a stream is not cut at that shorter wait, on /v1/chat/completions or on /v1/messages: its answer starts only when the generation ends.

The request's deadline is the X-Timeout you send, as the documentation describes, and a default when you send none.

When the only provider stays silent

A streamed request that sets no timeout of its own, to a model with one provider, and that provider sends nothing until the request's deadline, ends with a 408. It used to end earlier, at the fixed wait, with a server error. On /v1/chat/completions the body is:

{"error":{"code":"request_timeout","message":"Total request timeout (<ms>ms) exceeded after 1 channel attempts. Set X-Timeout header to adjust.","type":"timeout"}}

On /v1/messages the same 408 comes in Anthropic's error format:

{"error":{"message":"Total request timeout (<ms>ms) exceeded after 1 channel attempts. Set X-Timeout header to adjust.","type":"timeout_error"},"type":"error"}

<ms> is the request's deadline in milliseconds.

Terminal: the 408 on chat completions and on /v1/messages when a model's only provider sends nothing until the request's deadline