Part 3 of 6 · 2 min read
Talking to a local endpoint
A small client that handles timeouts, dropped replies, readiness and streaming.
Many local runtimes expose an HTTP server with an endpoint for chat requests, often modelled on the hosted chat APIs. Check your runtime's documentation for the exact path and fields. The example uses simplified ones. The client (client.py) uses only the standard library and does four things worth copying.
1. One error type with useful kinds.
| kind | meaning | what to do |
|---|---|---|
unreachable |
nothing is listening | tell the user to start the server |
timeout |
the model is too slow, including a stall in the middle of a reply | retry or use a smaller model |
dropped |
the reply was cut off or unreadable | retry |
not_loaded |
wrong model name | fail fast, fix the name |
loading |
the model is still starting | wait and retry |
2. A timeout on every request. A local model on a busy laptop can be very slow. Without a timeout the app freezes (module 5). Note that the timeout applies to each wait for data, not to the whole reply.
3. A readiness check. wait_until_ready retries only on loading and unreachable. A real error such as not_loaded stops immediately, because waiting will not help.
4. Streaming. stream() yields (seconds, piece) pairs, so you can measure:
for t, piece in client.stream(messages):
... # the first t is your time to first token
On the imitation server with 20 ms between pieces, the pieces arrive at 0.00, 0.02, 0.04, 0.06 and 0.08 seconds. A real local model usually has a noticeable wait for the first piece (the prompt is processed first) and then a steady rate. Both numbers depend on your hardware.
For the curious A bug the first version had
The client turned failures into LocalError only while opening the connection. Once the headers had arrived, two things escaped as raw Python exceptions: a model that stalled in the middle of a stream (a bare TimeoutError), and a server that died after sending half a reply (IncompleteRead). A caller that handled LocalError("timeout"), as the notes promised, would have crashed. Both are now converted, the second under the new dropped kind, and two tests reproduce them (one with a stalled stream, one with a hand-made socket that closes early). Lesson: errors can happen at any point of a request, not only at the start.