Neutral types and adapters: A thin provider layer

The types your own code speaks, and the small class that translates them to one provider's format and back.

Neutral types and adapters

Neutral request and response

The types your application speaks. They contain only what you need, in your naming, and know nothing about any provider.

@dataclass
class Request:  messages: list[dict]; system: str = ""; max_tokens: int = 100; model: str = ""
@dataclass
class Response: text: str; input_tokens: int; output_tokens: int; stop: str   # "done" | "length" | "other"

The neutral request leaves out things on purpose: there is no system role in messages (the system prompt is its own field), and the stop reason is normalised. Choosing the neutral shape is the design work. It is what the rest of the app depends on.

Keep it thin

Support only what you use: text in, text out, usage, stop reason, errors. Add tool calls when module 4's loop needs them. Every feature you add is one more thing to translate for every provider, and the lowest common denominator is a feature, not a bug.

Adapter

A small class with one method, complete(request) -> response, that translates your neutral request into one provider's format, makes the call, and translates the reply and any error back. (The name comes from the physical kind: a travel plug adapter changes the shape, not the electricity.)

The same neutral request becomes two different payloads:

One neutral request, two payloads. Real output of the example code; the red lines show where the system prompt lives.

class AdapterB:
    def complete(self, req):
        payload = {"model": ..., "messages": req.messages, "max_tokens": req.max_tokens}
        if req.system:
            payload["system"] = req.system            # provider B wants it at the top level
        raw = self.backend.post(payload)
        text = "".join(b["text"] for b in raw["content"] if b["type"] == "text")
        return Response(text, raw["usage"]["input_tokens"], raw["usage"]["output_tokens"], ...)

Things an adapter must do, and that the tests check:

Translate the system prompt to wherever the provider wants it.
Fill in required fields. Provider B refuses a call without max_tokens, so the adapter always sends one.
Normalise the reply. Pull the text out of blocks or choices, rename the usage fields, and map each provider's "ran out of tokens" value to "length".
Never mutate the caller's message list.
Do not hide the unknown. A finish reason the adapter does not recognise becomes "other", with the provider's original reply kept in raw.

For the curious Three bugs the first version had

1. An unknown stop reason looked like success. The adapter mapped "length" to "length" and everything else to "done". A reply cut off by a content filter (finish_reason: "content_filter") was reported as a normal finish, so the app would have used an empty or partial answer as complete. Now it is "other".

2. A network failure bypassed the fallback. Only the provider's own error responses were converted to ProviderError. A dropped connection raised ConnectionError straight through the router, so the second provider was never tried, which is exactly the case a fallback is for. Timeouts and connection errors are now converted to a retryable server error.

3. The neutral request let one provider's habit through. A message with the role system worked on provider A (where the system prompt is a message) and failed on provider B. Code written against A would break on the day B took over. Request now refuses unknown roles, empty message lists, a last message that is not from the user, and max_tokens below 1, with the same error for every provider. The cost: this thin layer also refuses a trailing assistant message (a "prefill"), which some providers allow.

Check yourself

Why does the neutral Request have no system role in messages? (Providers disagree about where the system prompt goes. One neutral place means each adapter decides how to translate it, and the rest of the app never has to know.)

One error type

Each adapter catches its provider's failure and raises the same exception, ProviderError, with a kind (rate_limit, server, bad_request, auth, other) and a retryable flag. The retry loop from module 5 then needs one rule instead of one per provider.

What happened	kind	Retryable?
429	rate_limit	yes
408, 5xx, or no reply at all (timeout, dropped connection)	server	yes
401, 403	auth	no (but see the router)
400, 404, 422	bad_request	no
anything else	other	no

Check yourself

A request times out and no HTTP status exists. Which kind does the example use, and why? (server. The call may succeed later or on another provider, so it is treated as retryable, like a 5xx.)