Part 5 of 6 · 3 min read
Exercises
Easy
Easy 1: Size three models (20 minutes)
Using weights_gb, compute the weight memory for models of 3, 13 and 70 billion parameters at 16, 8 and 4 bits. Put the nine numbers in a table and mark which fit in 16 GB (ignoring everything else).
Done when: the table is correct and you note what the 70B model needs.
Hint
70 billion at 4 bits is about 35 GB, so it does not fit in 16 GB at any of the three precisions.
Easy 2: Context doubling (15 minutes)
Using kv_cache_gb, show by calculation that doubling the context doubles the KV cache, and that four simultaneous users quadruple it.
Done when: two printed comparisons match the claims.
Easy 3: Pick the error response (20 minutes)
For each LocalError kind, write one sentence of what your app should tell the user and whether it should retry.
Done when: all four kinds have a sentence and a yes or no.
Medium
Medium 1: Measure the stream (60 minutes)
Write a function that consumes stream() and returns time to first token, mean time between pieces and total time. Test it against LocalServer with different delay and token_delay values.
Done when: tests check each of the three numbers within a tolerance.
Hint
use time.monotonic() differences between consecutive pieces.
Medium 2: AdapterLocal (60 minutes)
Write AdapterLocal for module 7's layer using the imitation server, with the error mapping from section 3 (dropped counts as retryable). Put it in a Router behind a hosted imitation provider that fails with 503.
Done when: a test shows fallback from the stopped local server to the hosted one, and the reverse when the hosted one fails.
Hint
map not_loaded to bad_request so the router does not fall back on it, but think about whether your app would prefer that for a local model.
Medium 3: Memory planner (60 minutes)
Write a command that takes memory, parameters, layers, kv heads, head dim and a list of contexts, and prints the largest context that fits for each quantisation level.
Done when: it runs, and a test checks one case by hand.
Hint
solve for context in the KV formula, context = (memory − weights − overhead) ÷ per-token-bytes.
Hard
Hard 1: A real local run (120 minutes)
Install a local runtime and a small model of your choice on your own machine, point LocalClient (adjusting the path and fields to match the runtime's documentation) at it, and measure time to first token and tokens per second for three prompt lengths.
Done when: you have a table of three measurements and a note on how close the sizing formulas were to what you saw. (I did not run this one, so your numbers are the ground truth.)
Hard 2: Concurrency (120 minutes)
Make the imitation server serve only one request at a time (a lock). Send 10 requests from 10 threads and measure total time and per-request wait. Then add a client-side queue with a limit of 3 waiting requests that returns a clear error when full.
Done when: a test shows the queue limit, and you explain how this relates to the KV cache growing per simultaneous request.
Hint
threading.Lock inside the handler, and queue.Queue(maxsize=3) on the client.
Hard 3: Cost model (120 minutes)
Build a function that, given monthly requests, average tokens, API prices (module 6), hardware cost, and a quality penalty (extra retries needed with the local model), returns the cheaper option. Plot cost against volume for both options and show where they cross.
Done when: the chart is produced from your function and a test checks the crossing point for one set of numbers.