CSE Roadmap

Common tier · about 3 hours · 12 parts

Part 8 of 12 · 2 min read

Answering pricing questions in an interview

How do you keep cost per resolved task under control?

"Measure it per task from traces, then work the biggest lever. In my support model failures and resolution rate mattered more than token price: raising resolution from 35 to 65 percent cut vendor cost per resolved task from $1.50 to $0.12. Model-side levers (caching, routing with a verifier, per-run caps) matter most where tokens dominate, such as a research agent at about $4.7 per run with 77 percent model cost. I report cost and quality together." Follow-up: "What if quality drops when you cut cost?" The eval gate and canary catch it (Level 11 module 8); cost per resolved task already includes the effect of lower resolution.

How would you price an agent product?

"Floor from cost per task at my target margin, ceiling from the customer's value, then choose a structure by who should bear which risk. Per seat is simple but heavy users can sink it; per task tracks cost; per resolved task aligns incentives but needs a clear definition of resolved; a platform fee with included usage and overage gives predictability. I would simulate my customer distribution before committing." Follow-up: "What about competitors?" They set the market range; my unit economics tell me whether I can survive at that price.

Why might a profitable-looking agent lose money?

"Heavy-tail customers on flat pricing, failures that cost credits or support, costs that scale with steps and retries, and prices set before measuring real token use. I track margin per customer, not only the blended margin." Follow-up: "How do you find it?" Join usage, cost and revenue per customer from traces and billing.

Should you charge for outcomes?

"It aligns incentives and customers like it, but the vendor takes the failure risk, so I need a measurable definition of resolved, protection against gaming, and a cost model that includes unresolved attempts. In the example's simulation the per-resolved price gave customers a uniform price but still left 5.5 percent of customers unprofitable because hard tasks cost more and resolve less." Follow-up: "Who decides if it resolved?" Customer confirmation, absence of reopen within a window, or a verifier; pick one and audit it.