What a framework decides for you: Raw SDK first, framework later

What a framework decides for you

The risk of a framework is not that it is bad. It is that it decides things about your agent that you did not decide. To make that concrete, autoagent.py is a small invented framework, "AutoAgent", with defaults that are individually reasonable:

Default	What it does	A reasonable reason	The cost
`max_iterations = 25`	stops after 25 loop turns	"a safety net"	a safety net is not a budget: a stuck run is allowed to burn 25 turns
`tool_retries = 3`	retries a tool on any exception	"flaky networks"	repeats side effects whose first attempt actually worked
`tool_output_limit = 80`	keeps the first 80 characters of a tool result	"protect the context window"	cuts facts off silently
`swallow_tool_errors = True`	an exception becomes `(no output)`	"do not crash the user's app"	the model cannot tell "empty" from "failed"
`log_retries = False`	the trace shows one call even if three ran	"less noise"	the trace no longer matches reality
a hidden system prompt	29 words added to every request	"good behaviour out of the box"	tokens and behaviour you did not write and cannot see in your own code

AutoAgent.describe_defaults() prints the settings, and that is the point of the experiment: every one of these is findable if you look. Then the same ten tasks, same model, same tools:

What the defaults did

Task	Raw loop	AutoAgent (defaults)	What happened
lookup	pass	fail	the 80-character cut turned "Dana Okafor" into "Dana Ok", and the agent answered confidently with the wrong name
write	pass	fail	the team file was cut before the `Launch` line, so the model could not read the dates
flaky	pass	fail	three real tool runs, one line in the trace; and the same cut as `lookup` ruined the answer
stubborn	pass	fail	25 turns instead of 3, no repeat check, and the "answer" was the last raw observation, returned silently
too-long	pass	fail by my rubric	it was allowed 6 turns and did produce a full answer: this one is a difference in policy, not an error. My rubric asked for a clean stop at 5 steps, and I would not call AutoAgent wrong here
the other five	pass	pass

Be careful what you take from this. It is a toy I designed to hit its own defaults, so the score (10 against 5) says nothing about any real framework. What it shows is the kind of failure: three of the five misses (lookup, flaky, stubborn) are silent: no exception, no warning, a plausible-looking answer. One (write) is an honest "I could not read the field", and one (too-long) is a policy difference.

One more experiment, on side effects. A tool saves a note and then times out before it can reply, so the caller sees an error although the note exists:

Setup	Notes saved
no blind retries	1
`tool_retries = 3`	4
raw loop (the model is shown `Error: send_note failed (TimeoutError ...)`)	1, and the model answers "the tool reported a timeout; check the log before retrying"

A retry is only safe for a tool that is idempotent (module 7: give each effect a key), and only the code that knows the tool can decide that. A framework default cannot.

What this does not say

It does not say real frameworks truncate silently or retry everything: some may, some do not, and most let you configure it. It says find out. The questions below take ten minutes and save a bad week.

Five questions to ask any framework

Before you trust a framework with a real task, answer these from its documentation or by running a small test:

What is the hard stop? Does a run end on a step count, a recursion limit, a budget, a timeout? Which is the default, and what does the caller get back when it triggers: an exception, a partial result, or nothing?
What does it retry, and how many times? Does it retry tools with side effects? Does the trace show each attempt?
What does it cut or rewrite? Truncated tool output, summarised history, trimmed messages. Is it logged?
What does it add to my prompts? Is there a hidden system prompt or an injected format? Can you print exactly what the model was sent?
What does it do with errors? Are they shown to the model, swallowed, or raised? Can you tell "no result" from "failed"?

Tip: keep your ten tasks and run them on the framework

The ten tasks in corpus.py are not a demo; they are a regression suite. Include a task that needs more steps than the limit, a model that repeats itself, a flaky tool and a "not found" question. Run them on a framework before and after every upgrade. This is how you find out answers 1 to 5 without trusting anyone's README.

Raw or framework? A decision guide

Neither is better. They answer different questions.

Situation	Lean toward	Why
You are learning, or the agent is under about a hundred lines of logic	Raw loop	you will understand every failure, and the loop is short
One agent, a few tools, strict control over limits, retries and logging	Raw loop (maybe on a thin SDK)	the framework would mostly add things you want to configure away
Long-running work that must pause, wait for a person, or survive restarts	A framework with checkpoints, or a workflow engine	durable state is hard to write well, and this is where graphs earn their keep
Many agents with handoffs, or a team of roles	A framework (Level 8)	the coordination code is large and common
You need several providers behind one interface	A thin wrapper or a framework	a wrapper exists for exactly this
Your team already knows one framework	That framework, with your regression tasks	shared vocabulary is worth a lot
Regulated or audited setting	Whichever lets you prove what ran	usually means raw control over logging, or a framework whose trace you have verified

A useful default: start raw, add a framework when you can name the specific thing it will do for you (durable state, human approval, handoffs, a tracing integration), not when you feel the loop is "too basic". The loop in this module is 74 lines. Most of an agent's complexity is in the tools, the prompts and the evaluation, not in the loop.