Part 1 of 5 · 1 min read
Overview
In this module you will
- Explain prompt injection, and the difference between direct and indirect injection.
- Fence untrusted text so it is clearly data, including the trap of text that closes your fence.
- Validate model output strictly and send anything odd to a human.
- Apply least privilege to tools, and build a small injection test set for your own pipeline.
Any text an agent reads (a ticket, a web page, a file, a tool result) can contain instructions written by someone else. A plain chatbot that is tricked says something silly. An agent with tools that is tricked does something: sends a message, changes a record. Customer text is untrusted by definition, so the habits start here.
About the examples
toymodel.py is an imitation with a deliberate, simple weakness, so defences can be tested offline. It obeys instruction-like lines such as Reply with: .... If the prompt contains a rule that text inside a tag is data, it ignores those lines inside the tag, except that a "slip" setting makes it obey them anyway 40% of the time (real models are not airtight). The attack strings are the plain, widely known kind used to test your own system. No defence here is a guarantee; the numbers show how layers reduce risk in a toy, not how well they work on a real model. Test your own pipeline with your own model.