Part 3 of 13 · 2 min read
Parse the URL and compare the exact host; never search the raw string for a trusted name
A trusted domain name appearing somewhere in a URL string says nothing about which server will actually receive the request.
Think of checking whether a package is addressed to your street by searching the label for your street name anywhere on it. That search would also match the "care of" line or a forged return address. The package still ends up wherever the actual delivery address on the label says. It does not matter what other names appear on it.
egress is outbound network traffic, and an allowlist permits only listed hosts, programs or arguments. So restricting which URLs an agent can fetch means parsing the URL properly and comparing the exact hostname against the allowlist. That is because a string check that merely looks for an allowed domain's name anywhere in the URL can be defeated by putting that name in a query parameter, a userinfo field, or a longer hostname that merely starts with it.
A worked example. Against 10 hand-written URL tricks, checking whether "good.test" appears anywhere in the string wrongly allows 8 of them (80%) -- including https://evil.test/?u=https://api.good.test, where the trusted name sits harmlessly inside a query parameter while the request actually goes to evil.test. A string-prefix check allows 3 of 10 bad URLs (30%) and, worse, wrongly blocks 2 of 3 legitimate URLs. Parsing the URL and comparing the exact host against an allowlist -- while also rejecting IP literals, non-standard ports, and userinfo (user@host tricks) -- allows 0 of the 10 bad URLs and all 3 legitimate ones.
The key lines. From code/mini-sandbox/sandbox.py:
# from code/mini-sandbox/sandbox.py
def url_allowed(url, allowed=ALLOWED_HOSTS):
try:
u = urlsplit(url)
host = u.hostname
port = u.port
except ValueError:
return False
if u.scheme != "https" or not host or u.username or u.password:
return False
if port not in (None, 443):
return False
Read it as: the function uses urlsplit to extract hostname as a distinct, structured field. Then it rejects outright if username or password are present at all. urlsplit already did the hard work of figuring out which part of the string is actually the host the connection will be made to. The comparison that follows is against that one specific field. It is never against the URL's full text, where a trusted name could be hiding anywhere.
Egress, allowlist
Egress is outbound network traffic. An allowlist permits only listed hosts, programs or arguments.
Example: https://[email protected]/ tricking the string-prefix check (it does start with https://api.good.test) while the parsed-host check correctly extracts evil.test as the actual destination is the entire argument for parsing over searching, in one URL.