Where chunking sits: Parsing and chunking

Where chunking sits

Retrieval-augmented generation (RAG)

Answering a question by first retrieving relevant pieces of text from your own collection, then putting them into the prompt so the model can answer from them. It lets a model use facts that were not in its training data and that you can update without retraining.

Chunk

One retrievable piece of a document: the text that gets indexed, plus the facts about where it came from (source, heading path, page). The retriever returns chunks, and the prompt is built from chunks.

The ingestion pipeline. Everything before "Index" happens once per document.

Two cuts are made on the way. Parsing decides what the text is (what is a heading, what is navigation noise). Chunking decides where the pieces end. Both change what the retriever can find.

Parsing: decide what the text is

Real documents carry things you do not want to index.

Source	Typical noise	What to keep
Web page (HTML)	menus, scripts, cookie banners, footers	headings, paragraphs, list items, table rows
PDF text extraction	page headers and footers, line-end hyphens, lines wrapped mid-sentence, columns merged	the body text in reading order
Markdown or code	front matter, badges, generated sections	headings, prose, code blocks as their own units

The example parser uses only the Python standard library:

blocks = parse_html(MANUAL_HTML)
# Block(kind='heading', text='Brightline R7 router', level=1)
# Block(kind='para', text='The R7 is a dual-band home router. ...', level=0)
# Block(kind='row', text='Blinking amber | No internet signal | Check the WAN cable, ...')

Three decisions are worth noticing:

Headings are kept as headings, with their level. Later they become a path ("Troubleshooting > Factory reset") that gives a chunk its context.
A table row stays together as one block, with its cells joined by |. If a table is flattened into one cell per line, "Blinking amber" and "No internet signal" end up in different chunks and the answer is lost.
Navigation, scripts, styles and footers are dropped at parse time. If you leave them in, "Copyright Brightline" will match half the questions that mention the brand.

For PDFs the example repairs the usual damage:

text = clean_pdf_pages(PDF_PAGES)
# "Page 1 of 3" and the repeated "Brightline R7 manual" header are gone,
# "com-\nputer" became "computer", wrapped lines are joined.

A line that is the first or last line of most pages is treated as a header or footer. That rule is simple and can be wrong (see exercise Medium 3 for a related case). It also glues a short heading onto the next line, as the first words of the cleaned text show ("Setup Plug the grey cable..."). Always read your parsed output. Pick ten documents, open the parsed text and look for lost tables, merged columns and junk.

Real PDFs are harder

The example PDF is plain text. Real PDFs may be scanned images (they need OCR), use two columns (a naive extractor interleaves the columns line by line), or hold tables as positioned text. Hard exercise 2 builds a column-repair function. Libraries and services exist for layout analysis, but whatever you use, verify its output on your own files.