steadydata

Recipe

Feed a RAG pipeline with clean page text

How do I turn a list of web pages into clean text for a RAG pipeline?

Why the obvious route disappoints

A browser-based crawler renders JavaScript you do not need, costs a few cents per page and returns HTML you still have to clean. For text you can do it without a browser.

The three steps

1

Sitemap URLs: every URL of a website from its XML sitemaps

Get every URL of the site from its XML sitemaps, so you know what there is to read before you fetch anything.

$0.50 per 1,000 · URL listed
2

Web Page to Markdown for LLMs: clean text from any URL

Turn each page into clean markdown: the main content, without menus, cookie bars and footers.

$1.00 per 1,000 · Page converted
3

PDF Text Extractor for LLMs: text per page from PDF URLs

Do the same for the PDFs you find, one row per page so the chunks stay small enough to embed.

$1.00 per 1,000 · Page extracted

What a run costs

A thousand pages costs about $1.00 in total, and pages that fail are free.