Skip to content

htmlText

Turning fetched HTML into the plain text a model can read.

Both scraping paths need this: runWebFetch’s full-page read and runScrapeSearch’s title/snippet extraction. The entity table is shared so the two cannot drift apart.

This is deliberately regex-based rather than a DOM parse. The inputs are whole pages fetched for summarization, not documents to be queried, so the cost of a real parser buys nothing - and a malformed page still degrades to readable text instead of throwing.