fetch
Page fetching over got-scraping
- a
gotwrapper that generates browser-like TLS + header fingerprints, so fetches survive the bot walls a plainfetchtrips. runWebFetch enforces the URL allow-list (an explicit fetch of a disallowed URL is refused, not silently emptied), fetches with the plugin’s timeout, and returns either the raw HTML or a readable plain-text reduction, capped at the configured length.
The HTML-to-text reduction is deliberately dependency-free (strip script/style, unwrap tags, decode a handful of entities, collapse whitespace): good enough to feed a model, and it keeps the add-on’s dependency surface to the two libraries the task called for.