scrape
Last-resort scraping fallback for web_search, used ONLY when the
workspace has no Databricks web-search-capable model deployed (no GPT /
Gemini serving endpoint). The native Databricks web-search tool is always
preferred (see search.ts); this exists so the tool still returns useful
results in an environment that can’t run it, rather than erroring on every
call.
It queries DuckDuckGo’s no-JS HTML endpoint through got-scraping
(browser-like fingerprints so the request isn’t blocked) via a GET with the
query in the query string - a POST to the same endpoint trips DDG’s bot
challenge (HTTP 202), while the GET returns normal result markup. It then
parses the result anchors + snippets. Unlike the native tool there is no
model synthesizing an answer, so answer is a short lead-in over the top
snippets and the substance rides in citations - the calling agent reads
those and writes its own answer.