Skip to content
dbx tools

@dbx-tools/appkit-web-search

Server-side web-search runtime, Mastra tools, and AppKit plugin.

Add web_search and web_fetch to an AppKit or Mastra backend. web_search runs on the Databricks Model Serving native web-search tool: the model searches the web server-side and returns a synthesized answer plus the sources it used - no third-party search API key. web_fetch reads a single page through got-scraping.

Quick Start

Section titled “Quick Start”

Register the plugin and give its tools to an agent:

import { createApp, server } from "@databricks/appkit";
import { webSearch } from "@dbx-tools/appkit-web-search";
import { agents, mastra } from "@dbx-tools/appkit-mastra";


const researcher = agents.createAgent({
  instructions: "Search the web, then read the most relevant sources.",
  async tools(plugins) {
    return { ...(await plugins["web-search"].toolkit()) };
  },
});


await createApp({
  plugins: [server(), webSearch(), mastra({ agents: { researcher } })],
});

Key features:

  • AppKit plugin registration that resolves and logs the effective policy at boot.
  • Two Mastra tools: web_search (answer + citations) and web_fetch (page contents as readable text or raw HTML), plus the same pair as AppKit agent tools through the plugin’s ToolProvider.
  • web_search resolves a dedicated web-search-capable model - defaulting to Gemini, then GPT - independently of the calling agent’s chat model (which may not support web search). Loose names ("gemini", "gpt") fuzzy-match the live catalogue via @dbx-tools/model.
  • A built-in provider -> request-fragment map (OpenAI Responses API {"tools":[{"type":"web_search"}]}, Gemini Chat Completions {"google_search":{}}), overridable per provider via the WEB_SEARCH_TOOLS setting.
  • A named URL policy (allowlist or unrestricted) over a glob allow-list built on @dbx-tools/path’s match matcher: web_search silently filters disallowed citations, web_fetch refuses a disallowed URL, and the mode in force is logged at boot.
  • Per-tool approval gating: gate every call, or (for web_fetch) only calls whose URL matches a pattern, mapped onto Mastra’s requireApproval.
  • Every outbound call runs through AppKit’s execute() chain - per-user cache, retry with jittered backoff, timeout, telemetry - and unwinds on an AbortSignal. Each plugin instance keeps its policy and executor separate, so multiple apps in one process remain isolated.
  • Page and fallback-result HTML use parser-backed text conversion and selectors, including complete HTML entity decoding and malformed-markup recovery.

Use With Native AppKit

Section titled “Use With Native AppKit”

AppKit has no first-party web-search or page-fetch surface. Use this package when an agent needs to look things up on the open web or read a URL the user pasted, with a policy layer (allow-list + optional approval) around it so a deployment controls which sites are reachable and which calls pause for a human. Crucially, the web-search tool resolves its own web-search-capable model, so an agent running on any chat model (including one without web search) can still search. It is a thin add-on in the same shape as @dbx-tools/email: a Mastra tool pair plus an AppKit plugin.

Configure The AppKit Plugin

Section titled “Configure The AppKit Plugin”
import { createApp, lakebase, server } from "@databricks/appkit";
import { webSearch } from "@dbx-tools/appkit-web-search";
import { agents, mastra } from "@dbx-tools/appkit-mastra";


const researcher = agents.createAgent({
  instructions: "Research questions using web_search, then read sources with web_fetch.",
  async tools(plugins) {
    return { ...(await plugins["web-search"].toolkit()) };
  },
});


await createApp({
  plugins: [
    server(),
    lakebase(),
    webSearch({
      model: "gemini", // defaults to Gemini, then GPT, when omitted
      allowedUrls: ["*.databricks.com", "docs.example.com"],
    }),
    mastra({ agents: researcher, storage: true }),
  ],
});

plugin.webSearch() resolves config (over env), compiles the URL policy, and provides the executor used by its tools. App-integrated Mastra agents can consume the native plugin toolkit as shown above, which preserves that plugin’s policy, OBO scope, and AppKit execution chain. Approval, when enabled, requires Mastra storage, so register lakebase() or configure storage in the Mastra plugin.

The plugin is also an AppKit ToolProvider, so a simple native code agent can take the same two tools without Mastra. Put the definition at server/agents/researcher/agent.ts so AppKit discovers it:

import { createAgent } from "@databricks/appkit/beta";


export default createAgent({
  name: "Researcher",
  default: true,
  model: "databricks-gpt-6-1-sol",
  instructions: "Research questions with web_search, then read sources with web_fetch.",
  tools: (plugins) => ({ ...plugins["web-search"].toolkit() }),
  skills: ["research-policy"],
});

Place the curated skill at server/agents/skills/research-policy/SKILL.md, register native agents() with no deprecated agent map, and build a compiled native-agent server with:

import { appkitServerConfig } from "@databricks/appkit/tsdown";


export default appkitServerConfig();

Both tools are annotated effect: "read" and require user context, and neither is auto-inheritable: an agent has to ask for them, because web_fetch reaches whatever URL the model produces.

Native volume skills currently execute as the service principal; the 0.81 skillCredentialMode: "obo" option is not wired. Keep request-scoped writable skills, remote sources, asynchronous Graphiti discovery, Sandbox/Monty, Mastra memory, and durable approvals on @dbx-tools/appkit-mastra.

Choose The Web-Search Model

Section titled “Choose The Web-Search Model”

The native web-search tool only runs on certain models, and it is provider- specific. This package resolves deployed endpoints through @dbx-tools/model and applies this order:

  1. Highest-ranked web-search-capable Gemini.
  2. Without Gemini, a capable GPT matching the calling/requested model.
  3. Highest-ranked capable GPT.
  4. Any additional modelFallbacks searches in order.

When Databricks reports that Gemini web search is unavailable because cross-region processing is disabled, the plugin suppresses Gemini for 24 hours in that runtime and transparently retries the same tool call with the next GPT candidate.

DATABRICKS_SERVING_ENDPOINT_NAME is AppKit’s standard name for a Model Serving binding, and it is the field the plugin declares in its manifest, so app.yaml / bundle wiring reaches this plugin the way it reaches any other. The request’s model, plugin model, and environment model values are used to match the calling GPT after Gemini availability is checked. When the tool runs through @dbx-tools/appkit-mastra, the host injects the actual resolved chat model, overriding any different model argument generated by the LLM.

Model resolution runs against the LIVE workspace catalogue (via @dbx-tools/model), so it only ever picks an endpoint that is actually deployed - a configured id that doesn’t exist is never called.

Scrape Fallback

Section titled “Scrape Fallback”

The native tool is always preferred. But when the workspace has NO deployed GPT/Gemini endpoint (so native web search can’t run at all), the tool falls back to a DuckDuckGo scrape (via got-scraping) so it still returns results instead of erroring. Fallback results set model to "scrape:duckduckgo" and carry the substance in citations (there is no model synthesizing an answer, so answer is a lead-in over the top results). It is enabled by default; set scrapeFallback: false (or WEB_SEARCH_SCRAPE_FALLBACK=0) to require a native model and error when none is deployed.

The right provider request fragment is selected automatically from the resolved model: OpenAI GPT uses the Responses API {"tools":[{"type":"web_search"}]}; Gemini uses the top-level Chat Completions field {"google_search":{}}. Override or extend that map per provider with the webSearchTools setting (env WEB_SEARCH_TOOLS, JSON) as the platform evolves:

plugin.webSearch({
  model: "databricks-gemini-3-pro",
  webSearchTools: { gemini: { request: { google_search: {} } } },
});

Search And Fetch Without An Agent

Section titled “Search And Fetch Without An Agent”
import { search, fetch, runtime, tool } from "@dbx-tools/appkit-web-search";
import { getExecutionContext } from "@databricks/appkit";


const webRuntime = runtime.createWebSearchRuntime({
  model: "gemini",
  allowedUrls: ["*.databricks.com"],
});


// web_search needs the OBO client + host from the active execution context:
const ctx = getExecutionContext();
const host = (await ctx.client.config.getHost()).toString();


const result = await search.runWebSearch(
  { query: "unity catalog lineage best practices" },
  webRuntime,
  { client: ctx.client, host },
);
// result.answer, result.citations, result.model


const page = await fetch.runWebFetch({ url: result.citations[0]!.url, format: "text" }, webRuntime);

Use direct calls for operational lookups, tests, or admin flows where a model is not involved. Standalone Mastra tool factories likewise accept runtime or config; they never discover or attach to a process-global plugin:

const webRuntime = runtime.createWebSearchRuntime({
  allowedUrls: ["*.databricks.com"],
});


const tools = {
  web_search: tool.webSearchTool({ runtime: webRuntime }),
  web_fetch: tool.webFetchTool({ runtime: webRuntime }),
};

Pass the runtime explicitly or provide the executor to createWebSearchRuntime.

Restrict Which URLs Are Reachable

Section titled “Restrict Which URLs Are Reachable”
import { allowlist } from "@dbx-tools/appkit-web-search";


const list = allowlist.toUrlAllowList(
  allowlist.parseAllowedUrls(["*.databricks.com", "docs.example.com/api/**"]),
);


list.allows("https://docs.databricks.com/aws/en/index.html"); // true
list.allows("https://evil.example.com/"); // false

Each entry is a glob compiled by @dbx-tools/path’s match matcher. A host entry (no path, e.g. databricks.com or *.databricks.com) matches the URL’s hostname, and a bare host also matches its subdomains; a path entry (docs.example.com/api/**) matches host + pathname. Enforcement is asymmetric by design: web_search citations are silently filtered to the permitted set (the model never surfaces a source it then can’t fetch), while an explicit web_fetch of a disallowed URL is refused with an error (a visible, correctable mistake). web_fetch accepts only HTTPS targets that resolve to public addresses, and reapplies both checks before every redirect request.

Which of the two modes is in force is named by urlPolicy, and the effective mode is in the boot log:

urlPolicy Effect
"allowlist" Only allowedUrls entries are reachable
"unrestricted" Any public HTTPS URL is reachable

Leave urlPolicy unset and it follows allowedUrls: "allowlist" when there are entries, "unrestricted" when there are none. Set it to say which one you meant. The two contradictions fail at startup rather than quietly picking a side: "allowlist" with no entries, and "unrestricted" with entries.

// An open deployment, said out loud:
plugin.webSearch({ urlPolicy: "unrestricted" });


// A closed one:
plugin.webSearch({ urlPolicy: "allowlist", allowedUrls: ["*.databricks.com"] });

Gate Calls For Approval

Section titled “Gate Calls For Approval”

Both tools run without approval by default. Pass approval to a tool (or set it plugin-wide) to require a human click. true gates every call; a URL pattern (or list) gates only calls whose URL matches - a web_fetch is evaluated precisely against its target URL, while a web_search (whose result URLs are unknown before the call) treats a pattern gate as “gate every call”.

import { tool } from "@dbx-tools/appkit-web-search";


// Every fetch requires approval:
tool.webFetchTool({ approval: true });


// Only fetches of an internal domain require approval:
tool.webFetchTool({ approval: "*.internal.example.com" });


// The same thing, with the mode named:
tool.webFetchTool({ approval: { mode: "urls", patterns: ["*.internal.example.com"] } });


// Plugin toolkit default:
plugin.webSearch({ approval: ["*.internal.example.com", "*.corp.example.com"] });


// Standalone tool config:
tool.webFetchTool({ config: { approval: ["*.internal.example.com"] } });

Every spelling normalizes to one of three modes - { mode: "none" }, { mode: "always" }, { mode: "urls", patterns } - which is what the resolved config carries and the boot log reports. Approval uses the same glob syntax as the allow-list, so the two policies read identically.

Configuration

Section titled “Configuration”

Resolution order is explicit config first, then the environment variable, then a built-in default.

Option Type Default Description
model string resolved from fallbacks Web-search endpoint: an endpoint name, a loose name, or a capability class.
modelFallbacks string | string[] Gemini, then GPT Ordered candidates tried when model is unset.
webSearchTools Record<string, unknown> built-in provider map Provider -> request-fragment overrides, merged over the defaults.
modelFuzzyMatch boolean true Fuzzy-match loose model names against the live catalogue.
modelFuzzyThreshold number 0.4 Fuse.js score below which a fuzzy match is accepted.
maxCitations number 10 Hard cap on citations returned from one search.
fetchMaxLength number 50000 Hard cap on web_fetch content length, in characters.
timeoutMs number 60000 Per-request network timeout.
scrapeFallback boolean true Scrape DuckDuckGo when no web-search-capable model is deployed.
urlPolicy "unrestricted" | "allowlist" follows allowedUrls Which URLs the tools may reach.
allowedUrls string | string[] none Allow-list globs or bare hosts.
approval boolean | string | string[] | ApprovalPolicy { mode: "none" } Default approval gate for both tools.
Environment variable Sets
WEB_SEARCH_MODEL model (dedicated override, wins over the binding below)
DATABRICKS_SERVING_ENDPOINT_NAME model (AppKit’s Model Serving binding, treated as a preference)
WEB_SEARCH_MODEL_FALLBACKS modelFallbacks (comma/space-separated)
WEB_SEARCH_TOOLS webSearchTools (JSON object; anything else fails at boot)
WEB_SEARCH_FUZZY modelFuzzyMatch
WEB_SEARCH_FUZZY_THRESHOLD modelFuzzyThreshold
WEB_SEARCH_MAX_CITATIONS maxCitations
WEB_SEARCH_FETCH_MAX_LENGTH fetchMaxLength
WEB_SEARCH_TIMEOUT_MS timeoutMs
WEB_SEARCH_SCRAPE_FALLBACK scrapeFallback
WEB_SEARCH_URL_POLICY urlPolicy
WEB_SEARCH_ALLOWED_URLS allowedUrls (comma/space-separated)

The plugin’s maxCitations / fetchMaxLength are hard caps: a per-call request may narrow them but never exceed them.

Every outbound call - the serving POST, a page fetch, the scrape fallback - runs through the plugin’s execute() chain, so all three are cached, retried with jittered backoff, timed out, and traced. The cache is namespaced per user, so an on-behalf-of result never crosses identities. See defaults.ts for the TTLs and retry budgets and why each was chosen.

OpenAI Responses web searches stream native search, page-read, and completion status through the Mastra tool writer. The common tool-session pill updates while the search runs; the completed Responses payload remains the tool result.

Requirements: the native web-search tool runs on pay-per-token GPT / Gemini serving endpoints with cross-region processing enabled; it is unavailable on provisioned throughput, for external models, or under HIPAA/BAA compliance. See the Databricks docs.

Modules

Section titled “Modules”
  • plugin - WebSearchPlugin, webSearch() AppKit plugin factory, and the AppKit ToolProvider implementation.
  • tool - webSearchTool() / webFetchTool() Mastra tools.
  • defaults - the cache / retry / timeout settings every outbound call runs with, and the reasoning behind each.
  • search - runWebSearch() over the Databricks native web-search tool.
  • provider - provider detection + the provider -> tool-spec map.
  • scrape - runScrapeSearch() DuckDuckGo fallback plus parser-backed parseDdgHtml() result extraction for workspaces with no native web-search model.
  • fetch - runWebFetch() over got-scraping.
  • html-text - htmlToText() / htmlFragmentToText() / decodeHtmlEntities(), shared by fetch and the DuckDuckGo scrape fallback.
  • runtime - isolated runtime construction and execution: createWebSearchRuntime(), createResolvedWebSearchRuntime(), and executeRead().
  • config - config types, JSON schema, resolveWebSearchConfig(), URL policy and approval helpers.
  • allowlist - URL allow-list parsing/compiling on top of @dbx-tools/path.
  • schema - zod tool contracts and inferred types.

Pair this package with @dbx-tools/appkit-mastra to spread the tools into an agent (and resolve its own model via @dbx-tools/model), and @dbx-tools/email for the sibling approval-gated tool pattern.

See more

Section titled “See more”

Open the complete API reference