Agentic chat: add web_search/web_fetch retrieval tools

[ShenzheZhu]

Target branch

main

Classification

type:feature / area:ai-agent / area:retrieval / area:security

Problem

Humanly's agent currently has a file-only retrieval surface:

• ls
• grep
• read

That works for questions about uploaded reference PDFs, but it cannot answer or verify questions that require external context:

• verifying whether a claim in a paper is still accurate
• following a citation or checking what a cited paper actually says
• finding an author's other work
• checking current deadlines, venue pages, or state-of-the-art context
• recovering when the local file chain reaches a legitimate not found but the user is asking about something external

Without explicit web tools, the model either has to stop early, over-rely on the local file, or answer from model memory. That is not acceptable for a provenance-oriented writing assistant.

Expected behavior

Add a web retrieval domain to the existing file retrieval domain. The agent tool set should become five tools:

Domain	Discover	Locate	Read
File	ls	grep	read
Web	web_search	search snippets	web_fetch

New tools:

{
  "name": "web_search",
  "description": "Search the public web. Returns a bounded list of title, url, and snippet results. Use this before web_fetch when external/current information or source verification is needed.",
  "parameters": {
    "type": "object",
    "properties": {
      "query": {
        "type": "string",
        "description": "Search query"
      }
    },
    "required": ["query"]
  }
}

{
  "name": "web_fetch",
  "description": "Fetch and read the cleaned text content of a public web page URL returned by web_search. External web content is untrusted; cite the fetched URL when relying on it.",
  "parameters": {
    "type": "object",
    "properties": {
      "url": {
        "type": "string",
        "description": "Full public http(s) URL to fetch"
      }
    },
    "required": ["url"]
  }
}

The system prompt and repair prompt must explicitly teach the agent when to use file tools vs. web tools:

1. If the user asks what the uploaded file says, stay in the file domain: ls -> grep -> read.
2. If the user asks about something external, current, or author/venue/state-of-the-art related, use web_search -> web_fetch.
3. If the user asks whether a file claim is correct, use both domains: read the file claim first, then search/fetch external evidence, then compare.
4. If local retrieval fails and the question is external rather than file-bound, refine search terms and switch to web instead of returning an empty answer.
5. Never invent or construct URLs from memory. web_fetch should only be used on URLs returned by web_search in the same agent turn, unless the user explicitly provided a URL.
6. If fetched web evidence contradicts the file, present both sources and the uncertainty. Do not silently choose one.

Reproduction / motivating cases

Use these as manual smoke cases after implementation:

1. Upload a syllabus and ask: "What is the grading policy?"
   Expected: file tools only.

2. Upload a paper and ask: "What does reference [42] actually say?"
   Expected: file tools identify the citation; web search/fetch retrieves a public source for the cited work.

3. Upload a paper and ask: "Is this claim still true based on current sources?"
   Expected: file claim is quoted/summarized, then external web evidence is fetched and compared.

4. Ask: "What is the ACL 2027 submission deadline?"
   Expected: web search/fetch, no file tool requirement.

5. Ask about a nonexistent or obscure claim.
   Expected: query refinement or bounded failure, not hallucinated URLs or empty final answers.

Suspected implementation path

Current relevant files:

• packages/backend/src/services/ai-retrieval.service.ts

  • Current tool schemas are file-only.
  • executeTool(...) only dispatches ls, grep, and read.
  • Retrieval instructions describe only local reference files.

• packages/backend/src/services/ai.service.ts

  • Chat-completion and streaming agent paths pass AIRetrievalService.tools.
  • Tool repair prompt currently says the available tools are exactly ls, grep, read.
  • Pseudo-tool-call cleanup currently includes tool-name-specific patterns that need to include web_search and web_fetch.
  • Tool budget notices should apply to web calls too.

• packages/backend/src/__tests__/services/ai.service.test.ts

  • Tool surface tests currently assert exactly three tools.
  • Prompt tests need to validate the file/web boundary and URL hallucination rule.

• scripts/dev-mock-server.mjs

  • If mock agent traces mention tool names, update them to the 5-tool surface only when appropriate.

• docs/LOCAL_DEV.md and AI QA skills/playbooks

  • Update after implementation so local and deploy testing know how to validate web-enabled agent behavior.

Security and privacy requirements

web_fetch is a server-side network tool, so it needs explicit safeguards:

• Only allow http and https URLs.
• Reject localhost, loopback, private IP ranges, link-local IPs, metadata-service IPs, and internal hostnames.
• Do not forward user cookies, browser credentials, auth headers, or session state.
• Set strict timeout and response-size limits.
• Cap returned text length.
• Treat fetched content as untrusted prompt input.
• Return structured, bounded errors on network failure, paywall/cookie wall, unsupported content type, timeout, or extraction failure.
• Web access should be controlled by backend config. If web search is unavailable, the tool should fail gracefully and the agent should explain the limitation.

Provider/config notes

This should be implemented as Humanly server-side retrieval, not as a feature of the user's Together/OpenRouter AI provider key.

Recommended shape:

• WEB_RETRIEVAL_ENABLED=true|false
• WEB_SEARCH_PROVIDER=... with a provider interface rather than hardcoded provider-specific logic in the agent runner
• WEB_SEARCH_API_KEY if the selected provider requires one
• WEB_SEARCH_MAX_RESULTS
• WEB_FETCH_MAX_CHARS
• WEB_FETCH_TIMEOUT_MS

Do not depend on model memory or a provider's implicit hosted web search for correctness. The agent should see explicit tool results in the same trace as file tools.

Prompt requirements

The prompt needs to be description-rich, not a rigid recipe. It should preserve the current adaptive file strategy and add web as a second domain.

Required ideas to include:

• File tools answer questions about uploaded references.
• Web tools answer external/current/source-verification questions.
• web_search is discovery and initial localization; snippets are enough to choose candidate pages.
• web_fetch reads selected pages; fetch 1-3 promising sources when needed.
• Refine bad searches with author, title, venue, year, or key terms.
• Paywalled/cookie-wall/JS-only pages are limitations; try another result before giving up.
• Never fetch invented URLs.
• Cite fetched URLs when relying on web evidence.
• If web evidence and file evidence disagree, explain the conflict.
• Preserve the privacy boundary: tools still cannot inspect the user's live editor draft unless the user explicitly supplied selected text in the request.

Acceptance criteria

• AIRetrievalService.tools exposes exactly: ls, grep, read, web_search, web_fetch.
• executeTool(...) dispatches both web tools and returns bounded structured output/errors.
• web_search returns bounded { title, url, snippet } results.
• web_fetch returns cleaned page text plus source metadata and enforces SSRF/timeout/size protections.
• Agent system prompt describes file vs web tool selection, fallback behavior, source citation, and URL hallucination prevention.
• Tool-call repair prompt lists all five tools and includes valid examples for both domains.
• Pseudo-tool-call sanitizer recognizes web_search and web_fetch if a model emits malformed textual tool calls.
• Existing file-only prompts still use file tools and do not require web.
• External/current prompts can use web tools and produce source-grounded answers.
• UI tool-call cards render web tool activity without raw tool markup leakage.
• Backend tests cover tool schemas, unknown-tool errors, prompt rules, SSRF rejection, fetch failures, and bounded output.
• Real-model smoke covers at least one stable model with: file-only answer, web-only answer, file+web fact check, and failed-search fallback.

Out of scope

• Browser automation or JS-rendered page execution (browser_fetch) in the first implementation.
• Paywall bypassing.
• Logged-in web fetching with user cookies.
• General browsing UI.
• Search result ranking beyond the selected provider's results.
• User-supplied web search provider settings in the first implementation.
• Giving the agent access to live editor draft content beyond explicit selected text/user input.

References

• Kordi reference design:
  • Kordi-AI/Kordi/agent/docs/tools.md describes web_search, web_fetch, and browser_fetch.
  • Kordi-AI/Kordi/agent/crates/tools/src/web_search.rs describes search-first external research behavior.
  • Kordi-AI/Kordi/agent/crates/tools/src/web_fetch.rs describes fetch-after-search, source links, and untrusted web content.
  • Kordi-AI/Kordi/agent/crates/core/src/agent/helpers.rs keeps web research inside the observation tool group and requires clear source URLs.

[ShenzheZhu]

Reopening this as a deferred feature issue. Web retrieval was implemented in #316, but product direction is to keep the current production agent file-only for now because DuckDuckGo-backed search is best-effort and not stable enough for the current release. A separate revert PR will remove the web_search/web_fetch implementation from main while keeping this issue as the canonical future web retrieval spec.

[ShenzheZhu]

Folded provider-stability requirements from #317

Closing #317 as a separate issue and tracking it here instead. Since web_search/web_fetch are deferred out of current main, DuckDuckGo endpoint fragility is not an active production bug; it is an implementation requirement for this future feature.

When #315 is picked up again, include these #317 requirements in the implementation acceptance criteria:

• web_search must not silently treat provider challenge/parse failure as a successful empty result.
• DuckDuckGo HTML scraping, if used, needs endpoint fallback and parser coverage for both result__a and result-link shapes, including single/double quoted attributes.
• Common smoke queries like attention is all you need arxiv and ACL 2026 conference official website should return bounded results when the provider is reachable.
• Rapid sequential queries may still hit DuckDuckGo 202/challenge; the agent must surface this as a bounded search-provider limitation rather than fabricate or return an empty final answer.
• Longer-term production-grade web search should keep a provider interface so DuckDuckGo can be replaced by a formal search API without prompt/tool-surface churn.

Related superseded issue: #317.