How to design a web fetch tool for AI agents

An agent does not browse. It calls a tool, and everything it learns about a page arrives as that tool's result. The design of that one tool, its name, description, parameters and output, decides whether the agent reads the right page, understands what came back and recovers when it fails. This guide gives a default contract you can copy and says where it stops working.

Start with one narrow tool

Give the agent one tool that reads one public URL. Not a "web" tool with a mode switch, and not five overlapping tools for fetch, scrape, render and extract. Anthropic's engineering write-up on tools for agents makes the same point in general terms: fewer, clearer tools with distinct purposes beat a thin wrapper around every endpoint, because overlapping tools create ambiguous choices for the model.

A narrow tool is also easier to evaluate. You can write a test set of URLs and expected reactions and check the agent against it. A tool that can also click, log in or search multiplies the cases you have to test.

Name and description: the part the model actually reads

The model sees the tool name, the description and the parameter schema. That is the whole interface.

• Name. Use a verb and an object: get_page, read_url. Avoid names that suggest more than the tool does, such as browse or web.
• Description. Say what the tool returns, when to use it and when not to. Two or three sentences are enough.

A description that works as a default:

Read one public web page and return its main content as Markdown, with the page
title and final URL. Use it when you have a specific URL and need what the page says
now. It cannot log in, fill forms or run JavaScript; if it returns an error, read the
message and choose another source or tell the user.

The last sentence matters most. It tells the model, before the first call, that failure is a normal result with instructions attached.

Parameters: expose the URL, fix the rest in code

A fetch API usually offers many options: output format, a character cap, a timeout, a freshness setting, a country. Most of them should not be parameters of the tool.

Every parameter you expose is a chance for the model to fill it wrongly, and a few tokens of schema on every turn. Add one only when you have seen the agent need it.

Output: content plus the metadata the model needs

Return a small object, not a raw page:

{
  "ok": true,
  "title": "Pricing FAQ",
  "url": "https://example.com/docs/faq",
  "content": "## Billing\n\nInvoices are issued on the first day of ...",
  "truncated": false
}

• Main content as Markdown. It keeps headings, lists and links and drops navigation, scripts and banners. HTML costs far more tokens and adds little for reading tasks.
• Final URL. Redirects happen; the model should cite the page it actually read.
• Title. It helps the model check that it landed on the page it wanted.
• A truncation flag. If the content was cut at the cap, say so, so the model does not treat the end of the text as the end of the page.

Claude's built-in web fetch tool follows a similar shape: it caps fetched content with a token limit and can restrict domains. The ideas carry over to any fetch tool you build or call.

Failures: same shape, different fields

A failed read should come back through the same channel as a successful one, as a tool result, not an exception that kills the loop:

{
  "ok": false,
  "error": "CRAWL_NOT_RENDERED",
  "message": "This page needs JavaScript to show its content. Try the public docs page or tell the user."
}

The tag is for your code (logging, retries, evals). The sentence is for the model. The error-handling guide covers which tags to retry in code and which to hand to the model.

One URL or many per call

One URL per call is the right default. The model reasons about one result at a time, and most frameworks run parallel tool calls when the model asks for several. A multi-URL tool makes sense when your code builds the list, for example the links on a page the agent already read. Then it must return one status per URL, so one dead link does not fail the whole call.

A worked example

Live Web Data API, made by the publisher of this site, is one way to back this contract. A request takes up to 10 public URLs and returns results[] with title, final URL and Markdown, plus statuses[] with one entry per URL, each with a status, the credits charged and, on failure, an error tag and message. The SDKs wrap this as getPage / get_page, which returns the content or raises an error carrying the tag. The SDK docs show the full option list; the tool above uses only url, with format and cap fixed in code.

Where this breaks

• Pages that need JavaScript or a login. A static fetch tool cannot read them, by design. See browser agent vs fetch.
• Tasks that are really search. If the model keeps guessing URLs, it needs a search tool first, not a better fetch tool.
• Huge pages. A cap keeps the context safe but can cut off the part the model needed. The token budget guide covers summarizing first.
• Untrusted text. A good contract does not make page content safe. Read the prompt injection guide before the agent gets any tool that acts.