Token budgets for agent web pages: caps, limits and costs

A web page is the largest thing most agents put into their context. It is easy to fetch and easy to forget, and it keeps costing tokens long after the model has used it. This guide covers three budgets, per page, per task and per context, with starting values you can copy and a way to measure your own.

Why pages are expensive

Two effects add up.

• Size. Raw HTML includes markup, scripts, styles, navigation and footers. The main content as Markdown is a fraction of that. Returning the cleaned content is the single largest saving available, and it usually helps answer quality too.
• Repetition. Chat-style model APIs resend the conversation on every turn, including earlier tool results. A page read at step 2 is billed as input again at every later step, unless your loop removes or shortens it. Anthropic's context-engineering write-up treats this as a core problem of long-running agents: context is a finite resource, and old tool results are often the first thing to trim.

Count with the tokenizer of the model you use. Providers publish token-counting endpoints for exactly this; general rules of thumb about characters per token vary by language and content.

Budget 1: a cap per page

Cut every page at a character cap in the tool, not in the prompt.

• Starting point: about 20,000 characters per page for reading tasks. That is a starting value, not a measurement for your workload.
• Tell the model when the page was cut ("truncated": true), so it does not assume the page ended there.
• Cut at the source when you can. A fetch API that applies the cap returns less data over the wire and avoids a second truncation step in your code. Claude's built-in web fetch tool offers a content token limit for the same reason.

If evals show answers that sit below the cap, raise it for that task type only, or let the tool return a specific section.

Budget 2: fetches per task

A per-task budget stops the slow leak of an agent that reads "just one more page".

class FetchBudget:
    def __init__(self, max_fetches=8, max_failures=4):
        self.left, self.failures_left = max_fetches, max_failures

    def check(self):
        if self.left <= 0 or self.failures_left <= 0:
            return {"ok": False, "error": "BUDGET_EXHAUSTED",
                    "message": "No more page reads for this task. Answer with what you have."}
        return None

Decrement left on every call and failures_left on every failed one. Return the budget error as a normal tool result, so the model finishes with what it has instead of crashing. Eight fetches and four failures are starting values; set them from logs of real tasks.

Budget 3: what stays in the main context

For tasks that read many or long pages, do not give the main agent the pages at all.

• A reader sub-agent fetches the page, answers a narrow question about it ("what does this page say about refunds?") and returns a short answer with the URL.
• The main agent works from those short answers and cites the URLs.
• Trim old tool results in long loops: once a page has been used, replace it with a one-line note of what it contained.

This also limits the spread of injected instructions, though summaries of untrusted pages remain untrusted; see the prompt injection guide.

Credits and money

Tokens are one bill; the fetch is another. With Live Web Data API, made by the publisher of this site, a page read with a plain fetch costs one credit, a read that needed a retry or browser rendering costs more (the API reference lists the credits per status), and failed, timed-out or restricted requests cost 0 credits, so failed reads add model tokens but no fetch cost. Your code can read the remaining balance with account() in the SDKs, or the MCP tool get_account_status, before starting a long run. Plans are on the pricing page.

Measure, then tighten

Log per task: pages fetched, characters returned, tokens in, failures and whether the answer was right. After a few hundred tasks you can see which caps are too loose (pages cut rarely, tokens high) and which too tight (answers missing). Change one budget at a time.

Where this breaks

• Pages where the answer is at the end, such as long changelogs or reference tables. A plain cap cuts it off; fetch a more specific URL or summarize in a sub-agent.
• Many small pages. A per-page cap does nothing when the agent reads thirty short pages; the per-task budget does.
• Retries the model sees. Each visible failure costs tokens; retry transient errors in code as the error-handling guide describes.