---
title: Fetch and index single web pages
description: Use the Browser Run /content endpoint to fetch a single web page's rendered HTML, then upload it to an AI Search instance's built-in storage so AI Search indexes it for search.
image: https://edgetunnel-b2h.pages.dev/dev-products-preview.png
---

> Documentation Index  
> Fetch the complete documentation index at: https://edgetunnel-b2h.pages.dev/ai-search/llms.txt  
> Use this file to discover all available pages before exploring further. 

[Skip to content](#%5Ftop) 

# Fetch and index single web pages

This guide builds a Worker that fetches a single web page's rendered HTML with the [Browser Run](https://edgetunnel-b2h.pages.dev/browser-run/) [/content endpoint](https://edgetunnel-b2h.pages.dev/browser-run/quick-actions/content-endpoint/) and uploads it to an [AI Search](https://edgetunnel-b2h.pages.dev/ai-search/) instance's [built-in storage](https://edgetunnel-b2h.pages.dev/ai-search/configuration/data-source/built-in-storage/) using the [Items API](https://edgetunnel-b2h.pages.dev/ai-search/api/items/workers-binding/). AI Search then indexes the page so it is searchable, the same as any other uploaded document. The Worker also exposes a `/search` endpoint that queries the indexed pages, so one service both indexes and searches.

## When to use this pattern

Use this pattern to index one page, or a small hand-picked set of pages, on demand. To crawl and continuously index an entire site, use the AI Search [website data source](https://edgetunnel-b2h.pages.dev/ai-search/configuration/data-source/website/) instead.

Both Browser Run and the AI Search instance are reached through bindings, so a single Worker can fetch a page and index it without a public endpoint in between.

## Prerequisites

1. Sign up for a [Cloudflare account ↗](https://dash.cloudflare.com/sign-up/workers-and-pages).
2. Install [Node.js ↗](https://docs.npmjs.com/downloading-and-installing-node-js-and-npm).

Node.js version manager

Use a Node version manager like [Volta ↗](https://volta.sh/) or [nvm ↗](https://github.com/nvm-sh/nvm) to avoid permission issues and change Node.js versions. [Wrangler](https://edgetunnel-b2h.pages.dev/workers/wrangler/install-and-update/), discussed later in this guide, requires a Node version of `16.17.0` or later.

You also need an AI Search instance to upload to. To create one, refer to [Get started](https://edgetunnel-b2h.pages.dev/ai-search/get-started/). This guide uploads to the instance's built-in storage, so the instance does not need an external data source.

## 1\. Create a Worker project

Create a new Worker project using the `create-cloudflare` CLI (C3). [C3 ↗](https://github.com/cloudflare/workers-sdk/tree/main/packages/create-cloudflare) is a command-line tool designed to help you set up and deploy new applications to Cloudflare.

Create a new project named `fetch-and-index` by running:

 npm  yarn  pnpm 

```
npm create cloudflare@latest -- fetch-and-index
```

```
yarn create cloudflare fetch-and-index
```

```
pnpm create cloudflare@latest fetch-and-index
```

For setup, select the following options:

* For _What would you like to start with?_, choose `Hello World example`.
* For _Which template would you like to use?_, choose `Worker only`.
* For _Which language do you want to use?_, choose `TypeScript`.
* For _Do you want to use git for version control?_, choose `Yes`.
* For _Do you want to deploy your application?_, choose `No` (we will be making some changes before deploying).

Go to your application directory:

```sh
cd fetch-and-index
```

## 2\. Configure Wrangler

Add both bindings to your [Wrangler configuration file](https://edgetunnel-b2h.pages.dev/workers/wrangler/configuration/): a [browser binding](https://edgetunnel-b2h.pages.dev/browser-run/reference/wrangler/#bindings) for Browser Run and an [AI Search namespace binding](https://edgetunnel-b2h.pages.dev/ai-search/api/items/workers-binding/) for uploads. The `/content` endpoint runs through the browser binding, so you do not need to install Puppeteer or any other package.

* [  wrangler.jsonc ](#tab-panel-7239)
* [  wrangler.toml ](#tab-panel-7240)

**JSONC**

```jsonc
{
  "$schema": "./node_modules/wrangler/config-schema.json",
  "name": "fetch-and-index",
  "main": "src/index.ts",
  // Set this to today's date
  "compatibility_date": "2026-07-20",
  "browser": {
    "binding": "BROWSER",
    "remote": true
  },
  "ai_search_namespaces": [
    {
      "binding": "AI_SEARCH",
      "namespace": "default",
      "remote": true
    }
  ]
}
```

**TOML**

```toml
name = "fetch-and-index"
main = "src/index.ts"
# Set this to today's date
compatibility_date = "2026-07-20"


[browser]
binding = "BROWSER"
remote = true


[[ai_search_namespaces]]
binding = "AI_SEARCH"
namespace = "default"
remote = true
```

The browser binding's `quickAction` method requires a compatibility date of `2026-03-24` or later, and is not supported in local development without remote mode. Setting `remote = true` on the browser binding enables remote mode for `wrangler dev`. The `remote` option on the AI Search binding proxies uploads to your deployed instance, since AI Search does not run locally.

## 3\. Add the Worker code

Update `src/index.ts`. This Worker has two routes: a request with a `?url=` parameter fetches that page's rendered HTML and indexes it, and a request to `/search?q=` queries the indexed content. Replace `my-instance` with the name of your instance.

* [  JavaScript ](#tab-panel-7241)
* [  TypeScript ](#tab-panel-7242)

**src/index.js**

```js
// The instance that indexes the fetched page.
const INSTANCE_ID = "my-instance";


// Build a stable item key that ends in .html, so AI Search converts the HTML
// to Markdown before indexing it.
function itemKey(pageUrl) {
  const slug = `${pageUrl.hostname}${pageUrl.pathname}`
    .replace(/[^a-zA-Z0-9]+/g, "-")
    .replace(/^-+|-+$/g, "");
  return `${slug || "index"}.html`;
}


export default {
  async fetch(request, env) {
    const url = new URL(request.url);


    // Search route: query the indexed content and return the matching chunks.
    if (url.pathname === "/search") {
      const query = url.searchParams.get("q");
      if (!query) {
        return new Response("Add a ?q= query parameter", { status: 400 });
      }
      const results = await env.AI_SEARCH.get(INSTANCE_ID).search({ query });
      return Response.json({
        query: results.search_query,
        results: results.chunks.map((chunk) => ({
          key: chunk.item.key,
          score: chunk.score,
          text: chunk.text,
        })),
      });
    }


    // Index route: fetch a URL's rendered HTML and index it.
    const target = url.searchParams.get("url");
    if (!target) {
      return new Response(
        "Add a ?url= parameter to index a page, or use /search?q= to search",
        { status: 400 },
      );
    }


    const pageUrl = new URL(target);


    // Fetch the fully rendered HTML with the Browser Run /content endpoint.
    // networkidle2 waits until the page has no more than two network
    // connections for at least 500 ms, giving client-side JavaScript time
    // to render the content.
    const response = await env.BROWSER.quickAction("content", {
      url: pageUrl.toString(),
      gotoOptions: {
        waitUntil: "networkidle2",
        timeout: 30000,
      },
    });


    if (!response.ok) {
      const detail = (await response.text()).slice(0, 500);
      return new Response(
        `Browser Run failed with ${response.status}: ${detail}`,
        { status: 502 },
      );
    }


    // The /content endpoint returns a JSON envelope with the rendered HTML
    // in the result field.
    const data = await response.json();


    if (!data.success || typeof data.result !== "string") {
      return new Response("Browser Run returned an unsuccessful response", {
        status: 502,
      });
    }


    const html = data.result;


    // Upload the rendered HTML to built-in storage. uploadAndPoll waits until
    // the page is indexed and searchable.
    const item = await env.AI_SEARCH.get(INSTANCE_ID).items.uploadAndPoll(
      itemKey(pageUrl),
      html,
      { timeoutMs: 60_000 },
    );


    return Response.json({ key: item.key, status: item.status });
  },
};
```

**src/index.ts**

```ts
export interface Env {
  BROWSER: BrowserRun;
  AI_SEARCH: AiSearchNamespace;
}


// The instance that indexes the fetched page.
const INSTANCE_ID = "my-instance";


// Build a stable item key that ends in .html, so AI Search converts the HTML
// to Markdown before indexing it.
function itemKey(pageUrl: URL): string {
  const slug = `${pageUrl.hostname}${pageUrl.pathname}`
    .replace(/[^a-zA-Z0-9]+/g, "-")
    .replace(/^-+|-+$/g, "");
  return `${slug || "index"}.html`;
}


export default {
  async fetch(request, env): Promise<Response> {
    const url = new URL(request.url);


    // Search route: query the indexed content and return the matching chunks.
    if (url.pathname === "/search") {
      const query = url.searchParams.get("q");
      if (!query) {
        return new Response("Add a ?q= query parameter", { status: 400 });
      }
      const results = await env.AI_SEARCH.get(INSTANCE_ID).search({ query });
      return Response.json({
        query: results.search_query,
        results: results.chunks.map((chunk) => ({
          key: chunk.item.key,
          score: chunk.score,
          text: chunk.text,
        })),
      });
    }


    // Index route: fetch a URL's rendered HTML and index it.
    const target = url.searchParams.get("url");
    if (!target) {
      return new Response(
        "Add a ?url= parameter to index a page, or use /search?q= to search",
        { status: 400 },
      );
    }


    const pageUrl = new URL(target);


    // Fetch the fully rendered HTML with the Browser Run /content endpoint.
    // networkidle2 waits until the page has no more than two network
    // connections for at least 500 ms, giving client-side JavaScript time
    // to render the content.
    const response = await env.BROWSER.quickAction("content", {
      url: pageUrl.toString(),
      gotoOptions: {
        waitUntil: "networkidle2",
        timeout: 30000,
      },
    });


    if (!response.ok) {
      const detail = (await response.text()).slice(0, 500);
      return new Response(
        `Browser Run failed with ${response.status}: ${detail}`,
        { status: 502 },
      );
    }


    // The /content endpoint returns a JSON envelope with the rendered HTML
    // in the result field.
    const data = (await response.json()) as {
      success: boolean;
      result?: string;
    };


    if (!data.success || typeof data.result !== "string") {
      return new Response("Browser Run returned an unsuccessful response", {
        status: 502,
      });
    }


    const html = data.result;


    // Upload the rendered HTML to built-in storage. uploadAndPoll waits until
    // the page is indexed and searchable.
    const item = await env.AI_SEARCH.get(INSTANCE_ID).items.uploadAndPoll(
      itemKey(pageUrl),
      html,
      { timeoutMs: 60_000 },
    );


    return Response.json({ key: item.key, status: item.status });
  },
} satisfies ExportedHandler<Env>;
```

The `.html` item key tells AI Search to run the content through [Markdown conversion](https://edgetunnel-b2h.pages.dev/workers-ai/features/markdown-conversion/), which strips boilerplate such as the header and footer before indexing.

Warning

Fetch only URLs you trust. A Worker that fetches arbitrary user-supplied URLs can become an open proxy. Consider restricting the accepted hostnames to an allowlist. Uploaded content is also limited to 4 MB per item.

## 4\. Attach metadata for filtering

This step is optional. Because this Worker controls the upload, you can enrich each page with structured [metadata](https://edgetunnel-b2h.pages.dev/ai-search/configuration/indexing/metadata/), such as its title and section, and then [filter searches](https://edgetunnel-b2h.pages.dev/ai-search/configuration/retrieval/filtering/) by those fields. This is something the built-in crawler cannot do on its own.

First, define the custom metadata fields on your instance. If you are creating the instance now, pass them to `create`:

```sh
npx wrangler ai-search create my-instance --type builtin --custom-metadata title:text --custom-metadata section:text
```

To add fields to an existing instance, use the dashboard under **Settings**, or the [update()](https://edgetunnel-b2h.pages.dev/ai-search/api/instances/workers-binding/#update) binding method. An instance supports up to five custom fields, and each field can be a `text`, `number`, `boolean`, or `datetime` type. Changing the schema re-indexes existing documents.

Next, use the Browser Run [/json endpoint](https://edgetunnel-b2h.pages.dev/browser-run/quick-actions/json-endpoint/) to extract those fields from the same page. It runs through the same browser binding and returns structured JSON that matches a schema you provide. In the `fetch` handler from step 3, after you have the rendered `html` and before the upload, add:

**TypeScript**

```ts
// Extract structured metadata from the page with the /json endpoint.
// response_format constrains the model to the fields you defined above.
// Treat extraction as best-effort: if it fails, index the page without metadata.
const metadata: Record<string, string> = {};
try {
  const jsonResponse = await env.BROWSER.quickAction("json", {
    url: pageUrl.toString(),
    prompt: "Extract the page title and its top-level section.",
    response_format: {
      type: "json_schema",
      json_schema: {
        type: "object",
        properties: {
          title: { type: "string" },
          section: { type: "string" },
        },
        required: ["title"],
      },
    },
  });


  const extracted = (await jsonResponse.json()) as {
    result?: Record<string, unknown>;
  };


  // Metadata values must be strings, so coerce each value and drop empty ones.
  for (const [key, value] of Object.entries(extracted.result ?? {})) {
    if (value) metadata[key] = String(value);
  }
} catch {
  // Ignore extraction errors and index the page without metadata.
}
```

Then pass `metadata` in the upload options:

**TypeScript**

```ts
const item = await env.AI_SEARCH.get(INSTANCE_ID).items.uploadAndPoll(
  itemKey(pageUrl),
  html,
  { timeoutMs: 60_000, metadata },
);
```

Once indexed, you can restrict queries to pages in a given section, for example. Refer to [Filtering](https://edgetunnel-b2h.pages.dev/ai-search/configuration/retrieval/filtering/) for the query syntax.

## 5\. Run and deploy

Start a local development server. Because `remote = true` is set on the browser binding, `wrangler dev` runs the `/content` endpoint in remote mode:

```sh
npx wrangler dev
```

Index a page by passing its URL:

```sh
curl "http://localhost:8787/?url=https://example.com/"
```

The response contains the item key and its status (`completed` once indexed):

```json
{ "key": "example-com.html", "status": "completed" }
```

Then query the indexed content through the same Worker's `/search` endpoint:

```sh
curl "http://localhost:8787/search?q=what+is+this+domain+for"
```

```json
{
  "query": "what is this domain for",
  "results": [
    {
      "key": "example-com.html",
      "score": 0.75,
      "text": "# Example Domain\nThis domain is for use in documentation examples..."
    }
  ]
}
```

Log in with your Cloudflare account, then deploy your Worker to make it accessible on the Internet:

```sh
npx wrangler login
npx wrangler deploy
```

## Next steps

[ Browser Run /content endpoint ](https://edgetunnel-b2h.pages.dev/browser-run/quick-actions/content-endpoint/) Fetch the fully rendered HTML of a page after JavaScript runs, with options for load behavior and blocking. 

[ Items Workers binding ](https://edgetunnel-b2h.pages.dev/ai-search/api/items/workers-binding/) Full reference for uploading, listing, and deleting documents in built-in storage. 

[ Website data source ](https://edgetunnel-b2h.pages.dev/ai-search/configuration/data-source/website/) Crawl and index a domain you own automatically, following its sitemap.

```json
{"@context":"https://schema.org","@type":"TechArticle","@id":"https://edgetunnel-b2h.pages.dev/ai-search/how-to/fetch-and-index-web-pages/#page","headline":"Fetch and index single web pages · Cloudflare AI Search docs","description":"Use the Browser Run /content endpoint to fetch a single web page's rendered HTML, then upload it to an AI Search instance's built-in storage so AI Search indexes it for search.","url":"https://edgetunnel-b2h.pages.dev/ai-search/how-to/fetch-and-index-web-pages/","inLanguage":"en","image":"https://edgetunnel-b2h.pages.dev/dev-products-preview.png","dateModified":"2026-07-08","publisher":{"@type":"Organization","name":"Cloudflare","url":"https://www.cloudflare.com/"},"isPartOf":{"@type":"WebSite","@id":"https://edgetunnel-b2h.pages.dev/#website","name":"Cloudflare Docs","url":"https://edgetunnel-b2h.pages.dev/"}}
{"@context":"https://schema.org","@type":"BreadcrumbList","itemListElement":[{"@type":"ListItem","position":1,"item":{"@id":"/directory/","name":"Directory"}},{"@type":"ListItem","position":2,"item":{"@id":"/ai-search/","name":"AI Search"}},{"@type":"ListItem","position":3,"item":{"@id":"/ai-search/how-to/","name":"How to"}},{"@type":"ListItem","position":4,"item":{"@id":"/ai-search/how-to/fetch-and-index-web-pages/","name":"Fetch and index single web pages"}}]}
```