> ## Documentation Index
> Fetch the complete documentation index at: https://docs.flowx.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Web Page Extractor

> Collect readable content from web page URLs with configurable crawling depth, link following, and scrape speed.

## Overview

The **Web Page Extractor** node is a workflow node that collects readable content from web page URLs. It supports static URL lists and dynamic URL generation, configurable crawling depth with link following, and adjustable scrape speed presets.

![Web Page Extractor node configuration with URLs, Crawl Depth, and Scrape Speed settings](https://s3.eu-west-1.amazonaws.com/docx.flowx.ai/5.6/web_page_extractor_node.png)

> [card] Static or dynamic URLs
>
> Provide a fixed list of URLs or generate them dynamically from workflow data

> [card] Link following
>
> Optionally follow links on pages up to a configurable depth

> [card] PDF processing
>
> Extract content from PDF files linked on the page

> [card] Scrape speed control
>
> Choose from speed presets or define custom rate limits and concurrency

> [card] File downloads
>
> Download attached files (.docx, .xlsx, .pdf) found during crawling and store them automatically

***

## Configuration

**1. Open your workflow**

Open your workflow in **Integration Designer**.

**2. Add the node**

Add a **Web Page Extractor** node from the **Tools** category in the left panel.

**3. Configure URL source and extraction settings**

Configure the settings described below.

***

### URL source

**URL Mode** `enum` *required*

How URLs are provided to the node.

| Mode | Description |
| - | - |
| **Static** | Provide a fixed list of up to 20 URLs |
| **Dynamic** | Generate URLs from a workflow data key using `${expression}` syntax |

**Default:** `Static`

**URLs** `string[]`

List of URLs to extract content from. Only available when **URL Mode** is `Static`.

**Maximum:** 20 URLs

URLs must use `http://` or `https://` protocol. Supports `${variable}` placeholders for dynamic values.

**Dynamic Link** `string`

A workflow data key or expression that resolves to a URL at runtime. Only available when **URL Mode** is `Dynamic`.

**Example:** `${inputData.targetUrl}`

***

### Crawl depth

**Follow Links** `boolean`

When turned on, the extractor follows links found on the page up to the configured depth.

**Default:** OFF

**Depth of Crawling** `number`

How many levels of links to follow from the starting page. Only available when **Follow Links** is turned on. Displayed in the panel as a slider with the current value shown to the right (for example, *2 levels*).

**Range:** 0–10

**Default:** `0`

**Crawl URLs Containing** `string[]`

Optional list of substring filters applied to discovered links — only URLs that contain at least one of the configured fragments are followed. Click **Set Filters** to open the filter editor. Only available when **Follow Links** is turned on.

**Process Linked PDFs** `boolean`

When turned on, extracts content from PDF files linked on the page. Only available when **Follow Links** is turned on.

**Default:** OFF

***

### Download attached files

When turned on, the extractor downloads attached files (.docx, .xlsx, .pdf) found during crawling and stores them using the configured document destination.

**Download Attached Files** `boolean`

When turned on, files discovered during crawling are downloaded and stored automatically. Supported file types include `.docx`, `.xlsx`, and `.pdf`.

**Default:** OFF

**Document Destination** `enum`

Where downloaded files are stored. Only available when **Download Attached Files** is turned on.

| Option | Description |
| - | - |
| **Document Plugin** | Store files through the FlowX Document Plugin. Requires **Folder Name**. |
| **S3 Protocol** | Store files directly using S3-compatible storage. |

**Folder Name** `string` *required*

Identifier used to associate the file with its business owner. Only available when **Download Attached Files** is turned on and **Document Destination** is `Document Plugin`.

***

### Scrape speed

**Scrape Speed** `enum` *required*

Controls how aggressively the node requests pages from the target server.

| Preset | Description |
| - | - |
| **Slow** | Conservative rate limiting — best for fragile or rate-limited servers |
| **Moderate** | Balanced speed and reliability |
| **Fast** | Aggressive crawling — assumes the target server can handle high traffic |
| **Custom** | Define your own rate limit and concurrency |

**Default:** `Moderate`

**Rate Limit** `number`

Maximum requests per second. Displayed in the panel as a slider labelled in `req/s`. Only available when **Scrape Speed** is `Custom`.

**Default:** `2 req/s`

**Concurrency** `number`

Number of concurrent requests. Displayed in the panel as a slider labelled in `parallel`. Only available when **Scrape Speed** is `Custom`.

**Default:** `3 parallel`

***

### Response key

**responseKey** `string` *required*

The key where extracted content is stored in the workflow data.

**Example:** `extractedContent`

***

### Timeout and retry

**Timeout** `number`

Request timeout in milliseconds. If the extraction exceeds this duration, the node fails.

**Retry Config** `object`

Optional retry strategy for failed requests.

| Field | Description | Default |
| - | - | - |
| **Retry Type** | `Fixed` or `Exponential` backoff | — |
| **Max Attempts** | Maximum retry attempts | `2` |
| **Backoff Period** | Delay between retries (ms) | `1000` |
| **Max Backoff Period** | Maximum delay for exponential backoff (ms) | `120000` |
| **Backoff Multiplier** | Multiplier for exponential backoff | `2` |

***

## Best practices

> [card] Start with Moderate speed
>
> Use the Moderate preset unless you know the target server's capacity. Switch to Fast only for internal servers you know can handle the load.

> [card] Limit crawl depth
>
> Keep **Depth of Crawling** low (1–3) to avoid excessive page requests. Deep crawls can be slow and may trigger rate limiting.

> [card] Use dynamic URLs for runtime flexibility
>
> When the target URL comes from user input or a previous workflow step, use Dynamic mode with `${expression}` placeholders.

> [card] Set timeouts for external sites
>
> Always configure a timeout when crawling external websites to avoid blocking the workflow on slow or unresponsive servers.

***

## Related resources

> [card] [Extract Data from File](https://docs.flowx.ai/6.x/ai-platform/agent-builder/extract-data-from-file.md)
>
> Extract text and data from documents and images

> [card] [AI node types](https://docs.flowx.ai/6.x/ai-platform/agent-builder/node-types.md)
>
> Overview of all AI workflow node types

> [card] [Integration Designer](https://docs.flowx.ai/6.x/docs/platform-deep-dive/integrations/integration-designer.md)
>
> Build and manage integration workflows


## Related topics

- [FlowX.AI 5.6.0 Release Notes](https://docs.flowx.ai/release-notes/v5.x/v5.6.0-march-2026/v5.6.0-march-2026.md)
- [FlowX.AI 5.9.0 Release Notes](https://docs.flowx.ai/release-notes/v5.x/v5.9.0-june-2026/v5.9.0-june-2026.md)
- [FlowX.AI 5.7.0 Release Notes](https://docs.flowx.ai/release-notes/v5.x/v5.7.0-april-2026/v5.7.0-april-2026.md)
- [Browser Automation](https://docs.flowx.ai/6.x/ai-platform/agent-builder/browser-automation.md)
- [Deployment guidelines v5.6.0](https://docs.flowx.ai/release-notes/v5.x/v5.6.0-march-2026/deployment-guidelines-v5.6.md)


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.