Web Content Fetcher MCP Server

The Web Content Fetcher provides capabilities for retrieving and processing content from web pages. It enables LLMs to access information from the internet by fetching URLs and converting HTML to markdown format for easier consumption. This tool supports advanced features like content truncation and pagination through the start_index parameter, allowing models to read webpages in manageable chunks until they find the information they need. It's an essential tool for LLMs that need to access and process web-based information.

Overview

The Web Content Fetcher is a Model Context Protocol server that allows LLMs to retrieve and process content from web pages. It fetches URLs from the internet and converts HTML to markdown format, making it easier for models to consume web content.

Security Warning: This server can access local/internal IP addresses and may represent a security risk. Exercise caution when using this MCP server to ensure it doesn't expose any sensitive data.

Installation

There are several ways to install and run the Web Content Fetcher:

Using uv (Recommended)

When using uv, no specific installation is needed. You can use uvx to directly run the server:

uvx mcp-server-fetch

Using pip

You can install the server via pip:

pip install mcp-server-fetch

After installation, run it as a script:

python -m mcp_server_fetch

Using Docker

You can also run the server using Docker:

docker run -i --rm mcp/fetch

Configuration

For Claude.app

Add to your Claude settings:

Using uvx:

"mcpServers": {
  "fetch": {
    "command": "uvx",
    "args": ["mcp-server-fetch"]
  }
}

Using Docker:

"mcpServers": {
  "fetch": {
    "command": "docker",
    "args": ["run", "-i", "--rm", "mcp/fetch"]
  }
}

Using pip installation:

"mcpServers": {
  "fetch": {
    "command": "python",
    "args": ["-m", "mcp_server_fetch"]
  }
}

For VS Code

Add the following to your User Settings (JSON) file in VS Code or to .vscode/mcp.json in your workspace:

Using uvx:

{
  "mcp": {
    "servers": {
      "fetch": {
        "command": "uvx",
        "args": ["mcp-server-fetch"]
      }
    }
  }
}

Using Docker:

{
  "mcp": {
    "servers": {
      "fetch": {
        "command": "docker",
        "args": ["run", "-i", "--rm", "mcp/fetch"]
      }
    }
  }
}

Advanced Configuration

robots.txt Handling

By default, the server obeys a website's robots.txt file for model-initiated requests but not for user-initiated requests. To disable this behavior, add --ignore-robots-txt to the args list in your configuration.

User-Agent Customization

The server uses different user-agents depending on whether the request came from the model or was user-initiated. You can customize this by adding --user-agent=YourUserAgent to the args list.

Proxy Configuration

Configure the server to use a proxy with the --proxy-url argument.

Debugging

Use the MCP inspector to debug the server:

npx @modelcontextprotocol/inspector uvx mcp-server-fetch

Or if you're developing on it:

cd path/to/servers/src/fetch
npx @modelcontextprotocol/inspector uv run mcp-server-fetch