drio
Open app

Crawl4AI MCP Server

Source

Crawl4AI-MCP: A powerful web crawling and content analysis server that combines targeted web scraping with Claude AI processing. Extract insights from specific websites with customizable depth, content selection, and AI analysis capabilities.

Catalog onlyCatalog onlySTDIO

Overview

Crawl4AI-MCP is a powerful web crawling and content analysis server that combines targeted web scraping with Claude AI processing, allowing users to extract insights from specific websites with customizable depth and content selection.

To use Crawl4AI-MCP, clone the repository, install dependencies, set up your Anthropic API key, and start the server. You can then use the REST API to crawl websites and process the content.

  • Customizable web crawling depth and content selectors - AI-powered analysis using Claude models - Simple REST API for integration - Supports various AI processing tasks like summarization and extraction - Privacy-focused with local server operation
  1. Researchers extracting structured information from academic resources.
  2. Content creators analyzing competitor content.
  3. Data analysts processing web data for business intelligence.
  4. Developers integrating web content analysis into applications.
  5. Digital marketers tracking industry trends.

Add to your AI client

Use these steps to connect Crawl4AI MCP Server in Cursor, Claude, VS Code, and other MCP-compatible apps. The same JSON appears in the Use with menu above for one-click copy.

Cursor

Add this to your .cursor/mcp.json file in your project root, then restart Cursor.

.cursor/mcp.json

{
  "mcpServers": {
    "crawl4ai-mcp-vistiqx": {
      "command": "npx",
      "args": [
        "-y",
        "@modelcontextprotocol/server-crawl4ai-mcp-vistiqx"
      ]
    }
  }
}

Claude Desktop

Add this server entry to the mcpServers object in your Claude Desktop config, then restart the app.

~/Library/Application Support/Claude/claude_desktop_config.json (macOS) or %APPDATA%\Claude\claude_desktop_config.json (Windows)

{
  "mcpServers": {
    "crawl4ai-mcp-vistiqx": {
      "command": "npx",
      "args": [
        "-y",
        "@modelcontextprotocol/server-crawl4ai-mcp-vistiqx"
      ]
    }
  }
}

Claude Code

Add this to your project's .mcp.json file. Claude Code will detect it automatically.

.mcp.json (project root)

{
  "mcpServers": {
    "crawl4ai-mcp-vistiqx": {
      "command": "npx",
      "args": [
        "-y",
        "@modelcontextprotocol/server-crawl4ai-mcp-vistiqx"
      ]
    }
  }
}

VS Code (Copilot)

Add this to your .vscode/mcp.json file. Requires the GitHub Copilot extension with MCP support enabled.

.vscode/mcp.json

{
  "servers": {
    "crawl4ai-mcp-vistiqx": {
      "type": "stdio",
      "command": "npx",
      "args": [
        "-y",
        "@modelcontextprotocol/server-crawl4ai-mcp-vistiqx"
      ]
    }
  }
}

Windsurf

Add this to your Windsurf MCP config file, then restart Windsurf.

~/.codeium/windsurf/mcp_config.json

{
  "mcpServers": {
    "crawl4ai-mcp-vistiqx": {
      "command": "npx",
      "args": [
        "-y",
        "@modelcontextprotocol/server-crawl4ai-mcp-vistiqx"
      ]
    }
  }
}

Cline

Open Cline settings, navigate to MCP Servers, and add this server configuration.

Cline MCP Settings (via UI)

{
  "mcpServers": {
    "crawl4ai-mcp-vistiqx": {
      "command": "npx",
      "args": [
        "-y",
        "@modelcontextprotocol/server-crawl4ai-mcp-vistiqx"
      ]
    }
  }
}

FAQ

Can Crawl4AI-MCP crawl any website?

Yes, it can crawl any website as long as it respects the robots.txt directives.

Is there a cost associated with using Crawl4AI-MCP?

The server is free to use, but you will need your own Claude API key for processing.

How can I customize the crawling process?

You can customize the crawling depth, content selectors, and AI processing tasks through the API.7:["$","div",null,{"className":"container mx-auto flex flex-col gap-4","children":["$L26","$L27",["$","$L28",null,{"currentProject":{"id":2789,"uuid":"5634bb3b-6578-4e9a-9b2d-a9d23d04cda2","name":"Crawl4AI-MCP","title":"Crawl4AI MCP Server","description":"Crawl4AI-MCP: A powerful web crawling and content analysis server that combines targeted web scraping with Claude AI processing. Extract insights from specific websites with customizable depth, content selection, and AI analysis capabilities.","avatar_url":"https://avatars.githubusercontent.com/u/197793535?v=4","created_at":"2025-03-14T01:54:45.238Z","updated_at":"2025-03-14T02:01:09.718Z","status":"created","author_name":"Vistiqx","author_avatar_url":"https://avatars.githubusercontent.com/u/197793535?v=4","tags":"[]","category":"research-and-data","is_featured":false,"sort":1,"url":"https://github.com/Vistiqx/Crawl4AI-MCP","target":"_self","content":"$29","summary":"$2a","img_url":null,"type":null,"metadata":"{\"star\":\"20\",\"license\":\"View license\",\"language\":\"JavaScript\",\"is_official\":false,\"latest_commit_time\":\"2025-04-02 11:59:21\"}","user_uuid":null,"tools":null,"sse_url":null,"sse_provider":null,"sse_params":null,"is_official":false,"server_command":null,"server_params":null,"server_config":null,"allow_call":false,"is_innovation":false,"is_dxt":false,"dxt_manifest":null,"dxt_file_url":null,"is_audit":false},"randomProjects":[],"currentServerKey":"$undefined"}]]}]

Research and data