drio
Open app

Crawlab MCP Server

Source

Catalog onlyCatalog onlySTDIO

Overview

Crawlab MCP Server is a Model Context Protocol (MCP) server designed for Crawlab, enabling AI applications to interact seamlessly with Crawlab's functionalities.

To use the Crawlab MCP Server, install it as a Python package or run it locally using Docker. After installation, start the server and configure your AI application to connect to it.

  • Spider management (create, read, update, delete) - Task management (run, cancel, restart) - File management (read, write) - Resource access (spiders, tasks) - Natural language interaction with Crawlab functionalities
  1. Automating web scraping tasks through AI commands.
  2. Managing and monitoring spider tasks in real-time.
  3. Integrating with AI applications for enhanced data extraction capabilities.

Add to your AI client

Use these steps to connect Crawlab MCP Server in Cursor, Claude, VS Code, and other MCP-compatible apps. The same JSON appears in the Use with menu above for one-click copy.

Cursor

Add this to your .cursor/mcp.json file in your project root, then restart Cursor.

.cursor/mcp.json

{
  "mcpServers": {
    "crawlab-mcp-server-crawlab-team": {
      "command": "npx",
      "args": [
        "-y",
        "@modelcontextprotocol/server-crawlab-mcp-server-crawlab-team"
      ]
    }
  }
}

Claude Desktop

Add this server entry to the mcpServers object in your Claude Desktop config, then restart the app.

~/Library/Application Support/Claude/claude_desktop_config.json (macOS) or %APPDATA%\Claude\claude_desktop_config.json (Windows)

{
  "mcpServers": {
    "crawlab-mcp-server-crawlab-team": {
      "command": "npx",
      "args": [
        "-y",
        "@modelcontextprotocol/server-crawlab-mcp-server-crawlab-team"
      ]
    }
  }
}

Claude Code

Add this to your project's .mcp.json file. Claude Code will detect it automatically.

.mcp.json (project root)

{
  "mcpServers": {
    "crawlab-mcp-server-crawlab-team": {
      "command": "npx",
      "args": [
        "-y",
        "@modelcontextprotocol/server-crawlab-mcp-server-crawlab-team"
      ]
    }
  }
}

VS Code (Copilot)

Add this to your .vscode/mcp.json file. Requires the GitHub Copilot extension with MCP support enabled.

.vscode/mcp.json

{
  "servers": {
    "crawlab-mcp-server-crawlab-team": {
      "type": "stdio",
      "command": "npx",
      "args": [
        "-y",
        "@modelcontextprotocol/server-crawlab-mcp-server-crawlab-team"
      ]
    }
  }
}

Windsurf

Add this to your Windsurf MCP config file, then restart Windsurf.

~/.codeium/windsurf/mcp_config.json

{
  "mcpServers": {
    "crawlab-mcp-server-crawlab-team": {
      "command": "npx",
      "args": [
        "-y",
        "@modelcontextprotocol/server-crawlab-mcp-server-crawlab-team"
      ]
    }
  }
}

Cline

Open Cline settings, navigate to MCP Servers, and add this server configuration.

Cline MCP Settings (via UI)

{
  "mcpServers": {
    "crawlab-mcp-server-crawlab-team": {
      "command": "npx",
      "args": [
        "-y",
        "@modelcontextprotocol/server-crawlab-mcp-server-crawlab-team"
      ]
    }
  }
}

FAQ

What is the MCP Server used for?

The MCP Server allows AI applications to communicate with Crawlab for managing web scraping tasks and resources.

How do I install the MCP Server?

You can install it as a Python package or run it using Docker. Detailed instructions are provided in the documentation.

Can I use the MCP Server with any AI application?

Yes! The MCP Server is designed to work with various AI applications that can send natural language queries.7:["$","div",null,{"className":"container mx-auto flex flex-col gap-4","children":["$L26","$L27",["$","$L28",null,{"currentProject":{"id":1767,"uuid":"58fb5c8a-21c9-41de-9fd8-e43748da95e3","name":"crawlab-mcp-server","title":"Crawlab MCP Server","description":"","avatar_url":"https://avatars.githubusercontent.com/u/49225390?v=4","created_at":"2025-03-05T05:13:43.516Z","updated_at":"2025-03-12T10:20:31.981Z","status":"created","author_name":"crawlab-team","author_avatar_url":"https://avatars.githubusercontent.com/u/49225390?v=4","tags":"crawlab,mcp-server,ai-integration","category":"developer-tools","is_featured":false,"sort":1,"url":"https://github.com/crawlab-team/crawlab-mcp-server","target":"_self","content":"$29","summary":"$2a","img_url":null,"type":null,"metadata":"{\"star\":\"1\",\"license\":\"\",\"language\":\"Python\",\"is_official\":false,\"latest_commit_time\":\"2025-03-06 17:45:44\"}","user_uuid":null,"tools":null,"sse_url":null,"sse_provider":null,"sse_params":null,"is_official":false,"server_command":null,"server_params":null,"server_config":null,"allow_call":false,"is_innovation":false,"is_dxt":false,"dxt_manifest":null,"dxt_file_url":null,"is_audit":false},"randomProjects":[],"currentServerKey":"$undefined"}]]}]

Developer tools