Analyze and generate robots.txt with AI crawler detection for 20+ bots like GPTBot and ClaudeBot.
Use this profile to copy client config, check auth requirements, review tools and resources, and compare related MCP servers before adding it to an AI client.
npx -y robotstxt-ai-mcpAdd this server entry to the mcpServers object in your Claude Desktop config, then restart the app.
{
"mcpServers": {
"io-github-sharozdawa-robotstxt-ai": {
"command": "npx",
"args": [
"-y",
"robotstxt-ai-mcp"
]
}
}
}~/Library/Application Support/Claude/claude_desktop_config.json%APPDATA%\Claude\claude_desktop_config.jsonNo remote HTTP endpoint is advertised. Use the package or stdio setup shown in Install.
robotstxt-ai is an MCP server for Analyze and generate robots.txt with AI crawler detection for 20+ bots like GPTBot and ClaudeBot.. It supports STDIO transport.
Use the generated config in Install. This server runs with npx -y robotstxt-ai-mcp; add any required environment variables before starting your client.
Choose the Claude Desktop tab in Install and copy the config for npx -y robotstxt-ai-mcp. Add required environment variables before starting Claude Desktop.
Choose the Claude Code tab in Install and copy the config for npx -y robotstxt-ai-mcp. Add required environment variables before starting Claude Code.
Choose the Codex tab in Install and copy the config for npx -y robotstxt-ai-mcp. Add required environment variables before starting Codex.
Choose the Cursor or VS Code tab in Install and copy the config for npx -y robotstxt-ai-mcp. Add required environment variables before starting Cursor or VS Code.
robotstxt-ai uses STDIO transport. Use the package or command config in Install.
robotstxt-ai inventory is listed when the MCP endpoint exposes tools, resources, or prompts. Some servers require auth first.
robotstxt-ai does not advertise a verified auth requirement. If discovery fails, it may still need provider login, an API key, a bearer token, or a session header.
| Package | Registry | Version | Inputs |
|---|---|---|---|
robotstxt-ai-mcpstdio | npm | 1.0.1 | None advertised |