Open Crawler
A web crawler and text extractor with robots.txt compliance, rate limiting, and page size protection.
Install
Install and configure the MCP from https://github.com/naoto24kawa/open-crawler-mcp-server now. Follow the repository's installation instructions, ask me for anything you can't complete yourself, and verify its tools load.Related servers
Composer Package READMEnaoto24kawaFetches Composer package README and usage information from Packagist.Open Custom Searchnaoto24kawaPerform web searches using Google's Custom Search API.Package README MCP Serversnaoto24kawaA collection of MCP servers for fetching READMEs from various package managers.Everythingmodelcontextprotocol91KReference / test server with prompts, resources, and tools