Skip to content
#

Crawler

Star
crawler logo

A Web crawler, sometimes called a spider or spiderbot and often shortened to crawler, is an Internet bot that systematically browses the World Wide Web and that is typically operated by search engines for the purpose of Web indexing (web spidering).

Here are 9,278 public repositories matching this topic...

Language: All
Filter by language
All 9,278 Python 4,202 JavaScript 1,039 Go 589 TypeScript 468 Java 438 PHP 325 HTML 300 Jupyter Notebook 271 C# 216 Rust 174
Sort: Most stars
Sort options
Most stars Fewest stars Most forks Fewest forks Recently updated Least recently updated
Scrapling

D4Vinci / Scrapling

Sponsor
Star 84.4k

🕷️ An adaptive Web Scraping framework that handles everything from a single request to a full-scale crawl! Don't be shy, join here: https://discord.gg/EMgGbDceNQ and follow here for daily tips and tricks: https://x.com/Scrapling_dev

python crawler data automation ai mcp scraping crawling web-scraper web-scraping selectors xpath data-extraction stealth webscraping crawling-python playwright web-scraping-python ai-scraping mcp-server
  • Updated Sep 28, 2026
  • Python

NaiboWang / EasySpider

Sponsor
Star 44.6k

A visual no-code/code-free web crawler/spider易采集：一个可视化浏览器自动化测试/数据采集/网页爬虫软件，可以无代码图形化的设计和执行爬虫任务。别名：ServiceWrapper面向Web应用的智能化服务封装系统。

visualization html crawler scraper gui web spider frontend robotics parameters visual www data-collection batch-script batch-processing layman rpa visualprogramming code-free input-parameters
  • Updated Sep 17, 2026
  • JavaScript

feder-cr / invisible_playwright_mcp

Sponsor
Star 31.7k

Playwright MCP server undetected by anti-bots and captchas: AI agent browses the web on anti-detect stealth Firefox, Python, undetected browser automation, scraping, computer use.

productivity crawler scraper browser autonomous-agents market-research job-search lead-generation web-automation job-hunting playwright ai-agent ai-automation llm-agent agentic-ai model-context-protocol mcp-server web-research claude-code-plugin gemini-cli-extension
  • Updated Sep 25, 2026
  • Python
crawlee

apify / crawlee

Star 25.9k

Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.

nodejs javascript npm crawler scraper automation typescript web-crawler headless scraping crawling web-scraping web-crawling headless-chrome apify puppeteer playwright
  • Updated Sep 26, 2026
  • TypeScript
Douyin_TikTok_Download_API

Evil0ctal / Douyin_TikTok_Download_API

Sponsor
Star 20.4k

🚀 抖音、TikTok 数据采集与无水印视频下载 API，自托管，支持 MCP 调用与 Docker 一键部署。| Self-hosted TikTok & Douyin scraper and no-watermark video downloader — async REST API, MCP server, CLI and web console for posts, profiles, comments and playlists. Self-healing identity pool, PostgreSQL archive, one docker compose up.

crawler scraper downloader social-media spider rest-api openapi video-downloader web-scraping asyncio data-collection douyin tiktok fastapi tiktok-scraper tiktok-api douyin-api tiktok-downloader model-context-protocol mcp-server
  • Updated Sep 28, 2026
  • Python
Followers
585 followers
Website
github.com/topics/crawler
Wikipedia
Wikipedia