TypeScript
Supercharge your AI agents with data from the web and beyond. Building the library for superintelligence. 🔥
954Curated repositories
44,614.2kTotal GitHub stars
Open sourceBuilt by the community
crawler repositories
Hand-picked tools to improve your workflow.
Python
Scrapy, a fast high-level web crawling & scraping framework for Python.
TypeScript · Trending
Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.
Go · Trending
ScopeSentry-Cyberspace mapping, subdomain enumeration, port scanning, sensitive information discovery, vulnerability scanning, distributed nodes