Home / Skills / crawlee-python
Crawlee-A web scraping and browser automation library for Python to build reliable crawlers. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Parsel, BeautifulSoup, Playwright, and raw HTTP. Both headful and headless mode. With proxy rotation.
What it is, and how people use it
Crawlee-A web scraping and browser automation library for Python to build reliable crawlers. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Parsel, BeautifulSoup, Playwright, and raw HTTP. Both headful and headless mode. With proxy rotation. It sits on the Dev Agents & Programs shelf of the catalog, maintained by @apify.
Why builders keep it installedDev Agents & Programs benefits
These CLIs and dashboards solve one job well and install with a single command. Value shows up in the first hour, not after a config weekend.
Most of them ship an MCP server, skill or hooks - so the same tool you use manually becomes an action your agent can call safely.
Star counts here track repos people actually run daily: the issue history shows bug classes fixed under production pressure, not demo-week polish.
Every tool on this page ships under an open license and can be self-hosted. Your usage data and your configs stay yours.
Contributorsloading from GitHub…
Languagesestimated from snapshot
- Python 80.0%
- Shell 12.0%
- Go 8.0%