A next-generation crawling and spidering framework.
17.6kstars
1.2kforks
17open issues
2026-09-29last pushed
MITlicense
What it is, and how people use it
A next-generation crawling and spidering framework. It sits on the Dev Agents & Programs shelf of the catalog, maintained by @projectdiscovery.
✓Drop-in for Claude Code, Codex CLI and Cursor - install and it just runs.
✓Read the SKILL.md / README first; most setups are one command from the docs.
✓Teams use it to standardise how agents behave across every repo in the org.
clicrawlergocrawlerhacktoberfestheadless
Why builders keep it installedDev Agents & Programs benefits
Day-one leverage
These CLIs and dashboards solve one job well and install with a single command. Value shows up in the first hour, not after a config weekend.
Composes with your agent
Most of them ship an MCP server, skill or hooks - so the same tool you use manually becomes an action your agent can call safely.
Built under real workloads
Star counts here track repos people actually run daily: the issue history shows bug classes fixed under production pressure, not demo-week polish.
Free and open source
Every tool on this page ships under an open license and can be self-hosted. Your usage data and your configs stay yours.
Contributorsloading from GitHub…
Languagesestimated from snapshot
- Go 87.8%
- Shell 12.2%
Community pulsesnapshot ratios, normalized
starsforksissues ×3watchers