katana
A command-line web crawler that visits pages and collects URLs, for security testing and automation.
- Stars
- 17.8k
- Forks
- 1.2k
- Licence
- MIT
- Language
- Go
- Last update
- 4 days ago
- Type
- CLI
What it is
Katana is a command-line tool that automatically visits a website, follows its links, and lists the URLs and endpoints it finds. It can run without a browser for speed, or in headless mode to handle pages that rely on JavaScript. People use it to map out a site's pages and inputs as a first step in reviewing it or feeding other tools.
- Crawl a single site to a set depth and save the discovered URLs to a file
- Run headless mode to discover URLs that appear only after JavaScript runs
- Use form extraction to list form, input, and select elements in JSONL output
- Limit crawling by scope rules, a crawl time limit, or a maximum number of pages per domain
Great for
- Security testers mapping the pages and endpoints of a site they are authorized to test
- Developers who need a scriptable crawler that fits into automation pipelines
- People who want to crawl JavaScript-heavy pages using headless Chrome
- Anyone who wants structured JSON output of discovered URLs and forms
Not ideal for
- Casual users wanting a point-and-click tool; katana is a terminal program with many flags
- Crawling sites you do not have permission to test, since aggressive crawling can be unwanted or unlawful
- Projects needing features the README does not describe, since the README here was truncated and some options are marked experimental
How to get it
- 1
Clone and build from source
You'll need: Go 1.26.10+ (per the README) · Google Chrome, only if using headless mode with a local browser
Install katana with Go
CGO_ENABLED=1 go install github.com/projectdiscovery/katana/cmd/katana@latest - 2
Run it
- Install katana using the command above, or download a pre-compiled binary from the release page
- Run katana -h to see the available flags
- Crawl a site with: katana -u https://example.com (replace with a site you are authorized to test)
- Add -d to change crawl depth, or -headless for JavaScript-heavy pages
- 3
Get the most out of it
- Use -js-crawl to find endpoints inside JavaScript files
- Use -jsonl or JSON output options to feed results into other tools via STDIN and STDOUT
- Use -known-files to crawl robots.txt and sitemap.xml, with a minimum depth of 3
- Use -fsu or -iqp to reduce duplicate-looking URLs
- Use the Docker image if you want to avoid a local Go install
Downloads
Strengths
- Offers both standard and headless (Chrome-based) crawling in one tool
- Flexible input and output options, including STDIN, lists, STDOUT, files, and JSON
- Many configuration options for depth, strategy, filtering, and scope
Trade-offs
- Many flags and experimental features make it harder to learn; some options are marked experimental
- Headless mode needs a Chrome installation and extra setup on some systems
- The README is truncated here, so some details and defaults could not be fully verified
How it compares
| Project | Stars | Type | Level | Updated | Best for |
|---|---|---|---|---|---|
| katana (this) | 17.8k | CLI | intermediate | 4 days ago | Security testers mapping the pages and endpoints of a site they are authorized to test |
| spider-flow | 11.4k | App | intermediate | 3.4 years ago | Non-programmers who need to collect data from websites and prefer visual flow definitions |
| lux | 31.8k | CLI | beginner | 7 months ago | People who want to save videos from supported sites to a local file from the command line |
| refine | 36k | Framework | intermediate | 2 months ago | Developers building internal admin panels or back-office tools in React |
| newspaper | 15.2k | Library | intermediate | 11 days ago | Python developers building news aggregation or article collection scripts |
| chromedp | 13.3k | Library | intermediate | 4 days ago | Go developers who want to automate or test web pages in code |
Stats synced from GitHub. The assessment above was drafted with AI from the project's README on 2026-10-29 and may miss recent changes — check the README before relying on a command. MIT licence: free to use, modify and distribute, including commercially, as long as you keep the copyright notice.