Guide · Agent skill

How to use the Firecrawl Site Crawler for SEO agent skill

Updated 2026-10-29 · written with AI from the project's own documentation and checked against our catalogue data

The seo-firecrawl skill teaches an AI agent to inspect a whole website through the Firecrawl service, instead of looking at one page at a time. It is an add-on to the Claude SEO toolkit and needs a Firecrawl connection set up before it works. This guide explains what the skill does, how to install it, how to ask for it in plain language, and what to watch for on cost, permission and safety.

Firecrawl Site Crawler for SEOAgriciDaniel/claude-seo · MIT Download the skill

Quick start

1Install the Firecrawl extension. Run the install script at extensions/firecrawl/install.sh from the Claude SEO repository. This connects the Firecrawl MCP server.
2Get a Firecrawl API key. Create a Firecrawl account and obtain an API key. The source lists a free tier of 500 credits per month.
3Add the skill files. Download the ZIP from our page and unzip it into ~/.claude/skills/ or a project's .claude/skills/ folder, or upload it under Settings → Capabilities → Skills in the Claude apps.
4Confirm the tools are available. Ask the agent to map a small site. If it reports that Firecrawl tools are missing, the MCP server is not connected and the install step needs repeating.
5Start with a small map. Map the site first to get the URL list. This is cheaper than a crawl and shows you the page count before you spend credits.
6Run a targeted crawl. Crawl one section with includePaths and a low limit, then review the results before widening the scope.

What is an agent skill, and what does this one do?

An agent skill is a folder of instructions and files that an AI agent such as Claude can load when a task calls for it. The skill's SKILL.md file tells the agent when to use it and how to work with the tools available to it.

This skill covers four jobs: listing every URL on a site (map), crawling many pages to pull their content and links (crawl), scraping a single JavaScript-heavy page that plain fetching may miss (scrape), and searching within a site (search).

Map: discover URLs quickly, without fetching page content
Crawl: extract page content, metadata and links for many pages
Scrape: render one page's JavaScript and extract its content
Search: find pages on a site that match a query

Install the Firecrawl extension and the skill

The skill requires the Firecrawl extension to be installed first. The source gives this install script from the Claude SEO repository: extensions/firecrawl/install.sh. You need a Firecrawl account and API key as well.

To add the skill files, download the ZIP from our page and either unzip it into ~/.claude/skills/ for your user account, or into a project's .claude/skills/ folder. In the Claude apps, you can instead upload the ZIP under Settings → Capabilities → Skills.

Personal install: unzip into ~/.claude/skills/
Project install: unzip into the project's .claude/skills/ folder
Claude apps: Settings → Capabilities → Skills, then upload the ZIP

./extensions/firecrawl/install.sh

Check that the Firecrawl tools are connected

Before using any Firecrawl tool, the agent checks whether the Firecrawl MCP server is connected by looking for tools such as firecrawl_scrape. If those tools are not available, the skill tells you the extension is not installed and gives install instructions.

If you see an error saying FIRECRAWL_API_KEY is not set, the source says the MCP server is not configured and the fix is to run the install script.

How do you trigger the skill with natural requests?

The skill activates on phrases such as "crawl site", "map site", "full crawl", "find all pages", "broken links", "site structure", "discover pages", "JS rendering", or any request for site-wide analysis. You can also invoke it directly with the slash form shown in its quick reference.

Give the agent the site's full URL and say what you want to find. Describing the goal, such as broken links or a page inventory, helps the agent choose between map, crawl and scrape.

"Map example.com and tell me how many pages are under /blog/."
"Crawl the /products/ section of example.com and list broken internal links."
"Scrape this React product page and extract the main content."
"Search example.com for pages that mention our pricing."

/seo firecrawl map <url>
/seo firecrawl crawl <url>
/seo firecrawl scrape <url>
/seo firecrawl search <query> <url>

What limits and costs should you know before a crawl?

Firecrawl uses credits. The source lists a free tier of 500 credits per month, with 1 credit per page crawled or scraped and about 0.5 credits per URL discovered by a map. The crawl command defaults to a limit of 100 pages and a maximum depth of 3 links, with a hard maximum of 500 pages.

The skill is instructed to tell you the estimated credit use before large crawls. Ask for a small limit first, such as 50 pages, when you are testing.

Crawl defaults: limit 100 pages (max 500), depth 3
Map defaults: up to 5000 URLs discovered
Use includePaths such as /blog/* to limit a crawl to one section
Use excludePaths such as /admin/* to skip sections

Safety: scripts, network and permission

This skill runs scripts and uses the network, because it connects to the Firecrawl service. Read the install script before running it, and only point the skill at sites you are allowed to crawl.

Large crawls can burden a server and may breach a site's terms of service. The skill's error table notes that a site may block crawling (403) and advises checking robots.txt. Treat those as signals to stop and reconsider, not obstacles to work around.

Only crawl sites you own or have permission to analyse
Check robots.txt and the site's terms before large crawls
Keep limits low until you know the cost and load

Troubleshoot common Firecrawl errors

The skill lists several errors. A 402 Payment Required response means credits are used up; check usage in your Firecrawl account and upgrade the plan if needed. A 429 Too Many Requests response means you are rate limited; the source advises waiting 60 seconds and reducing crawl concurrency.

A 408 timeout means the page was too slow to render, so the source suggests increasing the timeout or trying without JS rendering. If Firecrawl is unavailable altogether, the skill suggests fetch_page.py for single-page checks or the WebFetch tool.

402: credits exhausted, check usage and plan
429: wait 60 seconds, reduce concurrency
408: increase timeout or drop JS rendering
403: check robots.txt, skip the site if blocked

What people use it for

Finding broken internal links across a site section by crawling with links output and checking for 404s.
Comparing URLs found by a map against the XML sitemap to spot orphan or missing pages.
Building a content inventory of page titles, meta descriptions and H1s across a site.
Scraping JavaScript-rendered React, Vue or Angular pages that plain fetching misses.
Searching a site for a keyword to check whether content on a topic already exists.

Frequently asked questions

Is the Firecrawl site crawler for SEO free?

Firecrawl has a free tier listed at 500 credits per month, with 1 credit per page crawled or scraped. Larger crawls need paid credits, so the skill asks you to estimate usage before big runs.

What do I need to use the seo-firecrawl skill?

You need the Firecrawl MCP server connected to your agent, a Firecrawl account and API key, and the Claude SEO extension install script at extensions/firecrawl/install.sh. Without the MCP server, the skill will not work.

How do I install the Firecrawl agent skill?

Download the ZIP from our page and unzip it into ~/.claude/skills/ or a project's .claude/skills/ folder. In the Claude apps, you can upload it under Settings → Capabilities → Skills, after running the Firecrawl install script.

Can I use this skill for a single page?

You can, but the skill itself says the built-in fetch_page.py script is free and sufficient for simple one-page checks. Use Firecrawl scrape when the page relies on JavaScript or needs clean markdown.

What is the difference between map and crawl?

Map discovers URLs only and is fast and cheap, at about 0.5 credits per URL. Crawl returns page content, metadata and links for each page, costing 1 credit per page.

What should I do if I get a 429 or 402 error?

A 429 means you are rate limited, so wait 60 seconds and reduce concurrency. A 402 means credits are exhausted, so check your usage in Firecrawl and upgrade your plan if needed.

New agent skills, repos and AI tools — every week

The best new skills and open-source projects, the fastest-growing repos, and one free prompt. Five minutes to read. Unsubscribe anytime.

Our verdict

This skill suits SEO specialists and developers who need site-wide audits, broken link checks or a fast page inventory, especially on JavaScript-rendered sites, and who already have a Firecrawl account. If you only need to check one page, or you cannot or should not crawl the site, use the free built-in fetch tool or pick a different approach.

Download the skill