Skip to content
Get packs

Guide · Open source

How to use Anubis: setup, bot policies and tips

Updated · written with AI from the project's own documentation and checked against our catalogue data

Anubis is an open-source web AI firewall utility that sits in front of a website and runs challenges on incoming connections to filter out scraper bots. This guide explains what it does, how to get the repository and configure it, and where it can cause problems. It is aimed at people running small sites or self-hosted services who are being overwhelmed by AI crawler traffic.

AnubisTecharoHQ/anubis · ★ 23.9k · MIT

Quick start

  1. 1Decide if Anubis fits. Confirm you cannot or will not use Cloudflare, and that you are prepared to write bot policies. If your site needs search engine or archive crawling, plan the allowlists first.
  2. 2Clone the repository. Run the README's git clone command to get the source code from GitHub.
  3. 3Read the documentation. Open the Anubis documentation site and review the bot policy definitions page before deploying.
  4. 4Define bot policies. Write rules that allowlist the good bots you want to keep, so legitimate crawlers are not blocked.
  5. 5Test with real traffic. Watch for legitimate visitors or good bots being challenged or blocked, and adjust your policies.
  6. 6Get support if needed. Open a GitHub issue with full diagnostic details if you hit a problem.

What Anubis does and who it is for

Anubis weighs the connection of each incoming HTTP request using one or more challenges, in order to protect upstream resources from scraper bots. The project describes its goal as protecting small sites from the flood of requests from AI companies.

It is a fairly blunt tool. The project notes it can block smaller scrapers and may also block good bots such as the Internet Archive unless you configure bot policy definitions to allowlist them.

  • Best for operators of small websites or self-hosted services overwhelmed by AI crawler traffic.
  • Best for people who cannot or will not use Cloudflare to protect an origin server.
  • Best for admins willing to write bot policy rules to allowlist good bots they want to keep.
  • Not ideal if your site relies on search engines or archives crawling it.
  • Not ideal if you already have Cloudflare or a similar service in front of your origin, which the project says is probably sufficient.

Get the Anubis source code

The README's install instruction is to clone the repository from GitHub. The README does not list prerequisites, so check the project's documentation site for build and deployment details before you start.

The project's documentation is hosted at anubis.techaro.lol. The project is licensed under MIT and written in Go.

git clone https://github.com/TecharoHQ/anubis

Write bot policy definitions before going live

The README stresses that Anubis is not a plug-and-play setting. Because it can over-block, you should define bot policies that explicitly allowlist the crawlers you want to keep, such as search or archive bots.

The project says it is working on a curated set of known good bots to balance discoverability and uptime, but the README does not describe that set as available. Plan to maintain your own policy rules.

  • Read the bot policy definitions documentation linked from the project docs.
  • List the good bots your site depends on for discovery or archiving.
  • Test changes and watch for legitimate visitors or crawlers being challenged or blocked.

Compare Anubis with Cloudflare and other options

The README suggests that in most cases Cloudflare protecting the origin is probably enough. Anubis is positioned for situations where Cloudflare cannot or will not be used.

Other projects listed as comparable to Anubis are go-away and haproxy-protection, and the hosted option Cloudflare. Compare their features and configuration effort before choosing.

  • Cloudflare: the README suggests it is probably sufficient for most cases.
  • go-away: a comparable project listed alongside Anubis.
  • haproxy-protection: a comparable project listed alongside Anubis.

Get help if something goes wrong

If you run into issues running Anubis, open an issue on the project's GitHub issue tracker. The README asks you to include all the information needed to diagnose the problem.

For live chat, the README points to the maintainer's Patreon and a channel named #anubis in the Patron discord.

  • Include your configuration and the exact error or behaviour you see.
  • Describe which bots or visitors were affected.

What people use it for

  • Protecting a small self-hosted site from floods of AI crawler requests.
  • Shielding a origin server when Cloudflare cannot be used.
  • Allowlisting specific good bots while challenging scrapers.

Frequently asked questions

Is Anubis free and open source?

Yes. The repository is licensed under MIT, so you can use and modify it under those terms. Read the LICENSE file in the repository for the full terms.

What are the requirements to run Anubis?

The README does not list prerequisites. Check the project documentation at anubis.techaro.lol for build and deployment requirements.

How do I install Anubis?

The README's install step is to clone the repository with git clone https://github.com/TecharoHQ/anubis. Further setup steps are in the project documentation.

Does Anubis block Google or the Internet Archive?

It may. The README warns that Anubis can inhibit good bots such as the Internet Archive. You can use bot policy definitions to explicitly allowlist them.

Should I use Anubis or Cloudflare?

The README says that in most cases you should not need Anubis and can probably use Cloudflare to protect an origin. Anubis is intended for cases where Cloudflare cannot or will not be used.

What are alternatives to Anubis?

The project lists go-away, haproxy-protection and Cloudflare as comparable options. Compare how each handles bot policies and configuration effort.

New agent skills, repos and AI tools — every week

The best new skills and open-source projects, the fastest-growing repos, and one free prompt. Five minutes to read. Unsubscribe anytime.

Our verdict

Anubis suits operators of small sites or self-hosted services who cannot use Cloudflare and are willing to write and maintain bot policy rules. If you want a plug-and-play setting, need search engines or archives to crawl you without extra configuration, or already have Cloudflare in front of your origin, pick a different option.

Install options & downloads