Skip to content
Get packs

Guide · Open source

How to use arc-task-gen: generate private ARC-AGI-1-style tasks

Updated · written with AI from the project's own documentation and checked against our catalogue data

arc-task-gen is an open-source Python project (MIT licence) that generates new ARC-AGI-1-style puzzle tasks designed to look statistically similar to the public ARC-AGI-1 dataset. It is aimed at researchers and teams who want a private test set to compare against public benchmark results. This guide explains what the tool is for, how to get the code, what the output looks like, and where the public documentation stops.

BDH-CQpathwaycom/arc-task-gen · ★ 10.9k · MIT

Quick start

  1. 1Check Python. Confirm Python is installed, since it is the listed prerequisite.
  2. 2Clone the repository. Run git clone https://github.com/pathwaycom/arc-task-gen to download the code.
  3. 3Open instructions.md. Read the instructions.md file in the cloned repository for the generation steps.
  4. 4Generate tasks. Follow the generation steps in instructions.md to produce new ARC-style tasks.
  5. 5Check the tasks.json output. Confirm the file has the standard ARC structure with train and test lists.
  6. 6Run your evaluation. Feed tasks.json to an ARC evaluation harness and compare results with the public ARC-AGI-1 set.

What arc-task-gen does

ARC tasks ask a model to infer a visual rule from a few input-output examples. The public ARC-AGI-1 benchmark cannot fully rule out that a model has seen similar tasks before. arc-task-gen creates new tasks with similar properties so that results on the public set can be compared with results on newly generated puzzles.

Install arc-task-gen: clone and build

The project uses a clone-and-build install method and requires Python. The README gives this command to clone the repository:

  • Make sure Python is installed on your machine.
  • Clone the repository with the command below.
git clone https://github.com/pathwaycom/arc-task-gen

Generate tasks and read the output format

The generated tasks.json file follows the standard ARC format, with a train list and a test list. The README says it is compatible with existing ARC evaluation harnesses, so you can feed it to tools that already read ARC JSON.

The README does not include the generation commands. It points to a separate instructions.md file for the steps to start generating tasks. Follow that file once you have cloned the repository.

  • Output file: tasks.json
  • Structure: {"train": [], "test": []}
  • Generation steps: see instructions.md in the repository
{ "train": [], "test": [] }

Use the generated tasks to compare models

The main use is evaluation. Run a model on the public ARC-AGI-1 evaluation set and on your generated tasks, then compare the two. The generated set is meant to be private, so keep it out of public training data if you want the comparison to mean something.

  • Compare public benchmark scores against results on newly generated tasks.
  • Hold a private test set alongside the public ARC-AGI-1 evaluation set.
  • Check how the BDH-CQ paper's evaluation was set up.

Limits and what the README does not cover

arc-task-gen is a task generator, not a model and not a game. Its output is data for evaluation harnesses. The README does not provide a full install path beyond cloning, and the generation steps live in a separate file that was not included in the source reviewed here.

  • Not a chatbot or general-purpose AI model.
  • Not a ready-made puzzle game.
  • Install and generation details are in instructions.md, not the README.

What people use it for

  • Evaluating a few-shot reasoning model on ARC-style puzzles it is unlikely to have seen.
  • Building a private test set to sit beside the public ARC-AGI-1 evaluation set.
  • Comparing public benchmark scores with results on newly generated tasks.
  • Studying how the BDH-CQ paper's evaluation was set up.

Frequently asked questions

Is arc-task-gen free to use?

The repository is licensed under MIT. Check the LICENSE file in the repository for the full terms before you use or redistribute it.

What do I need to install arc-task-gen?

The README lists Python as the prerequisite and uses a clone-and-build method. Clone the repository with git clone https://github.com/pathwaycom/arc-task-gen.

What is the output of arc-task-gen?

It writes a tasks.json file in the standard ARC format, with train and test lists. The README says it is compatible with existing ARC evaluation harnesses.

Where are the generation steps?

The README points to instructions.md in the repository for the steps to start generating tasks. Those steps were not included in the source reviewed here.

Is arc-task-gen a chatbot or AI model?

No. The README describes it as a task generator. It creates puzzle data for evaluating models, not a model you chat with.

What are alternatives to arc-task-gen?

The README lists the ARC-AGI official repository (fchollet/ARC-AGI) and re-arc as comparable projects. Compare them on the task types and formats you need.

New agent skills, repos and AI tools — every week

The best new skills and open-source projects, the fastest-growing repos, and one free prompt. Five minutes to read. Unsubscribe anytime.

Our verdict

arc-task-gen suits researchers and teams who evaluate few-shot reasoning models and want a private ARC-style test set to compare with the public ARC-AGI-1 benchmark. It is not suitable if you want a chatbot, a puzzle game, or a fully documented install, because the generation steps sit in a separate file. If you only need the official public ARC data, use the ARC-AGI official repository instead.

Install options & downloads