Skip to content
Get packs

Bulk Ingestion Pipeline

Guides an agent through turning large data dumps into brain pages with trials, checks, and a resumable manifest.

Source Download skill
Repository
garrytan/gbrain
Stars
31,447
Licence
MIT
Files
2 files

What it does

When you ask an agent to import a large collection (audio libraries, email exports, document folders, chat logs, API dumps) into a personal knowledge brain, this skill makes it follow a disciplined process. It requires defining the page format and filing rules first, testing on 5-10 sample items and fixing problems before any full run, and tracking progress in a JSON manifest so a crash or session break can resume without duplicating pages. It also treats all ingested text as data rather than instructions.

Great for

  • Developers building a knowledge base from a large export or archive
  • Data engineers who need resumable, idempotent batch imports
  • Teams running multi-session or multi-subagent ingestion jobs

Skip it if

  • Single-item imports: the skill itself points to a simpler single-item skill, since this full lifecycle adds overhead
  • Unvetted third-party corpora without review: the skill expects untrusted text and flags imperatives, but a bulk run can still write large amounts of content into your brain if the trial phase is skipped or rushed

Try asking

  • “Bulk ingest my year of exported email into the brain, starting with a trial of 10 varied messages”
  • “Make a manifest for importing my 4,000 audio meeting recordings and track progress”
  • “Set up an ingestion pipeline for these chat exports with dedup against what's already in the brain”
Step-by-step guide: how to use Bulk Ingestion PipelineInstall, first run, examples, tips and FAQs

Install in 3 steps

  1. 1

    Download the skill

    The ZIP contains the 2 original files, the MIT licence and an attribution note.

    bulk-ingestion.zip
  2. 2

    Add it to your agent

    Claude Code (all projects)

    mkdir -p ~/.claude/skills && unzip bulk-ingestion.zip -d ~/.claude/skills/

    Claude Code / Agent SDK (this project only)

    mkdir -p .claude/skills && unzip bulk-ingestion.zip -d .claude/skills/

    Claude apps (web/desktop): Settings → Capabilities → Skills → Upload skill, and choose the ZIP.

  3. 3

    Ask for the task

    No special command needed — the agent reads the skill's description and loads it when your request matches. Try one of the prompts above.

Get better results

  • Write the page template, filing rules, and dedup key before the first trial, and review the 5-10 trial outputs yourself before approving a bulk run.
  • Check the manifest and the output directories yourself after a run; the skill is designed to verify completion from files on disk, but you should confirm.
  • Point the agent at a clearly scoped source (one folder or one export) first, since the skill writes into projects/ and sources/ directories.

What's inside

  • MANIFEST-PATTERN.md
  • SKILL.md

Redistributed unmodified from garrytan/gbrain under the MIT licence. The assessment was drafted with AI from the skill's files on 2026-10-29. Skills can instruct your agent to run code — read SKILL.md and any scripts before installing.