Skip to content
Get packs

NeMo Curator Data Curation

Clean, deduplicate and filter large text, image, video and audio datasets for LLM training using GPUs.

Source Download skill
Repository
Orchestra-Research/AI-Research-SKILLs
Stars
14,195
Licence
MIT
Files
3 files

What it does

This skill teaches an AI agent how to use NVIDIA's NeMo Curator toolkit to prepare large training datasets for AI models. Once installed, the agent can write pipelines that remove low-quality or repeated documents, find near-duplicate text using fast GPU methods, redact personal information such as emails and phone numbers, and filter images, video and audio. It mainly provides code patterns and API examples the agent can adapt to your data.

Great for

  • Machine learning engineers preparing large web-scraped text corpora for LLM training
  • Data teams who need to deduplicate or clean very large datasets on GPU clusters
  • Researchers curating multimodal (text, image, video, audio) datasets

Skip it if

  • Small datasets or one-off cleaning tasks, where simpler CPU tools are faster to set up and sufficient
  • Users without NVIDIA GPUs or CUDA, since the GPU features and most examples depend on them; the CPU install is described as slower
  • Anyone relying on the PII redaction or NSFW filters as a compliance guarantee, since the examples are basic and their accuracy is not verified in these files

Try asking

  • “Write a NeMo Curator pipeline that filters out short and repetitive documents from my Common Crawl parquet files”
  • “Set up fuzzy deduplication with MinHash and LSH on my 500GB text corpus”
  • “Redact emails and phone numbers from my dataset before training”
  • “Help me choose between exact, fuzzy and semantic deduplication for my data”
Step-by-step guide: how to use NeMo Curator Data CurationInstall, first run, examples, tips and FAQs

Install in 3 steps

  1. 1

    Download the skill

    The ZIP contains the 3 original files, the MIT licence and an attribution note.

    nemo-curator.zip
  2. 2

    Add it to your agent

    Claude Code (all projects)

    mkdir -p ~/.claude/skills && unzip nemo-curator.zip -d ~/.claude/skills/

    Claude Code / Agent SDK (this project only)

    mkdir -p .claude/skills && unzip nemo-curator.zip -d .claude/skills/

    Claude apps (web/desktop): Settings → Capabilities → Skills → Upload skill, and choose the ZIP.

  3. 3

    Ask for the task

    No special command needed — the agent reads the skill's description and loads it when your request matches. Try one of the prompts above.

Get better results

  • Test filters on a small sample first; the examples use placeholder thresholds that need tuning for your data.
  • Verify PII redaction output manually before relying on it, since the skill does not measure detection accuracy.
  • Check the reference files (deduplication.md and filtering.md) for more detail on specific methods before running large jobs on GPU clusters.

What's inside

  • references/deduplication.md
  • references/filtering.md
  • SKILL.md

Redistributed unmodified from Orchestra-Research/AI-Research-SKILLs under the MIT licence. The assessment was drafted with AI from the skill's files on 2026-10-29. Skills can instruct your agent to run code — read SKILL.md and any scripts before installing.