Airflow DAG Patterns
Build production Apache Airflow data pipelines with proven design, testing and scheduling patterns
- Repository
- wshobson/agents
- Stars
- 42,095
- Licence
- MIT
- Files
- 2 files
What it does
When installed, the agent knows established best practices for writing Apache Airflow workflows (called DAGs), which are scheduled sets of tasks that move and process data. Asked to build or debug a pipeline, it follows patterns such as idempotent, retry-safe tasks, sensors that free up workers, TaskFlow API usage, and avoiding hardcoded dates. It can produce example DAG code and guidance on testing and deployment, though it does not run Airflow itself.
Great for
- Data engineers building or maintaining scheduled ETL pipelines in Apache Airflow
- Developers new to Airflow who want code that follows common production conventions
- Teams reviewing existing DAGs for retry, catchup, and dependency issues
Skip it if
- Projects not using Apache Airflow, since the patterns are specific to it
- Anyone expecting the skill to deploy or run pipelines automatically; it provides guidance and example code only, and the generated DAGs still need review and testing before production use
Try asking
- “Create an Airflow DAG that extracts daily sales data at 6 AM with retries and failure emails”
- “Review my existing Airflow DAG for idempotency and bad practices like depends_on_past=True”
- “Write a sensor that waits for a file in S3 without blocking worker slots”
- “Show me how to unit test my Airflow DAGs locally”
Install in 3 steps
- 1
Download the skill
The ZIP contains the 2 original files, the MIT licence and an attribution note.
airflow-dag-patterns.zip - 2
Add it to your agent
Claude Code (all projects)
mkdir -p ~/.claude/skills && unzip airflow-dag-patterns.zip -d ~/.claude/skills/Claude Code / Agent SDK (this project only)
mkdir -p .claude/skills && unzip airflow-dag-patterns.zip -d .claude/skills/Claude apps (web/desktop): Settings → Capabilities → Skills → Upload skill, and choose the ZIP.
- 3
Ask for the task
No special command needed — the agent reads the skill's description and loads it when your request matches. Try one of the prompts above.
Get better results
- Describe your data source, schedule, and failure requirements so the agent can tailor the DAG rather than use the generic example
- Ask the agent to explain each retry, catchup, and timeout setting so you understand the trade-offs before deploying
- Have the agent write tests for the generated DAG and run them in your own environment before deploying to production
What's inside
- references/details.md
- SKILL.md
Redistributed unmodified from wshobson/agents under the MIT licence. The assessment was drafted with AI from the skill's files on 2026-10-29. Skills can instruct your agent to run code — read SKILL.md and any scripts before installing.