Guide · Agent skill
How to use the Spark Engineer agent skill
Updated · written with AI from the project's own documentation and checked against our catalogue data
The spark-engineer skill is a set of instructions and reference notes that makes an AI agent work like a Spark specialist when you ask for help with Apache Spark, PySpark or Spark SQL. This guide explains what the skill changes, how to install it from the ZIP on our page, how to trigger it with plain requests, and what to check before running the code. By the end you will know how to get the agent to write, debug and tune Spark jobs with a clear review step.
Spark EngineerJeffallan/claude-skills · MIT Download the skill Quick start
- 1Download the ZIP. Download the spark-engineer skill ZIP from our page.
- 2Install the skill. Unzip it into ~/.claude/skills/ for personal use or into a project's .claude/skills/ folder, or upload the ZIP under Settings → Capabilities → Skills in the Claude apps.
- 3Set up Spark access. Make sure Apache Spark or PySpark is installed and that you can reach the cluster, the Spark UI and any storage your code uses.
- 4Describe your Spark task. Ask in plain language, naming Spark, PySpark or Spark SQL, the data format and the goal or symptom.
- 5Review the generated code. Check schemas, input paths, output paths and write modes before running anything.
- 6Test at realistic scale. Run the job on production-scale data in a safe environment and check the Spark UI for shuffle spill and skew.
- 7Iterate on tuning. If spill or skew appears, ask the agent to revisit partitioning, shuffle settings or joins and test again.
What the spark-engineer skill makes the agent do
When the skill is active, the agent follows a structured workflow for Spark work: it analyses data volume, transformations, latency and cluster resources, then designs the pipeline. It chooses DataFrames over RDDs for structured data, plans partitioning and looks for broadcast join opportunities.
The agent writes code with explicit schemas, appropriate caching and error handling. Before calling a job done, it checks the Spark UI for shuffle spill and data skew, and verifies the partition count. For detailed topics it loads reference notes on Spark SQL and DataFrames, RDD operations, partitioning and caching, performance tuning and structured streaming.
Install the spark-engineer skill
Download the ZIP for the skill from our page. Then choose one of these options.
For a personal setup available across your projects, unzip the download into ~/.claude/skills/. To limit it to one project, unzip it into that project's .claude/skills/ folder.
In the Claude apps, you can instead upload the ZIP under Settings → Capabilities → Skills.
- Personal: unzip into ~/.claude/skills/
- Single project: unzip into the project's .claude/skills/
- Claude apps: Settings → Capabilities → Skills, then upload the ZIP
Trigger the skill with natural requests
You do not need a special command. Describe a Spark task in plain language and mention Spark, PySpark or Spark SQL. The skill's description covers requests such as writing DataFrame transformations, optimizing Spark SQL queries, tuning shuffle operations, configuring executor memory, processing .parquet files, handling data partitioning and building structured streaming analytics.
Good requests say what the data is, what you want to compute and what is going wrong. Include the symptom if you are debugging, for example slow joins or out-of-memory errors.
- Write a PySpark job that totals amounts per user from Parquet files in s3://bucket/events/
- My Spark join is slow and one task takes far longer than the rest. Find out why.
- Suggest executor memory and shuffle partition settings for a 100 GB Parquet workload.
Example: a starter pipeline the skill produces
The skill's quick-start example reads Parquet with an explicit schema, filters null amounts, groups by user and prints the partition count before writing. It sets spark.sql.shuffle.partitions and spark.sql.adaptive.enabled when creating the session.
Treat this as a starting point. Replace the example S3 paths with your own and confirm the schema matches your data.
df = spark.read.schema(schema).parquet("s3://bucket/events/")
result = df \
.filter(F.col("amount").isNotNull()) \
.groupBy("user_id") \
.agg(F.sum("amount").alias("total_amount"), F.count("*").alias("event_count"))
print(f"Partition count: {result.rdd.getNumPartitions()}")
result.write.mode("overwrite").parquet("s3://bucket/output/")Other patterns the skill covers
The skill includes patterns for common performance problems. A broadcast join hints that a small dimension table, under about 200 MB, should be sent to every executor. A salting pattern spreads a skewed key across several buckets so one task does not handle most of the rows. Caching is recommended only when a DataFrame is reused several times, and the example unpersists it afterwards.
- Broadcast join for small dimension tables (under 200 MB)
- Salting to handle data skew on a join key
- Cache only when reused, materialise with an action, then unpersist
What you need before using it
The skill assumes an Apache Spark or PySpark installation and a cluster or local Spark environment. It also assumes you can open the Spark UI to monitor jobs, since the workflow depends on checking shuffle spill and skew there. Your code will need access to any storage it reads or writes, such as S3 buckets, because the examples use s3:// paths.
The skill does not run scripts and does not use the network itself. It is guidance and code generation; your own environment does the running.
- Apache Spark / PySpark installed, locally or on a cluster
- Access to the Spark UI
- Access to storage such as S3 buckets used by your code
Safety: review code before running it
Generated code can write over data. The example writes output with mode overwrite, so review the output path before running it against production clusters or data. Test with production-scale data volumes in a safe environment first, and avoid calling collect() on large datasets, which can cause out-of-memory errors.
The skill's source does not describe any script execution or network access, but you should still read the code it produces before running it.
- Check every write path and mode, especially overwrite
- Do not run unreviewed code against production data
- Confirm schemas and storage permissions before running
Tips for better results
Give the agent real numbers where you have them: data volume, number of executors, cores per executor and the size of the dimension tables. The skill's guidance uses these to suggest partition counts and memory settings. Paste the relevant Spark UI observations, such as spill or a long-running task, when debugging.
Ask for the expected shuffle size and memory usage along with the code, since the skill's output format includes performance analysis and monitoring recommendations.
- Provide data size, cluster shape and symptoms
- Paste Spark UI details when debugging
- Ask for monitoring recommendations with the code
What people use it for
- Write a PySpark job that aggregates Parquet event data into per-user totals.
- Diagnose a slow join where one task takes much longer than others and suggest a fix.
- Choose executor memory, cores and shuffle partition settings for a large Parquet workload.
- Convert a slow Python UDF pipeline to built-in Spark SQL functions.
- Design a structured streaming analytics job with watermarks and a sink.
Frequently asked questions
How do I install the spark-engineer skill?
Download the ZIP from our page and unzip it into ~/.claude/skills/ for personal use, or into a project's .claude/skills/ folder for one project. In the Claude apps, you can upload the ZIP under Settings → Capabilities → Skills.
What do I need to use the spark-engineer skill?
You need Apache Spark or PySpark installed in a cluster or local environment, access to the Spark UI for monitoring, and access to any storage your code reads or writes, such as S3 buckets. The skill does not run scripts and does not use the network.
Is the spark-engineer skill free?
The skill is licensed under MIT, as stated on this page. Check the license file in the download for the full terms.
Does the skill work with pandas or small datasets?
The skill is aimed at big data work. For data that fits on one machine, pandas or plain SQL is usually simpler, and Spark adds overhead that the skill's guidance may not justify.
Can the skill work with Flink, Dask or warehouse SQL?
No, it is focused on Apache Spark. For Flink, Dask or warehouse-native SQL, the guidance may not apply, so use tools built for those systems.
Why does the agent check the Spark UI?
The skill's workflow checks the Spark UI for shuffle spill and data skew before a job is considered done. If spill or skew appears, the agent returns to tuning the partitioning, shuffle or joins.
New agent skills, repos and AI tools — every week
The best new skills and open-source projects, the fastest-growing repos, and one free prompt. Five minutes to read. Unsubscribe anytime.
Our verdict
This skill suits data engineers writing or fixing PySpark and Spark SQL pipelines, and developers whose Spark jobs are slow, skewed or running out of memory. It is a poor fit for small datasets that a single machine handles well, or for non-Spark tools such as Flink or Dask. Review all generated code, especially write paths, before running it against real clusters or data.
Download the skill