How to use the Spark Job Optimization agent skill

What does the spark-optimization skill make the agent do?

Once installed, the agent knows to check common Apache Spark performance factors: shuffle cost, data skew, partition sizes, serialization and memory settings. It can then suggest or write Spark code and configuration that follow the patterns in the skill.

The skill is a knowledge guide, not a tool that runs Spark for you. The agent reads your code and the symptoms you describe; it does not see your live Spark UI or cluster metrics unless you paste them in.

• Looks for wide transformations that cause shuffles and suggests ways to minimize them
• Checks for data skew and points to salting or broadcast joins
• Recommends partition sizes in the 128MB to 256MB range and suggests repartition or coalesce where appropriate
• Suggests settings such as adaptive query execution (AQE), Kryo serialization and columnar formats like Parquet or Delta
• Flags habits such as collecting large data to the driver, unnecessary UDFs, over-caching and using .count() to test for existence

How do you install the spark-optimization skill?

Download the ZIP for this skill from our page and unzip it. The skill folder contains SKILL.md and references/details.md. Place the folder in one of these locations:

For all of your projects, unzip it into ~/.claude/skills/. For a single project, unzip it into that project's .claude/skills/ folder.

In the Claude apps you can instead upload the ZIP under Settings → Capabilities → Skills.

How do you trigger the skill with natural requests?

You do not need a special command. Describe a Spark problem in plain language and the agent can match it to the skill's description, which covers improving Spark performance, debugging slow jobs and scaling data processing pipelines.

Good requests include the job's code, the symptom and the scale. Mention the data size, the cluster setup if you know it, and what is slow or failing.

• "My Spark job takes 3 hours on a join between two large tables. Here is the code. Where is the shuffle cost?"
• "This PySpark aggregation fails with executor out-of-memory errors. Suggest memory and partition changes."
• "Review this job before we scale it to ten times the data and tell me what to fix first."
• "Should I broadcast this lookup table in my join? Here is the query plan."

What examples does the skill cover?

The skill's quick start shows a PySpark session with AQE, skew join handling, Kryo serialization and a shuffle partition setting, followed by a filter, select and groupBy aggregation written to Parquet. The detailed reference adds a partition-count helper that targets 128MB to 256MB per partition, examples of repartition and coalesce, filter pushdown on Parquet, and writing partitioned output by year, month and day.

Use these as starting points. Check the numbers against your own cluster and data before relying on them.

What do you need before using it?

The skill lists no special requirements, and it does not run scripts or use the network. You do get better results if you provide the Spark code, the job's configuration and any symptoms or metrics you can share, since the agent cannot measure the job itself.

The examples are written in PySpark, so the guidance fits Apache Spark jobs. It would be misleading for other engines.

What are the safety notes for changing production jobs?

The skill does not run scripts and does not use the network, so installing it does not run code on your machine. However, the advice is general. The right settings depend on your cluster and data, so treat every suggested change as a hypothesis.

Benchmark any change on representative data before applying it to production. Review the code the agent writes before running it, especially settings that change how a job writes output.

• Test changes on a copy or sample of your data first
• Compare run time, spills and GC behaviour before and after
• Change one setting at a time so you can tell what helped

What tips help you get the most from it?

Give the agent real evidence. Paste the Spark UI details you have, such as stage shuffle sizes, spill amounts or task duration spread, because the skill advises checking skew, spills and GC in the UI but cannot see them for you.

Enable AQE and prefer Parquet or Delta as the skill's do's suggest, then ask the agent to explain each change so you learn which settings matter for your workload.

• Ask for one fix at a time, ranked by expected impact
• Ask the agent to explain why a broadcast join or repartition helps your specific query
• Keep the original code so you can compare results