How to use the Segment Anything Model (SAM) agent skill

What the skill makes the agent do

Without the skill, an agent has to work out SAM's API from general knowledge. With it installed, the agent knows the setup steps, how to choose a checkpoint, how to pass prompts and which code patterns to use. The skill mainly guides the agent to write Python code; it does not segment images by itself.

• Segment a specific object using a foreground point click, a background point, a bounding box, or a combination of these.
• Refine an existing mask with extra points.
• Generate all object masks in an image automatically.
• Filter the resulting masks by area, predicted IoU or stability score.
• Run SAM through either the segment-anything GitHub package or HuggingFace transformers.

Install the segment-anything-model skill

Download the ZIP for the skill from this page. The skill folder contains SKILL.md plus two reference files, references/advanced-usage.md and references/troubleshooting.md.

Option 1 (personal): unzip the folder into ~/.claude/skills/ so it is available in every project.

Option 2 (single project): unzip it into the .claude/skills/ folder inside your project so it is available only there.

Option 3 (Claude apps): upload the ZIP in the Claude apps under Settings → Capabilities → Skills.

Trigger the skill with natural requests

You do not need to name the skill. The skill description says it applies when you need to segment any object in images using points, boxes, or masks as prompts, or automatically generate all object masks in an image. Ask for that outcome in plain language and the agent can load it.

Be specific about the image and the result you want. Mention whether you have a point, a box or nothing to start from.

• Write Python that uses SAM to cut out the coffee mug in photo.jpg, using a click at pixel 500, 375.
• Use the Segment Anything Model to generate every object mask in this folder of product images.
• Given a bounding box around each car, produce precise masks with SAM.
• Help me build an annotation tool where a user clicks an object and gets its mask.

Example workflows

The skill's basic pattern loads a model, sets the image once so its embeddings are computed once, then predicts masks for each prompt. A single foreground point returns three candidate masks when multimask_output is True, and the agent picks the one with the highest score.

For automatic generation, SAM returns a list of masks, each with fields such as segmentation, bbox, area, predicted_iou and stability_score. The agent can sort by area or keep only high-scoring masks.

• Point prompt: one foreground point with label 1, or add a background point with label 0 to exclude a region.
• Box prompt: pass a rectangle as [x1, y1, x2, y2].
• Iterative refinement: pass the best previous mask back in as mask_input along with an extra point.

What you need before running it

You need Python with either the segment-anything package (installed from GitHub) or HuggingFace transformers version 4.30.0 or later, plus torch 1.7.0 or later. Optional packages are opencv-python, pycocotools and matplotlib.

Checkpoint files are downloaded from dl.fbaipublicfiles.com. They range from about 375MB for ViT-B to 2.4GB for ViT-H, so plan for disk space and download time. The examples call model.to('cuda'), so a CUDA GPU is strongly recommended.

• ViT-B (vit_b): about 375MB, fastest, good accuracy.
• ViT-L (vit_l): about 1.2GB, medium speed, good accuracy.
• ViT-H (vit_h): about 2.4GB, slowest, best accuracy.

Safety notes: scripts and network access

This skill runs scripts and uses the network. The guidance includes commands that download model checkpoints and install packages from GitHub and PyPI. Read any generated install or download command before you run it, especially on a shared machine or inside a production environment.

Keep the checkpoint filenames and source URLs that the agent uses, and check that the code it writes loads the model file you intended.

• Review shell commands such as pip install and wget before running them.
• Run in a virtual environment so package changes stay isolated.
• Confirm the checkpoint size matches the model variant you chose.

Tips for better results

Call set_image once and reuse the predictor for many prompts on the same image, since computing embeddings is the expensive step. Use multimask_output=True when a single click is ambiguous and multimask_output=False when your prompts are clear.

If CUDA is not detected, the troubleshooting reference covers checking torch.cuda.is_available() and installing a CUDA-enabled PyTorch build. For video, the advanced reference covers SAM 2 instead of SAM.

• Start with ViT-B while testing, then move to a larger checkpoint for final results.
• Tune points_per_side and the thresholds in SamAutomaticMaskGenerator if you get too many or too few masks.
• For real-time named-class detection, use YOLO or Detectron2 instead, as the skill itself recommends.