Guide · Agent skill
How to use the LangSmith LLM Observability agent skill
Updated · written with AI from the project's own documentation and checked against our catalogue data
The langsmith-observability skill is a reference guide that teaches an AI agent how to write Python code for LangSmith, the LLM observability platform. Once it is installed, the agent can add tracing to code that calls language models, build test datasets, run evaluations with scoring functions and set up production monitoring. This guide shows you how to install the skill, ask for the right things and check the results before you rely on them.
LangSmith LLM ObservabilityOrchestra-Research/AI-Research-SKILLs · MIT Download the skill Quick start
- 1Download and unzip the skill. Download the langsmith-observability ZIP from this page and unzip it into ~/.claude/skills/ or a project's .claude/skills/ folder.
- 2Install langsmith. Run pip install langsmith in a Python environment with version 0.2.0 or later.
- 3Set your API key and tracing flag. Export LANGSMITH_API_KEY with your key and set LANGSMITH_TRACING=true.
- 4Ask for a specific change. Describe the function or dataset you want to instrument or evaluate, in plain language.
- 5Review the generated code. Check filtering, key handling and which calls leave your machine before running it.
- 6Run and check traces. Run the code, then open your LangSmith project to confirm the runs appear with the expected inputs and outputs.
What the skill makes the agent do
Without the skill, an agent writing LLM code has no specific knowledge of LangSmith. With it, the agent follows the patterns in the skill's SKILL.md and reference files. It knows how to record every model call, chain step, retriever and tool use as runs inside a trace, so you can see inputs, outputs and latency for each step in the LangSmith platform.
The skill covers four jobs: tracing with decorators or wrappers, building datasets from examples, running systematic evaluations with scoring functions, and monitoring production systems. It is a guide for writing this code. It does not run LangSmith itself, and it does not connect to your account.
- Tracing with @traceable or the wrap_openai wrapper
- Datasets created with client.create_dataset and client.create_examples
- Evaluations with evaluate() and custom or built-in evaluators
- Production patterns such as feedback collection, sampling and tracing context
Install the skill (ZIP download or Claude apps)
Download the skill ZIP from this page and unzip it. For a personal setup, place the unzipped folder in ~/.claude/skills/. To scope it to one project, place it in that project's .claude/skills/ folder instead.
If you use the Claude apps, you can upload the skill instead. Open Settings, go to Capabilities, then Skills, and upload it there.
- Personal install: ~/.claude/skills/
- Project install: .claude/skills/ inside your project
- Claude apps: Settings → Capabilities → Skills
Set up the Python environment and API key
The examples need Python with the langsmith package, version 0.2.0 or later. You also need a LangSmith account and an API key. Install the package and set the environment variables the source shows. The skill's install notes say to use LANGSMITH_TRACING=true to turn tracing on.
Your model provider SDK, such as openai or langchain, is also needed for the code being instrumented.
- Python with langsmith>=0.2.0
- A LangSmith account and LANGSMITH_API_KEY
- An SDK for the model you call, such as openai or langchain
pip install langsmith
export LANGSMITH_API_KEY="your-api-key"
export LANGSMITH_TRACING=trueTrigger the skill with natural requests
You do not need to name the skill. Describe the task in plain language that matches what the skill is for: debugging an LLM application, evaluating outputs against a dataset, monitoring production, or building a testing pipeline. The skill's description lists these triggers, so requests that use these words are the most likely to load it.
Be specific about your code. Name the file, the function that calls the model, and the provider you use. The agent can then add tracing to the right place rather than guessing.
- "Add LangSmith tracing to the generate_response function in app/llm.py."
- "Build a LangSmith dataset from these 20 question and answer pairs and run an accuracy evaluator against my_model."
- "Help me debug why my LangChain agent gives wrong answers, using LangSmith traces."
- "Set up LangSmith feedback collection so users can rate each answer."
Review what the agent writes
The skill's examples include a few patterns worth checking. The @traceable decorator records a function as a run. The process_inputs option lets you redact fields such as passwords before they are sent. Sampling is set with LANGSMITH_TRACING_SAMPLING_RATE, so a value of 0.1 traces about 10% of calls.
Because the skill runs no scripts and the code it writes calls the LangSmith service over the network, read the generated code before you run it, and confirm that prompts and outputs being sent are acceptable to share with a third-party platform.
- Check that sensitive fields are filtered with process_inputs
- Confirm the API key comes from an environment variable, not a hard-coded string
- Test with a small dataset before running a full evaluation in CI
What people use it for
- Tracing a multi-step RAG pipeline so each retriever and LLM call shows up as a child run
- Building a regression dataset from question and answer pairs and scoring each new prompt version
- Collecting user ratings as feedback linked to specific run IDs
- Tracking latency and token usage in a production app through LangSmith projects
Frequently asked questions
How do I install the LangSmith agent skill?
Download the ZIP from this page and unzip it into ~/.claude/skills/ for personal use or a project's .claude/skills/ folder for one project. In the Claude apps, you can instead upload it under Settings → Capabilities → Skills.
Is the langsmith-observability skill free?
The skill is licensed under MIT. LangSmith itself is a separate service with its own account and API key, so check its pricing on the LangSmith site before you rely on it.
What do I need to use this skill?
You need Python with the langsmith package at version 0.2.0 or later, a LangSmith account with LANGSMITH_API_KEY, and the SDK for the model provider you are instrumenting, such as openai or langchain.
Does the skill run code or send data on my behalf?
The skill itself runs no scripts. The code it helps you write uses the network to send traces and evaluation data to LangSmith, so review what gets sent before running it.
Can I use this skill with languages other than Python?
The skill's examples are Python-specific, so it is a poor fit for projects that do not use Python. Use it only if your LLM code is written in Python.
What should I use instead of this skill for experiment tracking?
The skill points to Weights & Biases for deep learning experiment tracking and model training, MLflow for the general ML lifecycle and model registry, and Arize or WhyLabs for ML monitoring and data drift detection.
New agent skills, repos and AI tools — every week
The best new skills and open-source projects, the fastest-growing repos, and one free prompt. Five minutes to read. Unsubscribe anytime.
Our verdict
This skill suits Python developers who want to trace LLM chains or agents, and teams who need evaluation datasets and regression tests for AI features. Skip it if your project does not use Python or LangSmith, or if your main need is deep learning experiment tracking or model training, where the skill itself points to other tools.
Download the skill