Skip to content
Get packs

LangSmith LLM Observability

Trace, evaluate and monitor LLM apps with LangSmith's Python SDK

Source Download skill
Repository
Orchestra-Research/AI-Research-SKILLs
Stars
14,195
Licence
MIT
Files
3 files

What it does

Once installed, the agent knows how to instrument Python code that calls language models so every model call, step and tool use is recorded in the LangSmith platform. It can write code that adds tracing with decorators or wrappers, builds test datasets, runs systematic evaluations with scoring functions, and sets up production monitoring. It is a reference guide for writing this code; it does not itself run LangSmith.

Great for

  • Python developers debugging LLM chains, prompts or agents who want to see every step of a run
  • Teams building regression tests or evaluation datasets for AI features
  • Engineers who want to track latency, token usage and errors in production LLM apps

Skip it if

  • Projects that do not use Python or LangSmith, since the examples are Python-specific and need a LangSmith account and API key (sending traces to a third-party service)
  • Deep learning experiment tracking or model training, where the skill itself points to Weights & Biases or MLflow instead

Try asking

  • “Add LangSmith tracing to my OpenAI chatbot so I can see every call”
  • “Create an evaluation dataset from my production traces and score a new prompt version”
  • “Write a correctness evaluator for my QA pipeline and run it with langsmith evaluate”
  • “Sample only 10% of production requests for tracing and hide passwords in logged inputs”
Step-by-step guide: how to use LangSmith LLM ObservabilityInstall, first run, examples, tips and FAQs

Install in 3 steps

  1. 1

    Download the skill

    The ZIP contains the 3 original files, the MIT licence and an attribution note.

    langsmith.zip
  2. 2

    Add it to your agent

    Claude Code (all projects)

    mkdir -p ~/.claude/skills && unzip langsmith.zip -d ~/.claude/skills/

    Claude Code / Agent SDK (this project only)

    mkdir -p .claude/skills && unzip langsmith.zip -d .claude/skills/

    Claude apps (web/desktop): Settings → Capabilities → Skills → Upload skill, and choose the ZIP.

  3. 3

    Ask for the task

    No special command needed — the agent reads the skill's description and loads it when your request matches. Try one of the prompts above.

Get better results

  • Set LANGSMITH_TRACING and API keys through environment variables rather than hardcoding them in source files
  • Use the process_inputs option to redact sensitive fields such as passwords before traces are sent
  • Review generated evaluators carefully, since simple string-matching scoring can be misleading

What's inside

  • references/advanced-usage.md
  • references/troubleshooting.md
  • SKILL.md

Redistributed unmodified from Orchestra-Research/AI-Research-SKILLs under the MIT licence. The assessment was drafted with AI from the skill's files on 2026-10-29. Skills can instruct your agent to run code — read SKILL.md and any scripts before installing.