Agent skill · data analytics · davila7
langsmith-observability
LLM observability platform for tracing, evaluation, and monitoring. Use when debugging LLM applications, evaluating model outputs against datasets, monitoring production systems, or building systematic testing pipelines for AI applications.
Why this skill is useful
Provides domain-specific tools for monitoring and evaluating LLM applications that the AI wouldn't reliably generate on its own.
What it needs
Requires langsmith installed locally. About 4k tokens when loaded. Last updated 2026-08-06. 30,138 stars on the source repository.
What this skill does
LangSmith - LLM Observability Platform Development platform for debugging, evaluating, and monitoring language models and AI applications. When to use LangSmith Use LangSmith when: Debugging LLM application issues (prompts, chains, agents) Evaluating model outputs systematically against datasets Monitoring production LLM systems Building regression testing for AI features Analyzing latency, token usage, and costs Collaborating on prompt engineering Key features: Tracing: Capture inputs, outputs, latency for all LLM calls Evaluation: Systematic testing with built-in and custom evaluators Datasets: Create test sets from production traces or manually Monitoring: Track metrics, errors, and costs in production Integrations: Works with OpenAI, Anthropic, LangChain, LlamaIndex Use alternatives instead: Weights & Biases: Deep learning experiment tracking, model training MLflow: General ML lifecycle, model registry focus Arize/WhyLabs: ML monitoring, data drift detection Quick start Installation Basic tracing with @traceable OpenAI wrapper (automatic tracing) Core concepts Runs and traces A run is a single execution unit (LLM call, chain, tool). Runs form hierarchical traces showing the full execution flow. Projects Projects organize related runs. Set via environment or code: Client API Datasets and evaluation Create dataset Run evaluation Built-in evaluators Advanced tracing Tracing context Manual runs Process inputs/outputs Sampling LangChain integration Production monitoring Hub prompts Async client Feedback collection Testing integration Pytest integration Evaluation in CI/CD Best practices 1. Structured naming - Use consistent project/run naming conventions 2. Add metadata - Include version, environment, user info 3. Sample in production - Use sampling rate to control volume 4. Create datasets - Build test sets from interesting production cases 5. Automate evaluation - Run evaluations in CI/CD pipelines 6. …
How to use it
Reference it in AdaL, Claude Code, Cursor or any coding agent — nothing to install:
@skills davila7/observability-langsmith