Agent skill · software engineering · jeremylongshore
langfuse-ci-integration
Configure Langfuse CI/CD integration with GitHub Actions and automated
Why this skill is useful
Adds specific GitHub Actions workflows and Langfuse integration patterns that the AI wouldn't reliably generate on its own.
What it needs
Requires node installed locally. Requires langfuse, openai account access. About 4k tokens when loaded. Last updated 2026-08-07. 2,604 stars on the source repository.
What this skill does
Langfuse CI Integration Overview Integrate Langfuse into CI/CD pipelines: trace validation tests, prompt regression testing, experiment-driven quality gates, automated prompt deployment from version control, and score monitoring. Prerequisites Langfuse API keys stored as GitHub secrets (LANGFUSEPUBLICKEY, LANGFUSESECRETKEY) Test framework (Vitest or Jest) OpenAI API key for LLM tests Instructions Step 1: GitHub Actions Workflow for AI Quality Tests Step 2: Prompt Regression Tests Step 3: Experiment-Driven Quality Gates Step 4: Automated Prompt Deployment Step 5: Score Regression Monitoring CI Best Practices Practice Why ---------- ----- Use temperature: 0 in CI tests Deterministic outputs, fewer false failures Separate CI API keys Isolate test traces from production Run experiments on dataset changes Catch regressions before deploy Assert on ranges, not exact strings LLM output varies even at temp 0 Flush/shutdown in afterAll Ensure all traces reach Langfuse Error Handling Issue Cause Solution ------- ------- ---------- Traces not in dashboard No flush in CI Add sdk.shutdown() or afterAll flush Flaky quality tests Non-deterministic LLM Use temperature: 0, assert on ranges Prompt not found Not yet deployed Deploy prompts before running tests Missing secrets in CI Not configured Add to GitHub Settings > Secrets > Actions Output The pipeline reports the test suite result, experiment/run identifier, and aggregate quality score. A regression exits non-zero so the pull request is blocked; a pass leaves a traceable Langfuse run linked to the commit SHA. Examples Run the focused quality suite locally before opening a pull request: For an intentionally changed prompt, first update the expected dataset result, then run the experiment gate and inspect the resulting run in Langfuse before merging. Resources Experiments via SDK Prompt Management Scores via SDK
How to use it
Reference it in AdaL, Claude Code, Cursor or any coding agent — nothing to install:
@skills jeremylongshore/langfuse-ci-integration