Agent skill · research science · daymade
llm-eval-harness
Tests/benchmarks a third-party LLM endpoint (OpenAI- or Anthropic-compatible): availability, fidelity, speed, concurrency, protocol compliance, quality regression. Use before hand-rolling a curl loop, when onboarding a provider, or debugging "system prompt 不生效" / a tok/s claim — 测评/压测一个模型或渠道. Not for TTS/voice-clone supplier eval (use the audio skill).
Why this skill is useful
Adds six executable scripts for comprehensive LLM endpoint evaluation that aren't available in public documentation.
What it needs
Requires python installed locally. Requires anthropic, openai account access. About 11k tokens when loaded. Last updated 2026-08-07. 1,320 stars on the source repository.
What this skill does
LLM Eval Harness Overview Give this skill an endpoint (baseurl + model + an API key in an env var) and it measures whether the endpoint actually works and whether the model is fast, stable, protocol-correct, and good enough — instead of trusting the vendor's headline numbers. Choose from the following evaluation dimensions: Dimension Script Answers --- --- --- Availability scripts/availabilityprobe.py which model IDs work here, with explicit error classification Request fidelity scripts/fidelityprobe.py do system prompt / tools / history actually REACH the model? Speed scripts/speedprobe.py TTFT + sustained decode tok/s, thinking-aware Concurrency / stability scripts/concurrencyprobe.py success rate, p50/p90 latency, where it breaks Protocol compliance scripts/protocolprobe.py does the Anthropic thinking block actually fire when requested, AND does the endpoint accept one already sitting in history on a later turn (N≥10, both are separate checks — see Dimension 3)? Quality / use-case regression scripts/usecaserunner.py + blind judges does it pass your accumulated cases? Which dimensions to run Route from what the user is actually doing: "接入新模型/新网关先测一下" / onboarding a provider → Availability → Fidelity, then speed/concurrency if adoption looks likely "这个 ID 能不能用 / 写个支持列表" → Availability (and read disciplines §9 before writing any "unavailable" verdict) "模型不听 system prompt / agent 行为怪但没报错" → Fidelity (system canary) "快不快 / tok/s 是真的吗" → Speed; "扛不扛得住" → Concurrency "这个兼容端点是真兼容吗" → Protocol + Fidelity's tools/auth checks "这个模型/渠道用着用着就 400 了 / session 断了 / continue 不下去了" (extended-thinking client, multi-turn session) → Protocol --check history-replay specifically — this symptom is the generation check passing while history-replay silently doesn't; don't stop at generation "换模型质量会不会掉" → Quality regression "要给厂商报 bug" → run the relevant probe with --output, then follow references/vendorevidenceprotocol.md "部署闸门 / 常驻监控 / 告警老误报" → references/productiontestingpatterns.md Run …
How to use it
Reference it in AdaL, Claude Code, Cursor or any coding agent — nothing to install:
@skills daymade/llm-eval-harness