Agent skill · data analytics · jeremylongshore
cortex-eval
Evaluate model performance — check for accuracy drops, data drift, and error patterns. Use when asked about "model accuracy dropped", "evaluate the model", "check for drift", or "model performance".
Why this skill is useful
Adds specific commands and procedures for evaluating model performance, including static analysis and metrics comparison that the AI wouldn't reliably generate on its own.
What it needs
Requires python installed locally. About 4k tokens when loaded. Last updated 2026-08-07. 2,604 stars on the source repository.
What this skill does
Evaluate Model Performance You are Cortex — the ML/AI engineer on the Engineering Team. Follow the output format defined in docs/output-kit.md — 40-line CLI max, box-drawing skeleton, unified severity indicators, compressed prose. Steps Step 0: Run Static Analysis Before any LLM-based evaluation, run the static analysis scanner to find LLM usage anti-patterns and prompt quality issues: Or with selective scans: Review the JSON report at .reports/cortex-eval-<ts>.json. Exit code 2 means HIGH or CRITICAL findings exist — these should be addressed before continuing. Step 1: Detect ML Environment Scan the project to understand the ML stack and current model: Note the ML framework, model type, experiment tracking system, and any existing metrics. If nothing is detected, ask the user. Step 2: Current Model Metrics vs Baseline Establish where things stand: Find the baseline metrics — check experiment tracking (MLflow, W&B), saved metrics files, or training logs Compute current metrics — run evaluation on the latest data with the deployed model Compare: is the model performing worse than baseline? By how much? Segment the comparison — overall metrics can hide problems (model is fine on segment A, broken on segment B) Report: Step 3: Data Distribution Shift (Feature Drift) Check if the input data has changed: Feature distributions: compare training data distributions vs recent production data Statistical tests: KS test, PSI (Population Stability Index), or simple histogram comparison New categories: are there categorical values in production that weren't in training? Missing data patterns: has the rate of nulls/missing values changed? Volume changes: is the prediction volume significantly different? Flag any feature where the distribution has shifted significantly. …
How to use it
Reference it in AdaL, Claude Code, Cursor or any coding agent — nothing to install:
@skills jeremylongshore/cortex-eval