Agent skill · data analytics · colbymchenry
agent-eval
Benchmark CodeGraph retrieval quality on a real codebase by comparing agent behavior with vs without CodeGraph. Use when the user runs /agent-eval or asks to test, benchmark, audit, or validate a codegraph version (the local dev build or a published npm version) against a language's repo.
Why this skill is useful
Provides specific benchmarking scripts and workflows for evaluating CodeGraph's performance against traditional methods, which the AI wouldn't generate on its own.
What it needs
Requires git, node, tmux installed locally. Requires claude account access. About 2k tokens when loaded. Last updated 2026-08-07. 65,083 stars on the source repository.
What this skill does
CodeGraph Quality Audit Measures how much CodeGraph helps an agent versus plain grep/read, for a chosen codegraph version on a chosen real-world repo. Drives the harness in scripts/agent-eval/. Prerequisites tmux 3+, a logged-in claude CLI, node, git (macOS/Linux). Run from the codegraph repo root. Workflow Copy this checklist: Step 1 — version. Ask with AskUserQuestion: which codegraph version to test. Offer "Local dev build" and "Latest published"; the free-text "Other" lets the user type a specific version (e.g. 0.7.10). Map the answer to a VERSION token: "Local dev build" → local "Latest published" → latest a typed version → that string (e.g. 0.7.10) Step 2 — language. Read .claude/skills/agent-eval/corpus.json. Ask with AskUserQuestion which language to test, listing the languages that have entries. Step 3 — repo. From the chosen language's entries, ask which repo. Label each option with its size and file count, e.g. excalidraw — Medium (~600 files). Each entry carries the repo URL and a representative question. Step 4 — harness. Ask with AskUserQuestion which harness to run, and map the answer to a MODE token: "Headless" → headless — claude -p with stream-json: exact tokens/cost and a clean tool sequence (2 runs, fast, no TTY). "Interactive (tmux)" → tmux — drives the real Claude TUI in tmux: faithful Explore-subagent behavior, metrics from session logs (2 runs, slower). "Both" → all — headless + interactive (4 runs). Step 5 — run. Launch in the background (sets the version, clones if missing, wipes + re-indexes, runs the chosen arms — several minutes): Step 6 — report. When the job finishes, read the log and report per arm: Headless (parse-run.mjs): total tool calls, file Reads, Grep/Bash, codegraph-tool calls, duration, total cost. Interactive (parse-session.mjs): the VERDICT: codegraphexplore used Nx Read N Grep/Bash N and TOKENS: lines. …
How to use it
Reference it in AdaL, Claude Code, Cursor or any coding agent — nothing to install:
@skills colbymchenry/agent-eval