---
name: n8n:create-instance-ai-eval
description: >-
  Authors a new Instance AI workflow eval case — written locally as JSON,
  calibrated against a real build, then pushed to the LangTracer suite CI runs
  — build cases, behaviour/process cases, credential cases, and seeded
  (mid-conversation) cases — with intent-driven expectations. Use when adding
  or changing an Instance AI workflow eval, or debugging why one is flaky.
---

# Create an Instance AI workflow eval

Each eval is **one JSON case** — authored locally as a file in
`packages/@n8n/instance-ai/evaluations/data/workflows/` (the disk loader
auto-discovers `*.json`, no registration step), with a LangTracer suite as its
durable home. Cases validate against
[`harness/schema.ts`](../../../packages/@n8n/instance-ai/evaluations/harness/schema.ts)
(`.strict()` — unknown keys fail at load). The eval
[README](../../../packages/@n8n/instance-ai/evaluations/README.md) is the
exhaustive field reference; this skill is the opinionated *how*.

> **Committing new case JSONs into the repo is no longer the recommended
> approach.** Author the file locally (uncommitted), calibrate it against a real
> build, then **push it to a lang-tracer suite** with `eval:langtracer-push`
> (see [Push to a lang-tracer suite](#push-to-a-lang-tracer-suite)) —
> `--suite baseline` for the consolidated corpus n8n CI runs, or a dedicated
> capability suite like `agents`.
> The suite is the home for the case; the eval CLI reads it back via
> `--source langtracer`. You still write the JSON file — it's just the input to
> the push, not a committed artifact.
>
> **Seeded cases.** An `inline` seed pushes with the case — the case-write API
> stores it verbatim, so the suite is its home like any other case. Only a
> `seed.mode: "replay"` case is refused (listed under `skipped:`): it's
> reconstructed from a LangSmith trace at run time, so it dies when that trace is
> pruned and has no durable home. Don't commit a replay case either — derive a
> synthetic case from it. See [`case-shapes.md`](case-shapes.md).

## Set the autonomy level first

**Before you source, draft, or run anything, decide how hands-on the driver
wants to be — and say it back.** This skill runs at one of two autonomy levels.
If the request makes the level clear ("just author and calibrate it yourself" vs.
"stop me at each step", or an explicit mode), adopt it, state it in one line, and
note how to override (e.g. "say 'stop me at calibration' to add a checkpoint").
**If it's not clear, ask the driver one question** offering the two levels
*before* doing any work.

The skill has four natural decision **gates** — **selection** (which real
failure to encode), **shape + expectations** (archetype, must-haves, scope
trim), **calibration** (classify each red and resolve keep/loosen/drop), and
**push** (kind + tier). The level decides what happens at each gate:

| Level | Who decides when to stop | Behaviour |
|---|---|---|
| **autonomous** | agent | Runs all four gates start-to-finish; reports a **decision log** at the end for the driver to review — with the pushed case, its suite, and the source thread as **links** ([Share links, never bare ids](#share-links-never-bare-ids)); includes the observation id when the driver opted to record one; and includes a Linear ticket **proposal** for any kept capability-gap red ([Capability gap → propose a Linear ticket](#capability-gap--propose-a-linear-ticket)). |
| **checkpoint** | driver, per gate | Stops at each gate with a compact **proposal + recommendation**; driver says "go" or redirects. At the **calibration** gate, hands the driver a link to the just-built thread on the live instance plus login credentials so they can review the real conversation and workflow themselves before confirming (below). |

**Calibration is special-cased at both levels.** A calibration verdict that
flips a case's *meaning* — a real capability-gap red vs. a harness-caused red, or
any loosening that would let a known-bad build pass — is **surfaced explicitly**
(interactively in checkpoint; in the decision log in autonomous), never silently
committed. It's the one call where a quiet mistake corrupts the suite, so it
never fully auto-commits.

**Recording a source observation is optional, not a fifth gate.** After a real
thread passes selection, offer to save why it was selected and what the developer
observed with LangTracer's `create_observation` tool (see
[`sourcing-cases.md`](sourcing-cases.md#optional-record-the-selection-as-an-observation)).
If the driver declines, has not stated a preference in autonomous mode, or the
write fails, continue with the eval. Never block drafting, calibration, or push
on an observation. `create_observation` and `update_observation` write only to
LangTracer; do not add a LangSmith feedback or sync step.

**Checkpoint calibration — review the real thread on the instance.** Because the
calibration verdict is trust-critical, in checkpoint mode you don't ask the
driver to trust your reading of the run. You built the case against a live
instance with `--keep-workflows` (step 4), so the thread and the workflow are
still there — hand the driver a direct link and let them look:

- **Thread:** `<base-url>/assistant/<threadId>` — the exact conversation the case
  ran (the run prints the `threadId`; the built workflow prints as `BUILT (<id>)`
  and opens at `<base-url>/workflow/<id>`).
- **Login:** the email + password the eval signs in with (the owner you seeded on
  the instance — see [`running-evals.md`](running-evals.md); the default local
  seed is `nathan@n8n.io` / `PlaywrightTest123`).

Present, per red: the assertion, whether it went green/red, your proposed
classification (real capability gap / harness limitation / noise) and
keep/loosen/drop, and the review link. The driver logs in, reads the thread and
the workflow, and confirms or redirects before you write the verdict back into
the case `description`.

## Share links, never bare ids

Every lang-tracer entity has a shareable web page, but the CLI and the MCP hand
you **numeric ids** — `eval:langtracer-push` prints `+ created <slug> (#621)`,
`get_eval_run` returns a run number, `list_conversations` returns thread ids.
An id is unclickable: the driver has to go find it. **Whenever you name a case,
suite, thread, cluster, or run in anything a human reads** — a checkpoint
proposal, the end-of-run decision log, a PR description, a Linear ticket, a Slack
message — render it as a link, keeping the id in the label:

```
pushed as [#621](https://lang-tracer.n8n-maintenance.workers.dev/test-cases/621)
```

Build links off the **web base** (`LANGTRACER_URL`, in production
`https://lang-tracer.n8n-maintenance.workers.dev`). Never off the API bases —
`${LANGTRACER_URL}/api/v1` and `/api/mcp` are machine endpoints, and a link into
either 404s for the driver or dumps JSON.

| Entity | URL | Where the id comes from |
|---|---|---|
| Test case | `<base>/test-cases/<id>` | push output `(#<id>)`; `create_test_case` / `search_test_cases` |
| Suite | `<base>/suites/<suiteId>` | push header `Suite "<slug>" (#<id>)`; `list_suites` |
| Source conversation | `<base>/conversations/<threadId>` | `list_conversations` / `get_conversation` |
| Cluster report | `<base>/clusters/<id>` | `list_cluster_runs` / `get_latest_cluster_run` |
| Eval run (sweep) | `<base>/results?sweep=<runId>` | `list_eval_runs` / `get_eval_run` (`runId` *is* `sweeps.id`, the "run #N") |

Two links that are **not** lang-tracer and don't take this base: the built thread
(`<base-url>/assistant/<threadId>`) and workflow (`<base-url>/workflow/<id>`) live
on the **n8n instance** the eval ran against. When both are relevant — reviewing a
calibration red, writing a capability-gap ticket — give both, labelled, so nobody
has to guess which host a link points at.

## Where the best cases come from

The strongest cases encode a **real** failure, not an invented premise. Two
connections help you find and verify one: **LangTracer** clusters real
conversations into capability-gap themes (discover what actually fails, at
scale), and **LangSmith** holds the raw traces (verify exactly what happened in a
run). LangTracer is the discovery layer; the durable artifact is almost always a
synthetic case you author from what you learn (use `seed.mode: "replay"` only per
[`case-shapes.md`](case-shapes.md)). See
[`sourcing-cases.md`](sourcing-cases.md) for connecting the MCPs and the
discover → verify → encode workflow.

## Pick the case shape first

The corpus is four archetypes. Decide which you're writing before you draft — it
determines the fields, the grading, and how you validate. They compose (a seeded
case can still assert outcome), but the primary shape drives the work.

| Archetype | Question it answers | Primary fields |
|---|---|---|
| **Build** (default) | Does the workflow the agent builds actually *work*? | `outcomeExpectations` + `executionScenarios` |
| **Behaviour / process** | Does the agent *converse* correctly (ask the right clarifying question, not re-ask, honour a correction, respect plan approval)? | `processExpectations` + multi-turn director script; often **build-only** |
| **Credential** | Does the build behave correctly given a specific credential view? | `credentials[]` |
| **Seeded** | Start mid-thread, with prior work already in place, and drive the turn under test | `seed` (authored `mode: "inline"`; `"replay"` for a local check) |

**Build** is documented in full below. The other three, the director-script
vocabulary, and the seeding modes are in [`case-shapes.md`](case-shapes.md).

## Core principle (all shapes)

**Write expectations from intent, then calibrate against a real build.** Decide
up front what makes *any* correct solution correct — the must-haves implied by
what the user actually said — then build the workflow once for real to calibrate
granularity: loosen what's over-specified, confirm the must-haves are
achievable, and catch requirements the agent legitimately satisfies a different
way. Don't transcribe one observed build into assertions — that overfits the
eval into "did the agent reproduce that run" instead of "did it solve the
problem."

**Keep the conversation in the user's voice.** State the goal and real
constraints the way a user would — don't name node types, wire up the structure,
or restate your `outcomeExpectations` in the prompt. If the conversation spells
out the build, the case only tests whether the agent can follow instructions and
the expectations become tautological; the gap between what the user asks for and
how a correct workflow realizes it is the capability under test. Even when the
anchor *is* honoring a user's stated technical preference, phrase it as their
need + constraint ("I need field X and the built-in node doesn't expose it, so
pull it straight from the API") — not as an implementation spec ("use an HTTP
Request node").

**Write the conversation in English** unless the user asked otherwise (or the
case exists specifically to test non-English handling). Sourced real threads are
frequently non-English — translate the intent into English when you rewrite the
prompt in the user's voice; the failure mode is the anchor, not the original
language.

**Trim to the smallest multi-turn conversation that reproduces the issue.**
Real sourced threads are long (dozens of turns of setup, debugging, and
tangents) — do **not** transcribe them. Distill to the fewest turns that still
drive the build or behaviour under test. Every retained turn must earn its place:
a turn stays only if it is *load-bearing* — a value the agent must ask for
(withheld until asked, via a director note), a correction/push-back the case
exists to test, or a plan approval that gates the build. If removing a turn
doesn't change what's tested, remove it. **Collapse to a single turn** whenever
the whole request can be stated at once without a load-bearing exchange; keep it
multi-turn *only* for those exchanges, and keep each director script in one turn
(don't fabricate assistant "done" turns to sequence steps — see
[`case-shapes.md`](case-shapes.md)). A minimal conversation isolates the
capability; a transcribed one buries it in noise and tests instruction-following.

**Size the build, not just the assertions.** Real sourced prompts are often
kitchen-sink ("production-ready, runs forever, 3 feed posts *and* 8 stories a
day", "generate 50 articles daily") and reliably blow the ~900s build budget (see
"Known harness limitations"). A **faithful trim is a legitimate authoring move**:
reduce batch sizes, drop one of several parallel pipelines, or merge adjacent AI
steps so the case builds within budget — then note the reduction in the case
`description` ("the original request also asked for an 8-stories/day pipeline;
scoped to feed posts so it builds in budget"). Keep the capability under test; cut
the combinatorial bulk. A case that never builds tests nothing.

## Workflow

These steps map to the four gates from [Set the autonomy level first](#set-the-autonomy-level-first):
sourcing (before step 1) is the **selection** gate, including the optional offer
to record a source observation; steps 1–2 are the **shape +
expectations** gate; steps 5–6 are the **calibration** gate; steps 7–8 are the
**push** gate. In *autonomous* mode you flow through all of them and summarize in
a decision log; in *checkpoint* mode you pause at each with a proposal, and at
calibration you hand the driver the thread link + login to review the real build
(see [Set the autonomy level first](#set-the-autonomy-level-first)). Calibration
(step 6) always surfaces meaning-flipping verdicts explicitly regardless of level.

1. **State the must-haves first.** From the conversation alone, list what every
   correct workflow must do (trigger type, essential operations, gating
   condition) — those become draft `outcomeExpectations`. Required fields:
   `conversation` (≥1 turn, first `user`), `complexity`, `tags`, and **at least
   one** of `executionScenarios` / `processExpectations` / `outcomeExpectations`.
2. **Draft the case** from the template below; validate it loads (see
   "Validate").
3. **Smoke-test the *environment* with one case before any batch.** Run a single
   case end-to-end first. This validates auth / model / `--base-url` / the built
   dist for ~1/Nth the cost — distinct from validating a *case*. If that one case
   crashes at execution (especially with an identical error you'd expect to hit
   every case), fix the environment before running the batch (see "A red is
   signal" → environment check). Running 15 cases only to discover a stale-dist
   crash on all of them wastes a full run.
4. **Build it once** against a running instance (see
   [`running-evals.md`](running-evals.md)) with `--keep-workflows` so the built
   workflow stays for inspection.
5. **Inspect** — read the built workflow (the run prints `BUILT (<id>)`; fetch
   via `GET /rest/workflows/<id>`) and the HTML report's transcript to see what
   the agent actually did.
6. **Calibrate — sharpen assertions; never dull them to force a green.** Fix
   assertions that are genuinely mis-sized: relax one that pins a choice the
   conversation left open (so a valid *alternative* build wrongly fails), tighten
   one a wrong build would slip past, and phrase `executionScenarios` to match how
   the workflow runs on mocked data. But when a scenario goes red because the
   build has a real gap, or because the harness can't exercise it, **that red is
   the result — keep it and surface why** (see "A red is signal", below). Never
   delete a scenario, weaken an assertion, or drop to build-only just to make the
   run green. And when the case comes back **green**, that is a result to earn,
   not to accept: confirm the precondition actually fired, then re-derive it from
   the raw thread before calling the case a regression guard (see
   [First reproduce, then reclassify](#first-reproduce-then-reclassify)).
7. **Push to the suite — do NOT commit the JSON.** Once calibrated, push the case
   into its curated lang-tracer suite with `eval:langtracer-push` (see
   [Push to a lang-tracer suite](#push-to-a-lang-tracer-suite)); the suite is the
   case's home, not the repo. Leave the `data/workflows/*.json` file uncommitted
   (or delete it once it's in the suite). Committing new case JSONs into the repo
   is no longer the approach. (An `inline` seed pushes with the case; only a
   `replay` case is refused — it's a local throwaway; see
   [`case-shapes.md`](case-shapes.md).) For a sourced case,
   finish by **linking it to its source thread/finding** over the MCP — see
   [Link the pushed case to its source](#link-the-pushed-case-to-its-source-provenance-step--always-do-this).
8. **Hand back links, and a ticket proposal if the case found a gap.** Report the
   pushed case as `<base>/test-cases/<id>`, not `#<id>` ([Share links, never bare
   ids](#share-links-never-bare-ids)), and if calibration kept a real
   capability-gap red, propose a Linear ticket for it ([Capability gap → propose a
   Linear ticket](#capability-gap--propose-a-linear-ticket)) rather than leaving the
   gap as a red case nobody owns.

`--iterations N` is available to measure flakiness (pass@k / pass^k) — reach for
it when you suspect a case is non-deterministic or before promoting it to a
gated tier, not as a routine step (each iteration is a full build + execution).

Gut-check: if you can't picture a plausible *wrong* build that this case
reliably turns **red**, the assertions are too loose to guard anything.

**Confirm the precondition fired, not just the green.** For any *conditional*
assertion — "when X happened, the agent did Y" (most `processExpectations`, and
any behaviour case) — a pass has two readings: the agent did Y, or **X never
happened** and the assertion passed vacuously. A behaviour case that hinges on
the mock producing a specific failure (e.g. an AI node simulated to empty so a
downstream parse node fails) is the classic trap: if the mock instead returns
parseable data, the failure never occurs and the case guards nothing while
showing green. Calibration must read the execution trace and the agent's
`finalText` (`buildTrace.finalText` in the verifier snapshot, or the HTML report)
and verify X actually materialised — the direct-loop `eval-results.json` does not
persist per-expectation judge reasoning, so pass/fail alone can't tell you which
reading you got.

**The negative form is the easiest to fool yourself with.** An assertion phrased
as "the agent did NOT call `X` with a bad argument" passes when the agent called
`X` correctly *and* when **it never called `X` at all**. Those are opposite
results and the judge reports the same green. So for any assertion about tool
misuse, confirm the tool was actually invoked before believing the pass: parse
`testCases[].transcriptPerRun[][].steps[]` for the call. Note the transcript
groups multi-action tools under a bare `toolName` (`nodes`, `workflows`,
`credentials`), so read `args.action` to get the real one — filtering on
`nodes[explore-resources]` finds nothing and looks like a clean pass. Measured on
this corpus: a batch of five tool-misuse cases scored 100% on its first
calibration run, and three of them were passing vacuously because the tool under
test was never called.

**A sourced failure that does not reproduce is not yet a regression guard — first
re-derive the precondition.** Behaviour does drift across versions, and a case
that flips from *capability-gap* (red) to *regression guard* (green, catches a
re-introduction) is a legitimate and valuable outcome. But reach it by
elimination, not by default: a green far more often means *your case never set up
the situation* than *the builder improved*. See below.

## First reproduce, then reclassify

A case built from a real failure that comes back green is the most common
outcome of a first calibration run, and "the build must have improved" is the
most common wrong conclusion. T