Agent skill · operations · jeremylongshore
clickhouse-incident-runbook
ClickHouse incident response — triage, diagnose, and remediate server issues using system tables, kill stuck queries, and execute recovery procedures. Use when ClickHouse is slow, unresponsive, OOM-killed, out of disk, backing up merges, or producing errors in production and you need an on-call playbook. Trigger with "clickhouse incident", "clickhouse outage", "clickhouse down", "clickhouse emergency", "clickhouse on-call", "clickhouse broken".
Why this skill is useful
Provides specific SQL commands and shell procedures for diagnosing and resolving ClickHouse incidents that the AI wouldn't reliably generate on its own.
What it needs
Requires curl installed locally. About 4k tokens when loaded. Last updated 2026-08-07. 2,604 stars on the source repository.
What this skill does
ClickHouse Incident Runbook Overview Step-by-step procedures for triaging and resolving ClickHouse incidents using built-in system tables and SQL commands. Start here: assess severity, run quick triage, walk the decision tree, then jump to the matching remediation procedure. Prerequisites Network access to the ClickHouse HTTP interface (default port 8123) or a working clickhouse-client. A user with rights to read system. tables and issue KILL QUERY / ALTER. Shell access to the host or container for P1 restarts (systemctl, docker, or kubectl). Severity Levels Level Definition Response Examples ------- ------------ ---------- ---------- P1 ClickHouse unreachable / all queries failing < 15 min Server down, OOM, disk full P2 Degraded performance / partial failures < 1 hour Slow queries, merge backlog P3 Minor impact / non-critical errors < 4 hours Single table issue, warnings P4 No user impact Next business day Monitoring gaps, optimization Instructions Work the incident top to bottom: triage, classify with the decision tree, then apply the matching procedure. 1. Quick triage (run first) 2. Decision tree — classify the failure 3. Apply the matching remediation Each branch maps to a full procedure (SQL + shell, copy-paste ready) in references/remediation-procedures.md: P1: Server down / OOM — inspect dmesg/journalctl, restart, verify. P1: Disk full — find largest tables, drop old partitions, check system.disks. P2: Stuck queries — inspect system.processes, KILL QUERY by id/user/elapsed. P2: Too many parts — check part counts, raise partstothrowinsert, batch inserts. P2: Memory pressure — rank by memoryusage, kill the largest, cap maxmemoryusage. P3: Replication lag — inspect system.replicas for queue and replica gaps. 4. Collect evidence and communicate Once mitigated, export the error window and post status updates. Templates and INTO OUTFILE exports are in references/evidence-and-comms.md. …
How to use it
Reference it in AdaL, Claude Code, Cursor or any coding agent — nothing to install:
@skills jeremylongshore/clickhouse-incident-runbook