Agent skill · data analytics · jeremylongshore
clickhouse-webhooks-events
Ingest data into ClickHouse from webhooks, Kafka, and streaming sources with batching, dedup, and exactly-once patterns. Use when building data ingestion pipelines, consuming webhook payloads, or integrating Kafka topics into ClickHouse. Trigger with "clickhouse ingestion", "clickhouse webhook", "clickhouse Kafka", "stream data to clickhouse", "clickhouse data pipeline".
Why this skill is useful
Provides specific patterns and code for ingesting data into ClickHouse from various sources, which the AI wouldn't reliably generate on its own.
What it needs
Requires @clickhouse/client installed locally. About 3k tokens when loaded. Last updated 2026-08-07. 2,604 stars on the source repository.
What this skill does
ClickHouse Data Ingestion Overview Build data ingestion pipelines into ClickHouse from HTTP webhooks, Kafka, and streaming sources with proper batching, deduplication, and error handling. The core rule: ClickHouse hates one-row-at-a-time inserts — buffer events and flush them in batches. This skill covers four ingestion paths (application-side webhook receiver, server-side Kafka engine, managed ClickPipes, and HTTP bulk loads) plus idempotent dedup and insert monitoring. Prerequisites A ClickHouse table with an appropriate engine already exists (a MergeTree variant, e.g. analytics.events) — see clickhouse-core-workflow-a. The @clickhouse/client package is installed and connected via CLICKHOUSEHOST. For the Kafka paths, a reachable Kafka broker and topic. Instructions Step 1: Webhook Receiver with Batched Inserts Buffer incoming events in memory, flush on a size threshold or a timer, and re-queue the batch on failure so no event is lost. This is the application-side core of the skill: Step 2: Choose a Server-Side or Managed Path For high-volume streams, prefer a path that needs no application consumer: Kafka table engine — ClickHouse consumes a topic directly and a materialized view pipes rows into your MergeTree table. No consumer to run. ClickPipes — ClickHouse Cloud's managed, code-free ingestion for Kafka, Confluent, Amazon MSK, S3, and GCS. HTTP interface — bulk-load CSV / NDJSON / Parquet from files, remote URLs, or S3 with plain curl, no client library. Full DDL and configuration for all three: see Ingestion methods. Step 3: Make Ingestion Idempotent and Observable Webhook retries and Kafka reprocessing deliver duplicates. Use a ReplacingMergeTree keyed on a unique eventid so re-delivered events collapse to one row, and query system.querylog to watch insert throughput and errors. Full DDL, monitoring queries, and the batch-tuning matrix: Deduplication & monitoring. …
How to use it
Reference it in AdaL, Claude Code, Cursor or any coding agent — nothing to install:
@skills jeremylongshore/clickhouse-webhooks-events