Agent skill · data analytics · jeremylongshore
databricks-streaming-guardian
Guard production Databricks data pipelines — Delta Lake, Liquid Clustering, Structured Streaming, Auto Loader, and DLT — against the twelve foot-guns that fire at scale: OPTIMIZE/auto-compaction conflicts, Liquid-Clustering merge conflicts, VACUUM breaking a streaming checkpoint, RocksDB OOM, Auto Loader schema-evolution stops, and DLT refresh data loss. Includes a PreToolUse hook that blocks DROP/CREATE-OR-REPLACE/VACUUM on a table with active streaming consumers. Use when a Delta MERGE/OPTIMIZE fails with a concurrency exception, a stream breaks after VACUUM or a table replace, an Auto Loader stream stops on a new column, a DLT refresh drops data, or before running a destructive op on a streamed-from table. Trigger with "ConcurrentAppendException", "ConcurrentDeleteDeleteException", "DELTA_FILE_NOT_FOUND", "streaming checkpoint broke", "vacuum broke my stream", "autoloader UnknownFieldException", "dlt full refresh".
Why this skill is useful
Adds a PreToolUse hook that prevents destructive operations on active streaming tables, enhancing data pipeline reliability.
What it needs
Requires bash, databricks, jq, python3 installed locally. About 7k tokens when loaded. Last updated 2026-08-07. 2,604 stars on the source repository.
What this skill does
Databricks Streaming Guardian The data-ops spine of the pack. Delta Lake, Liquid Clustering, Structured Streaming, and DLT each ship a different set of foot-guns that fire most visibly when production data flows through them at scale — and most of them are documented platform decisions that surprise engineers, not bugs. This skill's job is friction at trigger time (a hook that blocks the genuinely-irreversible op) plus deterministic recovery when something already broke. Overview Twelve foot-guns, grouped by the surface that triggers them. Eleven are owned outright (D01–D10, D12); the twelfth — D11, DLT rebuild cost — is shared with databricks-cost-leak-hunter: this skill checks the rebuild cost as part of pre-refresh safety, that skill owns ongoing cost optimization. Delta write conflicts. D01 ConcurrentDeleteDeleteException — a manual OPTIMIZE colliding with auto-compaction, which is silently enabled on any table touched by MERGE/UPDATE/DELETE. D02 ConcurrentAppendException after moving to Liquid Clustering — LC keeps file-set-level writer conflicts; a fan-out MERGE breaks unless its predicate is narrowed to the clustering keys. Streaming + checkpoint. D03 DELTAFILENOTFOUNDDETAILED — VACUUM deletes files the checkpoint pins to. D04 silent checkpoint corruption / reset to batch 0. D05 RocksDB state-store off-heap OOM (the heap looks fine while off-heap state pins multi-GB). D12 DIFFERENTDELTATABLEREADBYSTREAMINGSOURCE — CREATE OR REPLACE mints a new UUID and kills every active consumer. Migration + evolution. D06 Liquid-Clustering migration's hidden full-rewrite cost + downstream partition-predicate breakage. D07 time travel breaking silently when VACUUM crosses the retention boundary. D10 Auto Loader UnknownFieldException stopping the stream on every new column. DLT. D08 the @dlt.table thread race (out-of-order registration). D09 full refresh silently dropping data from a non-replayable source. …
How to use it
Reference it in AdaL, Claude Code, Cursor or any coding agent — nothing to install:
@skills jeremylongshore/databricks-streaming-guardian--a4d0f0