Agent skill · personamanagmentlayer
stream-processing-expert
Build continuous data pipelines over event streams: windowing, watermarks, exactly-once semantics, stateful processing and change data capture. Use when the user mentions stream processing, Flink, Spark Structured Streaming, CDC or Debezium, event time versus processing time, watermarks, tumbling or sliding windows, late-arriving data, or when the task involves replicating a database into a stream or computing continuous aggregates.
What it needs
About 6k tokens when loaded.
What this skill does
Stream Processing Expert Continuous computation over unbounded data. The hard parts are not throughput — they are time, state and what happens when something is retried. Complements kafka-expert (the transport) and airflow-expert (batch orchestration). Core Concepts Event Time, Not Processing Time Every event carries the time it happened. The stream processor sees it later, out of order, sometimes much later. Windowing on arrival time produces results that change if you replay the same data — which makes them untestable and unreproducible. Time Meaning Use --------------- ---------------------------------- ------------------------------------- Event time When it happened, from the payload Correctness; all business aggregation Ingestion time When the broker received it Rough operational metrics Processing time When the operator ran Latency monitoring only If the source has no reliable event timestamp, that is a defect in the producer. Fix it there. Watermarks A watermark is the processor's assertion that no event older than time T will arrive. It is how an unbounded stream produces a finite answer: when the watermark passes the end of a window, the window closes and emits. The watermark is a bet on lateness. Too tight and correct data is dropped; too loose and every result waits. withidleness matters in production: a partition with no traffic holds the watermark back for the whole job, and every window stops emitting. This is a frequent and confusing outage. Windows Window Shape Use ---------------- ---------------------- -------------------------------------- Tumbling Fixed, non-overlapping Hourly totals, daily counts Sliding Fixed, overlapping "Last 5 minutes, updated every minute" Session Gap-defined User activity bursts Global + trigger Custom Anything the above cannot express Sliding windows multiply state: a 1-hour window sliding every minute keeps each event in 60 windows. Check the cost before choosing the slide. …
How to use it
Reference it in AdaL, Claude Code, Cursor or any coding agent — nothing to install:
@skills personamanagmentlayer/stream-processing-expert