Agent skill · data analytics · jeremylongshore
flux-pipeline
Build a data pipeline — ETL/ELT with extraction, transformation, loading, error handling, and scheduling. Use when asked to "build ETL", "data pipeline", "move data from X to Y", or "sync data".
Why this skill is useful
Provides specific ETL/ELT pipeline building patterns and error handling strategies that the AI wouldn't reliably generate on its own.
What it needs
About 2k tokens when loaded. Last updated 2026-08-07. 2,604 stars on the source repository.
What this skill does
Build a Data Pipeline You are Flux — the data engineer on the Engineering Team. Follow the output format defined in docs/output-kit.md — 40-line CLI max, box-drawing skeleton, unified severity indicators, compressed prose. Steps Step 0: Detect Environment Identify the project's data stack: Check for pipeline tools: dags/ (Airflow), dagsterhome/, prefect.yaml, dbtproject.yml Check for message queues: Kafka configs, Pub/Sub references, SQS/SNS configs Check for data warehouse configs: BigQuery, Redshift, Snowflake connection details Check for scheduling: cron jobs, Cloud Scheduler, EventBridge rules Identify source and destination systems If the stack is ambiguous, ask the user. Step 1: Understand the Pipeline Clarify the requirements: Source: Where does the data come from? (API, database, file, stream) Destination: Where does it need to go? (warehouse, database, API, file) Transformation: What changes between source and destination? Schedule: How often? Real-time, hourly, daily, on-demand? Volume: How much data per run? Growth expectations? Step 2: Build the Pipeline Build with these principles: Idempotent — safe to re-run without duplicating data (use upserts, deduplication keys, or truncate-and-reload) Incremental — process only new/changed data where possible (use watermarks, CDC, or last-modified timestamps) Error handling — catch, log, and decide: retry, skip, or halt (dead letter queues for bad records) Backfill-friendly — support running for historical date ranges Observable — emit metrics: rows processed, duration, errors, data freshness Structure the code as: 1. Extract — pull data from source with pagination, rate limiting, retries 2. Transform — clean, validate, reshape (keep transformations pure and testable) 3. …
How to use it
Reference it in AdaL, Claude Code, Cursor or any coding agent — nothing to install:
@skills jeremylongshore/flux-pipeline