Agent skill · data analytics · aradotso
realtime-cinema-data-engineering-pipeline
Build end-to-end real-time data pipelines with Kafka, PostgreSQL, Airflow, and Streamlit using Medallion Architecture for streaming analytics.
Why this skill is useful
Adds executable scripts for Kafka event streaming and PostgreSQL data management that aren't in public docs.
What it needs
Requires docker, docker-compose, python installed locally. About 7k tokens when loaded. Last updated 2026-08-05. 5 stars on the source repository.
What this skill does
CinéWorld Real-Time Data Engineering Pipeline Skill Skill by ara.so — Data Skills collection. Overview This project implements an end-to-end real-time data engineering pipeline using Apache Kafka for event streaming, PostgreSQL for data warehousing with Medallion Architecture (Bronze/Silver/Gold layers), Apache Airflow for ELT orchestration, and Streamlit for live visualization. Perfect for learning how to build production-grade streaming data pipelines that process 1M+ events. Installation Prerequisites Docker and Docker Compose Python 3.8+ Virtual environment (recommended) Setup Steps Important: Wait 2-3 minutes for Airflow to fully initialize before proceeding. Architecture Components 1. Medallion Architecture Layers Bronze Layer: Raw JSON event data ingested from Kafka Silver Layer: Normalized 3NF tables (Customers, Movies, Showtimes, Transactions) Gold Layer: Materialized views for analytics 2. Kafka Event Producer Generate and stream synthetic cinema transaction events: 3. Kafka Consumer (Bronze Layer Ingestion) Consume events from Kafka and insert into PostgreSQL Bronze layer: 4. Airflow ELT DAG (Bronze → Silver → Gold) Orchestrate data transformation pipeline: 5. Streamlit Dashboard Real-time analytics visualization: Configuration Environment Variables Create a .env file for configuration: Docker Compose Services Common Workflows Starting the Complete Pipeline Manual DAG Trigger Query Gold Layer Directly Troubleshooting Kafka Consumer Not Receiving Messages Airflow DAG Not Running PostgreSQL Connection Issues Streamlit Dashboard Not Updating Performance Optimization Batch Processing Size Adjust batch size in Airflow DAG: Kafka Consumer Parallelization Materialized View Incremental Refresh This skill enables AI agents to help developers build production-grade real-time streaming data pipelines with proper architecture patterns, orchestration, and visualization.
How to use it
Reference it in AdaL, Claude Code, Cursor or any coding agent — nothing to install:
@skills aradotso/realtime-cinema-data-engineering-pipeline