Agent skill · data analytics · aradotso
harvard-art-museums-data-engineering-pipeline
Build end-to-end data engineering pipelines with Harvard Art Museums API, ETL, SQL analytics, and Streamlit dashboards
Why this skill is useful
Provides a complete ETL pipeline with executable scripts for data extraction, transformation, and loading from the Harvard Art Museums API into a SQL database.
What it needs
Requires mysql-connector-python, pandas, plotly, python-dotenv, requests, streamlit installed locally. Requires harvardartmuseums account access. About 6k tokens when loaded. Last updated 2026-08-05. 5 stars on the source repository.
What this skill does
Harvard Art Museums Data Engineering Pipeline Skill by ara.so — Data Skills collection. What This Project Does The Harvard-Artifacts-Collection-Data-Engineering-Analytics-App is a complete data engineering solution that demonstrates: API Integration: Fetching artifact data from Harvard Art Museums API with pagination and rate limiting ETL Pipeline: Extracting, transforming, and loading nested JSON into normalized SQL tables Database Design: Multi-table relational schema with proper foreign keys SQL Analytics: 20+ predefined analytical queries for insights Interactive Visualization: Streamlit dashboards with Plotly charts This project serves as a reference architecture for building production-grade data pipelines. Installation Required packages: Configuration Environment Variables Create a .env file in the project root: Getting Harvard API Key 1. Visit https://www.harvardartmuseums.org/collections/api 2. Request an API key (free for non-commercial use) 3. Add to .env file Database Setup The application supports MySQL and TiDB Cloud. Create the database: Tables are created automatically by the ETL pipeline. Running the Application The app will be available at http://localhost:8501 Core Components 1. API Data Extraction 2. ETL Pipeline Extract and Transform: Load to Database: 3. Complete ETL Execution 4. SQL Analytics Queries 5. Streamlit Dashboard Common Patterns Rate Limiting API Requests Incremental Data Loading Error Handling in ETL Troubleshooting API Key Issues Database Connection Errors Missing Dependencies Streamlit Port Already in Use Large Dataset Memory Issues This skill enables AI agents to help developers build complete data engineering pipelines using the Harvard Art Museums API, from data extraction through visualization.
How to use it
Reference it in AdaL, Claude Code, Cursor or any coding agent — nothing to install:
@skills aradotso/harvard-art-museums-data-engineering-pipeline