Agent skill · data analytics · aradotso
harvard-artifacts-collection-etl-analytics
Build ETL pipelines and analytics dashboards for Harvard Art Museums API data using Python, SQL, and Streamlit
Why this skill is useful
Adds a complete ETL pipeline and analytics dashboard with executable scripts for data extraction, transformation, and visualization that are specific to the Harvard Art Museums API.
What it needs
Requires mysqlclient, python, streamlit installed locally. Requires harvard account access. About 5k tokens when loaded. Last updated 2026-08-05. 5 stars on the source repository.
What this skill does
Harvard Artifacts Collection ETL Analytics Skill by ara.so — Data Skills collection. This project provides a complete data engineering and analytics solution for the Harvard Art Museums API. It demonstrates ETL pipeline construction, relational database design, SQL analytics, and interactive visualization using Streamlit. The application extracts artifact metadata, transforms nested JSON into structured tables, loads data into SQL databases, and provides 20+ analytical queries with auto-generated visualizations. Installation Project Architecture API → ETL → SQL → Analytics → Visualization Data Source: Harvard Art Museums API ETL Engine: Python (requests, pandas) Database: MySQL / TiDB Cloud Visualization: Streamlit + Plotly Getting Harvard API Key 1. Visit Harvard Art Museums API 2. Register for a free API key 3. Store in environment variable: HARVARDAPIKEY Core Components 1. API Data Collection Fetch artifacts with pagination and rate limiting: 2. ETL Pipeline Transform nested JSON into relational tables: 3. Database Schema and Loading 4. SQL Analytics Queries 5. Streamlit Dashboard Running the Application Common Patterns Incremental Data Loading Error Handling and Logging Troubleshooting API Rate Limiting: Add time.sleep(1) between requests or reduce page size Database Connection Errors: Verify environment variables and database credentials Missing Data: Check API response structure; use .get() with defaults for optional fields Memory Issues: Process data in smaller batches; use chunked DataFrame operations Foreign Key Violations: Ensure parent records exist before inserting child records; use transactions This skill enables AI agents to help developers build complete data engineering pipelines from API to analytics dashboard.
How to use it
Reference it in AdaL, Claude Code, Cursor or any coding agent — nothing to install:
@skills aradotso/harvard-artifacts-collection-etl-analytics