Agent skill · data analytics · personamanagmentlayer
databricks-expert
Expert-level Databricks platform, Apache Spark, Delta Lake, MLflow, notebooks, and cluster management. Use when the user mentions Apache Spark, Delta Lake, MLflow, lakehouse architecture, or PySpark, or when the task involves Cluster Configuration and Management, Delta Lake Architecture, PySpark Data Processing, or MLflow Integration.
Why this skill is useful
Adds executable scripts for cluster management and Delta Lake operations that enhance data processing capabilities on Databricks.
What it needs
Requires databricks-sdk, pyspark installed locally. About 2k tokens when loaded. Last updated 2026-03-30. 41 stars on the source repository.
What this skill does
Databricks Expert You are an expert in Databricks with deep knowledge of Apache Spark, Delta Lake, MLflow, notebooks, cluster management, and lakehouse architecture. You design and implement scalable data pipelines and machine learning workflows on the Databricks platform. Best Practices 1. Cluster Configuration Use job clusters for scheduled workflows (lower cost) Use instance pools for faster cluster startup Enable autoscaling with appropriate min/max workers Set autotermination to 15-30 minutes for interactive clusters Use Photon-enabled clusters for SQL workloads 2. Delta Lake Optimization Enable auto-optimize for write and compaction Use Z-ordering for columns in filter predicates Partition large tables by date or high-cardinality columns Run VACUUM regularly but respect retention periods Use Change Data Feed for incremental processing 3. Performance Tuning Use broadcast joins for small dimension tables Enable adaptive query execution (AQE) Cache DataFrames that are reused multiple times Use partition pruning in queries Optimize shuffle operations with appropriate partition counts 4. Cost Optimization Use Spot/Preemptible instances for fault-tolerant workloads Terminate idle clusters automatically Use table properties to enable auto-compaction Monitor cluster utilization metrics Use Delta caching for frequently accessed data 5. Security and Governance Use Unity Catalog for centralized governance Implement fine-grained access control Store secrets in Databricks secret scopes Enable audit logging Use service principals for production jobs Anti-Patterns 1. Collecting Large DataFrames 2. Not Using Delta Lake Optimization 3. Inefficient Joins 4. …
How to use it
Reference it in AdaL, Claude Code, Cursor or any coding agent — nothing to install:
@skills personamanagmentlayer/databricks-expert