Agent skill · data analytics · aj-geddes
Model Monitoring
Monitor model performance, detect data drift, concept drift, and anomalies in production using Prometheus, Grafana, and MLflow
Why this skill is useful
Provides specific implementation patterns and metrics tracking for monitoring model performance that the AI wouldn't reliably generate on its own.
What it needs
Requires grafana, mlflow, prometheus installed locally. About 5k tokens when loaded. Last updated 2026-03-04. 307 stars on the source repository.
What this skill does
Model Monitoring Overview Monitoring deployed machine learning models ensures they continue to perform well in production, detecting data drift, concept drift, and performance degradation. When to Use When models are deployed in production environments serving real users When detecting data drift or concept drift in input features When tracking model performance metrics over time When ensuring model reliability, accuracy, and operational health When implementing ML observability and alerting systems When establishing thresholds for model retraining or intervention Monitoring Components Performance Metrics: Accuracy, latency, throughput Data Drift: Distribution changes in input features Concept Drift: Changes in target variable relationships Output Drift: Changes in prediction distribution Feature Drift: Individual feature distribution changes Anomaly Detection: Unusual samples in production Monitoring Tools Prometheus: Metrics collection and storage Grafana: Visualization and dashboarding MLflow: Model tracking and registry TensorFlow Data Validation: Data statistics Evidently: Drift detection and monitoring Great Expectations: Data quality assertions Python Implementation Drift Detection Techniques Kolmogorov-Smirnov Test: Statistical test for distribution change Population Stability Index: Measures feature distribution shifts Chi-square Test: Categorical feature drift Wasserstein Distance: Optimal transport-based distance Jensen-Shannon Divergence: Information-theoretic metric Alert Thresholds Critical: >5% accuracy drop, immediate action High: Data drift in 3+ features, investigation needed Medium: Output drift, monitoring increased Low: Single feature drift, log and track Deliverables Monitoring dashboards Alert configuration Drift detection reports Performance degradation analysis Retraining recommendations Action playbook
How to use it
Reference it in AdaL, Claude Code, Cursor or any coding agent — nothing to install:
@skills aj-geddes/model-monitoring