Agent skill · data analytics · microsoft
flaky-smoke-tests
Diagnose intermittent VS Code Electron smoke-test failures from the Azure DevOps Flaky Smoke Tests pipeline (definition 700). Covers finding failed iterations, downloading task logs and platform artifacts with Azure CLI, correlating cumulative runner logs, tracing the introducing commit, and queueing focused validation runs.
Why this skill is useful
Provides specific Azure CLI commands and diagnostic procedures for troubleshooting flaky smoke tests in Azure DevOps that the AI wouldn't reliably generate on its own.
What it needs
Requires azure-cli installed locally. About 7k tokens when loaded. Last updated 2026-08-07. 188,455 stars on the source repository.
What this skill does
Diagnosing Flaky Smoke Tests Use this skill for failures from the Azure DevOps Flaky Smoke Tests pipeline: Organization: https://dev.azure.com/monacotools Project: Monaco Definition ID: 700 Pipeline YAML: build/azure-pipelines/product-smoke-flaky.yml Scheduled runs: twice daily on main The pipeline builds VS Code from source and runs the complete Electron smoke suite once per entry in the iterations parameter. Each platform runs its iterations sequentially in one job. The exception is Windows unit and integration tests, which are each split into two parallel jobs: WindowsSmokeFlakyUnitodd / WindowsSmokeFlakyIntegrationodd run the odd iterations and WindowsSmokeFlakyUniteven / WindowsSmokeFlakyIntegrationeven run the even ones. Each shard has its own artifacts (logs-windows-x64-<unit integration>-<odd even>-<attempt>) and its own Slack tally line. Important Pipeline Behavior Each iteration is a separate timeline task named Smoke test iteration <N>/<total> (Electron). Iteration tasks use continueOnError: true, so a failed iteration normally has result succeededWithIssues and later iterations still run. Do not infer smoke-test health from the overall build or job result. Inspect every iteration task. Logs are published once per platform job from the shared .build/logs directory. The artifact is job-scoped, not iteration-scoped: smoke-test-runner.log can contain multiple iterations, and suite directories are not separated by iteration. Do not cancel a platform job while investigating if you still need its logs artifact. Cancellation can skip the publish steps entirely, even when one or more failed iteration task logs are already available. If a validation run has become predictably red, either let the current job finish or queue a single diagnostic iteration and let that job publish before iterating. Later iterations can overwrite or truncate files in the shared logs directory. …
How to use it
Reference it in AdaL, Claude Code, Cursor or any coding agent — nothing to install:
@skills microsoft/flaky-smoke-tests