--- name: opentelemetry-net-instrumentation description: Provides guidance for implementing OpenTelemetry instrumentation in .NET codebases, covering tracing (Activities/Spans), metrics, logs, naming conventions, error handling, performance, SDK setup, resources, context propagation, and API design best practices. version: 2.0.0 tags: - opentelemetry - dotnet - observability - tracing - metrics - logs - performance --- # OpenTelemetry .NET Instrumentation Skill ## When to Use - Adding OpenTelemetry instrumentation to .NET code (traces, metrics, logs) - Creating or modifying ActivitySources, Meters, or ILogger usage - Setting up the OpenTelemetry SDK, resources, exporters, or sampling - Reviewing telemetry implementations for spec compliance - Optimizing instrumentation performance - Designing telemetry APIs that become part of the public surface - Implementing context propagation across service boundaries ## Architecture: .NET Is Different **CRITICAL**: The .NET OpenTelemetry implementation is fundamentally different from other platforms. .NET provides tracing, metrics, and logging APIs **in the framework itself**. That means **OTel does not provide a separate instrumentation API** — it uses the built-in .NET APIs and acts as the collection/export layer. ### The Three Built-in .NET APIs (Primary — Zero Dependencies) | Signal | .NET Framework API | Namespace | |--------|-------------------|-----------| | **Tracing** | `ActivitySource` / `Activity` | `System.Diagnostics` | | **Metrics** | `Meter` / `Counter` / `Histogram` / etc. | `System.Diagnostics.Metrics` | | **Logging** | `ILogger` | `Microsoft.Extensions.Logging` | These are **the primary and only APIs** library authors should use for instrumentation. They ship with the .NET runtime — **no NuGet packages required**. ### The OTel Collection/Export Layer (Secondary — Application Root Only) OTel NuGet packages are the **collection and export layer**, added only at the application composition root (not in libraries): | Package | Purpose | When to add | |---------|---------|-------------| | `OpenTelemetry.Extensions.Hosting` | DI integration for ASP.NET Core / generic host | Application only | | `OpenTelemetry.Exporter.Console` | Console exporter (dev/testing) | Application only | | `OpenTelemetry.Exporter.OpenTelemetryProtocol` | OTLP exporter (production) | Application only | | `OpenTelemetry.Exporter.Prometheus*` | Prometheus metrics endpoint | Application only | | `OpenTelemetry.Instrumentation.AspNetCore` | Auto-instrument ASP.NET Core requests | Application only | | `OpenTelemetry.Instrumentation.Http` | Auto-instrument HttpClient calls | Application only | | `OpenTelemetry.Instrumentation.SqlClient` | Auto-instrument SQL calls | Application only | ### Package Decision Guide **Before adding ANY OpenTelemetry NuGet package, discuss the trade-off with the user:** > "You're about to add an OTel NuGet package. Is this an application where you need to > export telemetry to an observability backend (Jaeger, Prometheus, OTLP collector)? > If you're writing a library, you likely need **zero** OTel packages — just use > `System.Diagnostics.ActivitySource` / `System.Diagnostics.Metrics.Meter` and let the > consuming application configure the export pipeline. Do you want to proceed?" **Library authors**: Add **nothing**. Use only `System.Diagnostics.*` and `ILogger`. The consuming application wires up the SDK and exporters. **Application authors**: Add `OpenTelemetry.Extensions.Hosting` + the exporters and instrumentation libraries you need. See [sdk-resources-and-logs-reference.md](sdk-resources-and-logs-reference.md) for full setup patterns. **Never add** `OpenTelemetry.Api` to a library — `System.Diagnostics.*` IS the API. For SDK setup, resource configuration, exporters, sampling, and logs integration, see [sdk-resources-and-logs-reference.md](sdk-resources-and-logs-reference.md). ## Core Principles ### Resiliency First **CRITICAL**: Exceptions in diagnostic/tracing/metrics logic MUST NEVER impact application processing. - Assume Activity instances can be null. Always protect against null Activity references except in Activity extension methods (use `activity?.ExtensionMethod()`) - Guard all instrumentation code with appropriate null checks ### API Surface Awareness - Any telemetry emitted becomes part of the public API surface - Changes are subject to breaking changes guidelines - Telemetry should be emitted by default (users opt-in to collection via OpenTelemetry extensions) - Exception: High-cardinality metric dimensions may require explicit opt-in ### Standards Compliance - Follow Microsoft best practices for [distributed tracing instrumentation](https://learn.microsoft.com/en-us/dotnet/core/diagnostics/distributed-tracing-instrumentation-walkthroughs) - Follow [OpenTelemetry semantic conventions](https://opentelemetry.io/docs/specs/semconv/) - Attribute values support: **string, boolean, double (IEEE 754), int64, byte arrays, and homogeneous arrays** of these primitive types. Null/empty values are valid and meaningful per the [OTel AnyValue spec](https://opentelemetry.io/docs/specs/otel/common/#anyvalue) — they MUST be stored and passed to exporters. - Attribute **keys** must be non-null, non-empty strings ## Traces / Spans (Activities) ### ActivitySource Setup ```csharp // ✅ CORRECT: Use ActivitySource, not DiagnosticSource public class MyFeature { // Primary ActivitySource - name typically matches the component or NuGet package name private static readonly ActivitySource ActivitySource = new("MyApp.MyComponent", "1.0.0"); // Specialized ActivitySource for opt-in scenarios private static readonly ActivitySource DetailedActivitySource = new("MyApp.MyComponent.Detailed", "1.0.0"); } ``` **Rules**: - Every component defines a primary `ActivitySource` for mainstream activities - Name typically matches the component or NuGet package (e.g., `"MyCompany.MyLibrary"`) - Version the ActivitySource using SemVer - Create separate `ActivitySource`s for specialized or opt-in scenarios. Use hierarchical source names, e.g. `MyCompany.MyLibrary` and `MyCompany.MyLibrary.Detailed`, so consuming applications can subscribe only to the sources they want via `AddSource(...)` and backends can filter by instrumentation scope. ### Creating Activities ```csharp // ✅ Check HasListeners, null-check, then guard expensive work behind IsAllDataRequested if (ActivitySource.HasListeners()) { using var activity = ActivitySource.StartActivity("ProcessItem", ActivityKind.Internal); if (activity != null && activity.IsAllDataRequested) { activity.DisplayName = "Processing order #12345"; activity.SetTag("app.item_id", itemId); activity.SetTag("app.item_type", itemType); } } // ❌ WRONG: Don't start activities in fire-and-forget tasks where the // using scope ends before the async work completes (AsyncLocal context is lost) async Task HelperAsync() { using var activity = ActivitySource.StartActivity("Helper"); _ = Task.Run(() => DoWorkAsync()); // ❌ activity disposed before task completes } ``` **Rules**: - Check `ActivitySource.HasListeners()` before creating (zero-allocation fast path) - Always null-check Activity after creation (listener may filter or sample it out) - Never start activities in async helper methods (`Activity.Current` uses `AsyncLocal`) - Guard expensive tag computation behind `activity.IsAllDataRequested` - Use W3C TraceContext. .NET Core 3.0+ / .NET 5+ uses it by default; older TFMs or .NET Framework apps may set `Activity.DefaultIdFormat = ActivityIdFormat.W3C` at startup and use `Activity.ForceDefaultIdFormat = true` to override hierarchical parents. ### Activity Naming ```csharp // ✅ Unique operation name, friendly display name (null-check before accessing) using var activity = ActivitySource.StartActivity( name: "ProcessItem", // Unique, identifies class of spans kind: ActivityKind.Internal ); if (activity != null) activity.DisplayName = "Processing order #12345"; // User-friendly, can be specific // ❌ WRONG: Don't include runtime data in operation name using var badActivity = ActivitySource.StartActivity($"Process_{itemId}"); // ❌ ``` **Rules**: - Each span type has unique `OperationName` (identifies statistically interesting class of spans) - Operation name should NOT contain runtime data (only compile/config-time info) - Use human-readable `DisplayName` for specifics - Follow [OpenTelemetry span naming conventions](https://opentelemetry.io/docs/specs/otel/trace/api/#span) ### SpanKind Selection Choose the correct `ActivityKind` to clarify the span's role in distributed tracing: | `ActivityKind` | OTel SpanKind | When to use | |----------------|---------------|-------------| | `Internal` | `INTERNAL` | Default — in-process operations not crossing a remote boundary | | `Server` | `SERVER` | Processing an incoming request/response call (HTTP server, gRPC server, RPC server) | | `Client` | `CLIENT` | Making an outgoing request/response call (HTTP client, database client, RPC call) | | `Producer` | `PRODUCER` | Enqueuing/publishing deferred work (message queue publish, event emit, job enqueue) | | `Consumer` | `CONSUMER` | Dequeuing/processing deferred work (message queue receive, event handle, job dequeue) | **Rules**: - A single span SHOULD NOT serve more than one purpose - Create the outgoing span **before** injecting its `SpanContext` into the request. If you inject first, the parent's context propagates instead and the outgoing span ends up dangling (no connection to the downstream call). - See [traces-and-propagation-reference.md](traces-and-propagation-reference.md) for detailed SpanKind guidance with examples ### Span Attributes (Tags) ```csharp // ✅ Application code: use your own namespace activity?.SetTag("myapp.order_id", orderId); activity?.SetTag("myapp.payment.status", "confirmed"); // ✅ Manual infrastructure instrumentation: use semantic conventions // activity?.SetTag("db.system.name", "postgresql"); // custom database client // activity?.SetTag("http.request.method", "GET"); // custom HTTP transport // Values can be strings, numbers, booleans, or homogeneous arrays activity?.SetTag("app.item_count", 42); activity?.SetTag("app.related_ids", new int[] { 1, 2, 3 }); // ❌ WRONG: PascalCase, hyphen delimiter, plural, or unrelated namespace activity?.SetTag("MyApp.OrderId", orderId); // ❌ Wrong case activity?.SetTag("myapp.order-id", orderId); // ❌ Wrong delimiter ``` **Rules**: - Namespace prefix matching your component: `myapp.*`, `myapp.db.*` - All lowercase, underscore (`_`) delimiters, singular form - Attribute values: string, boolean, double, int64, byte arrays, homogeneous arrays (null/empty valid per [AnyValue spec](https://opentelemetry.io/docs/specs/otel/common/#anyvalue)) - **Business/domain attributes**: use your own namespace (`myapp.*`). - **HTTP, database, messaging, or RPC concepts you manually instrument**: use [semantic conventions](https://opentelemetry.io/docs/specs/semconv/). Do not duplicate attributes already emitted by auto-instrumentation. Do not use OTel namespaces as prefixes for custom attributes. ### Activity Status and Errors ```csharp try { await ProcessItemAsync(); // ✅ success: leave status Unset, do not call SetStatus(Ok) — see rules below } catch (Exception ex) { if (activity != null) { activity.SetStatus(ActivityStatusCode.Error, ex.Message); // modern API activity.SetTag("error.type", ex.GetType().FullName); } throw; } ``` **Rules** (per [Recording Errors](https://opentelemetry.io/docs/specs/semconv/general/recording-errors/) and the [Set Status API spec](https://opentelemetry.io/docs/specs/otel/trace/api/#set-status)): - Leave span status **Unset** on success — do not call `SetStatus(ActivityStatusCode.Ok)`. - `Ok` is for application code, not instrumentation libraries. The trace API spec: "Instrumentation Libraries SHOULD NOT set the status code to `Ok`, unless explicitly configured to do so (...) Application developers and Operators may set the status code to `Ok`" — typically to override a library-reported `Error` they've decided isn't a real failure (e.g. suppressing a noisy 404). Once set, `Ok` is final; later calls are ignored." - On failure: **SHOULD** set `ActivityStatusCode.Error`, **SHOULD** set the `error.type` tag, **SHOULD** set the status description to the exception message. - Use `SetStatus` — legacy `otel.status_code`/`otel.status_description` tags are no longer needed. - Do **not** record errors that were retried or handled and let the operation complete gracefully — `Error` status and `error.type` describe operations that failed, not ones that recovered. ### Recording Exceptions — Keep Emitting Span Events, Add the Logs Opt-In `exceptions-spans` — the convention behind `Activity.AddEvent(new ActivityEvent("exception", ...))` — carries a **Deprecated** status in favor of `exceptions-logs`, but the .NET ecosystem has not followed yet: no `OpenTelemetry.*` .NET package implements the spec's `OTEL_SEMCONV_EXCEPTION_SIGNAL_OPT_IN` opt-in (verified against `OpenTelemetry.Api` 1.17.0), and `OpenTelemetry.Instrumentation.AspNetCore` 1.17.0 itself still records exceptions via `Activity.AddException`, i.e. span events. That means the exporters, backends, and dashboards a typical .NET user has wired up today are built to read exception span events, not log-based exceptions. **Keep implementing span events as the default** — dropping them because the underlying spec document is marked Deprecated will silently break exception visibility for anyone still on today's tooling. Layer the newer logs-based path on top as an opt-in, exactly as the spec's own transition guidance describes for instrumentations moving off span events: ```csharp // Mirrors the spec's OTEL_SEMCONV_EXCEPTION_SIGNAL_OPT_IN values: unset/anything else → spans only (today's default), "logs/dup" → both, "logs" → logs only. private static readonly string? ExceptionSignalOptIn = Environment.GetEnvironmentVariable("OTEL_SEMCONV_EXCEPTION_SIGNAL_OPT_IN"); private static readonly bool EmitSpanEvents = ExceptionSignalOptIn != "logs"; private static readonly bool EmitLogs = ExceptionSignalOptIn is "logs" or "logs/dup"; try { await ProcessItemAsync(); } catch (Exception ex) { activity?.SetStatus(ActivityStatusCode.Error, ex.Message); activity?.SetTag("error.type", ex.GetType().FullName); if (EmitSpanEvents) // ✅ default — what current .NET tooling and dashboards actually consume today { activity?.AddEvent(new ActivityEvent("exception", tags: new ActivityTagsCollection { ["exception.type"] = ex.GetType().FullName, ["exception.message"] = ex.Message, ["exception.stacktrace"] = ex.ToString() })); } if (EmitLogs) // opt-in — the spec's forward direction; pass the exception instance while `activity` is still Activity.Current so the SDK derives trace_id/span_id and exception.type/message/stacktrace from it { logger.LogError(ex, "Item processing failed"); } throw; } ``` **Rules**: - Don't set `exception.escaped` on the span event — it's deprecated outright: "no longer recommended to record exceptions that are handled and do not escape the scope of a span." - Support `OTEL_SEMCONV_EXCEPTION_SIGNAL_OPT_IN` (`logs` / `logs/dup`) so consumers who are ready can opt into logs or dual-emission, but default to spans-only — this is the spec's own transition guidance, not an optional nicety, and it exists precisely so nobody has to choose one signal over the other before they're ready. - When emitting logs (opt-in or dual), pick severity per `exceptions-logs`: `ERROR` for unhandled exceptions (especially on `SERVER`/`CONSUMER` spans), `WARN` for exceptions expected to be handled by the caller (especially on `CLIENT`/`PRODUCER` spans), `DEBUG` for exceptions that don't indicate an actual issue (e.g., a request cancelled client-side), `FATAL` only when the exception causes application shutdown. - The log call **MUST** happen while the span it's associated with is still current — the spec requires "exception events emitted by instrumentations that also record spans for the same operation MUST be associated with the corresponding span context." Logging from a detached error-reporting callback after the `using` activity scope has ended silently correlates to the wrong span, or none. See [Accessing Activities](#accessing-activities) below. - Only move to `logs`-only once you've held dual-emission for at least six months on a stable major version (the spec's own minimum) **and** confirmed your actual consumers read log-based exceptions — not on a fixed timeline alone. See [traces-and-propagation-reference](traces-and-propagation-reference.md) for the full pattern, including how a library with no logging dependency in its core can still expose a callback so a separate OTel integration package can capture exception details while the right span is active. ### Accessing Activities ```csharp var current = Activity.Current; // ❌ may be a user-created ambient span using var ownedActivity = ActivitySource.StartActivity("MyOperation"); // ✅ captured reference ownedActivity?.SetTag("myapp.key", value); ``` **Rules**: Do not rely on `Activity.Current` for spans you own; user code can replace it via `AsyncLocal`. Pass/store captured `Activity` only while alive. Store `ActivityContext` for propagation identity. ### Span Links Links connect a span to other spans that are causally related but not in a direct parent-child relationship — batch processing, scatter/gather, trace boundary crossings. ```csharp var links = new List { new(activityContext1), new(activityContext2), }; var activity = ActivitySource.StartActivity( ActivityKind.Internal, name: "batch-process", links: links); ``` See [traces-and-propagation-reference.md](traces-and-propagation-reference.md) for full link patterns including batch processing, scatter/gather, and trace boundary crossing. ### Context Propagation Distributed tracing requires propagating trace context across process boundaries (HTTP calls, message queues, etc.) using W3C `traceparent` headers. In .NET, this is handled by `DistributedContextPropagator`. The OTel SDK configures W3C TraceContext propagation by default. See [traces-and-propagation-reference.md](traces-and-propagation-reference.md) for propagation patterns, custom propagators, and manual inject/extract for non-standard transports. ## Metrics ### Meter and Metrics Class Setup ```csharp public sealed class OrderProcessingMetrics : IDisposable { private readonly Meter meter = new("MyApp.OrderProcessing", "1.0.0"); private readonly Histogram processingDuration = meter.CreateHistogram("myapp.order.processing.duration", unit: "s"); private readonly Counter itemsProcessed = meter.CreateCounter("myapp.order.processing.count", unit: "{order}"); public void Dispose() => meter.Dispose(); } ``` **Naming Conventions** (follow [OTel semantic conventions](https://opentelemetry.io/docs/specs/semconv/general/metrics/)): - Singular names, nested hierarchy: `myapp.order.processing.duration` - Define units (s, ms, {item}, {connection}); avoid technical suffixes (`_counter`, `_histogram`) - Start with pre-1.0.0 version until adoption proven ### Instrument Type Overview .NET provides 7 metric instrument types. Choose the right one for your measurement: | Instrument | .NET API | Behavior | Typical Use | |------------|---------|----------|-------------| | **Counter** | `CreateCounter` | Monotonically increasing | Request counts, error counts | | **UpDownCounter** | `CreateUpDownCounter` | Increases or decreases | Queue size, active connections | | **Histogram** | `CreateHistogram` | Distribution of values | Du