Diagnostics and observability in production

Counters, traces and dumps answer different questions. Knowing which tool fits a symptom is most of production debugging.

The diagnostic toolset

Tool	Answers	Typical symptom
`dotnet-counters`	Live rates and totals for a running process	CPU high, GC thrashing, thread pool starvation
`dotnet-trace`	Where time goes, as a sampled profile	Slow requests, hot method, lock contention
`dotnet-dump`	What is in memory right now	Memory growth, hung process
`dotnet-gcdump`	Managed heap by type, without a full dump	Steady memory growth
`dotnet-stack`	Managed stacks of every thread	Deadlock, blocked request handling
EventPipe counters via OpenTelemetry	The same signals exported to your platform	Everything, at scale

# Install the global tools once
dotnet tool install --global dotnet-counters
dotnet tool install --global dotnet-trace
dotnet tool install --global dotnet-dump

# Live counters: GC, thread pool and request rates
dotnet-counters monitor --process-id 4242 \
  --counters System.Runtime,Microsoft.AspNetCore.Hosting,Microsoft.AspNetCore.Server.Kestrel

# A CPU profile for 30 seconds, then analyse the speedscope file
dotnet-trace collect --process-id 4242 --profile cpu-sampling --duration 00:00:30

# Heap by type, with a diff against a later capture
dotnet-gcdump collect --process-id 4242 -o before.gcdump
dotnet-gcdump report before.gcdump | head -40

# Every thread's managed stack, when a process stops responding
dotnet-stack report --process-id 4242

Reading the signals

using System.Diagnostics;
using System.Diagnostics.Metrics;
using OpenTelemetry.Metrics;
using OpenTelemetry.Trace;

public sealed class Orders
{
    // A counter for events worth counting
    private static readonly Counter<long> Placed =
        Diagnostics.Meter.CreateCounter<long>("orders.placed", "orders");

    // A histogram for a distribution, which is what latency actually is
    private static readonly Histogram<double> Value =
        Diagnostics.Meter.CreateHistogram<double>("orders.value", "GBP");

    public void Record(decimal amount, string currency)
    {
        Placed.Add(1, new KeyValuePair<string, object?>("currency", currency));
        Value.Record((double)amount,
            new KeyValuePair<string, object?>("currency", currency));
    }
}

public static class Diagnostics
{
    public const string ServiceName = "orders-api";
    public static readonly ActivitySource Source = new(ServiceName);
    public static readonly Meter Meter = new(ServiceName, "2.3.1");
}

var builder = WebApplication.CreateBuilder(args);

builder.Services.AddOpenTelemetry()
    .ConfigureResource(r => r.AddService(Diagnostics.ServiceName))
    .WithTracing(t => t
        .AddAspNetCoreInstrumentation()
        .AddHttpClientInstrumentation()
        .AddEntityFrameworkCoreInstrumentation()
        .AddOtlpExporter())
    .WithMetrics(m => m
        .AddAspNetCoreInstrumentation()
        .AddHttpClientInstrumentation()
        .AddRuntimeInstrumentation()          // GC, thread pool, exceptions
        .AddMeter(Diagnostics.ServiceName)
        .AddOtlpExporter());

Thread pool starvation shows as a low thread pool completion rate with a rising queue length, and it is almost always caused by blocking on async calls.
A high Gen0 rate with a stable Gen2 and low CPU is allocation churn: find the allocation rather than tuning the GC.
A gcdump diff between two points in time identifies the type that is growing, which is a much better starting point than reading a full dump.
A sampled trace tells you where threads were, not what was slow in wall-clock terms. Combine it with a latency histogram to confirm the impact.
Correlate across services with the trace identifier in every log line, so a log search and a trace search converge on the same request.
Do not sample away the errors: keep error traces at 100 percent and sample the successes.

💡

Instrument for the questions you will ask at 3am: what is the request rate, what is the error rate, what is the latency distribution, and which dependency is slow. A service without those four signals cannot be operated, only restarted.

FAQ

Can I attach these tools to production?

The dotnet-* tools use EventPipe and are low overhead when the profile is light, but the process must be running with the diagnostics socket available. In containers, run the tool in the same container or share the socket.

Why do traces show every request but the logs show nothing?

Because the trace context is not being written into the log scope. Add the TraceId and SpanId to the log scope so the two signals join up.

Configuration, options and logging Performance: allocation, Span and benchmarking

Last refreshed 2026-09-18.

Diagnostics and observability in production

The diagnostic toolset

Reading the signals

FAQ

Related