.NET Event Counters

.NET event counters are a relatively new API for collecting metrics from .NET applications. They are part of the EventSource and EventCounter namespace since .NET (Core) 3.0.

They are extremely important for understanding how an application really works in a production environment and are the key functionality for building dashboards.

Many frameworks and libraries already offer ready-made EventCounters, but these are not usually activated by default.

Metrics

Usually, an application has two types of metrics: Counters and measured values. Counters are incrementable values that count, for example, the number of requests, the number of emails sent or simply errors. Measured values, on the other hand, are values that change over time, such as the number of active users or the available memory of a disk. The .NET counters are called "rate" counters and "snapshot" counters.

Both categories have two different ways of collecting data:

• Polling counters, which provide a snapshot of the metric. The data is usually queried at a fixed interval from a third-party source (e.g. the storage API).
• Non-polling counters that collect, manage and provide the data themselves within the counter class.

Own Counter

A counter is always part of an event source, and an event source is ultimately the "domain" of a source. For example, if I want to measure the number of e-mails sent, then I create an event source EMailCounterSource and not EMailSentCouterSource. Within the EMailCounterSource there are then counters such as Sent, SentFailed, SentCancelled etc...

Custom EMailCounterSource

A very simple, non-optimized implementation of the EMailCounterSource, which nevertheless shows the important components of such a CounterSource, can look as follows:

[EventSource(Name = "MyApp.EMail")]
public sealed class EMailCounterSource : EventSource
{
    private EMailCounterSource() { }

    public static readonly EMailCounterSource Log = new();

    [Event(1)]
    public void MailSent() => WriteEvent(1);

    [Event(2)]
    public void MailSentFailed() => WriteEvent(2);
}

Here we see the class EMailCounterSource, which contains two events. These two events have unqique IDs, which are later used to identify the event itself. The methods MailSent and MailSentFailed are the methods that trigger the event or are called from outside to register the event. Outside is the keyword: it must be ensured that the EventSource is unique in the Runtime; therefore the constructor is private and the instance is made available as a static field. The name Log its a bit weird but is actually common here.

So if an e-mail is sent, the implementation of sending can look like this:

// EMailProvider.cs

public async Task<..> SendMail(...)
{
    try
    {
        await _client.SendMail(...);
            
        // send mail
        EMailCounterSource.Log.MailSent();
    }
    catch
    {
        EMailCounterSource.Log.MailSentFailed();
    }
}

Counters

The EMailCounterSource class now has the events, but the counters are still missing.

Both events MailSent and MailSentFailed fire the event (WriteEvent), but in order to feed the counters with data, we need two fields that represent the values that we want to make available via the counter.

This

[Event(1)]
public void MailSent() => WriteEvent(1);

[Event(2)]
public void MailSentFailed() => WriteEvent(2);

becomes this

// Sent
private long _totalSent;

[MethodImpl(MethodImplOptions.NoInlining)]
[Event(1, Level = EventLevel.Informational)]
private void Sent() => WriteEvent(1);

[NonEvent]
public void MailSent()
{
    Interlocked.Increment(ref _totalSent);
    if (IsEnabled()) { Sent(); }
}

// Sent Failed
private long _totalSentFailed;

[MethodImpl(MethodImplOptions.NoInlining)]
[Event(2, Level = EventLevel.Informational)]
private void SentFailed() => WriteEvent(2);

[NonEvent]
public void MailSentFailed()
{
    Interlocked.Increment(ref _totalSentFailed);
    if (IsEnabled()) { SentFailed(); }
}

As can be seen, the actual event methods are no longer an event due to the NonEvent attribute, but are used purely to count the values. However, there are two new, internal methods Sent and SentFailed, which trigger the actual events. These methods have the attribute Event and are therefore our actual event methods - but are now only called if the EventSource is also activated.

We now have our internal counters, which we want to make available to the counters.

private PollingCounter? _totalSentCounter;
private PollingCounter? _totalSentFailedCounter;
protected override void OnEventCommand(EventCommandEventArgs command)
{
    if (command.Command == EventCommand.Enable)
    {
        _totalSentCounter ??= new PollingCounter("sent",
            this, () => Interlocked.Exchange(ref _totalSent, 0))
        {
            DisplayName = "Total Sent",
            DisplayUnits = "Mails"
        };

        _totalSentFailedCounter ??= new PollingCounter("sent-failed",
        this, () => Interlocked.Exchange(ref _totalSentFailed, 0))
        {
            DisplayName = "Total Failed",
            DisplayUnits = "Mails"
        };
    }
}

protected override void Dispose(bool disposing)
{
    _totalSentCounter?.Dispose();
    _totalSentCounter = null;

    _totalSentFailedCounter?.Dispose();
    _totalSentFailedCounter = null;

    base.Dispose(disposing);
}

In our case, both counters are simple PollingCounter. They are created by calling OnEventCommand. When they are created, the name of the counter is declared as well as a delegate, which is executed when the counter is called. The call is controlled externally; the more often the counter is queried, the more often the delegate is executed.

The time span of the call is ultimately the resolution in which the values are later displayed in a diagram; in the case of Azure Application Insights, the standard resolution is 1 minute - the counters are therefore queried every 1 minute. The names and units are important for the later display in a diagram; the name is also the name that is displayed in Azure Application Insights (e.g. then MyApp.EMail|sent or MyApp.EMail|sent-failed).

As a best practice, each separate EventSource class should indicate the counters it contains:

public static IEnumerable<string> GetCounters()
{
    yield return "sent";
    yield return "sent-failed";
}

Full Sample

The full, optimized EventSource can look like this:

using System.Diagnostics.Tracing;
using System.Runtime.CompilerServices;

namespace BenjaminAbt.Samples.Diagnostics;

[EventSource(Name = EventSourceName)]
public sealed class EMailCounterSource : EventSource
{
    private EMailCounterSource() { }

    public static readonly EMailCounterSource Log = new();

    public const string EventSourceName = "MyApp.EMail";

    // Sent
    private const int SentId = 1;
    private long _totalSent;

    [MethodImpl(MethodImplOptions.NoInlining)]
    [Event(SentId, Level = EventLevel.Informational)]
    private void Sent() => WriteEvent(SentId);

    [NonEvent]
    public void LogSent()
    {
        Interlocked.Increment(ref _totalSent);
        if (IsEnabled()) { Sent(); }
    }

    // Sent Failed
    private const int SentFailedId = 2;
    private long _totalSentFailed;

    [MethodImpl(MethodImplOptions.NoInlining)]
    [Event(SentFailedId, Level = EventLevel.Informational)]
    private void SentFailed() => WriteEvent(SentFailedId);

    [NonEvent]
    public void LogSentFailed()
    {
        Interlocked.Increment(ref _totalSentFailed);
        if (IsEnabled()) { SentFailed(); }
    }

    // Counters
    private const string TotalSentCounterName = "sent";
    private PollingCounter? _totalSentCounter;

    private const string TotalSentFailedCounterName = "sent-failed";
    private PollingCounter? _totalSentFailedCounter;

    protected override void OnEventCommand(EventCommandEventArgs command)
    {
        if (command.Command == EventCommand.Enable)
        {
            _totalSentCounter ??= new PollingCounter("sent",
                this, () => Interlocked.Exchange(ref _totalSent, 0))
            {
                DisplayName = "Total Sent",
                DisplayUnits = "Mails"
            };

            _totalSentFailedCounter ??= new PollingCounter("sent-failed",
                this, () => Interlocked.Exchange(ref _totalSentFailed, 0))
            {
                DisplayName = "Total Failed",
                DisplayUnits = "Mails"
            };
        }
    }

    public static IEnumerable<string> GetCounters()
    {
        yield return TotalSentCounterName;
        yield return TotalSentFailedCounterName;
    }

    protected override void Dispose(bool disposing)
    {
        _totalSentCounter?.Dispose();
        _totalSentCounter = null;

        _totalSentFailedCounter?.Dispose();
        _totalSentFailedCounter = null;

        base.Dispose(disposing);
    }
}

Built-In Counters

As mentioned, many frameworks and libraries come with counters. There are a large number of counters that are inherently part of .NET, ASP.NET Core or Entity Framework. Examples from the System.Runtime namespaces include counters for CPU utilization, memory utilization or the number of loaded assemblies. But more detailed information such as GC Heap Size, Allocation Rate or Gen 0/1/2 Collections etc etc are also available.

These can usually be found in the documentation; in the case of Entity Framework I use this:

public static class EFCoreEventCounters
{
    public static void Add(EventCounterCollectionModule module)
    {
        public const string EventSourceName = "Microsoft.EntityFrameworkCore";

        public static IEnumerable<string> GetCounters()
        {
            yield return "active-db-contexts";
            yield return "total-queries";
            yield return "queries-per-second";
            yield return "total-save-changes";
            yield return "save-changes-per-second";
            yield return "compiled-query-cache-hit-rate";
            yield return "total-execution-strategy-operation-failures";
            yield return "execution-strategy-operation-failures-per-second";
            yield return "total-optimistic-concurrency-failures";
            yield return "optimistic-concurrency-failures-per-second";
        }
    }
}

Consume Event Counters

There are various ways in which event counters can be consumed; one of these ways is dotnet-trace, which attaches itself to the process and reads the event counters.
I usually use the NuGet tool dotnet counters:

dotnet counters monitor Microsoft.EntityFrameworkCore -p <PID>
dotnet counters monitor MyApp.EMail -p <PID>

Consume Event Counters with Application Insights

The Application Insights SDK comes with an EventCounterCollectionModule where you can register the EventCounters so that they are sent to Application Insights.

public static class MyMonitoringRegistration
{
    public static IServiceCollection AddMyMonitoring(this IServiceCollection services)
    {
        // add app insights
        services.AddApplicationInsightsTelemetry();

        // configure telemetry
        services.ConfigureTelemetryModule<EventCounterCollectionModule>((module, _) =>
        {
            // add ef core event counters
            InternalAddEventCounters(module, EFCoreEventCounters.EventSourceName, EFCoreEventCounters.GetCounters());

            // add my custom email event counters
            InternalAddEventCounters(module, EMailCounterSource.EventSourceName, EMailCounterSource.GetCounters());
        });

        return services;
    }

    private static void InternalAddEventCounters(EventCounterCollectionModule module, 
        string sourceName, IEnumerable<string> counterNames)
    {
        foreach (string eventCounterName in counterNames)
        {
            module.Counters.Add(new EventCounterCollectionRequest(sourceName, eventCounterName));
        }
    }
}

---

Have fun collecting the metrics!