Prometheus not Scraping Custom Metrics (prometheus_demo_request_total) from ASP.NET Core Application with OpenTelemetry

Question

I have implemented a custom middleware in my ASP.NET Core application to log HTTP request metrics using Prometheus. I implemented the middleware using the link. However, despite the middleware being triggered for each HTTP request, these custom metrics are not appearing when accessing the /metrics endpoint.

I have confirmed that the counter.Labels(...).Inc() calls are executed during the middleware's invocation, but the expected metrics are missing in the Prometheus metrics output. Other default metrics from OpenTelemetry are present.

I seek assistance in identifying the root cause of this issue and ensuring that the custom metrics are correctly scraped and displayed by Prometheus.

My Startup.cs class looks like this:

public void ConfigureServices(IServiceCollection services)
{
    ...

    services.AddOpenTelemetry()
                .WithMetrics(builder => builder
                    .AddRuntimeInstrumentation()
                    .AddAspNetCoreInstrumentation()
                    .AddHttpClientInstrumentation()
                    .AddPrometheusExporter());
}

public void Configure(IApplicationBuilder app, IWebHostEnvironment env, CustomDbContext dbContext, ILogger<RedactableDbContext> logger)
{
    app.UseRequestMiddleware();

    ...
}

The following are the libraries that I am using for OpenTelemetry and Prometheus:

<PackageReference Include="OpenTelemetry.Exporter.Console" Version="1.6.0" />
<PackageReference Include="OpenTelemetry.Exporter.OpenTelemetryProtocol" Version="1.6.0" />
<PackageReference Include="OpenTelemetry.Exporter.Prometheus.AspNetCore" Version="1.6.0-rc.1" />
<PackageReference Include="OpenTelemetry.Extensions.Hosting" Version="1.6.0" />
<PackageReference Include="OpenTelemetry.Instrumentation.AspNetCore" Version="1.5.1-beta.1" />
<PackageReference Include="OpenTelemetry.Instrumentation.Http" Version="1.5.1-beta.1" />
<PackageReference Include="OpenTelemetry.Instrumentation.RunTime" Version="1.5.1" />
<PackageReference Include="prometheus-net.AspNetCore" Version="8.2.0" />

Not familiar with this specific implementation, but it's common for custom metrics to be "registered" in metrics registry, to be later exposed on metrics page. Could it be that you missed that step? — markalex
– markalex, Commented Dec 28, 2023 at 23:03

Matt Hensley · Accepted Answer · 2023-12-30 01:40:19Z

3

Custom metrics need to be registered with the OpenTelemetry SDK.

Modify your existing setup code to include a call to AddMeter:

services.AddOpenTelemetry()
    .WithMetrics(builder => builder
        .AddRuntimeInstrumentation()
        .AddAspNetCoreInstrumentation()
        .AddHttpClientInstrumentation()
        .AddPrometheusExporter());
        .AddMeter("YourMeterName"));

Without this, the Prometheus exporter will not be aware of your custom metric.

answered Dec 30, 2023 at 1:40

Matt Hensley

4994 silver badges8 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

Prometheus not Scraping Custom Metrics (prometheus_demo_request_total) from ASP.NET Core Application with OpenTelemetry

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related