appmetrics / health Goto Github PK

Hello,

Our internal API guidelines require using the following strings to represent application health status:

• green
• orange
• red

What would be the best way (if any) to use these health statuses instead the ones defined in the library (Healthy, Degraded and Unhealthy)?

Thank you.

Fix could be to only report new unhealthy checks or when they change status

Hi. I use version 2.0.0 of App.Metrics.Health.Checks.Sql. When I pass to the AddSqlCheck method NpgsqlConnection from the Npgsql.EntityFrameworkCore.PostgreSQL version 2.1.2 package, I get an error in runtime:

"unhealthy": {
"PostgreSQL": "EXCEPTION: InvalidCastException - Unable to cast object of type 'System.Int32' to type 'System.Int64'.\r\n    InvalidCastException: Unable to cast object of type 'System.Int32' to type 'System.Int64'.\r\n    at App.Metrics.Health.Checks.Sql.SqlHealthCheckBuilderExtensions.<>c__DisplayClass0_0.<AddSqlCheck>b__0(CancellationToken cancellationToken) in C:\\projects\\health\\src\\App.Metrics.Health.Checks.Sql\\SqlHealthCheckBuilderExtensions.cs:line 48\r\n"
}

This is most likely due to the fact that ExecuteScalar in NpgsqlCommand returns an int instead of a long.

Initially the slack health reporter continuously alerted on every report run which is by default every 10secs, way to noisy. This was changed to only report on status changes.

Ideally an alert would be sent out whilst unhealthy but via a configured interval which would be longer than each report run.

Add a flag to the slack alert configuration to enable/disable alerting health

How the 2.1.0 version of the health check inherits from the project and outputs it to influxdb？

This is the error in health endpoint

{
healthy: {
CPU Health Check: "OK",
Drive Health Check: "OK"
},
unhealthy: {
DB Connection: "EXCEPTION: InvalidCastException - Unable to cast object of type 'System.Int32' to type 'System.Int64'. InvalidCastException: Unable to cast object of type 'System.Int32' to type 'System.Int64'. at App.Metrics.Health.Checks.Sql.SqlHealthCheckBuilderExtensions.<>c__DisplayClass0_0.<AddSqlCheck>b__0(CancellationToken cancellationToken) in C:\projects\health\src\App.Metrics.Health.Checks.Sql\SqlHealthCheckBuilderExtensions.cs:line 48 ",
Private Memory Size: "FAILED. 87666688 > 1000 bytes",
Virtual Memory Size: "FAILED. 2218767613952 > 1000 bytes",
Working Set: "FAILED. 81408000 > 1000"
},
status: "Unhealthy"
}

and the configure for health with deaults :

public static IWebHostBuilder CreateWebHostBuilder(string[] args)
        {
            return WebHost.CreateDefaultBuilder(args)
                .ConfigureHealthWithDefaults(
                builder =>
                {
                    const int threshold = 1000;
                    // Check that the current amount of private memory in bytes is below a threshold
                    builder.HealthChecks.AddProcessPrivateMemorySizeCheck("Private Memory Size", threshold);
                    // Check that the current amount of virtual memory in bytes is below a threshold
                    builder.HealthChecks.AddProcessVirtualMemorySizeCheck("Virtual Memory Size", threshold);
                    // Check that the current amount of physical memory in bytes is below a threshold
                    builder.HealthChecks.AddProcessPhysicalMemoryCheck("Working Set", threshold);

                    // Check db status
                    builder.HealthChecks.AddSqlCheck("DB Connection", () => new SqlConnection("myconnectionstring"), TimeSpan.FromSeconds(10));
                })
                .UseHealth()
                .UseHealthEndpoints()
                .UseContentRoot(Directory.GetCurrentDirectory())
                .UseConfiguration(Configuration)
                .UseSerilog()
                .UseIISIntegration()
                .UseKestrel()
                .UseStartup<Startup>();
        }

Background
For many services the dependencies (sql server, third party web services etc) will frequently be implicitly tested in the normal course of operation. This means that extra explicit checks to those dependencies are redundant.
Proposal
This feature would provide a way for the service code to "manually" update the cached health check result. This result would remain valid for some cache expiry time and would be served out to requests for health checks. So under normal operating conditions we would never have to run the explicit health check. The explicit health check would remain of course as a fall back for the situation where the load on the server is too light to provide a sufficiently frequently updated check.
Implemention
We are happy to implement something and provide a pull request. The design of the HealthCheck class is not extensible but we could implement this feature modifying HealthCheck. By exposing a IManualResultUpdate interface on HealthCheck we would provide access to update the cached result.

It could be helpful to use some sort of SQL Checker as in preview of standard asp.net health-check - see details here: https://github.com/dotnet-architecture/HealthChecks

Some failing health checks aren't critical enough to wake up those on on call, but failures are important to alert on during reasonable times.

An example could be the number of messages in a dead letter queue, where it may not be important to alert on a threshold at 2am but would be something that should be investigated and fixed during work hours.

Allow a retry count and delay between retries to be specified on HTTP health checks.

• Test projects package updates
• Benchmark projects package updates
• Library projects package updates

SqlHealthCheckBuilderExtensions should check that a sql connection is not already open before trying to open it. If you attempt you open an already open DbConnection, it will throw an invalid operation exception.

Implement a health check reporter which posts an incoming webhook to slack notifing of unhealthy and degraded health checks

Support https://github.com/ctaggart/SourceLink which enables the source code repo to be the source server. Requires portable PDBs so will have to sacrifice OpenCover support for now.

Health status is currently reported every 10 secs and is not configurable. Allow the interval to be configured per registered reporter.

I want to report the overall health status to Prometheus. I've understood that in AppMetrics 2.0 it is not done by default anymore. So how can I access that current HealthStatus from an ASP.NET Core app ?
Should I have to call the DefaultHealthCheckRunner ? I would have prefered not to run the health checks myself.

Reporting health check results as metrics using App Metrics then automatically provides to ability to report health check results to any of the App Metrics supported TSDB reporters

I am using health checks as part of a Windows Service (using Topshelf). Is there a best practive way of scheduling health checks on a regular basis, e.g. every 5 minutes. It is of course possible to use Quartz or Rx to do something like this but I am just curious to know if there is some build-in way of doing this. I looked in the documentation but did not find anything. Thanks.

Allow scanning and auto registering of health check implementations in .net full framework

Repro steps:

1. In appsettings.json specify HealthEndpointOptions with PingEndpointEnabled=false parameter:

"HealthEndpointsOptions": { "HealthEndpointEnabled": true, "PingEndpointEnabled": false, "Timeout": "0:0:10" }

2. call in Configure method of Startup class

app.UseHealthAllEndpoints();

3. call in ConfigureServices method of Startup class:

services.AddHealth();
services.AddHealthEndpoints();

Result: [applicationpath]/ping returns 'pong'
Expected result: [applicationpath]/ping returns 404

Tests fail when attempting to get the startup assembly for health check scanning due to microsoft/vstest#649

Rather than return stack traces on HTTP failures using the HTTP Health Check extension, log the exception details but return an friendly error message as the health check status message.

I am implementing an IReportHealthStatus health reporter, with the latest health packages 2.1.0-alpha -0118 and when setting up the reporter :

healthReportingBuilder.Using(new HealthNLogReporter());

using the IHealthReporting Builder created by:

AppMetricsHealth.CreateDefaultBuilder()

I get the error:

System.TypeLoadException: Method 'CanReport' in type 'App.Metrics.Health.Builder.HealthBuilder' from assembly 'App.Metrics.Health.Core, Version=2.1.0.0, Culture=neutral, PublicKeyToken=0d5193a913d1b812' does not have an implementation.
at App.Metrics.Health.AppMetricsHealth.CreateDefaultBuilder()

I assume I must be somehow using the wrong health builder?

Hi. I use version 2.1.0--alpha-0071. A added services.AddHealthReportingHostedService(). What do i need else to send health status to prometheus. Or version with this feature does not yet released?

1. For now all the health checks discovered during startup are run sequentially. This could cause a problem, when there are many 'heavy' checks configured. E.g. database connection, service availability checks. These two examples are blocking I/O operations which are perfectly parallelilizable.

2. Also, it would be really nice to have a setting for health check mechanism, which allows to skip all the checks, if at least one of them returned unhealthy.

The Grafana HTTP API supports annotations/events as do many other TSDBs. http://docs.grafana.org/http_api/annotations/

Providing an event subscription on health check failures would be useful to write such annotations and also provide the ability for webhooks for example

In earlier versions of App Metrics health and metrics where included in the same package whereby health check results where captured as metrics allowing results to be reported to the underlying TSDB.

In decoupling health and metrics this ability was removed, since there has been several requests to allow for this.

Allow specific health checks to cache their results for a configured period of time

App.Metrics.AspNetCore.Health 2.0.0