Giter VIP home page Giter VIP logo

serilog-sinks-loki's Introduction

Serilog.Sinks.Loki

.NET Core

This is a Serilog Sink for Grafana's new Loki Log Aggregator.

What is Loki?

Loki is a horizontally-scalable, highly-available, multi-tenant log aggregation system inspired by Prometheus. It is designed to be very cost effective and easy to operate, as it does not index the contents of the logs, but rather a set of labels for each log stream.

You can find more information about what Loki is over on Grafana's website here.

Loki Screenshot

Current Features:

  • Formats and batches log entries to Loki via HTTP
  • Ability to provide global Loki log labels
  • Comes baked with an HTTP client, but your own can be provided
  • Provides contextual log labels

Coming soon:

  • Write logs to disk in the correct format to send via Promtail
  • Send logs to Loki via HTTP using Snappy compression

Installation

The Serilog.Sinks.Loki NuGet package can be found here. Alternatively you can install it via one of the following commands below:

NuGet command:

Install-Package Serilog.Sinks.Loki

.NET Core CLI:

dotnet add package Serilog.Sinks.Loki

Basic Example:

// var credentials = new BasicAuthCredentials("http://localhost:3100", "<username>", "<password>");
var credentials = new NoAuthCredentials("http://localhost:3100"); // Address to local or remote Loki server

Log.Logger = new LoggerConfiguration()
        .MinimumLevel.Information()
        .Enrich.FromLogContext()
        .WriteTo.LokiHttp(credentials)
        .CreateLogger();

var exception = new {Message = ex.Message, StackTrace = ex.StackTrace};
Log.Error(exception);

var position = new { Latitude = 25, Longitude = 134 };
var elapsedMs = 34;
Log.Information("Message processed {@Position} in {Elapsed:000} ms.", position, elapsedMs);

Log.CloseAndFlush();

Adding global labels

Loki indexes and groups log streams using labels, in Serilog.Sinks.Loki you can attach labels to all log entries by passing an implementation ILogLabelProvider to the WriteTo.LokiHttp(..) configuration method. This is ideal for labels such as instance IDs, environments and application names.

See Serilog.Sinks.Loki.Example/LogLabelProvider.cs for a basic LogLabelProvider implementation.

var credentials = new BasicAuthCredentials("http://localhost:3100", "<username>", "<password>");
var log = new LoggerConfiguration()
        .MinimumLevel.Verbose()
        .Enrich.FromLogContext()
        .WriteTo.LokiHttp(credentials, new LogLabelProvider())
        .CreateLogger();

Local, contextual labels

In some occasions you'll want to add context to your log stream within a particular class or method, this can be achieved using contextual labels:

using (LogContext.PushProperty("A", 1))
{
    log.Warning("Warning with Property A");
    log.Fatal("Fatal with Property A");
}

Custom HTTP Client

Serilog.Loki.Sink is built on top of the popular Serilog.Sinks.Http library to post log entries to Loki. With this in mind you can you can extend the default HttpClient (LokiHttpClient), or replace it entirely by implementing IHttpClient.

// ExampleHttpClient.cs

public class ExampleHttpClient : LokiHttpClient
{
    public override Task<HttpResponseMessage> PostAsync(string requestUri, HttpContent content)
    {
        return base.PostAsync(requestUri, content);
    }
}
// Usage

var credentials = new BasicAuthCredentials("http://localhost:3100", "<username>", "<password>");
var log = new LoggerConfiguration()
        .MinimumLevel.Verbose()
        .Enrich.FromLogContext()
        .WriteTo.LokiHttp(credentials, new LogLabelProvider(), new ExampleHttpClient())
        .CreateLogger();

Configure using appsettings.json

Serilog-Sinks-Loki can be configured with appsettings.json using Serilog.Settings.Configuration.
It support the following arguments serverUrl, username, password, credentials, labelProvider, httpClient, outputTemplate and formatProvider.
Not all fields can be used in combination look in LokiSinkExtensions.cs for the supported combinations.
credentials, labelProvider, httpClient, and formatProvider are classes and must be specified using the Namespace.ClassName, Assembly syntax.

"Serilog": {
  "Using": [ "Serilog.Sinks.Loki" ],
  "MinimumLevel": {
    "Default": "Verbose"
  },
  "Enrich": [ "FromLogContext" ],
  "WriteTo": [      
    {
      "Name": "LokiHttp",
      "Args": {
        "serverUrl": "https://loki:3000",
        "labelProvider": "Namespace.ClassName, Assembly"
      }
    }
  ]
}

Missing a feature or want to contribute?

This package is still in its infancy so if there's anything missing then please feel free to raise a feature request, either that or pull requests are most welcome!

serilog-sinks-loki's People

Contributors

anderssonpeter avatar aragas avatar falco20019 avatar jincod avatar josephwoodward avatar michaelosthege avatar senritsu avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar

serilog-sinks-loki's Issues

Duplicated Labels

Hi @josephwoodward
I recently updated the library and logs stopped coming to grafana.
Investigating I saw that my code for various reasons generates duplicate labels.

HTTP/1.1 400 Bad Request
Content-Type: text/plain; charset=utf-8
...

stream '{app="siu-api", app="siu-statics", username="anonymous", username="dvanhaaster", version="1.5.0"}' has duplicate label name: 'app'

In a previous version of the source code, I think, that this behavior was allowed and the effect is that the last added label prevailed.
The question: Is this behavior correct? Should I change my code?

Thank you

ETA on new feature

Hi. Do you have an ETA on the upcomming feature "Write logs to disk in the correct format to send via Promtail# ?
Sounds like a feature we need

LogLabelProvider appsettings

Hi,

i have following settings in appsettings.yml

Serilog:
  MinimumLevel:
    Default: Information
    Override:
        Microsoft: Warning
        Microsoft.Hosting.Lifetime: Information
        Microsoft.EntityFrameworkCore.Database.Command: Error
  Using:
    - Serilog.Sinks.Loki
  Enrich:
    - FromLogContext
  WriteTo:
    - Name: LokiHttp
      Args:
        serverUrl: http://loki_url:3100
        labelProvider: UserManager.Api.LogLabelProvider, UserManager.Api
    - Name: Console

and following LogLabelProvider

using System.Collections.Generic;
using Serilog.Sinks.Loki.Labels;

namespace UserManager.Api
{
    public class LogLabelProvider : ILogLabelProvider
    {
        public IList<LokiLabel> GetLabels()
        {
            return new List<LokiLabel>
            {
                new ("microservice", "user-manager")
            };
        }
    }
}

Everything works great except labels. LogLabelProvider stays ignored and no global labels are attached. Any suggestions?

Is version 4.0.0-beta3 stable?

Hi, I'm trying to upgrade from version 3.0.0 which on nuget.org shows as "Deprecated" due to various bugs detected, but I still don't see version 4 as productive, just as pre-release. I know I can use that version, however in production a directive does not allow publishing pre-releases.

Strongly Naming Assembly

Curently this package doesn't appear to have a strong name.

Using it in other projects that do have a strong name is causing issues.

Could not load file or assembly 'Serilog.Sinks.Loki, Version=2.2.0.0, Culture=neutral, PublicKeyToken=null' or one of its dependencies. A strongly-named assembly is required. (Exception from HRESULT: 0x80131044)

I'm happy to sign it myself, but perhaps it should be published with a Strong name?

latest non-beta nuget package v3.0.0 is deprecated?

The newest (non-beta) package version on nuget is marked as deprecated as it 'has critical bugs' ...would it be an idea to de-list this v3.0.0 package if it's no longer safe and force users to downgrade to a pre v3.0.0 version?

image

Or is it time for the v3.0.1 package to be bumped to a full release version?

Labels with invalid identifiers and multline text

Hello everyone and thanks for the great work!

I am facing a problem while using the sink (v2.1.0) with loki server (v1.5). When I log something the following exception happens:

2020-07-03T12:14:39.2871483Z Exception while emitting periodic batch from Serilog.Sinks.Http.Private.Sinks.HttpSink: Serilog.Debugging.LoggingFailedException: Received failed result BadRequest when posting events to http://localhost:3100/api/prom/push
   at Serilog.Sinks.Http.Private.Sinks.HttpSink.EmitBatchAsync(IEnumerable`1 logEvents) in C:\projects\serilog-sinks-http\src\Serilog.Sinks.Http\Sinks\Http\Private\Sinks\HttpSink.cs:line 94
   at Serilog.Sinks.PeriodicBatching.PeriodicBatchingSink.OnTick()

The loki server returns the following:

error parsing labels: parse error at line 1, col 34: literal not terminated

I use a library called fo-dicom, and when it logs something, it adds some properties that are problematic. For instance, one of the properties (which are turned into labes by Serilog-Sinks-Loki) contains a name that is an invalid identifier (fo-dicom which contains a -) and the other contains line breaks. When the following requests are sent, they return BadRequest (400):

{"streams":[{"labels":"{level=\"info\",app=\"demo\", label1=\"test \n multiline\", namespace=\"prod\",MyPropertyName=\"MyPropertyValue\",ThreadId=\"1\"}","entries":[{"ts":"2020-07-03T13:50:55.6289463-03:00","line":"Linha Teste\n"}]}]}

This one fails because label1 has a multiline text. Other issue I found is:

{"streams":[{"labels":"{level=\"info\",app=\"demo\", invalid-label=\"test \n multiline\", labelinvalid=\"teste\", namespace=\"prod\",MyPropertyName=\"MyPropertyValue\",ThreadId=\"1\"}","entries":[{"ts":"2020-07-03T13:50:55.6289463-03:00","line":"Linha Teste\n"}]}]}

Since "invalid-label" contains a -, loki server returns:
error parsing labels: parse error at line 1, col 32: syntax error: unexpected -, expecting = or != or =~ or !~

I believe it is necessary to remove any linebreaks from labels, and also make sure they property names are valid identifiers. I've done a workaround to make it work, but it would be nice if the sink itself handled these cases.

Hangs when unable to connect to Loki

I only recently encountered this, and it requires a specific set of circumstances it appears.
For quite a few days now my Loki server has been broken, and during all this time the Loki sink was fine, it didn't seem to have any noticeable performance impact and my program would shut down properly.

However yesterday I brought my Loki server up. Then took it down about 5 minutes later. Of course, the Loki sink worked great when the Loki server was up, but it's what happened that when it went down is interesting:
My program still ran without any noticeable performance hit, but this time required a force close to exit.

As far as I can tell this change of behaviour is because previously, the Loki server was down, but it also did not have a DNS record. After bringing it up briefly and then back down, the server had a DNS entry, but the sink was unable to contact it.
This seems to occur at CloseAndFlush().

Versions:

  • Windows 10 1909
  • dotnet core v3.1.7
  • sink v2.2.0
  • serilog v2.9.0

If there's any other info I can provide let me know.

Configuration via appsettings.json

Howdy,

Firstly, I'm using the 4.0.0-beta3 release as I understand the 3.x releases have a critical bug.

I have tried to follow the readme to configure the sink via appsettings.json, however I receive the following error:

"Unable to find a method called LokiHttp for supplied arguments: labels, serverUrl. Candidate methods are:
LokiHttp(configFactory)"

I attribute this to a pull request that removed the logging configuration extension overloads and replaced them with a single argument. However, I cannot figure out whether it's even possible to leverage this argument from appsettings.json as Serilog complains it cannot instantiate a Func.

I have worked around this issue by adding my own extension method, however, I would like to bring it to your attention as it breaks for anybody using the existing configuration file approach.

v4.0.0-beta1 not using new build

Hi @josephwoodward,

I just tried the new NuGet package, but sadly it still contains the DLLs for 3.0.1-beta1+27e063c30c3411b74b2f33788dcd4116e14db3f1. So there definitely went something wrong while releasing. It also does not contain any of the changes. Can you please re-release (or try the build chain from @michaelosthege)?

Convert EventId to labels

EventIds are extremely useful when searching through logs for specific events.

Currently they are being pasted as json and the label parser only interprets the first {

image

It would be great if EventId.Id and EventId.Name were parsed as individual labels so we can quickly search through logs using the label indexers.

cc @josephwoodward

Log entries can be sent out of order

I am using the code in the master branch.
I enabled full logging to Loki, at boot it generates a huge log post to LOKI, but two of the entries are in the incorrect order.

{
	"ts": "2021-01-26T15:53:52.4747391+01:00",
	"line": "Connection id \"\"0HM6218P723BS\"\" accepted.\n level=debug ConnectionId=\"0HM6218P723BS\" EventId={ Id: 39, Name: \"ConnectionAccepted\" } SourceContext=\"Microsoft.AspNetCore.Server.Kestrel\" MachineName=\"TEST\" ApplicationVersion=\"2.0.0.0\""
}, {
	"ts": "2021-01-26T15:53:52.4751428+01:00",
	"line": "Connection id \"\"0HM6218P723BR\"\" received FIN.\n level=debug ConnectionId=\"0HM6218P723BR\" EventId={ Id: 6, Name: \"ConnectionReadFin\" } SourceContext=\"Microsoft.AspNetCore.Server.Kestrel.Transport.Sockets\" MachineName=\"TEST\" ApplicationVersion=\"2.0.0.0\""
}, {
	"ts": "2021-01-26T15:53:52.4751372+01:00",
	"line": "Connection id \"\"0HM6218P723BS\"\" received FIN.\n level=debug ConnectionId=\"0HM6218P723BS\" EventId={ Id: 6, Name: \"ConnectionReadFin\" } SourceContext=\"Microsoft.AspNetCore.Server.Kestrel.Transport.Sockets\" MachineName=\"TEST\" ApplicationVersion=\"2.0.0.0\""
}, {
	"ts": "2021-01-26T15:53:52.4764007+01:00",
	"line": "Connection id \"\"0HM6218P723BR\"\" started.\n level=debug ConnectionId=\"0HM6218P723BR\" EventId={ Id: 1, Name: \"ConnectionStart\" } SourceContext=\"Microsoft.AspNetCore.Server.Kestrel\" MachineName=\"TEST\" ApplicationVersion=\"2.0.0.0\""
}

So the order is

  • 2021-01-26T15:53:52.4747391+01:00
  • 2021-01-26T15:53:52.4751428+01:00 <-- this should be moved down one step
  • 2021-01-26T15:53:52.4751372+01:00
  • 2021-01-26T15:53:52.4764007+01:00

Logs are been sent as a batch once in 10 minutes

I couldn't find the way to set the time interval definition.
In grafana I can see the logs arriving each 10 minutes. Their 'parsed timestamp' is almost identical because of that,
is there any way to set to the time interval value?

MessageTemplate Property

Is there any way to get the message template sent along (may as a label?)? I would like to filter in loki based on all the entries with a specific Message Template

BAD REQUEST(400) - error parsing labels: parse error at line {0}, col {1}: invalid char escape

Hi Folks,

I added the Loki Serilog Sink on my Project, but I noticed(when I enable Visual Studio to Stop on all Exceptions), that I am getting a BadRequest response such as the one bellow:

(Note that my Loki is running on port 11187)....
REQUEST BODY:
`
POST http://localhost:11187/api/prom/push HTTP/1.1
Host: localhost:11187
Content-Type: application/json
Content-Length: 782

{"streams":[{"labels":"{level="info",app="demoapp",environment="production",CallerAssembly="Lalalal.Fab.Framework.WebApi.Service, Version=6.1.1.0, Culture=neutral, PublicKeyToken=null",CallerClassName="Lalalal.Fab.Framework.WebApi.Service.Extensions.XaLogExtensions",CallerMemberName="InitializeXaLog",CallerFileName="C:\Lalalal\LalalalFab_Framework\WebApi.Service\Source\Extensions\XaLogExtensions.cs",CallerFileLineNumber="88",Application="ILoggerWebAPI",ApplicationVersion="{ Major: 1, Minor: 0, Patch: 0 }",CorrelationId="null",ProcessId="1308",ThreadId="1",MachineName="DEPC009734"}","entries":[{"ts":"2020-05-18T14:50:44.4335725+02:00","line":"Logging initialized with Threshold: 'Debug'. Log setttings were read by 'APP_SETTING_JSON'.\r\n"}]}]}
`

RESPONSE BODY:
`
HTTP/1.1 400 Bad Request
Content-Type: text/plain; charset=utf-8
X-Content-Type-Options: nosniff
Date: Mon, 18 May 2020 12:50:44 GMT
Content-Length: 74

error parsing labels: parse error at line 1, col 293: invalid char escape
`

Visual Studio $exception:
$exception.ToString() = Serilog.Debugging.LoggingFailedException: Received failed result BadRequest when posting events to http://localhost:11187/api/prom/push at Serilog.Sinks.Http.Private.Sinks.HttpSink.EmitBatchAsync(IEnumerable1 logEvents)
`

What I think is curious, is that the Log entry ends up landing properly on Loki.... So I am not completely sure what is going on... Does anyone else faced this issue? What am i missing?

BR
Pedro

Release 3.0.0 contains 2.1.1 DLLs

The latest release 3.0.0 sadly only contains the 2.1.1 DLLs. Please re-release the version with the correct DLLs packaged.

Thanks for including #22. I would have liked the approach from #15 more, since some users might want to include all properties dynamically set through scopes which use-case was broken now with this PR. But even this version still has problems with double-quotes being replaced unnecessarily as mentioned in #15 (review) (which breaks appended fields containing whitespaces). So for now, even 3.0.0 would not work for me sadly.

Should I create another issue ticket for this or will you give #15 another look?

Too many labels?

Hi Joseph,

First and foremost: Thank you for this package! It certainly saved me a lot of time.

I've added it as a package to an asp.net core application and it's successfully sending logs to a loki instance that we run. The downside is that there are a lot of labels being sent with each log line. This is due to the call to .Enrich.FromLogContext() that we add to the Serilog configuration. This context information is great to have but not always necessary as a stream in loki. Reading through the loki design documents I came across this little gem: https://grafana.com/blog/2020/04/21/how-labels-in-loki-can-make-log-queries-faster-and-easier/ which states that each label:value combination constitutes a stream and that having too many streams will kill performance.

Would you accept PRs that add a configuration option to allow the user to specify a list of log event properties (if they are present) that should be used as labels and all other properties from the context be appended to the log message. Or better yet, two lists of properties, one to be used as labels, one to be appended to the log message and all other properties from the log context be discarded?

Thanks ,
Bo

LogEntry timestamp

Hi Joseph,

When I run the included test program and check the logs with Grafana, I find that many log entries have exactly the same timestamp - down to the millisecond.

So my question: Why don't you use LogEvent.Timestamp instead of DateTime.Now?

Regards,
Mathias

Define custom Loki endpoint

@josephwoodward can we extend the LokiRoutes with an additional method that will take a custom URL. We have a case where we do not use the default Loki push endpoint. Like this:

    public static class LokiRouteBuilder
    {
        public static string BuildPostUri(string host)
        {
            return host.Substring(host.Length - 1) != "/" ? $"{host}{PostDataUri}" : $"{host.TrimEnd('/')}{PostDataUri}";
        }

        public static string BuildPostUri(string host, string uri)
        {
            return host.Substring(host.Length - 1) != "/" ? $"{host}{uri}" : $"{host.TrimEnd('/')}{uri}";
        }

        public const string PostDataUri = "/api/prom/push";
    }

Let me know, and I can create a pull request.

TypeLoadException

Hi,
I'm getting exceptions when attemting to do

var credentials = new NoAuthCredentials(lokiUrl);
var llp = new LogLabelProvider();
Console.WriteLine("Creating logger config");
Log.Logger = new LoggerConfiguration()
    .MinimumLevel.Verbose()
    .WriteTo.Console()
    .WriteTo.LokiHttp(credentials, llp)
    .CreateLogger();

grafik

Version numbers:

  • .NET Framework 4.6.2 (console application)
  • Serilog 2.10.0 (latest stable)
  • Serilog.Sinks.Http 7.2.0 (latest stable)
  • Serilog.Sinks.Loki 3.0.0 (latest stable)

I attempted to use the prerelease version 3.0.1, but then I get errors on the LogLabelProvider (copied from the readme) not implementing the interface correctly.

Looks like I have a bad combination of versions?

Ooops in Console Output

With each error, an "Ooops" from LokiBatchFormatter is written to the console.
Why?
Can you please remove this behaviour?

Replace Newtonsoft.Json with System.Text.Json

Hi. I'd like to know whether you would be open to a PR to change the usage of Newtonsoft.Json to System.Text.Json instead? It'd allow us to cross it out from our dependency list which would be pretty nice.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.