Not sure if this functionality is planned, but it would be really nice to have an ability to track transactions which are not originated from HTTP requests in Asp.Net Core application. For instance there can be a Asp.Net Core microservice which is part of event-driven system. Requests in this case are coming from Event Bus and cannot be monitored as Web transactions.

By looking at the code-base I'd say there are too many things that are public.

E.g. on ITransaction there is a public Spans array, and this is an example why this can be a problem in user code:

class MySpan : ISpan
{
  //...all the ISpan stuff
}
		
public static void MyMethod()
{
   Agent.Tracer.CurrentTransaction.Spans[0] = new MySpan { Id = -1 , Duration = -10};
  //leave everything else empty, and send id -1 and everything else as empty values...
}

The server won't like this...And there are many other examples.

I'd say generally for the first release we should make sure we only make things public that 100% should be public. Adding things later won't break, so if there is something missing we can add it later easily. On the other hand removing things would break.

Therefore: I'll go on and make sure that we only have things in public that will very-very likely remain public forever, and make things that are questionable internal, just to be on the safe side.

This is more pressuring for writing tests.

Currently code like this does not unsubscribe the agent from receiving events and creating spans:

using (agent.Subscribe(new HttpDiagnosticsSubscriber()))
{
	//do stuff			
}
//we should not be subscribed here, but we are.

Therefore subsequent tests that assume the HttpDiagnosticsSubscriber is not registered will fail.

We would like to start working on Full .NET Framework support in .NET agent so we will need Windows CI environment to build and run tests.

Projects requiring Full Framework should be excluded from builds on non-Windows environments to avoid false build failures.

As a test we can implement #42.

The Activity class of System.Diagnostics.DiagnosticSource is already designed for tracking code execution and is a similar idea to your Span. It would mean that the apm agent could just create diagnostic listeners for the scopes that you want to track.

The main benefit I can see from this is that it is built into and used by the framework, which means that I can track things inside the framework without needing custom code. i.e. all requests made by HttpClient. The other benefit is that AspNet already handles the distributed tracing aspect. If you set the Request-Id header and have a diagnostics listener configured it will automatically create an activity using the Request-Id as it's parent.

There is a whole bunch of documentation on using this here: https://github.com/dotnet/corefx/tree/master/src/System.Diagnostics.DiagnosticSource/src

Scope:

• Add the Error API and the infrastructure around it to send errors to the server
• Research how to integrate it for the current ASP.NET Core request monitoring + implement
• Report errors for outgoing HTTP Requests

Since EF Core support is a little bit behind, error capturing will be implemented in the scope of #3

It should be possible to monitor ASP.NET (IIS hosted) applications using full framework with the following additional requirements:

• Adding tracing by APM Agent to an application should require minimal manual effort - that is minimal amount of changes to application code and/or configuration
• The following technologies should be traced automatically when used by application during transaction processing:
  • HttpClient
  • Entity Framework Core and/or Entity Framework (the old one) - investigate which one requires less effort to start with

Subtasks:

• Trace transaction start/end automatically for ASP.NET applications (Full .NET Framework, IIS hosted) #113
• Investigate how to monitor Entity Framework 6 calls with “library-only” approach #335
• Add support for tracing calls via HttpClient on Full Framework #115

Currently the logging interface is very simple. It's basically 1 method, with it's string parameter (which contains the log level, all the parameters, etc). Users have access to the log, and they can redirect it wherever they want by doing this:

Agent.Config.Logger = new CustomLogCollector();

class CustomLogCollector : AbstractLogger
{
	protected override void PrintLogline(string logline)
	{
		//send the logline or process it...
	}
}

@SeanFarrow suggested here that:

Even if I derive from AbstractLogger to send the log somewhere else, I’m stuck with a string and would have to use something else to extract data using regex or an alternative input such as LogStash.

Let's discuss if we could make this a nicer interface! I'm totally open to provide an interface with more details.

@SeanFarrow feel free to suggest a specific interface if you have something in mind.

Thanks

See elastic/apm-agent-go#325.

We should have a signing key and sign our assemblies: https://docs.microsoft.com/en-us/dotnet/standard/library-guidance/strong-naming

elastic/apm-server#1523

For .NET this could be useful to differentiate sync and async calls.

Scope
Send stack traces for Spans.

Issues

In case of async calls the stack trace is very hard to read. In .NET Core 2.1 this was improved, but only for exceptions (at least this is what I see...). Capturing a stack with System.Diagnostics.StackTrace().GetFrames() still returns the "real", hard-to-read stack trace and I did not find any .NET Standard 2.0 API which would demystify those stack traces.

I tried using Ben.Demystifier, but within the DiagnosticSource OnNext() method it threw an exception - so it'd be an additional dependency + it does not work for this use case out of the box.

Another problem related to .NET is that we seem to show this format on the UI:
[FileName] in [Function] at [Lineno].

Currently the agent sends the assembly name as filename and since the stacktrace_frame definition doesn't have a "class" or "type" field I send the type name in the "module" field, but that is not shown on the UI.
So we end up showing [AssemblyName] in [Function (aka method)] at [lineno].
For .NET a better format would be something like this:
[Module (aka .NET type name)] in [Function (aka method)] at [Lineno]. A workaround would be to send the Typename in the filename field, but to me that does not sound correct.

We don't always have lineno, this topic is currently discussed here.

Current status
In case of outgoing HTTP calls I already send callstacks, we just need to discuss how/what fields I need to fill.

From #86:

• we should have nightly builds published as NuGet packages.
• @Mpdreamz`s summary about what the client does (...which we probably should do similarly) #86 (comment)

We should think about how our service naming (and in general the Agent) works on Full Framework with multiple AppDomains.

Currently we assume 1 service per process, so probably we will send the data from each AppDomain with the same service.name with the same service.agent values.

I don't know if this is what we want. Either way, I think we should test this, think about what our goal is in such a (not so uncommon) scenario and document it.

Scope
Write code that parses and apply settings (e.g. log level).

Settings can come from:

• Environment variables
• appsettings.json
• TBD

See elastic/apm#8

Will this agent support .net core as well? and 2.1?

.net core 2.x allows us to run code in linux on kubernetes, but thats not the same bowl of wax as classic .net.

https://docs.microsoft.com/en-us/dotnet/core/linux-prerequisites?tabs=netcore2x
https://www.infoq.com/news/2017/10/dotnet-core-standard-difference

If set to true, the agent will capture request and response headers, including cookies.
NOTE: Setting this to true can increase the network and disk requirements [and increases allocations].

If false the agent doesn't capture headers.

Default: true

This issue is about a design decision and for discussion about this specific decision. Relates to #26.

To save time/space in the discussion we focus on Span, but the same discussion is valid for Transaction.

Description of the current situation

The default pattern to wrap a particular piece of code into a span is this:

var span = transaction.StartSpan("SampleSpan", Span.TYPE_DB);
try
{
    //code you want to capture as a span
}
catch(Exception e)
{
    span.CaptureException(e);
    throw; //don't use `throw e` to preserve the call stack.
}
finally
{
    span.End();
}

This is very similar to the Java Agent API.

Currently both ISpan and ITransaction offer a convenient method to start, stop spans/transactions and also to automatically report unhandled exceptions. The intention is to avoid having to write too many additional lines just to capture something as a span or transaction.

So, the equivalent of the code from above with the convenient API is this:

transaction.CaptureSpan("TestSpan", Span.TYPE_DB, () =>
{
   //code you want to capture as a span
});

There are overloads for async methods, and also for methods with return value.

Alternative solution, IDisposable

Lots of similar tools, like MiniProfiler solve this by implementing IDisposable, and not by having a method that takes Action or Func as we do. Therefore some users may expect that our Span and Transaction types also implement IDisposable and disposing those would do basically the same as calling the End() method.

With this the code would be something like this:

using(var span = transaction.StartSpan("SampleSpan", Span.TYPE_DB))
{
   //code you want to capture as a span
   //+ capturing exceptions somehow
}

Therefore: Should our Span and Transaction types implement IDisposable?

Pros:

• Other tools do this (OpenTracing C# API, MiniProfiler).
• Wrapping these kings of things into a using block seems very natural.
• using avoids closures, which we have with the methods taking a lambda. This can cause overhead in perf. critical scenarios due to allocations. (On the other hand the StartX() and End() combination still can be used to avoid this problem).

Cons:

• The using block only partially solves the problem: it can automatically call End(), but it does not capture unhandled exceptions. Actually tools that typically use IDisposable for similar things don't have a catch(Exception e){capture(e)} part in the pattern that they simplify with the using block. So having the using block plus an additional exception handling logic would make the API more noisy than the plain, original API from the beginning of the discussion.
• In the original API ending a span happens by calling the End() method, and this is common across all agents. If we implement IDisposable then we will call the End() method from the Dispose() method and from that point we basically have 2 methods that end a span, and that does not feel right. The current API does not have this confusion.
• Adding IDisposable is a pattern to wrap unmanaged resources. We don't really wrap unmanaged resources here. So automatically calling the End() method is a similar problem to the problem that IDisposable solved, but it's not the same. See IDisposable documentation : "Provides a mechanism for releasing unmanaged resources." That's not what we do by calling End().

Currently we don't implement IDisposable and that seems to be the better decision, but this can be of course revisited. Happy to hear opinions!

In order to test the ApmMiddleware we need tests that exercise that code like a real world application.

For this an extra test project will be created based on Microsoft.AspNetCore.Mvc.Testing. With this we can simulate incoming HTTP calls and dependency inject a fake payloadsender that simply keeps the generated spans, transactions, errors, etc. in memory which we can use to assert.

Motivation: Having request specific data (user ID, name, etc.) in transaction name makes metrics aggregation very problematic. For example, let's say we saw two GET requests with URL paths: /user/123/info and /user/456/info and we would like to calculate average transaction latency - if we just use URL path as the transaction name for each request we will calculate average latency separately for transaction name GET /user/123/info and separately for GET /user/456/info - it would be much more useful to deduce transaction name without request specific data (in this case without user ID). For example, if the transaction name is GET /users/{id}/info then average latency will be calculated using both requests.

If the user does not give us a service name we automatically use the name of the entry assembly:

            get
            {
                if(service == null)
                {
                    service = new Service
                    {
                        Name = Assembly.GetEntryAssembly()?.GetName()?.Name,
                        Agent = new Model.Payload.Agent
                        {
                            Name = Consts.AgentName,
                            Version = Consts.AgentVersion
                        }
                    };
                }

                return service;
            }

This bugs, because Assembly.GetEntryAssembly() can return null as the docu says:

The GetEntryAssembly method can return null when a managed assembly has been loaded from an unmanaged application.

This happens e.g. on Full Framework when the app runs in IIS.

Stack overflow has an MIT licensed profiler for dotnet called MiniProfiler that ya'll could pull providers and/or inspiration from: https://github.com/MiniProfiler/dotnet
Thought I'd mention it. Looking forward to seeing this progress!

We use IdentityServer4 for our Authentication provider. It is currently configured to use EntityFrameworkCore with SqlServer. A piece of functionality that comes out of the box with IdentityServer4 and EFCore as the backing store is a periodic task that runs in the background that deletes expired tokens from the database.

When this background task runs it triggers the OnNext method of the EfCoreDiagnosticListener. However because this task is not executed within the context of an HttpRequest, no transaction is created. Therefore when the background task executes a null reference exception is thrown because the TransactionContainer.Transactions.Value is null.

Scope
Add logging infrastructure in the core project, including:

• log levels
• adding a prefix to differentiate components (e.g. to easily separate logs from capturing ASP.NET Core requests from logs from EF Core capturing)

Idea: also do it by subscribing to DiagnosticSource events

Scope:
Create the public API. E.g. this is the Java Agent API.

Add support for elastic/apm-server#1781

We have sometimes failing tests like this:

Error
Expected 9 to be greater or equal to: 10
Expected: True
Actual: False
Stacktrace
at Elastic.Apm.Tests.ApiTests.ConvenientApiTransactionTests.AssertWith1TransactionAsync(Func`2 func) in /var/lib/jenkins/workspace/tnet_apm-agent-dotnet-mbp_master/src/github.com/elastic/apm-agent-dotnet/test/Elastic.Apm.Tests/ApiTests/ConvenientApiTransactionTests.cs:line 487
at Elastic.Apm.Tests.ApiTests.ConvenientApiTransactionTests.AsyncTaskWithReturnType() in /var/lib/jenkins/workspace/tnet_apm-agent-dotnet-mbp_master/src/github.com/elastic/apm-agent-dotnet/test/Elastic.Apm.Tests/ApiTests/ConvenientApiTransactionTests.cs:line 265
--- End of stack trace from previous location where exception was thrown ---

This is the assert:

var duration = payloadSender.FirstTransaction.Duration;
Assert.True(duration >= SleepLength, $"Expected {duration} to be greater or equal to: {SleepLength}");

This is how we calculate the duration (very very bad!):
if (!Duration.HasValue) Duration = (long)(DateTimeOffset.UtcNow - Start).TotalMilliseconds;

We have a little bit of a data type chaos here and we should heavily overthink this! - also we should see what datatypes other agents use for duration and probably stick with that, but each time we convert we should be more careful.

Example:
ASP.NET Core request started -> Microsoft.EntityFrameworkCore diagnostic events are sent (no access to HttpContext there, create span, link it to the transaction) -> HttpClient diagnostic events are sent, same again: create span, link it to the transaction.

Potential solution: AsyncLocal

From elastic/apm#30

PR: #46

Generally speaking, .NET Tasks set to LongRunning have a far superior error handling system and still run on their own dedicated thread, so the core loop inside PayloadSender. Threads scheduled on their own can still completely crash the containing application when using async void.

So the recommendation would be to use a Task for this, as outlined in this StackOverflow thread: https://stackoverflow.com/questions/26921191/how-to-pass-longrunning-flag-specifically-to-task-run/26931368#26931368

Currently we have 4 projects under the src folder and each should be released as a nuget package referencing each other.

So we should have nuspec files doing this. Additionally the release process should be also defined.

Main requirement for sampling feature

"Non-sampled" means that spans and detailed transaction context aren't reported. So for each downstream service, we will still report a transaction with minimal context (for now -- may change to metrics later)

Additional requirements for integration between distributed tracing and sampling features

1. recorded flag is used to signal sampling decision which means that the agent that starts the trace should decide whether to sample and set recorded flag in outgoing calls accordingly. If an agent sees that it's a downstream agent for the current trace then it should sample current transaction based on the incoming recorded flag.
2. In order to enforce the same sampling decision for the whole trace agents should inject distributed tracing header even if the current trace is nonsampled. The difference in in the way distributed tracing should behave for nonsampled transactions is that for transaction ID should used for inject header value instead of span ID (see elastic/apm#16).
3. Optionally consider covering requirements in #44 (Add transaction.sampled to Errors) by this implementation.

From elastic/apm-server#1660

While trying to pull in packages for ElasticApm that I built locally into a project I ran into some dependency issues.

Currently the libraries reference these versions:
Microsoft.AspNetCore.Http.Abstractions Version 2.1.1
Microsoft.Extensions.Configuration.Abstractions Version 2.0.2

Our IdentityServer4 impelementation currently references these dependency versions:
Microsoft.AspNetCore.Http.Abstractions Version 2.0.0
Microsoft.Extensions.Configuration.Abstractions Version 2.0.0

We created our project right after 2.0.0 came out and haven't gotten around to updating. (We can and should update.) I was able to tweak the version numbers manually and build a new package and get the application running. However, it made me wonder if it would be worth targeting those dependency versions as a decision to try and support 2.0.0 outright.

I'll submit a PR with the change and let you decide if it makes sense.

Subscribe to "Microsoft.EntityFrameworkCore" events and create spans based on those events.

There should be .NET tests here: https://github.com/elastic/apm-integration-testing.

Prerequisite: #14

• Create opbeans repository
• Implement opbeans sample
• Add test to https://github.com/elastic/apm-integration-testing

By implementing IHostingStartup and using the following two environment variables we could load the agent into an ASP.NET Core app without any code change or even recompilation:

• DOTNET_ADDITIONAL_DEPS
• ASPNETCORE_HOSTINGSTARTUPASSEMBLIES

I blogged about this feature here.

• Create NuGet packages (#85)
• Make sure we can upload to nuget.org
• Make sure the documentation is ok
• Create blogpost
• Create walkthrough video

The ASP.NET Core sample app already does database and outgoing web request calls, but this is only in the code and the HTML doesn't really show that. (it's basically just the default ASP.NET Core template)

We should build a UI telling what happens in the specific request and what that request would look like on the Kibana UI.

With this if by cloning the repo and starting the sample app people can already have an idea what to do with the repo and what to expect from the agent.

No matter what I put in I always end up with a localhost:8200 as address.

{

"Logging": {
"IncludeScopes": false,
"Debug": {
"LogLevel": {
"Default": "Warning"
}
},
"Console": {
"LogLevel": {
"Default": "Warning"
}
}
},
"AllowedHosts": "*",
"ElasticApm": {
"LogLevel": "Debug",
"ServerUrls": "172.26.xxx.xxx"
}
}

based on the sample..

I would like to try this as my error logging from an asp.net api service

With kind regards,

Michael

Since the Elastic.Apm project is the core of the agent it can end up basically in any types of application on any types of framework that we support. Therefore we should be super careful about what other packages/assemblies this package depends on.

We already had suggestions from the community to add fancy logging libraries and I expect some other ideas in the future. All those are valid ideas, but one of the main design goals regarding this package should be portability (also beyond just .NET Standard 2.0). If we lock ourself into a specific framework or into a relatively high framework version then it can block our productivity and limit the supported frameworks/platforms heavily.

Therefore:

• There should be a documentation describing this and each time someone suggests to add a new dependency to Elastic.Apm then we should just check if against or design goals and make the decision based on that. (hint: probably most of the suggestions to add a dependency will be declined...)

• We should revisit the existing dependencies and think about solutions.

• Json.NET (the elasticsearch client already solves this, we should reuse this solution)

• System.Threading.Tasks.Dataflow (caused problems with our full framework sample on @SergeyKleyman`s setup, we should check if this is ok)

• System.Diagnostics.DiagnosticSource

I have the APM up and running in my kubernetes environment with the use of global variables..
So thanks again for that explanation.. Very happy with it!!!

But now the next step,
How to activate tracing.. or how do activate tracing in my code.., what do I need to get them filled..
I looked at the samples but that bring me any steps closer.

Sorry if I might have missed it somewhere..

with kind regards

Threads doing work for the monitored application should be affected as little as possible.

E.g. in case of ASP.NET Core a request should not wait to send telemetry data back to the server.

Proposed solution: create 1 specific thread and dispatch agent related work to that thread.

Is there a reason the ElasticApm class is static.

This makes the ElasticApm agent unmockable/untestable.
cheers
Sean.