aaronontheweb / actor-benchmarker Goto Github PK

Is your feature request related to a problem? Please describe.

I ran into a problem with a misconfigured ShardRegion in #7 which caused my clustered benchmarks not to run at all. That issue would have been very easy to diagnose if I could have found a way to inject the xUnit ITestOutputHelper into our logging infrastructure. This is something that we do regularly inside the Akka.TestKit, but the current design of the ActorSystemConfigurator<TActor> doesn't make this particularly easy at the moment.

Describe the solution you'd like

I would like some way to configure the logging both of the IHost and the ActorSystem itself. This will probably require two separate optional configuration methods on the API.

Describe alternatives you've considered

In #7 I tried to implement this on the IBenchmarkConfig interface via a special startup method that attached to the IHostBuilder but that wasn't super workable - I suppose I could have made it workable if I instantiated my configuration inside my test, so maybe this approach is worth revisiting.

Additional context

Probably need to make this work both for unit tests and Benchmark.NET execution.

Is your feature request related to a problem? Please describe.

Ideally, we should try to leverage Benchmark.NET for the actual execution and running of these benchmarks. The IActorBenchmark type is practically set up for this automatically:

public interface IActorBenchmark
{
    IBenchmarkConfiguration Configuration { get; }
    
    Task<(int totalActors, int messagesPerActor)> Setup(CancellationToken ct);
    
    Task Run(CancellationToken ct);
    
    Task Teardown(CancellationToken ct);
}

Describe the solution you'd like

I would like to do the following:

1. 1 Benchmark should be computed per IActorBenchmarkDefintion;
2. 1 BenchmarkCase should be computed per unique IBenchmarkConfiguration; and
3. The InvocationsPerOperation number for each BenchmarkCase should be set programmatically via the `IActorBenchmark.Setup~ computed value, if that's possible.

I believe we should be able to do this by tapping into the Benchmark.NET APIs directly and passing them into the BenchmarkSwitcher - but it will take a bit of meta-programming magic to make this work.

Essentially, we need to make sure this function returns true for whatever wrapping we do around Benchmark.NET:

https://github.com/dotnet/BenchmarkDotNet/blob/570eeefa20c8692662722cd4bf8a1b663bd361b7/src/BenchmarkDotNet/Running/TypeFilter.cs#L12-L45

Describe alternatives you've considered

It might actually be a lot easier to instrument this with NBench, as it allows for dynamic computation of operation invocations in a much more simplified manner than Benchmark.NET.