Bench as a global .NET tool about azure-signalr-bench HOT 5 CLOSED

I agree it's something we should do.
@ccic

from azure-signalr-bench.

@bradygaster Let us discuss it further about how to ship it on NuGet.

Something else about this tool:

Current features:
This tool currently supports performance evaluation on echo/broadcast/sendtoclient/sendtogroup on all supported transports and protocols of SignalR. In addition, it supports Azure SignalR's REST API (serverless mode).
In addition, it can be used as stability test since it allows long time running.

Feature plan:
There are some feature lists in my mind:

1. Friendly usage:
   (1) Build a container web app which can be configured through the browser, and run it in K8S environment. Customer does not need to install anything. The behavior is similar as SignalR crank, but it does not collect CPU/Memory metrics.
   (2) Expose REST API to control the benchmark master, like start/stop/query benchmark. Allow the customers to develop further based on those information.
   (3) Enrich the client with support other development languages. Currently it only support C# client.

2. Extensible:
   The benchmark is in fact a master-slave mode, and the framework is just responsible to communicate between master and slave nodes. Developers can customize their own plugins to replace the default SignalR plugin.

3. Deployment:
   Cloud deployment: Azure SignalR service deployed this tool in Azure VMs currently. In fact, it support other deployment, like in K8S, or even in other kind of cloud.

4. Stress load test service:
   The load test is a requirement for many products, it is better deploy it as a load service.

from azure-signalr-bench.

This tool currently supports performance evaluation on echo/broadcast/sendtoclient/sendtogroup on all supported transports and protocols of SignalR. In addition, it supports Azure SignalR's REST API (serverless mode).

We should definitely make it as effective for SignalR users as it is for Azure SignalR Service users; I think optimizing this as "a SignalR testing tool" is going to give us more customers to work with than it would if it were ASRS only.

What that means is, we'd need the global tool to be set up in such a way that a user could pass it a YAML file to parse, OR allow for parameterized calls, so I could pass it a hub name and a hub method to call from the command line.

1. Friendly usage:
   (1) Build a container web app which can be configured through the browser, and run it in K8S environment. Customer does not need to install anything. The behavior is similar as SignalR crank, but it does not collect CPU/Memory metrics.

I don't dislike this idea, but I think we should first optimize for creating a global tool. This is a great idea, but my suggestion (this issue) was more around the idea of compartmentalizing what we can out of Bench to create a global tool that SignalR users could use to test their SignalR Hub performance and stress responses.

(2) Expose REST API to control the benchmark master, like start/stop/query benchmark. Allow the customers to develop further based on those information.

I think this is a good idea on the service side, but that we should focus on the SignalR benchmarking as a pri 0 and evaluate stretch features like this later.

(3) Enrich the client with support other development languages. Currently it only support C# client.

For the purpose of building a global tool, C# is fine - it'll be used primarily by .NET developers. Later on, once we have the tool authored, these additional features would be great to have, but I think they inflate the goals of having "a SignalR hub benchmarking global CLI tool."

2. Extensible:
   The benchmark is in fact a primary/secondary (reworded) mode, and the framework is just responsible to communicate between primary/secondary (reworded) nodes. Developers can customize their own plugins to replace the default SignalR plugin.

I do love this idea. Again though, I think this might conflate the basic idea of a global tool. That said, if there was a way to pass in the "plugin-containing assembly" via a parameter, that'd be great. This way, a developer who authors a plugin can feed it to the tool at run-time.

3. Deployment:
   Cloud deployment: Azure SignalR service deployed this tool in Azure VMs currently. In fact, it support other deployment, like in K8S, or even in other kind of cloud.

For a tool, I think this is orthogonal. But later, definitely.

4. Stress load test service:
   The load test is a requirement for many products, it is better deploy it as a load service.

For a tool, I think this is orthogonal. But later, definitely.

from azure-signalr-bench.

@bradygaster

we'd need the global tool to be set up in such a way that a user could pass it a YAML file to parse

This benchmark supports simple command and YAML file to do performance test like this, but currently lack of one command to do build and run all the components like dotnet run perf -C config.yaml. Is it what you want?

from azure-signalr-bench.

We should definitely make it as effective for SignalR users as it is for Azure SignalR Service users; I think optimizing this as "a SignalR testing tool" is going to give us more customers to work with than it would if it were ASRS only.

I agree. This tool is for both SignalR and ASRS. It has already supports SignalR. The internal web app uses environment variable to enable/disable self host SignalR .

What that means is, we'd need the global tool to be set up in such a way that a user could pass it a YAML file to parse, OR allow for parameterized calls, so I could pass it a hub name and a hub method to call from the command line.

For a general test, the global tool needs to know the appserver endpoint, hub name, and scenario (hub method is included in scenario). For example, send to group, which requires the connection to join a specific group before calling the hub method. In addition, there are some other parameters, like transport type, protocol, message size, etc. That is why a YAML file is required. Of course, the tool provides many default settings. So, user does not need to input many parameters to run this test.

I don't dislike this idea, but I think we should first optimize for creating a global tool. This is a great idea, but my suggestion (this issue) was more around the idea of compartmentalizing what we can out of Bench to create a global tool that SignalR users could use to test their SignalR Hub performance and stress responses.

Agree. Let us ship the global tool first.

from azure-signalr-bench.