aspnet / benchmarks Goto Github PK

Currently, if the Driver is killed (Ctrl+C) or crashes, the server and client jobs will never be deleted and block the server until manually killed. This should be improved by either automatically finishing the job after some timeout (say 5 minutes), or perhaps by doing cleanup in the Driver shutdown (if this can be done after Ctrl+C).

Right now, if you try to use a repo path of C:\git\KestrelHttpServer.git, the benchmark server barfs :(. You can work around it by using C:/git/KestrelHttpServer.git, but it'd be good to eventually fix the parsing logic that fails :).

Stacktrace:

[02:55:11.298] Error starting job '2': System.InvalidOperationException: Couldn't parse repository name from C:\git\Kest
relHttpServer.git
   at BenchmarkServer.Startup.GetRepoName(Source source) in C:\code\github\aspnet\benchmarks\src\BenchmarksServer\Startu
p.cs:line 317
   at BenchmarkServer.Startup.SourceRepoComparer.GetHashCode(Source obj) in C:\code\github\aspnet\benchmarks\src\Benchma
rksServer\Startup.cs:line 334
   at System.Collections.Generic.HashSet`1.InternalGetHashCode(T item)
   at System.Collections.Generic.HashSet`1.AddIfNotPresent(T value)
   at System.Collections.Generic.HashSet`1.UnionWith(IEnumerable`1 other)
   at System.Collections.Generic.HashSet`1..ctor(IEnumerable`1 collection, IEqualityComparer`1 comparer)
   at BenchmarkServer.Startup.CloneAndRestore(String path, ServerJob job) in C:\code\github\aspnet\benchmarks\src\Benchm
arksServer\Startup.cs:line 172
   at BenchmarkServer.Startup.<ProcessJobs>d__9.MoveNext() in C:\code\github\aspnet\benchmarks\src\BenchmarksServer\Star
tup.cs:line 121

At a minimum, you have to upgrade a few scripts and project.json files just like here: aspnet/Configuration@9cfd22f

cc @DamianEdwards

https://www.nuget.org/packages/Jil/2.13.0-beta1

/cc @NickCraver @mgravell @kevin-montrose

• Setup cluster of VMs that matches the TechEmpower "Cloud" environment as closely as possible
  • VM Size: D3v2
  • Roles
    • Load Generator (Ubuntu 14.04)
    • Linux Web Server (Ubuntu 14.04)
    • Linux PostgreSQL Server (Ubuntu 14.04)
    • Windows Web Server (Windows Server 2016 Datacenter)
    • Windows SQL Server (Windows Server 2016 Datacenter)
• After every build, run all TechEmpower scenarios (Plaintext, JSON, Fortunes, DB queries and updates) using the same parameters as the official runs.
  • Add parameter to Driver to specify server type ("physical" or "virtual")
  • Add column to SQL
  • Scenarios
    • Plaintext
    • JSON
    • Fortunes
    • SingleQuery
    • MultiQuery
    • MultiUpdate
• Display results in a series of charts, as the first tab at https://aka.ms/aspnet/benchmarks

Was hyper threading enabled in your benchmarking tests?

Could you break out the ASP.NET 5 on Kestrel in to more flavours?

So something like

Maybe a pretty over time graph at each change point? Like https://arewefastyet.com/

It would be great to see golang results from TechEmpower benchmarks on the README.md for the following tests:

• go-std
• fasthttp

This is a regression due my change yesterday. This is due to aspnet/Mvc#4660. HttpResponseStreamWriter doesn't scale at this number of cores due to contention in the ArrayPool.

The repo description needs to be updated:

Benchmarks for ASP.NET 5 -> Benchmarks for ASP.NET Core

Many developers on windows platform may work on this group

We should enable Server GC in project.json, once this feature is added to the CLI (https://github.com/dotnet/cli/issues/2121).

Our workload project files use wildcarding to specify their version of the SDK/runtime https://github.com/aspnet/benchmarks/blob/dev/src/Benchmarks/project.json#L25

This means that the version our projects need is dependent on our feeds. This also means that the local copy of dotnet on the server can get out of date with what we need to build/run our workload projects.

We need a solution so that every time we take a new CLI it doesn't hose the benchmark server

Prior to submitting to Techempower - dual sockets with NUMA have different performance characteristics and Techempower runs the peak benchmark on a dual socket machine (40 HT cores)

https://h2o.examp1e.net/

• Add parameter to Driver, Server, and ServerJob to specify .NET Framework or .NET Core
• Update Server to run Benchmarks on .NET Framework or .NET Core based on parameter
• Add column to SQL
• Add rows to makefile.shade for the following scenarios on .NET Framework
  • Plaintext
  • Plaintext with HTTPS
  • JSON
  • JSON with HTTPS
  • MvcPlaintext
  • MvcJSON
• Update PowerBI report to compare .NET Framework and .NET Core

• Install and configure Windows and Linux servers on new hardware
• Add parameter to Driver to specify server OS
• Add column to SQL
• Add rows to makefile.shade to test all available scenarios on Linux
  • .NET Framework and WebListener are Windows-only
• Update PowerBI report to compare Windows and Linux

http://www.serverframework.com

A few enhancements are needed so we can run in TechEmpower's automation.

See #95.

Which version of TechEmpower is used here? Is it Round12?

I notice that the number of ASP.net connection per thread (256) is much lower than that of Scala (1024). Why they are set differently? How will that affect the final result?

Response caching would be better, but as that's not there yet using https://github.com/aspnet/Caching with the static files #32 (comment) as a byte[]+headerDetails structure would also work.

Although its likely to just saturate the network; so the CPU would probably be more interesting.

/cc @DamianEdwards

It's under the desk of the SmurfLab. Needs to replace the existing server running SQL Server. The new one has the same 6-core CPU as the web servers so should hopefully reduce the bottleneck for the DB tests which is currently the DB server.

• Dual-boot
  • Ubuntu Server 14.04 LTS
  • Windows Server 2012 R2
• On Windows
  • SQL Server Enterprise 2014
  • PostgreSQL 9.5.1
• On Linux
  • MySql
  • PostgreSQL 9.5.1

https://msdn.microsoft.com/en-us/library/aa712982(v=vs.71).aspx

I am interested in returning back & learning asp.net.
I would like to see how it compares to java & js, at least for the single query & update tests

I have been analyzing the Techempower benchmark results with dotnet core (https://github.com/aspnet/benchmarks/blob/master/results/Results.xlsx). Wanted to check if pipelining was enabled for the mentioned scenarios (Plaintext, Json in particular). If so, what was the pipelining depth used? Also, I believe the number of wrk threads was set to the number of cores in the server. Would be great if I could also get the thread counts.

• Add parameter to Driver and ServerJob to specify WebListener or Kestrel
• Update Server to run Benchmarks on WebListener or Kestrel based on parameter
• Add column to SQL
• Add rows to makefile.shade for the following scenarios on WebListener
  • Plaintext
  • Plaintext with HTTPS
  • JSON
  • JSON with HTTPS
• Update PowerBI report to compare WebListener and Kestrel

Should be similar to the changes required for adding HTTPS: f69541f

In dev branch f92dca7 after successfully restoring following README.md and then calling dnx --configuration Release run then it fails with:

Error: E:\oss\aspnet\benchmarks\src\Benchmarks\Controllers\FortunesController.cs(6,28): DNXCore,Version=v5.0 error CS023
4: The type or namespace name 'Mvc' does not exist in the namespace 'Microsoft.AspNetCore' (are you missing an assembly
reference?)
E:\oss\aspnet\benchmarks\src\Benchmarks\Controllers\HomeController.cs(7,28): DNXCore,Version=v5.0 error CS0234: The type
 or namespace name 'Mvc' does not exist in the namespace 'Microsoft.AspNetCore' (are you missing an assembly reference?)

E:\oss\aspnet\benchmarks\src\Benchmarks\Controllers\FortunesController.cs(11,39): DNXCore,Version=v5.0 error CS0246: The
 type or namespace name 'Controller' could not be found (are you missing a using directive or an assembly reference?)
E:\oss\aspnet\benchmarks\src\Benchmarks\Controllers\FortunesController.cs(10,6): DNXCore,Version=v5.0 error CS0246: The
type or namespace name 'Route' could not be found (are you missing a using directive or an assembly reference?)
E:\oss\aspnet\benchmarks\src\Benchmarks\Controllers\FortunesController.cs(24,10): DNXCore,Version=v5.0 error CS0246: The
 type or namespace name 'HttpGet' could not be found (are you missing a using directive or an assembly reference?)
E:\oss\aspnet\benchmarks\src\Benchmarks\Controllers\FortunesController.cs(25,27): DNXCore,Version=v5.0 error CS0246: The
 type or namespace name 'IActionResult' could not be found (are you missing a using directive or an assembly reference?)

E:\oss\aspnet\benchmarks\src\Benchmarks\Controllers\FortunesController.cs(30,10): DNXCore,Version=v5.0 error CS0246: The
 type or namespace name 'HttpGet' could not be found (are you missing a using directive or an assembly reference?)
E:\oss\aspnet\benchmarks\src\Benchmarks\Controllers\FortunesController.cs(31,27): DNXCore,Version=v5.0 error CS0246: The
 type or namespace name 'IActionResult' could not be found (are you missing a using directive or an assembly reference?)

E:\oss\aspnet\benchmarks\src\Benchmarks\Controllers\FortunesController.cs(36,10): DNXCore,Version=v5.0 error CS0246: The
 type or namespace name 'HttpGet' could not be found (are you missing a using directive or an assembly reference?)
E:\oss\aspnet\benchmarks\src\Benchmarks\Controllers\FortunesController.cs(37,27): DNXCore,Version=v5.0 error CS0246: The
 type or namespace name 'IActionResult' could not be found (are you missing a using directive or an assembly reference?)

E:\oss\aspnet\benchmarks\src\Benchmarks\Controllers\HomeController.cs(12,35): DNXCore,Version=v5.0 error CS0246: The typ
e or namespace name 'Controller' could not be found (are you missing a using directive or an assembly reference?)
E:\oss\aspnet\benchmarks\src\Benchmarks\Controllers\HomeController.cs(11,6): DNXCore,Version=v5.0 error CS0246: The type
 or namespace name 'Route' could not be found (are you missing a using directive or an assembly reference?)
E:\oss\aspnet\benchmarks\src\Benchmarks\Controllers\HomeController.cs(15,16): DNXCore,Version=v5.0 error CS0246: The typ
e or namespace name 'IActionResult' could not be found (are you missing a using directive or an assembly reference?)
E:\oss\aspnet\benchmarks\src\Benchmarks\Controllers\HomeController.cs(28,16): DNXCore,Version=v5.0 error CS0246: The typ
e or namespace name 'ViewResult' could not be found (are you missing a using directive or an assembly reference?)
E:\oss\aspnet\benchmarks\src\Benchmarks\Controllers\HomeController.cs(14,10): DNXCore,Version=v5.0 error CS0246: The typ
e or namespace name 'HttpGet' could not be found (are you missing a using directive or an assembly reference?)
E:\oss\aspnet\benchmarks\src\Benchmarks\Controllers\HomeController.cs(20,10): DNXCore,Version=v5.0 error CS0246: The typ
e or namespace name 'HttpGet' could not be found (are you missing a using directive or an assembly reference?)
E:\oss\aspnet\benchmarks\src\Benchmarks\Controllers\HomeController.cs(21,10): DNXCore,Version=v5.0 error CS0246: The typ
e or namespace name 'Produces' could not be found (are you missing a using directive or an assembly reference?)
E:\oss\aspnet\benchmarks\src\Benchmarks\Controllers\HomeController.cs(27,10): DNXCore,Version=v5.0 error CS0246: The typ
e or namespace name 'HttpGet' could not be found (are you missing a using directive or an assembly reference?)
E:\oss\aspnet\benchmarks\src\Benchmarks\Controllers\HomeController.cs(33,47): DNXCore,Version=v5.0 error CS0246: The typ
e or namespace name 'IActionResult' could not be found (are you missing a using directive or an assembly reference?)
E:\oss\aspnet\benchmarks\src\Benchmarks\Controllers\HomeController.cs(37,44): DNXCore,Version=v5.0 error CS0246: The typ
e or namespace name 'ActionContext' could not be found (are you missing a using directive or an assembly reference?)
E:\oss\aspnet\benchmarks\src\Benchmarks\Startup.cs(84,43): DNXCore,Version=v5.0 error CS1061: 'IServiceCollection' does
not contain a definition for 'AddMvcCore' and no extension method 'AddMvcCore' accepting a first argument of type 'IServ
iceCollection' could be found (are you missing a using directive or an assembly reference?)
E:\oss\aspnet\benchmarks\src\Benchmarks\Startup.cs(173,21): DNXCore,Version=v5.0 error CS1061: 'IApplicationBuilder' doe
s not contain a definition for 'UseMvc' and no extension method 'UseMvc' accepting a first argument of type 'IApplicatio
nBuilder' could be found (are you missing a using directive or an assembly reference?)
E:\oss\aspnet\benchmarks\src\Benchmarks\Controllers\HomeController.cs(30,20): DNXCore,Version=v5.0 error CS0103: The nam
e 'View' does not exist in the current context
E:\oss\aspnet\benchmarks\src\Benchmarks\Controllers\FortunesController.cs(27,20): DNXCore,Version=v5.0 error CS0103: The
 name 'View' does not exist in the current context
E:\oss\aspnet\benchmarks\src\Benchmarks\Controllers\FortunesController.cs(33,20): DNXCore,Version=v5.0 error CS0103: The
 name 'View' does not exist in the current context
E:\oss\aspnet\benchmarks\src\Benchmarks\Controllers\FortunesController.cs(39,20): DNXCore,Version=v5.0 error CS0103: The
 name 'View' does not exist in the current context

Something different?

What is your pipeline.lua looks like? Can you share the script?

Current benchmark automation is totally fine with socket errors occurring and will ignore them. The system needs to totally crash and burn (without reporting perf numbers) if there's any error from the client.

We think the current round of errors we're seeing are an artifact of wrk and not of the server, but that's beside the point 😉

https://github.com/aspnet/KestrelHttpServer

Registered RIO //build/ announce from 2011
http://channel9.msdn.com/events/Build/BUILD2011/SAC-593T

Microsoft Windows 8 and Windows Server 2012 introduce new Windows Sockets programming elements.

A set of high-speed networking extensions are available for increased networking performance with lower latency and jitter. These extensions targeted primarily for server applications use pre-registered data buffers and completion queues to increase performance.

The following are new Windows Sockets functions added to support Winsock high-speed networking Registered I/O extensions:

https://msdn.microsoft.com/en-us/library/windows/desktop/ms740642(v=vs.85).aspx

Should be able to easily run the benchmark using a script (DNX console app?) in this repo (on both Windows & *nix). It would do something like the following:

1. Allow specifying which DNX to run with (all variants applicable for the OS, e.g. x86, x64, CLR, CoreCLR, Mono, etc.).
2. Allow specifying the *nix server to drive load from using wrk
3. Allow specifying which test scenario to run (e.g. plaintext, json, single-query, multiple-queries, mvc-plaintext, mvc-json, mvc-view, etc.)
4. Update to latest DNX
5. Restore packages & build the /src/Benchmarks project
6. Start the server (dnx --configuration Release --project src/Benchmarks kestrel)
7. Start an SSH session to *nix load server
8. Run wrk on *nix load server (following taken from https://www.techempower.com/benchmarks/#section=motivation&hw=peak&test=fortune)
   1. Run a 5-second primer at 8 client-concurrency to verify that the server is in fact running. These results are not captured.
   2. Run a 15-second warmup at 256 client-concurrency to allow lazy-initialization to execute and just-in-time compilation to run. These results are not captured.
   3. Run a 15-second captured test for each of the concurrency levels (or iteration counts) exercised by the test type. Concurrency-variable test types are tested at 8, 16, 32, 64, 128, and 256 client-side concurrency. The high-concurrency plaintext test type is tested at 256, 1,024, 4,096, and 16,384 client-side concurrency.
9. Results from measured runs should be captured and output to a results file
10. Stop the server process

I wanted to try this but when I run Benchmarks asp vnext web app, I get the following error in visual studio. I get the same error with dnx with and beta5 (clr\coreclr x86\x64).

I have the latest VS 2015 and beta5 and I am not behind proxy. dnu restore works just fine.

C:\Projects\git\benchmarks\src\Benchmarks>dnx . web
System.TypeLoadException: Could not load type 'Microsoft.Framework.Runtime.IRuntimeServices' from assembly 'Microsoft.Fr
amework.Runtime.Abstractions, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null'.
   at Microsoft.AspNet.Hosting.WebHostBuilder.BuildHostingServices()
   at Microsoft.AspNet.Hosting.WebHostBuilder.Build()
   at Microsoft.AspNet.Hosting.Program.Main(String[] args)
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.Framework.Runtime.Common.EntryPointExecutor.Execute(Assembly assembly, String[] args, IServiceProvider s
erviceProvider)
   at Microsoft.Framework.ApplicationHost.Program.ExecuteMain(DefaultHost host, String applicationName, String[] args)
   at Microsoft.Framework.ApplicationHost.Program.Main(String[] args)
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.Framework.Runtime.Common.EntryPointExecutor.Execute(Assembly assembly, String[] args, IServiceProvider s
erviceProvider)
   at dnx.host.Bootstrapper.RunAsync(List`1 args, IRuntimeEnvironment env)
   at dnx.host.RuntimeBootstrapper.ExecuteAsync(String[] args)
   at dnx.host.RuntimeBootstrapper.Execute(String[] args)

We should transition the database scenarios to no longer rely on EF migrations. For instance, if one were to try and only run the Dapper scenario there's no need to use EF.

Lets come up with a unified set of migration-like sql scripts 😄

/cc @DamianEdwards

The DB Update test (#5 at this link) should also be implemented with the same set of tests as the query tests.

See requirements at https://www.techempower.com/benchmarks/#section=code

We should have a middleware version and an MVC version.

@davidfowl mentioned on slack that there are now automated runs that publish results at least for the plain text benchmarks, but that these are only published internally. Please consider making the results available externally or at least updating the readme. It seems there's been some pretty significant progress :)

Also, is the techempower hardware the same as smurflab or is that another setup?

cc @DamianEdwards

@Eilon @Tratcher
I am not sure where to put my original issue so I just leave it here.

My original issue is: aspnet/Announcements#133
@Eilon I am recently update the target framework from dnxcore50 to dotnet5.4 based on your annoucement "Switching to new ‘dotnet’ target framework monikers", when we are updating to rc1-final/update1

One of our class library using TcpClient in System.Net.Sockets. When I update to dotnet5.4, it won't build because it cannot find TcpClient. After change our project.json from dotnet5.4 to dotnet5.5 it working fine. So what is the best practice for this? Should I change my project.json to dotnet5.5 and change #if DNXCORE50 to #if DOTNET5_5 for ALL my other library?

It seems on nuget.org only System.Net.Sockets support 5.4 and 5.5, all other packages only support dotnet5.4, what is the compatibility here?

The benchmark says ASP.NET Core on Kestrel perfsvr = 1,174,881...is this RC2 speed?
If so how fast would RC1 run in the same scenario or did I miss something?

Maybe it's a good idea to put the benchmark in azure VM

My laptop (local)

Azure machine A0

42ms to 8.82s :(

Requirements are at https://www.techempower.com/benchmarks/#section=code

We should have a middleware version and an MVC version.