aspnet / benchmarks Goto Github PK
View Code? Open in Web Editor NEWBenchmarks for ASP.NET Core
License: Other
Benchmarks for ASP.NET Core
License: Other
Currently, if the Driver is killed (Ctrl+C) or crashes, the server and client jobs will never be deleted and block the server until manually killed. This should be improved by either automatically finishing the job after some timeout (say 5 minutes), or perhaps by doing cleanup in the Driver shutdown (if this can be done after Ctrl+C).
Right now, if you try to use a repo path of C:\git\KestrelHttpServer.git
, the benchmark server barfs :(. You can work around it by using C:/git/KestrelHttpServer.git
, but it'd be good to eventually fix the parsing logic that fails :).
Stacktrace:
[02:55:11.298] Error starting job '2': System.InvalidOperationException: Couldn't parse repository name from C:\git\Kest
relHttpServer.git
at BenchmarkServer.Startup.GetRepoName(Source source) in C:\code\github\aspnet\benchmarks\src\BenchmarksServer\Startu
p.cs:line 317
at BenchmarkServer.Startup.SourceRepoComparer.GetHashCode(Source obj) in C:\code\github\aspnet\benchmarks\src\Benchma
rksServer\Startup.cs:line 334
at System.Collections.Generic.HashSet`1.InternalGetHashCode(T item)
at System.Collections.Generic.HashSet`1.AddIfNotPresent(T value)
at System.Collections.Generic.HashSet`1.UnionWith(IEnumerable`1 other)
at System.Collections.Generic.HashSet`1..ctor(IEnumerable`1 collection, IEqualityComparer`1 comparer)
at BenchmarkServer.Startup.CloneAndRestore(String path, ServerJob job) in C:\code\github\aspnet\benchmarks\src\Benchm
arksServer\Startup.cs:line 172
at BenchmarkServer.Startup.<ProcessJobs>d__9.MoveNext() in C:\code\github\aspnet\benchmarks\src\BenchmarksServer\Star
tup.cs:line 121
At a minimum, you have to upgrade a few scripts and project.json files just like here: aspnet/Configuration@9cfd22f
Was hyper threading enabled in your benchmarking tests?
Could you break out the ASP.NET 5 on Kestrel
in to more flavours?
So something like
Stack | Runtime | Server |
---|---|---|
ASP.NET 5 on Kestrel | clr x86 | Win 2012 R2 |
ASP.NET 5 on Kestrel | coreclr x86 | Win 2012 R2 |
ASP.NET 5 on Kestrel | coreclr x86 | Linux |
ASP.NET 5 on Kestrel | clr x64 | Win 2012 R2 |
ASP.NET 5 on Kestrel | coreclr x64 | Win 2012 R2 |
ASP.NET 5 on Kestrel | coreclr x64 | Linux |
Maybe a pretty over time graph at each change point? Like https://arewefastyet.com/
This is a regression due my change yesterday. This is due to aspnet/Mvc#4660. HttpResponseStreamWriter doesn't scale at this number of cores due to contention in the ArrayPool.
The repo description needs to be updated:
Benchmarks for ASP.NET 5 -> Benchmarks for ASP.NET Core
Many developers on windows platform may work on this group
We should enable Server GC in project.json, once this feature is added to the CLI (https://github.com/dotnet/cli/issues/2121).
Our workload project files use wildcarding to specify their version of the SDK/runtime https://github.com/aspnet/benchmarks/blob/dev/src/Benchmarks/project.json#L25
This means that the version our projects need is dependent on our feeds. This also means that the local copy of dotnet on the server can get out of date with what we need to build/run our workload projects.
We need a solution so that every time we take a new CLI it doesn't hose the benchmark server
Prior to submitting to Techempower - dual sockets with NUMA have different performance characteristics and Techempower runs the peak benchmark on a dual socket machine (40 HT cores)
A few enhancements are needed so we can run in TechEmpower's automation.
See #95.
Which version of TechEmpower is used here? Is it Round12?
I notice that the number of ASP.net connection per thread (256) is much lower than that of Scala (1024). Why they are set differently? How will that affect the final result?
Response caching would be better, but as that's not there yet using https://github.com/aspnet/Caching with the static files #32 (comment) as a byte[]
+headerDetails structure would also work.
Although its likely to just saturate the network; so the CPU would probably be more interesting.
/cc @DamianEdwards
It's under the desk of the SmurfLab. Needs to replace the existing server running SQL Server. The new one has the same 6-core CPU as the web servers so should hopefully reduce the bottleneck for the DB tests which is currently the DB server.
I am interested in returning back & learning asp.net.
I would like to see how it compares to java & js, at least for the single query & update tests
I have been analyzing the Techempower benchmark results with dotnet core (https://github.com/aspnet/benchmarks/blob/master/results/Results.xlsx). Wanted to check if pipelining was enabled for the mentioned scenarios (Plaintext, Json in particular). If so, what was the pipelining depth used? Also, I believe the number of wrk threads was set to the number of cores in the server. Would be great if I could also get the thread counts.
Should be similar to the changes required for adding HTTPS: f69541f
In dev
branch f92dca7 after successfully restoring following README.md and then calling dnx --configuration Release run
then it fails with:
Error: E:\oss\aspnet\benchmarks\src\Benchmarks\Controllers\FortunesController.cs(6,28): DNXCore,Version=v5.0 error CS023
4: The type or namespace name 'Mvc' does not exist in the namespace 'Microsoft.AspNetCore' (are you missing an assembly
reference?)
E:\oss\aspnet\benchmarks\src\Benchmarks\Controllers\HomeController.cs(7,28): DNXCore,Version=v5.0 error CS0234: The type
or namespace name 'Mvc' does not exist in the namespace 'Microsoft.AspNetCore' (are you missing an assembly reference?)
E:\oss\aspnet\benchmarks\src\Benchmarks\Controllers\FortunesController.cs(11,39): DNXCore,Version=v5.0 error CS0246: The
type or namespace name 'Controller' could not be found (are you missing a using directive or an assembly reference?)
E:\oss\aspnet\benchmarks\src\Benchmarks\Controllers\FortunesController.cs(10,6): DNXCore,Version=v5.0 error CS0246: The
type or namespace name 'Route' could not be found (are you missing a using directive or an assembly reference?)
E:\oss\aspnet\benchmarks\src\Benchmarks\Controllers\FortunesController.cs(24,10): DNXCore,Version=v5.0 error CS0246: The
type or namespace name 'HttpGet' could not be found (are you missing a using directive or an assembly reference?)
E:\oss\aspnet\benchmarks\src\Benchmarks\Controllers\FortunesController.cs(25,27): DNXCore,Version=v5.0 error CS0246: The
type or namespace name 'IActionResult' could not be found (are you missing a using directive or an assembly reference?)
E:\oss\aspnet\benchmarks\src\Benchmarks\Controllers\FortunesController.cs(30,10): DNXCore,Version=v5.0 error CS0246: The
type or namespace name 'HttpGet' could not be found (are you missing a using directive or an assembly reference?)
E:\oss\aspnet\benchmarks\src\Benchmarks\Controllers\FortunesController.cs(31,27): DNXCore,Version=v5.0 error CS0246: The
type or namespace name 'IActionResult' could not be found (are you missing a using directive or an assembly reference?)
E:\oss\aspnet\benchmarks\src\Benchmarks\Controllers\FortunesController.cs(36,10): DNXCore,Version=v5.0 error CS0246: The
type or namespace name 'HttpGet' could not be found (are you missing a using directive or an assembly reference?)
E:\oss\aspnet\benchmarks\src\Benchmarks\Controllers\FortunesController.cs(37,27): DNXCore,Version=v5.0 error CS0246: The
type or namespace name 'IActionResult' could not be found (are you missing a using directive or an assembly reference?)
E:\oss\aspnet\benchmarks\src\Benchmarks\Controllers\HomeController.cs(12,35): DNXCore,Version=v5.0 error CS0246: The typ
e or namespace name 'Controller' could not be found (are you missing a using directive or an assembly reference?)
E:\oss\aspnet\benchmarks\src\Benchmarks\Controllers\HomeController.cs(11,6): DNXCore,Version=v5.0 error CS0246: The type
or namespace name 'Route' could not be found (are you missing a using directive or an assembly reference?)
E:\oss\aspnet\benchmarks\src\Benchmarks\Controllers\HomeController.cs(15,16): DNXCore,Version=v5.0 error CS0246: The typ
e or namespace name 'IActionResult' could not be found (are you missing a using directive or an assembly reference?)
E:\oss\aspnet\benchmarks\src\Benchmarks\Controllers\HomeController.cs(28,16): DNXCore,Version=v5.0 error CS0246: The typ
e or namespace name 'ViewResult' could not be found (are you missing a using directive or an assembly reference?)
E:\oss\aspnet\benchmarks\src\Benchmarks\Controllers\HomeController.cs(14,10): DNXCore,Version=v5.0 error CS0246: The typ
e or namespace name 'HttpGet' could not be found (are you missing a using directive or an assembly reference?)
E:\oss\aspnet\benchmarks\src\Benchmarks\Controllers\HomeController.cs(20,10): DNXCore,Version=v5.0 error CS0246: The typ
e or namespace name 'HttpGet' could not be found (are you missing a using directive or an assembly reference?)
E:\oss\aspnet\benchmarks\src\Benchmarks\Controllers\HomeController.cs(21,10): DNXCore,Version=v5.0 error CS0246: The typ
e or namespace name 'Produces' could not be found (are you missing a using directive or an assembly reference?)
E:\oss\aspnet\benchmarks\src\Benchmarks\Controllers\HomeController.cs(27,10): DNXCore,Version=v5.0 error CS0246: The typ
e or namespace name 'HttpGet' could not be found (are you missing a using directive or an assembly reference?)
E:\oss\aspnet\benchmarks\src\Benchmarks\Controllers\HomeController.cs(33,47): DNXCore,Version=v5.0 error CS0246: The typ
e or namespace name 'IActionResult' could not be found (are you missing a using directive or an assembly reference?)
E:\oss\aspnet\benchmarks\src\Benchmarks\Controllers\HomeController.cs(37,44): DNXCore,Version=v5.0 error CS0246: The typ
e or namespace name 'ActionContext' could not be found (are you missing a using directive or an assembly reference?)
E:\oss\aspnet\benchmarks\src\Benchmarks\Startup.cs(84,43): DNXCore,Version=v5.0 error CS1061: 'IServiceCollection' does
not contain a definition for 'AddMvcCore' and no extension method 'AddMvcCore' accepting a first argument of type 'IServ
iceCollection' could be found (are you missing a using directive or an assembly reference?)
E:\oss\aspnet\benchmarks\src\Benchmarks\Startup.cs(173,21): DNXCore,Version=v5.0 error CS1061: 'IApplicationBuilder' doe
s not contain a definition for 'UseMvc' and no extension method 'UseMvc' accepting a first argument of type 'IApplicatio
nBuilder' could be found (are you missing a using directive or an assembly reference?)
E:\oss\aspnet\benchmarks\src\Benchmarks\Controllers\HomeController.cs(30,20): DNXCore,Version=v5.0 error CS0103: The nam
e 'View' does not exist in the current context
E:\oss\aspnet\benchmarks\src\Benchmarks\Controllers\FortunesController.cs(27,20): DNXCore,Version=v5.0 error CS0103: The
name 'View' does not exist in the current context
E:\oss\aspnet\benchmarks\src\Benchmarks\Controllers\FortunesController.cs(33,20): DNXCore,Version=v5.0 error CS0103: The
name 'View' does not exist in the current context
E:\oss\aspnet\benchmarks\src\Benchmarks\Controllers\FortunesController.cs(39,20): DNXCore,Version=v5.0 error CS0103: The
name 'View' does not exist in the current context
Something different?
What is your pipeline.lua looks like? Can you share the script?
Current benchmark automation is totally fine with socket errors occurring and will ignore them. The system needs to totally crash and burn (without reporting perf numbers) if there's any error from the client.
We think the current round of errors we're seeing are an artifact of wrk
and not of the server, but that's beside the point π
Registered RIO //build/ announce from 2011
http://channel9.msdn.com/events/Build/BUILD2011/SAC-593T
Microsoft Windows 8 and Windows Server 2012 introduce new Windows Sockets programming elements.
A set of high-speed networking extensions are available for increased networking performance with lower latency and jitter. These extensions targeted primarily for server applications use pre-registered data buffers and completion queues to increase performance.
The following are new Windows Sockets functions added to support Winsock high-speed networking Registered I/O extensions:
https://msdn.microsoft.com/en-us/library/windows/desktop/ms740642(v=vs.85).aspx
Should be able to easily run the benchmark using a script (DNX console app?) in this repo (on both Windows & *nix). It would do something like the following:
wrk
dnx --configuration Release --project src/Benchmarks kestrel
)wrk
on *nix load server (following taken from https://www.techempower.com/benchmarks/#section=motivation&hw=peak&test=fortune)
I wanted to try this but when I run Benchmarks asp vnext web app, I get the following error in visual studio. I get the same error with dnx with and beta5 (clr\coreclr x86\x64).
I have the latest VS 2015 and beta5 and I am not behind proxy. dnu restore works just fine.
C:\Projects\git\benchmarks\src\Benchmarks>dnx . web
System.TypeLoadException: Could not load type 'Microsoft.Framework.Runtime.IRuntimeServices' from assembly 'Microsoft.Fr
amework.Runtime.Abstractions, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null'.
at Microsoft.AspNet.Hosting.WebHostBuilder.BuildHostingServices()
at Microsoft.AspNet.Hosting.WebHostBuilder.Build()
at Microsoft.AspNet.Hosting.Program.Main(String[] args)
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at Microsoft.Framework.Runtime.Common.EntryPointExecutor.Execute(Assembly assembly, String[] args, IServiceProvider s
erviceProvider)
at Microsoft.Framework.ApplicationHost.Program.ExecuteMain(DefaultHost host, String applicationName, String[] args)
at Microsoft.Framework.ApplicationHost.Program.Main(String[] args)
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at Microsoft.Framework.Runtime.Common.EntryPointExecutor.Execute(Assembly assembly, String[] args, IServiceProvider s
erviceProvider)
at dnx.host.Bootstrapper.RunAsync(List`1 args, IRuntimeEnvironment env)
at dnx.host.RuntimeBootstrapper.ExecuteAsync(String[] args)
at dnx.host.RuntimeBootstrapper.Execute(String[] args)
We should transition the database scenarios to no longer rely on EF migrations. For instance, if one were to try and only run the Dapper scenario there's no need to use EF.
Lets come up with a unified set of migration-like sql scripts π
/cc @DamianEdwards
The DB Update test (#5 at this link) should also be implemented with the same set of tests as the query tests.
See requirements at https://www.techempower.com/benchmarks/#section=code
We should have a middleware version and an MVC version.
@davidfowl mentioned on slack that there are now automated runs that publish results at least for the plain text benchmarks, but that these are only published internally. Please consider making the results available externally or at least updating the readme. It seems there's been some pretty significant progress :)
Also, is the techempower hardware the same as smurflab or is that another setup?
@Eilon @Tratcher
I am not sure where to put my original issue so I just leave it here.
My original issue is: aspnet/Announcements#133
@Eilon I am recently update the target framework from dnxcore50 to dotnet5.4 based on your annoucement "Switching to new βdotnetβ target framework monikers", when we are updating to rc1-final/update1
One of our class library using TcpClient in System.Net.Sockets. When I update to dotnet5.4, it won't build because it cannot find TcpClient. After change our project.json from dotnet5.4 to dotnet5.5 it working fine. So what is the best practice for this? Should I change my project.json to dotnet5.5 and change #if DNXCORE50 to #if DOTNET5_5 for ALL my other library?
It seems on nuget.org only System.Net.Sockets support 5.4 and 5.5, all other packages only support dotnet5.4, what is the compatibility here?
The benchmark says ASP.NET Core on Kestrel perfsvr = 1,174,881...is this RC2 speed?
If so how fast would RC1 run in the same scenario or did I miss something?
Requirements are at https://www.techempower.com/benchmarks/#section=code
We should have a middleware version and an MVC version.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
π Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. πππ
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google β€οΈ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.