kitura / kitura-benchmarks Goto Github PK

In TechEmpower, the queries and updates routes fail with the queries parameters set to a value greater than 1 (say, queries=5 or queries=10).

These are the failure messages with Kitura-Net:

[2018-11-28T09:46:41.649Z] [WARNING] [RouterResponse.swift:440 send] RouterResponse send(str:) invoked after end() for http://localhost/updates?queries=5
[2018-11-28T09:46:41.649Z] [WARNING] [RouterResponse.swift:185 end()] RouterResponse end() invoked more than once for http://localhost/updates?queries=5
[2018-11-28T09:46:43.058Z] [ERROR] [main.swift:164 TechEmpowerPostgres] OtherError("Timed out waiting for a DB connection from the pool")
[2018-11-28T09:46:43.058Z] [WARNING] [RouterResponse.swift:440 send] RouterResponse send(str:) invoked after end() for http://localhost/updates?queries=5

... and these are the failure messages with Kitura-NIO:

2018-11-28T10:08:40.909Z] [WARNING] [RouterResponse.swift:185 end()] RouterResponse end() invoked more than once for http://localhost:8080/updates?queries=5
[2018-11-28T10:08:40.909Z] [WARNING] [RouterResponse.swift:185 end()] RouterResponse end() invoked more than once for http://localhost:8080/updates?queries=5
[2018-11-28T10:08:40.908Z] [ERROR] [HTTPServerResponse.swift:123 end()] No channel available to end the response
[2018-11-28T10:08:40.907Z] [ERROR] [HTTPServerResponse.swift:90 write(from:)] No channel available to write
[2018-11-28T10:08:40.908Z] [ERROR] [HTTPServerResponse.swift:90 write(from:)] No channel available to write
[2018-11-28T10:08:40.907Z] [ERROR] [HTTPServerResponse.swift:90 write(from:)] No channel available to write

cc @djones6

Includes:

• Modifying the TechEmpower docker files to use Swift 5
• Adopting the new CouchDB API

Context & Description

Evaluate the asynchronous changes to Kuery and Kuery-PostgreSQL using the updated TechEmpower benchmarks.

This will allow the /queries and /updates routes to function recursively, rather than the current workaround of using /updatesParallel and /queriesParallel (executing multiple finds in parallel, and using a semaphore to keep track of when all have completed - which blocks a thread).

Background

The issue with the recursive definition for updates and queries is that, because closures are currently invoked synchronously, the connection used for query N is not returned to the pool until the end of that closure - unavailable for query N+1 (invoked from within the closure).

For queries > 1 and many concurrent clients, eventually the benchmark will deadlock, because there are a number of clients waiting for a connection that they can never have because the pool has become empty, and preventing the connections that were just used from returning to the pool.

eg:

djones6@gruffalo:~$ wrk -c256 -t2 -d10s -i1s http://localhost:8080/queries?queries=3
Running 10s test @ http://localhost:8080/queries?queries=3
  2 threads and 256 connections
   1s: 7337 requests in last 1s (   7337.0 req/sec)
   2s: 7501 requests in last 1s (   7501.0 req/sec)
   3s: 7524 requests in last 1s (   7524.0 req/sec)
   4s: 7472 requests in last 1s (   7472.0 req/sec)
   5s: 7434 requests in last 1s (   7434.0 req/sec)
   6s: 7369 requests in last 1s (   7369.0 req/sec)
   7s: 7203 requests in last 1s (   7203.0 req/sec)
   8s: 6991 requests in last 1s (   6991.0 req/sec)
   9s: 80 requests in last 1s (     80.0 req/sec)
  10s: 0 requests in last 1s (      0.0 req/sec)
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency    33.53ms    3.87ms 234.91ms   81.76%
    Req/Sec     3.68k   296.36     4.03k    95.65%
  58911 requests in 10.03s, 256 connections, 15.08MB read
Requests/sec:   5873.08
Requests/conn:   230.12
Transfer/sec:      1.50MB

Clear & Actionable Tasks

• Test TechEmpower using https://github.com/IBM-Swift/Swift-Kuery-PostgreSQL/tree/issue_async
• Evaluate queries and updates under stress, up to 20 queries and 256 concurrent clients

Exit Criteria:

This issue is complete once the following criteria are met:

• Code...
• Tests...

As discussed and decided upon, this is the priority in terms of performance testing for Kitura-NIO:

Measurements with microbenchmarks with Kitura-NIO vs. Kitura-net
Measurements with the same microbenchmarks on Vapor
Measurements with the same microbenchmarks on Express (All the above can be on Ubuntu 16.04)
Measurements of comparing 18.04 to 16.04
Getting TechEmpower running on Kitura-NIO and comparing to Kitura-net

The handling of CPU affinity within the driver scripts is a bit of a mess.

Originally this was hacked together to work on specific NUMA machines where it is desirable for the process under test to be affinitized to a subset of physical resources (all or part of a single socket). Recently (#3) this hard-coded affinity was replaced with a host-specific script which should define how the process(es) are affinitized.

Now what we have is a CPUS variable used by compare.sh, which defaults to 0-3, passed in to a CPULIST variable used by drive.sh. The CPULIST is used for two purposes: to determine which CPUs are included in activities such as monitoring CPU utilisation, and to actually restrict the process to those CPUs. The latter only works if either drive.sh itself is customized, or a host-specific script has been provided, and that file references CPULIST (as the provided example does).

This is all very fragile and can easily lead to someone running these scripts manually but getting no CPU affinity because they forgot to include a host script. Or CPU utilisation information that is inconsistent with the affinity, because it didn't correctly reflect CPULIST.

We should detect and then warn (or fail) in cases where we have (explicitly or implicitly) asked for CPU affinity, but are not going to get it because the host-specific commands have not been provided.