tailcallhq / graphql-benchmarks Goto Github PK

Hi, it seems that since #28 the Nginx proxy doesn't work as expected, which means that the graphql endpoints are returning errors. This largely explains why both Taillcall and Gqlgen saw a massive increase in throughput and RPS after the PR was merged in, since their underlying server / client is more performant than Caliban's. Also likely explains why DGS is crashing (probably doesn't handle the client failures well).

Steps to reproduce:

./nginx/run.sh
./graphql/tailcall/run.sh &
sleep 2
./test_query.sh

Running the above returns:

{"data":null,"errors":[{"message":"IOException: Request error: error sending request for url (http://jsonplaceholder.typicode.com/posts): error trying to connect: tcp connect error: Connection refused (os error 111)","locations":[{"line":1,"column":2}]}]}

Running the build on PRs is constantly causing merge conflicts because all of them keep updating the README and the image files.

Technical Requirements

• Run the benchmarks as it is without committing anything into git.
• Once the results are ready print them on the CI logs, as a separate step.

Update the benchmark results in the README.md file into the following format

This issue lists Renovate updates and detected dependencies. Read the Dependency Dashboard docs to learn more.

Open

These updates have all been created already. Click a checkbox below to force a retry/rebase of any.

• fix(deps): update dependency @tailcallhq/tailcall to v0.96.12
• fix(deps): update dependency graphql to v16

Detected dependencies

• Check this box to trigger a request for Renovate to run again on this repository

Like we have the new N + 1 benchmark add a simple "Hello World" benchmark

query { greet }

{
  data: {greet: "Hello World!"}
}

Technical Requirements

• Ensure that there is no duplication in this PR, any thing that can be abstracted to a common utility should be.
• Results should be embedded into the README file.
• Update all existing implementation for the "Hello World" benchmark.

• Use consistent naming convention for files. Stick to kebab case.
• Move images to a separate folder
• Display images on the readme
• Update readme with steps to run it on code spaces
• Add a link to setup and run on codespaces
• Double the resolution of the images
• Add the actual data in tabular format also on the Readme
• Fix the legend getting clipped.
• Add website links instead of github links
• Need a way to configure load externally
• Add architecture diagram
• Add benchmark description

Add Hasura to the benchmarks to compare GraphQL performance

Currently the benchmarks run in sequence one after the other. We want to change that so that all the benchmarks can run in parallel. For each server start with the "greet" query, moving on to a list of posts and then to posts with users.

Technical Requirements

• Ensure each GraphQL server runs independent of the other in a single runner.

A GH action has been recently introduced to run when a PR is raised. But, this is failing at checkout if the PR is from forked repos.

Description

The CI workflow requires enhancements to handle benchmark execution and updating the README file correctly. We need to address the following scenarios:

1. Main Branch: When changes are pushed to the main branch, the CI should run benchmarks and update the README file with the results, committing these changes back to the main branch.
2. Pull Requests: For pull requests, the CI should run benchmarks but not update the README or commit any changes. This step is to ensure benchmarks run successfully without altering the repository.
3. Failure Handling: The CI should fail if it encounters any issues in generating or updating benchmark results, making these failures visible for prompt resolution.

Expected Behavior

• On Main Branch:

  • Run benchmarks.
  • If benchmarks run successfully, update the README file with the results and commit these changes to the main branch.
  • If there is a failure in running benchmarks or updating the README, the CI should fail and report the error.

• On Pull Requests:

  • Run benchmarks without updating the README or committing any changes.
  • CI should fail if there is an issue in generating benchmark results.

Acceptance Criteria

1. CI updates to run benchmarks on the main branch and update the README file with the results, followed by committing these changes.
2. CI updates to run benchmarks on pull requests without updating the README or committing any changes.
3. Ensure CI fails and reports errors if benchmark generation or README update fails.

Seems that tailcall went from ~7k reqs/s to ~60k reqs/s even for the baseline query that doesn't contain N+1 queries.

Is this an issue with the benchmark or shall I add global request batching to Caliban as well?

Also, what's the reason for not using the batching endpoint of users such as /users?id=1&id=2 etc? I run the benchmark locally and more than 50-60% of the CPU was being used by the nginx proxy, meaning that we're no longer benchmarking graphql performance

Technical Requirements

• Use latest version of all the libraries
• Use connection pooling for the http client
• Make sure localhost:3000 is used as a proxy for http client
• Implement the graphQL schema as explained here https://github.com/tailcallhq/graphql-benchmarks#graphql-schema
• Ensure not batching and caching is happening in the http client.

Here is the log on the benchmark job:

[2023-11-17T14:55:22.168Z] #8 0.296 
#8 0.296   The NodeSource Node.js Linux distributions GitHub repository contains
#8 0.296   information about which versions of Node.js and which Linux distributions
#8 0.296   are supported and how to install it.
#8 0.296   https://github.com/nodesource/distributions
#8 0.296 
#8 0.296 
#8 0.296                           SCRIPT DEPRECATION WARNING
#8 0.296 
#8 0.296 ================================================================================
#8 0.297 ▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓
#8 0.297 ================================================================================
#8 0.297 
#8 0.297 TO AVOID THIS WAIT MIGRATE THE SCRIPT

This slows down the benchmark as the script sleeps for 60 seconds before continuing.

Add benchmarks for GraphQL JIT

Add all the existing benchmarks

Shell files are hard to manage, test and maintain. Re-write the complete logic in JS.

The user fetched for the posts seems to be incorrect in the gqlgen. I noticed this inconsistency in the CI.

{
  "data": {
    "posts": [
      {
        "id": 1,
        "userId": 1,
        "title": "sunt aut facere repellat provident occaecati excepturi optio reprehenderit",
        "user": {
          "id": 6,
          "name": "Mrs. Dennis Schulist",
          "email": "[email protected]"
        }
      }, ...
    ]
  }
}

Technical Requirements

• Make a request to the respective servers with

   posts id userId title user { id name email }

• Ensure the existing benchmarks don't break
• Run the benchmark for 30 seconds
• Use a data-loader to batch requests to /user/:id
• NOTE: Do not make a bulk API call to load all users.
• Test results should be combined with the previous tests and
  1. Comment on the commit like we do right now for each PR
  2. Update README with the new set of tests

Currently all servers start on different ports.

.devcontainer/Dockerfile

EXPOSE 8080 8081 8082 8083 8084 3000

Ideally we would like to expose every service on a single port 8000. While running the tests, we will start the service, run the benchmark and stop the service completely only after which move on to the next service.

Technical Requirements

• Run benchmarks for all PRs on benchmarking-runner runner.
• It should follow the steps we have documented and automatically generate
  1. The current README.md with a table sorted highest to lowest in terms by RPS.
  2. A histogram of RPS
  3. A histogram of Latency

Update all the links of the GraphQL servers to their open-source Github repository.