Description

The CI workflow requires enhancements to handle benchmark execution and updating the README file correctly. We need to address the following scenarios:

1. Main Branch: When changes are pushed to the main branch, the CI should run benchmarks and update the README file with the results, committing these changes back to the main branch.
2. Pull Requests: For pull requests, the CI should run benchmarks but not update the README or commit any changes. This step is to ensure benchmarks run successfully without altering the repository.
3. Failure Handling: The CI should fail if it encounters any issues in generating or updating benchmark results, making these failures visible for prompt resolution.

Expected Behavior

• On Main Branch:

  • Run benchmarks.
  • If benchmarks run successfully, update the README file with the results and commit these changes to the main branch.
  • If there is a failure in running benchmarks or updating the README, the CI should fail and report the error.

• On Pull Requests:

  • Run benchmarks without updating the README or committing any changes.
  • CI should fail if there is an issue in generating benchmark results.

Acceptance Criteria

1. CI updates to run benchmarks on the main branch and update the README file with the results, followed by committing these changes.
2. CI updates to run benchmarks on pull requests without updating the README or committing any changes.
3. Ensure CI fails and reports errors if benchmark generation or README update fails.

Running the build on PRs is constantly causing merge conflicts because all of them keep updating the README and the image files.

Technical Requirements

• Run the benchmarks as it is without committing anything into git.
• Once the results are ready print them on the CI logs, as a separate step.

Here is the log on the benchmark job:

[2023-11-17T14:55:22.168Z] #8 0.296 
#8 0.296   The NodeSource Node.js Linux distributions GitHub repository contains
#8 0.296   information about which versions of Node.js and which Linux distributions
#8 0.296   are supported and how to install it.
#8 0.296   https://github.com/nodesource/distributions
#8 0.296 
#8 0.296 
#8 0.296                           SCRIPT DEPRECATION WARNING
#8 0.296 
#8 0.296 ================================================================================
#8 0.297 ▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓
#8 0.297 ================================================================================
#8 0.297 
#8 0.297 TO AVOID THIS WAIT MIGRATE THE SCRIPT

This slows down the benchmark as the script sleeps for 60 seconds before continuing.

Currently all servers start on different ports.

.devcontainer/Dockerfile

EXPOSE 8080 8081 8082 8083 8084 3000

Ideally we would like to expose every service on a single port 8000. While running the tests, we will start the service, run the benchmark and stop the service completely only after which move on to the next service.

Technical Requirements

• Run benchmarks for all PRs on benchmarking-runner runner.
• It should follow the steps we have documented and automatically generate
  1. The current README.md with a table sorted highest to lowest in terms by RPS.
  2. A histogram of RPS
  3. A histogram of Latency

Hi, it seems that since #28 the Nginx proxy doesn't work as expected, which means that the graphql endpoints are returning errors. This largely explains why both Taillcall and Gqlgen saw a massive increase in throughput and RPS after the PR was merged in, since their underlying server / client is more performant than Caliban's. Also likely explains why DGS is crashing (probably doesn't handle the client failures well).

Steps to reproduce:

./nginx/run.sh
./graphql/tailcall/run.sh &
sleep 2
./test_query.sh

Running the above returns:

{"data":null,"errors":[{"message":"IOException: Request error: error sending request for url (http://jsonplaceholder.typicode.com/posts): error trying to connect: tcp connect error: Connection refused (os error 111)","locations":[{"line":1,"column":2}]}]}

A GH action has been recently introduced to run when a PR is raised. But, this is failing at checkout if the PR is from forked repos.

This issue lists Renovate updates and detected dependencies. Read the Dependency Dashboard docs to learn more.

Open

These updates have all been created already. Click a checkbox below to force a retry/rebase of any.

• fix(deps): update dependency @tailcallhq/tailcall to v0.87.0

Detected dependencies

• Check this box to trigger a request for Renovate to run again on this repository

• Use consistent naming convention for files. Stick to kebab case.
• Move images to a separate folder
• Display images on the readme
• Update readme with steps to run it on code spaces
• Add a link to setup and run on codespaces
• Double the resolution of the images
• Add the actual data in tabular format also on the Readme
• Fix the legend getting clipped.
• Add website links instead of github links
• Need a way to configure load externally
• Add architecture diagram
• Add benchmark description

Technical Requirements

• Use latest version of all the libraries
• Use connection pooling for the http client
• Make sure localhost:3000 is used as a proxy for http client
• Implement the graphQL schema as explained here https://github.com/tailcallhq/graphql-benchmarks#graphql-schema
• Ensure not batching and caching is happening in the http client.