Show that locust can saturate the gas limits and describe chain behavior about nearcore HOT 5 OPEN

I'm still unable to get blocks with >1000Tgas :(

So far I resolved 2 problem that prevented me from hitting the gas limit:

• Load generator CPU bottleneck: Swarming locust across enough threads (I ended up using 32 threads) where each has it's own funding account to avoid Nonce collisions. (resolved by #9111)
  • This problem shows up as a warning by locust that CPU usage is > 90%
• RPC node bottleneck: If all requests are going thorugh the same RPC node, his node will become a bottleneck in accepting more TXs. (resolved by using different -H args for different workers)
  • This problem shows up as TIMEOUT_ERROR on the requests to RPC nodes, reported as 'No result returned' in locust statistics

But even with that, I am not quite able to saturate even a single shard the way I was hoping to do.

I ran a 4 shard, 4 node localnet. And locust with 6000 users spanning across 32 workers, each with 4 separate FT contracts. The 32 workers send their request to 2 different RPC nodes. This setup peaked around 900 TPS, with only about 75% of gas capacity on each shard. (evenly distributed)

Note: 900TPS peak throughput corresponds to 900 * ~5 Tgas = 4500Tgas per second. With a block time of 1.3s that means 4500Tgas / 1.3s = 3461 Tgas / block which is about 86% of the 4000Tgas capacity.
The expected throughput at 100% gas capacity would be around 1050TPS.

Looking at the response time going up significantly starting at around 3800 users, it suggests that we are hitting a bottleneck there. But this has only about 750 TPS, far below the 1050TPS I want to see. So I need to figure out what the current bottleneck is. Trying more than 2 RPC nodes next.

cc @akhi3030 maybe you have some ideas regarding the bottleneck, or see flaws in my reasoning?

from nearcore.

I've repeated the experiment with more RPC nodes - same results.

Then I run with just a single shard. (thanks @akhi3030 for the idea!) Then I was hitting a limit at around 900 users, with again chunks never filling up. They are stuck at around 750Tgas again.

But after that, I figured out one big factor: Compute Costs! FT calls are doing a decent amount of storage requests, which means they are charged a higher compute cost than the gas cost. Removing compute cost parameters gives me almost full chunks, but sadly still not quite.

With ~4200 users I'm getting close to ~900 TPS with still a mostly stable median response time of 2.5s.
Going up all the way to 7000 users, I see short spikes of up to 1000 TPS and chunks filled up to 910 Tgas. The response time goes up to ~5.5s median, so things must be queuing up somewhere. But still it's not quite the gas limit we are hitting.

Next week I'll integrate it with Prometheus and Grafana to get more data about what the nodes are doing.

from nearcore.