Horizon Prerequisites Benchmarking about go HOT 6 OPEN

Created https://github.com/stellar/ops/issues/2536 request for provisioning new instances.

Hello @urvisavla , I left a comment for considertaion of using k8s for provisioning new instances rather than ec2:
https://github.com/stellar/ops/issues/2536#issuecomment-1728517511

from go.

Some unknowns to investigate here. Notably, how much API load we assume that users will have. Potentially use existing goreplay setup but filtered/reduced depending on what we decide.

from go.

@urvisavla , during verification of compute resources, wanted to mention it should include ENABLE_CAPTIVE_CORE=true and CAPTIVE_CORE_USE_DB=true, I think those are the defaults at this point. Since, captive core with disk db usage will dramatically lower the amount of RAM used by captive, current pre-reqs in docs mention 32GB of ram required, but with on-disk usage, that should be well under 8GB in almost all cases if not lower - #4092 (comment)

from go.

@urvisavla , during verification of compute resources, wanted to mention it should include ENABLE_CAPTIVE_CORE=true and CAPTIVE_CORE_USE_DB=true, I think those are the defaults at this point. Since, captive core with disk db usage will dramatically lower the amount of RAM used by captive, current pre-reqs in docs mention 32GB of ram required, but with on-disk usage, that should be well under 8GB in almost all cases if not lower - #4092 (comment)

@sreuland
We observed RAM usage on the ingestion instance (dev cluster) to remain below 8GB, usually hovering around 6GB. However, during state-verification, the RAM usage spikes to 11GB and remain so for the entire duration of state-verification. Meanwhile, the memory usage on the API instance (prod cluster) remains consistently below 3GB. I believe that the main contributor to memory usage is the in-memory graph for path payments.

Considering these observations and given that our recommendations are for an instance serving all functions (API + ingestion), 16GB RAM should be adequate. I will update our documentation to reflect this recommendation.

from go.

Update:

• Shared a document with the team detailing observations from EC2 and RDS instances from the dev and prod clusters

• Updated the hardware specifications, including CPU, memory, and disk, in our public docs within the partner-experience branch (to be merged to the main branch soon).

• Unfortunately, couldn't obtain hardware benchmarks for running an API instance due to the absence of API traffic in both staging and dev clusters.

• Explored options like using the 'go-replay' tool for mirroring traffic, but it proved to be unfeasible #2461.

• Next steps:
  Explore developing a custom tool to simulate requests from prod (using logs from AWS) and replay them on dev cluster. And for that we'd want to use instances with specifications similar to what we plan to recommend in our public docs. Created ops request for provisioning new instances.

from go.

@urvisavla , you mentioned a performance benchmarks doc was shared, can it be linked or summ'd here also? Thanks!

from go.