Idea: Introduce additional parallelism within application of a single shard chunk about nearcore HOT 4 OPEN

In an effort to implement this, the very first thing that will need to happen is a refactor of the VM runner.

There are a couple of issues to resolve:

• Split up the near_vm_runner::run method into an instantiate and run_method;
  • not quite as written, but #11667 implements this in a way that's agnostic over where exactly the split occurs.
• Instantiation must be split up so that the start method can be run at a later time. This may be particularly troublesome for VMs that we don't control implementation of. For instance wasmtime provides "pre-instantiate" sort of method, but that does less work than we'd ideally like.
• In near_vm_runner we have the split here already, but the instantiation is mixed up together with the loading of the executable, which is probably something we should fix as well.
  • All the caches make things more difficult than they ideally should be...
  • We may need to keep the cache of loaded contracts for use-cases like ReadRPC which will not be able to predict upcoming requests, but we could possibly drop that cache in nearcore.
• Figure out what type represents an instance: do we store it within a VM (and make the VM only able to run a single method?) or do we do some type erasure shenanigans to obtain a type that makes sense to return from an object-safe trait?
• Implementing this requires significant-enough changes to the code structure to require a protocol version change. How do we maintain backward compatibility when the old runtimes cannot allow for the same sort of logic flow...?

from nearcore.

cc #11808

from nearcore.

I have a question: instead of thinking about dynamic vs. static, could we just use an approach where at the beginning of apply, we eagerly go over receipts and load contracts into a queue, then we when need to execute a function call receipt, we pop prepared contracts from the front of queue. We need a concurrent queue here, but otherwise it should be easier to implement.

from nearcore.

I would argue that the described solution is a form of a static approach where N of preloaded contracts is implicitly bounded by available memory or the maximum number of delayed+incoming+local receipts. If we don't carefully manage the number of prepared receipts we'd be looking at >1 TiB of memory to store the prepared contracts (assuming ~20k of delayed receipts and 64MiB of contract data memory per prepared contract.)

Traditional queues are somewhat poor of a data structure for this problem too. For instance local receipts are constructed within apply and processed right after they are constructed before delayed or incoming receipts. But then if those local receipts do not fit into the current chunk, they get pushed to the back of the delayed receipt queue. As thus the ordering in which the receipts get executed is quite dynamic in the grand scheme of things, and is somewhat ambiguous at the beginning of apply.

For these reasons I don't think ordered data structures or starting preparation work in apply are the right way forward.

from nearcore.