Comments (15)
Ekleog-NEAR
commented on August 30, 2024
1
Here is what the numbers of the parameter estimator out of 12adab4 (with the log line removed) look like:
(The full data for the runs used for these graphs is available here)
So memtrie does bring in very significant benefits on estimator results! Though it also seems to make writing a bit slower at times. Thanksfully we already have a very large safety margin on writes, so we should be able to just ignore the increase in costs here. And the most significant benefit of memtrie is that it removes all disk access from the code path, meaning that the variance should be much lower, and thus that we can take a much smaller safety margin compared to the previous 15G -> 200G safety margin that we needed to account for disk limits under load.
With all that in mind, I suggest targeting a roughly 20~30% safety margin above estimator 90th percentile, because memtrie should have a pretty low variance:
StorageHasKeyBase: 200,000,000,000 -> 8,000,000,000, without 4k split because checking whether the key exists should AFAIU never require falling back to disk (⚠️ assumption that needs to be confirmed by someone familiar with memtrie)StorageHasKeyByte: 30,790,845 -> 9,000,000, without 4k split (⚠️ same assumption)StorageReadBase: 200,000,000,000 -> same if the value is 4k or more; 9,000,000,000 if the value is strictly less than 4kStorageReadKeyByte: 30,952,533 -> same if the value is 4k or more; 10,000,000 otherwiseStorageReadValueByte: 5,611,005 -> same if the value is 4k or more; 2,500,000 otherwiseStorageWrite*: do not changeStorageRemove*: do not changeTouchingTrieNode: 110,000,000,000 -> 6,000,000,000, without 4k split (⚠️ same assumption)ReadCachedTrieNode: 2,280,000,000 -> 1,400,000,000, we probably do not need the 2x safety margin that we originally had, even though memtrie did not make things much faster the estimator is still around 1,070,000,000 only
My next step will be making the protocol version change that does this split. It'll be my first time adding a cost so it might take a bit longer than initially expected.
@Longarithm / @tayfunelmas can you check in particular the assumptions in three places that have the
Also, what do you think about these numbers? They seem to be reasonable to me, but are they to you too?
Next steps
Considering the recent discussion, I'm removing 4 and 5 from the next steps listed above. The next steps thus are:
- Get feedback on whether the numbers look fine, and the assumptions seem reasonable
- Implement the protocol change that updates the current costs and adds the less-than-4k costs where applicable
- Land the PR and rely on release mocknet testing to validate that it does not break too much.
- In parallel, make sure #11963 lands
from nearcore.
Ekleog-NEAR
commented on August 30, 2024
Notes from today's meeting with @tayfunelmas and @akhi3030:
- What are the costs we expect should change with memtrie?
- Storage reads:
- Read base/key byte/value byte, has key base/byte, TTN
- Storage writes: it will first go through memtrie then to disk
- Write *, Remove *
- Don’t reevaluate the storage write costs for now, it’ll wait for when writes get out of the critical path, or even get eliminated completely altogether
- Storage reads:
- What are their expected new order of magnitude, eg. compared to memory reads?
- Between 1.3x and 10x faster probably
- Estimator uses directly ShardTries, so it could be by default, but how can I confirm with certainty that I’m not measuring without memtrie enabled?
- Action: send link to code to tayfun
- How can I make sure memtrie is enabled in the parameter estimator eg. in a profiler results?
- There should be a function that we can check for in the profiler
- Just look for this function:
nearcore/core/store/src/trie/mod.rs
Line 1366 in 3d4f554
- We can ignore qemu-based computation
- Run with and without shard cache
- What hardware to run on? N2-standard-8?
- Yes for MVP, then consider running on multiple machines later maybe
from nearcore.
Ekleog-NEAR
commented on August 30, 2024
I ended up not being able to run on n2-standard-8 due to some illegal instructions, most likely the newer sha256 instructions due to n2 being too old a cpu architecture. Switched to the amd-based n2d-standard-8, and got a few runs out.
Command used, always with 1 warmup iter but with a number of actual iterations that varies:
time cargo run --release --package runtime-params-estimator --features required --bin runtime-params-estimator -- --accounts-num 20000 --additional-accounts-num 200000 --iters 1 --warmup-iters 1 --metric time --costs StorageWriteBase,StorageWriteKeyByte,StorageWriteValueByte,StorageWriteEvictedByte,StorageReadBase,StorageReadKeyByte,StorageReadValueByte,StorageRemoveBase,StorageRemoveKeyByte,StorageRemoveRetValueByte,StorageHasKeyBase,StorageHasKeyByte,TouchingTrieNode,ReadCachedTrieNode
1 iter
StorageHasKeyBase 12_900_400_629 gas [ 12.9µs] UNCERTAIN (computed in 140.18s) HIGH-VARIANCE: runtime/runtime-params-estimator/src/utils.rs:354:24
StorageHasKeyByte 32_974_229 gas [ 33ns] UNCERTAIN (computed in 48.62s) HIGH-VARIANCE: runtime/runtime-params-estimator/src/utils.rs:354:24
StorageReadBase 13_176_595_629 gas [ 13.177µs] UNCERTAIN (computed in 37.94s) HIGH-VARIANCE: runtime/runtime-params-estimator/src/utils.rs:354:24
StorageReadKeyByte 32_950_083 gas [ 33ns] UNCERTAIN (computed in 48.77s) HIGH-VARIANCE: runtime/runtime-params-estimator/src/utils.rs:354:24
StorageReadValueByte 1_996_159 gas [ 2ns] UNCERTAIN (computed in 42.34s) HIGH-VARIANCE: runtime/runtime-params-estimator/src/utils.rs:354:24
StorageWriteBase 5_894_729_584 gas [ 5.895µs] UNCERTAIN (computed in 38.77s) HIGH-VARIANCE: runtime/runtime-params-estimator/src/utils.rs:354:24
StorageWriteKeyByte 6_054_362 gas [ 6ns] UNCERTAIN (computed in 66.78s) HIGH-VARIANCE: runtime/runtime-params-estimator/src/utils.rs:354:24
StorageWriteValueByte 2_372_167 gas [ 2ns] UNCERTAIN (computed in 40.39s) HIGH-VARIANCE: runtime/runtime-params-estimator/src/utils.rs:354:24
StorageWriteEvictedByte 3_502_758 gas [ 4ns] UNCERTAIN (computed in 42.64s) HIGH-VARIANCE: runtime/runtime-params-estimator/src/utils.rs:354:24
StorageRemoveBase 13_325_130_629 gas [ 13.325µs] UNCERTAIN (computed in 38.65s) HIGH-VARIANCE: runtime/runtime-params-estimator/src/utils.rs:354:24
StorageRemoveKeyByte 12_766_994 gas [ 13ns] UNCERTAIN (computed in 50.31s) HIGH-VARIANCE: runtime/runtime-params-estimator/src/utils.rs:354:24
StorageRemoveRetValueByte 1_660_624 gas [ 2ns] UNCERTAIN (computed in 42.21s) HIGH-VARIANCE: runtime/runtime-params-estimator/src/utils.rs:354:24
TouchingTrieNode 8_087_650_976 gas [ 8.088µs] UNCERTAIN (computed in 39.79s) HIGH-VARIANCE: runtime/runtime-params-estimator/src/utils.rs:354:24
ReadCachedTrieNode 838_062_500 gas [ 838ns] UNCERTAIN (computed in 120.42s) HIGH-VARIANCE: runtime/runtime-params-estimator/src/utils.rs:369:13
Finished in 879.35s, output saved to:
2 iters
StorageHasKeyBase 15_193_494_159 gas [ 15.193µs] UNCERTAIN (computed in 116.28s) HIGH-VARIANCE: runtime/runtime-params-estimator/src/utils.rs:354:24
StorageHasKeyByte 35_605_815 gas [ 36ns] UNCERTAIN (computed in 50.08s) HIGH-VARIANCE: runtime/runtime-params-estimator/src/utils.rs:354:24
StorageReadBase 13_538_829_159 gas [ 13.539µs] UNCERTAIN (computed in 35.86s) HIGH-VARIANCE: runtime/runtime-params-estimator/src/utils.rs:354:24
StorageReadKeyByte 33_160_425 gas [ 33ns] UNCERTAIN (computed in 49.22s) HIGH-VARIANCE: runtime/runtime-params-estimator/src/utils.rs:354:24
StorageReadValueByte 2_006_289 gas [ 2ns] UNCERTAIN (computed in 41.24s) HIGH-VARIANCE: runtime/runtime-params-estimator/src/utils.rs:354:24
StorageWriteBase 6_010_015_762 gas [ 6.01µs] UNCERTAIN (computed in 34.66s) HIGH-VARIANCE: runtime/runtime-params-estimator/src/utils.rs:354:24
StorageWriteKeyByte 6_186_676 gas [ 6ns] UNCERTAIN (computed in 91.99s) HIGH-VARIANCE: runtime/runtime-params-estimator/src/utils.rs:354:24
StorageWriteValueByte 2_401_317 gas [ 2ns] UNCERTAIN (computed in 38.72s) HIGH-VARIANCE: runtime/runtime-params-estimator/src/utils.rs:354:24
StorageWriteEvictedByte 3_662_605 gas [ 4ns] UNCERTAIN (computed in 41.88s) HIGH-VARIANCE: runtime/runtime-params-estimator/src/utils.rs:354:24
StorageRemoveBase 13_380_131_159 gas [ 13.38µs] UNCERTAIN (computed in 35.62s) HIGH-VARIANCE: runtime/runtime-params-estimator/src/utils.rs:354:24
StorageRemoveKeyByte 12_818_790 gas [ 13ns] UNCERTAIN (computed in 52.65s) HIGH-VARIANCE: runtime/runtime-params-estimator/src/utils.rs:354:24
StorageRemoveRetValueByte 1_669_679 gas [ 2ns] UNCERTAIN (computed in 40.42s) HIGH-VARIANCE: runtime/runtime-params-estimator/src/utils.rs:354:24
TouchingTrieNode 6_168_517_526 gas [ 6.169µs] UNCERTAIN (computed in 35.43s) HIGH-VARIANCE: runtime/runtime-params-estimator/src/utils.rs:354:24
ReadCachedTrieNode 938_437_500 gas [ 938ns] (computed in 158.17s)
Finished in 867.08s, output saved to:
3 iters
StorageHasKeyBase 13_065_329_313 gas [ 13.065µs] (computed in 109.02s)
StorageHasKeyByte 32_983_184 gas [ 33ns] (computed in 55.80s)
StorageReadBase 13_752_679_313 gas [ 13.753µs] (computed in 36.14s)
StorageReadKeyByte 33_064_852 gas [ 33ns] (computed in 55.15s)
StorageReadValueByte 2_080_969 gas [ 2ns] (computed in 45.52s)
StorageWriteBase 5_988_201_695 gas [ 5.988µs] (computed in 36.06s)
StorageWriteKeyByte 6_486_408 gas [ 6ns] (computed in 119.46s)
StorageWriteValueByte 2_391_904 gas [ 2ns] (computed in 39.74s)
StorageWriteEvictedByte 3_559_623 gas [ 4ns] (computed in 45.74s)
StorageRemoveBase 13_027_891_147 gas [ 13.028µs] (computed in 35.79s)
StorageRemoveKeyByte 12_733_717 gas [ 13ns] (computed in 60.51s)
StorageRemoveRetValueByte 1_692_055 gas [ 2ns] (computed in 43.00s)
TouchingTrieNode 8_995_546_152 gas [ 8.996µs] UNCERTAIN (computed in 35.47s) HIGH-VARIANCE: runtime/runtime-params-estimator/src/utils.rs:354:24
ReadCachedTrieNode 897_291_666 gas [ 897ns] UNCERTAIN (computed in 198.09s) HIGH-VARIANCE: runtime/runtime-params-estimator/src/utils.rs:369:13
Finished in 960.90s, output saved to:
4 iters
StorageHasKeyBase 14_058_590_550 gas [ 14.059µs] (computed in 79.23s)
StorageHasKeyByte 33_011_596 gas [ 33ns] (computed in 61.74s)
StorageReadBase 14_448_919_175 gas [ 14.449µs] UNCERTAIN (computed in 36.10s) HIGH-VARIANCE: runtime/runtime-params-estimator/src/utils.rs:354:24
StorageReadKeyByte 32_979_505 gas [ 33ns] (computed in 60.98s)
StorageReadValueByte 2_017_662 gas [ 2ns] (computed in 48.73s)
StorageWriteBase 6_040_652_050 gas [ 6.041µs] (computed in 36.30s)
StorageWriteKeyByte 6_248_627 gas [ 6ns] (computed in 148.50s)
StorageWriteValueByte 2_347_736 gas [ 2ns] (computed in 42.35s)
StorageWriteEvictedByte 3_590_032 gas [ 4ns] (computed in 48.67s)
StorageRemoveBase 13_096_787_050 gas [ 13.097µs] (computed in 35.98s)
StorageRemoveKeyByte 12_601_924 gas [ 13ns] (computed in 68.06s)
StorageRemoveRetValueByte 1_640_915 gas [ 2ns] (computed in 45.55s)
TouchingTrieNode 6_211_604_656 gas [ 6.212µs] (computed in 36.96s)
ReadCachedTrieNode 971_890_625 gas [ 972ns] UNCERTAIN (computed in 240.09s) HIGH-VARIANCE: runtime/runtime-params-estimator/src/utils.rs:369:13
Finished in 1034.30s, output saved to:
5 iters
StorageHasKeyBase 13_483_639_212 gas [ 13.484µs] UNCERTAIN (computed in 75.35s) HIGH-VARIANCE: runtime/runtime-params-estimator/src/utils.rs:354:24
StorageHasKeyByte 32_955_855 gas [ 33ns] UNCERTAIN (computed in 66.64s) HIGH-VARIANCE: runtime/runtime-params-estimator/src/utils.rs:354:24
StorageReadBase 14_311_978_112 gas [ 14.312µs] UNCERTAIN (computed in 36.33s) HIGH-VARIANCE: runtime/runtime-params-estimator/src/utils.rs:354:24
StorageReadKeyByte 33_267_940 gas [ 33ns] UNCERTAIN (computed in 66.08s) HIGH-VARIANCE: runtime/runtime-params-estimator/src/utils.rs:354:24
StorageReadValueByte 2_027_850 gas [ 2ns] UNCERTAIN (computed in 10.66s) HIGH-VARIANCE: runtime/runtime-params-estimator/src/utils.rs:354:24
StorageWriteBase 6_025_332_562 gas [ 6.025µs] UNCERTAIN (computed in 9.66s) HIGH-VARIANCE: runtime/runtime-params-estimator/src/utils.rs:354:24
StorageWriteKeyByte 6_184_661 gas [ 6ns] UNCERTAIN (computed in 35.09s) HIGH-VARIANCE: runtime/runtime-params-estimator/src/utils.rs:354:24
StorageWriteValueByte 2_391_077 gas [ 2ns] UNCERTAIN (computed in 11.00s) HIGH-VARIANCE: runtime/runtime-params-estimator/src/utils.rs:354:24
StorageWriteEvictedByte 3_774_544 gas [ 4ns] UNCERTAIN (computed in 10.63s) HIGH-VARIANCE: runtime/runtime-params-estimator/src/utils.rs:354:24
StorageRemoveBase 14_726_878_112 gas [ 14.727µs] UNCERTAIN (computed in 9.43s) HIGH-VARIANCE: runtime/runtime-params-estimator/src/utils.rs:354:24
StorageRemoveKeyByte 12_753_175 gas [ 13ns] UNCERTAIN (computed in 13.90s) HIGH-VARIANCE: runtime/runtime-params-estimator/src/utils.rs:354:24
StorageRemoveRetValueByte 1_704_211 gas [ 2ns] UNCERTAIN (computed in 10.34s) HIGH-VARIANCE: runtime/runtime-params-estimator/src/utils.rs:354:24
TouchingTrieNode 6_588_516_675 gas [ 6.589µs] (computed in 10.42s)
ReadCachedTrieNode 1_069_487_500 gas [ 1.069µs] UNCERTAIN (computed in 254.29s) HIGH-VARIANCE: runtime/runtime-params-estimator/src/utils.rs:369:13
Finished in 664.60s, output saved to:
10 iters
StorageHasKeyBase 13_578_744_617 gas [ 13.579µs] UNCERTAIN (computed in 26.07s) HIGH-VARIANCE: runtime/runtime-params-estimator/src/utils.rs:354:24
StorageHasKeyByte 32_979_644 gas [ 33ns] UNCERTAIN (computed in 23.39s) HIGH-VARIANCE: runtime/runtime-params-estimator/src/utils.rs:354:24
StorageReadBase 13_407_996_617 gas [ 13.408µs] UNCERTAIN (computed in 8.48s) HIGH-VARIANCE: runtime/runtime-params-estimator/src/utils.rs:354:24
StorageReadKeyByte 33_113_673 gas [ 33ns] UNCERTAIN (computed in 20.91s) HIGH-VARIANCE: runtime/runtime-params-estimator/src/utils.rs:354:24
StorageReadValueByte 2_069_821 gas [ 2ns] UNCERTAIN (computed in 12.42s) HIGH-VARIANCE: runtime/runtime-params-estimator/src/utils.rs:354:24
StorageWriteBase 6_324_529_522 gas [ 6.325µs] UNCERTAIN (computed in 9.55s) HIGH-VARIANCE: runtime/runtime-params-estimator/src/utils.rs:354:24
StorageWriteKeyByte 6_161_299 gas [ 6ns] UNCERTAIN (computed in 61.94s) HIGH-VARIANCE: runtime/runtime-params-estimator/src/utils.rs:354:24
StorageWriteValueByte 2_388_617 gas [ 2ns] UNCERTAIN (computed in 14.21s) HIGH-VARIANCE: runtime/runtime-params-estimator/src/utils.rs:354:24
StorageWriteEvictedByte 3_691_684 gas [ 4ns] UNCERTAIN (computed in 12.58s) HIGH-VARIANCE: runtime/runtime-params-estimator/src/utils.rs:354:24
StorageRemoveBase 13_050_943_167 gas [ 13.051µs] UNCERTAIN (computed in 8.59s) HIGH-VARIANCE: runtime/runtime-params-estimator/src/utils.rs:354:24
StorageRemoveKeyByte 12_708_137 gas [ 13ns] UNCERTAIN (computed in 19.37s) HIGH-VARIANCE: runtime/runtime-params-estimator/src/utils.rs:354:24
StorageRemoveRetValueByte 1_690_279 gas [ 2ns] UNCERTAIN (computed in 11.39s) HIGH-VARIANCE: runtime/runtime-params-estimator/src/utils.rs:354:24
TouchingTrieNode 7_055_293_690 gas [ 7.055µs] (computed in 10.44s)
ReadCachedTrieNode 1_080_375_000 gas [ 1.08µs] UNCERTAIN (computed in 462.98s) HIGH-VARIANCE: runtime/runtime-params-estimator/src/utils.rs:369:13
Finished in 747.11s, output saved to:
20 iters
StorageHasKeyBase 13_534_877_379 gas [ 13.535µs] UNCERTAIN (computed in 26.32s) HIGH-VARIANCE: runtime/runtime-params-estimator/src/utils.rs:354:24
StorageHasKeyByte 32_954_422 gas [ 33ns] UNCERTAIN (computed in 36.62s) HIGH-VARIANCE: runtime/runtime-params-estimator/src/utils.rs:354:24
StorageReadBase 13_596_178_804 gas [ 13.596µs] UNCERTAIN (computed in 9.27s) HIGH-VARIANCE: runtime/runtime-params-estimator/src/utils.rs:354:24
StorageReadKeyByte 33_125_687 gas [ 33ns] UNCERTAIN (computed in 34.16s) HIGH-VARIANCE: runtime/runtime-params-estimator/src/utils.rs:354:24
StorageReadValueByte 2_107_915 gas [ 2ns] UNCERTAIN (computed in 16.41s) HIGH-VARIANCE: runtime/runtime-params-estimator/src/utils.rs:354:24
StorageWriteBase 6_532_413_647 gas [ 6.532µs] UNCERTAIN (computed in 11.38s) HIGH-VARIANCE: runtime/runtime-params-estimator/src/utils.rs:354:24
StorageWriteKeyByte 6_229_705 gas [ 6ns] UNCERTAIN (computed in 320.09s) HIGH-VARIANCE: runtime/runtime-params-estimator/src/utils.rs:354:24
StorageWriteValueByte 2_400_606 gas [ 2ns] UNCERTAIN (computed in 86.49s) HIGH-VARIANCE: runtime/runtime-params-estimator/src/utils.rs:354:24
StorageWriteEvictedByte 3_641_082 gas [ 4ns] UNCERTAIN (computed in 101.73s) HIGH-VARIANCE: runtime/runtime-params-estimator/src/utils.rs:354:24
StorageRemoveBase 13_505_353_779 gas [ 13.505µs] UNCERTAIN (computed in 41.04s) HIGH-VARIANCE: runtime/runtime-params-estimator/src/utils.rs:354:24
StorageRemoveKeyByte 12_778_403 gas [ 13ns] UNCERTAIN (computed in 192.44s) HIGH-VARIANCE: runtime/runtime-params-estimator/src/utils.rs:354:24
StorageRemoveRetValueByte 1_705_733 gas [ 2ns] UNCERTAIN (computed in 86.82s) HIGH-VARIANCE: runtime/runtime-params-estimator/src/utils.rs:354:24
TouchingTrieNode 7_165_741_437 gas [ 7.166µs] UNCERTAIN (computed in 38.87s) HIGH-VARIANCE: runtime/runtime-params-estimator/src/utils.rs:354:24
ReadCachedTrieNode 990_625_000 gas [ 991ns] UNCERTAIN (computed in 906.51s) HIGH-VARIANCE: runtime/runtime-params-estimator/src/utils.rs:369:13
Finished in 1952.98s, output saved to:
30 iters
StorageHasKeyBase 13_672_466_157 gas [ 13.672µs] UNCERTAIN (computed in 28.76s) HIGH-VARIANCE: runtime/runtime-params-estimator/src/utils.rs:354:24
StorageHasKeyByte 32_923_901 gas [ 33ns] UNCERTAIN (computed in 185.66s) HIGH-VARIANCE: runtime/runtime-params-estimator/src/utils.rs:354:24
StorageReadBase 13_992_991_990 gas [ 13.993µs] UNCERTAIN (computed in 46.02s) HIGH-VARIANCE: runtime/runtime-params-estimator/src/utils.rs:354:24
StorageReadKeyByte 32_970_357 gas [ 33ns] UNCERTAIN (computed in 214.10s) HIGH-VARIANCE: runtime/runtime-params-estimator/src/utils.rs:354:24
StorageReadValueByte 2_122_544 gas [ 2ns] UNCERTAIN (computed in 134.66s) HIGH-VARIANCE: runtime/runtime-params-estimator/src/utils.rs:354:24
StorageWriteBase 6_609_279_470 gas [ 6.609µs] UNCERTAIN (computed in 54.12s) HIGH-VARIANCE: runtime/runtime-params-estimator/src/utils.rs:354:24
StorageWriteKeyByte 6_356_957 gas [ 6ns] UNCERTAIN (computed in 923.27s) HIGH-VARIANCE: runtime/runtime-params-estimator/src/utils.rs:354:24
StorageWriteValueByte 2_400_035 gas [ 2ns] UNCERTAIN (computed in 117.73s) HIGH-VARIANCE: runtime/runtime-params-estimator/src/utils.rs:354:24
StorageWriteEvictedByte 3_977_237 gas [ 4ns] UNCERTAIN (computed in 135.38s) HIGH-VARIANCE: runtime/runtime-params-estimator/src/utils.rs:354:24
StorageRemoveBase 14_193_024_007 gas [ 14.193µs] UNCERTAIN (computed in 47.10s) HIGH-VARIANCE: runtime/runtime-params-estimator/src/utils.rs:354:24
StorageRemoveKeyByte 12_636_652 gas [ 13ns] UNCERTAIN (computed in 268.56s) HIGH-VARIANCE: runtime/runtime-params-estimator/src/utils.rs:354:24
StorageRemoveRetValueByte 1_692_422 gas [ 2ns] UNCERTAIN (computed in 112.93s) HIGH-VARIANCE: runtime/runtime-params-estimator/src/utils.rs:354:24
TouchingTrieNode 7_553_665_014 gas [ 7.554µs] UNCERTAIN (computed in 43.51s) HIGH-VARIANCE: runtime/runtime-params-estimator/src/utils.rs:354:24
ReadCachedTrieNode 929_081_250 gas [ 929ns] UNCERTAIN (computed in 1323.45s) HIGH-VARIANCE: runtime/runtime-params-estimator/src/utils.rs:369:13
Finished in 3679.74s, output saved to:
40 iters
StorageHasKeyBase 13_589_294_563 gas [ 13.589µs] (computed in 65.51s)
StorageHasKeyByte 34_641_766 gas [ 35ns] UNCERTAIN (computed in 481.06s) HIGH-VARIANCE: runtime/runtime-params-estimator/src/utils.rs:354:24
StorageReadBase 14_583_977_776 gas [ 14.584µs] (computed in 47.28s)
StorageReadKeyByte 32_810_104 gas [ 33ns] (computed in 274.82s)
StorageReadValueByte 2_199_171 gas [ 2ns] (computed in 168.16s)
StorageWriteBase 6_687_153_364 gas [ 6.687µs] (computed in 59.30s)
StorageWriteKeyByte 6_298_567 gas [ 6ns] (computed in 1233.53s)
StorageWriteValueByte 2_415_817 gas [ 2ns] (computed in 150.57s)
StorageWriteEvictedByte 3_950_634 gas [ 4ns] UNCERTAIN (computed in 169.31s) HIGH-VARIANCE: runtime/runtime-params-estimator/src/utils.rs:354:24
StorageRemoveBase 13_956_983_851 gas [ 13.957µs] (computed in 51.10s)
StorageRemoveKeyByte 12_773_246 gas [ 13ns] (computed in 345.62s)
StorageRemoveRetValueByte 1_709_642 gas [ 2ns] (computed in 138.50s)
TouchingTrieNode 13_769_501_151 gas [ 13.77µs] UNCERTAIN (computed in 45.87s) HIGH-VARIANCE: runtime/runtime-params-estimator/src/utils.rs:354:24
ReadCachedTrieNode 980_657_812 gas [ 981ns] UNCERTAIN (computed in 1764.48s) HIGH-VARIANCE: runtime/runtime-params-estimator/src/utils.rs:369:13
Finished in 5040.11s, output saved to:
50 iters
StorageHasKeyBase 13_776_535_151 gas [ 13.777µs] UNCERTAIN (computed in 129.48s) HIGH-VARIANCE: runtime/runtime-params-estimator/src/utils.rs:354:24
StorageHasKeyByte 32_789_001 gas [ 33ns] (computed in 335.74s)
StorageReadBase 14_081_979_751 gas [ 14.082µs] (computed in 51.99s)
StorageReadKeyByte 33_072_709 gas [ 33ns] (computed in 335.58s)
StorageReadValueByte 2_285_782 gas [ 2ns] UNCERTAIN (computed in 200.27s) HIGH-VARIANCE: runtime/runtime-params-estimator/src/utils.rs:354:24
StorageWriteBase 6_764_527_654 gas [ 6.765µs] (computed in 67.03s)
StorageWriteKeyByte 6_185_901 gas [ 6ns] (computed in 1551.28s)
StorageWriteValueByte 2_697_848 gas [ 3ns] (computed in 186.35s)
StorageWriteEvictedByte 5_259_693 gas [ 5ns] UNCERTAIN (computed in 201.39s) HIGH-VARIANCE: runtime/runtime-params-estimator/src/utils.rs:354:24
StorageRemoveBase 16_254_930_461 gas [ 16.255µs] UNCERTAIN (computed in 55.22s) HIGH-VARIANCE: runtime/runtime-params-estimator/src/utils.rs:354:24
StorageRemoveKeyByte 14_017_706 gas [ 14ns] (computed in 422.84s)
StorageRemoveRetValueByte 1_818_455 gas [ 2ns] (computed in 164.58s)
TouchingTrieNode 7_465_078_177 gas [ 7.465µs] UNCERTAIN (computed in 48.77s) HIGH-VARIANCE: runtime/runtime-params-estimator/src/utils.rs:354:24
ReadCachedTrieNode 1_019_361_250 gas [ 1.019µs] UNCERTAIN (computed in 2203.93s) HIGH-VARIANCE: runtime/runtime-params-estimator/src/utils.rs:369:13
Finished in 5999.32s, output saved to:
75 iters
StorageHasKeyBase 14_014_411_258 gas [ 14.014µs] (computed in 198.15s)
StorageHasKeyByte 32_971_225 gas [ 33ns] (computed in 494.61s)
StorageReadBase 14_365_610_898 gas [ 14.366µs] (computed in 67.05s)
StorageReadKeyByte 33_022_814 gas [ 33ns] (computed in 496.56s)
StorageReadValueByte 2_350_673 gas [ 2ns] UNCERTAIN (computed in 373.15s) HIGH-VARIANCE: runtime/runtime-params-estimator/src/utils.rs:354:24
StorageWriteBase 7_296_011_225 gas [ 7.296µs] UNCERTAIN (computed in 124.21s) HIGH-VARIANCE: runtime/runtime-params-estimator/src/utils.rs:354:24
StorageWriteKeyByte 6_315_680 gas [ 6ns] (computed in 2434.43s)
StorageWriteValueByte 2_538_901 gas [ 3ns] (computed in 292.03s)
StorageWriteEvictedByte 4_342_591 gas [ 4ns] (computed in 295.30s)
StorageRemoveBase 13_523_278_652 gas [ 13.523µs] (computed in 70.04s)
StorageRemoveKeyByte 12_850_151 gas [ 13ns] (computed in 623.07s)
StorageRemoveRetValueByte 1_765_050 gas [ 2ns] (computed in 234.38s)
TouchingTrieNode 7_156_997_956 gas [ 7.157µs] UNCERTAIN (computed in 61.62s) HIGH-VARIANCE: runtime/runtime-params-estimator/src/utils.rs:354:24
ReadCachedTrieNode 981_690_833 gas [ 982ns] UNCERTAIN (computed in 3445.15s) HIGH-VARIANCE: runtime/runtime-params-estimator/src/utils.rs:369:13
Finished in 9295.41s, output saved to:
100 iters
StorageHasKeyBase 19_351_911_321 gas [ 19.352µs] UNCERTAIN (computed in 220.39s) HIGH-VARIANCE: runtime/runtime-params-estimator/src/utils.rs:354:24
StorageHasKeyByte 33_052_763 gas [ 33ns] (computed in 657.38s)
StorageReadBase 17_326_255_431 gas [ 17.326µs] UNCERTAIN (computed in 77.65s) HIGH-VARIANCE: runtime/runtime-params-estimator/src/utils.rs:354:24
StorageReadKeyByte 33_028_469 gas [ 33ns] (computed in 651.71s)
StorageReadValueByte 2_226_692 gas [ 2ns] (computed in 382.95s)
StorageWriteBase 7_246_144_383 gas [ 7.246µs] UNCERTAIN (computed in 108.05s) HIGH-VARIANCE: runtime/runtime-params-estimator/src/utils.rs:354:24
StorageWriteKeyByte 6_034_886 gas [ 6ns] UNCERTAIN (computed in 3135.38s) HIGH-VARIANCE: runtime/runtime-params-estimator/src/utils.rs:354:24
StorageWriteValueByte 2_500_958 gas [ 3ns] (computed in 400.69s)
StorageWriteEvictedByte 4_125_927 gas [ 4ns] (computed in 386.37s)
StorageRemoveBase 16_513_323_191 gas [ 16.513µs] UNCERTAIN (computed in 83.15s) HIGH-VARIANCE: runtime/runtime-params-estimator/src/utils.rs:354:24
StorageRemoveKeyByte 12_870_466 gas [ 13ns] (computed in 817.01s)
StorageRemoveRetValueByte 2_041_085 gas [ 2ns] UNCERTAIN (computed in 299.20s) HIGH-VARIANCE: runtime/runtime-params-estimator/src/utils.rs:354:24
TouchingTrieNode 8_713_369_479 gas [ 8.713µs] UNCERTAIN (computed in 69.04s) HIGH-VARIANCE: runtime/runtime-params-estimator/src/utils.rs:354:24
ReadCachedTrieNode 960_275_000 gas [ 960ns] (computed in 4728.84s)
Finished in 12105.58s, output saved to:
150 iters
StorageHasKeyBase 14_480_452_840 gas [ 14.48µs] UNCERTAIN (computed in 284.72s) HIGH-VARIANCE: runtime/runtime-params-estimator/src/utils.rs:354:24
StorageHasKeyByte 33_187_697 gas [ 33ns] (computed in 956.61s)
StorageReadBase 18_059_659_117 gas [ 18.06µs] UNCERTAIN (computed in 104.38s) HIGH-VARIANCE: runtime/runtime-params-estimator/src/utils.rs:354:24
StorageReadKeyByte 33_280_597 gas [ 33ns] (computed in 986.92s)
StorageReadValueByte 2_325_098 gas [ 2ns] (computed in 576.71s)
StorageWriteBase 8_026_454_400 gas [ 8.026µs] UNCERTAIN (computed in 153.31s) HIGH-VARIANCE: runtime/runtime-params-estimator/src/utils.rs:354:24
StorageWriteKeyByte 6_700_411 gas [ 7ns] UNCERTAIN (computed in 5587.91s) HIGH-VARIANCE: runtime/runtime-params-estimator/src/utils.rs:354:24
StorageWriteValueByte 2_529_798 gas [ 3ns] (computed in 639.25s)
StorageWriteEvictedByte 4_247_221 gas [ 4ns] (computed in 582.29s)
StorageRemoveBase 19_840_170_697 gas [ 19.84µs] UNCERTAIN (computed in 111.16s) HIGH-VARIANCE: runtime/runtime-params-estimator/src/utils.rs:354:24
StorageRemoveKeyByte 13_153_151 gas [ 13ns] (computed in 1202.99s)
StorageRemoveRetValueByte 1_979_379 gas [ 2ns] (computed in 430.68s)
TouchingTrieNode 7_543_821_926 gas [ 7.544µs] UNCERTAIN (computed in 86.86s) HIGH-VARIANCE: runtime/runtime-params-estimator/src/utils.rs:354:24
ReadCachedTrieNode 1_095_268_333 gas [ 1.095µs] UNCERTAIN (computed in 7482.10s) HIGH-VARIANCE: runtime/runtime-params-estimator/src/utils.rs:369:13
Finished in 19273.83s, output saved to:
from nearcore.
Ekleog-NEAR
commented on August 30, 2024
Data out of the runs (without memtrie enabled yet)
Skill unlocked: using chatgpt to turn unstructured data into structured data:
| Iteration | StorageHasKeyBase (Gas) | StorageHasKeyByte (Gas) | StorageReadBase (Gas) | StorageReadKeyByte (Gas) | StorageReadValueByte (Gas) | StorageWriteBase (Gas) | StorageWriteKeyByte (Gas) | StorageWriteValueByte (Gas) | StorageWriteEvictedByte (Gas) | StorageRemoveBase (Gas) | StorageRemoveKeyByte (Gas) | StorageRemoveRetValueByte (Gas) | TouchingTrieNode (Gas) | ReadCachedTrieNode (Gas) | Computation Time (Seconds) |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 1 | 12,900,400,629 | 32,974,229 | 13,176,595,629 | 32,950,083 | 1,996,159 | 5,894,729,584 | 6,054,362 | 2,372,167 | 3,502,758 | 13,325,130,629 | 12,766,994 | 1,660,624 | 8,087,650,976 | 838,062,500 | 879.35 |
| 2 | 15,193,494,159 | 35,605,815 | 13,538,829,159 | 33,160,425 | 2,006,289 | 6,010,015,762 | 6,186,676 | 2,401,317 | 3,662,605 | 13,380,131,159 | 12,818,790 | 1,669,679 | 6,168,517,526 | 938,437,500 | 867.08 |
| 3 | 13,065,329,313 | 32,983,184 | 13,752,679,313 | 33,064,852 | 2,080,969 | 5,988,201,695 | 6,486,408 | 2,391,904 | 3,559,623 | 13,027,891,147 | 12,733,717 | 1,692,055 | 8,995,546,152 | 897,291,666 | 960.9 |
| 4 | 14,058,590,550 | 33,011,596 | 14,448,919,175 | 32,979,505 | 2,017,662 | 6,040,652,050 | 6,248,627 | 2,347,736 | 3,590,032 | 13,096,787,050 | 12,601,924 | 1,640,915 | 6,211,604,656 | 971,890,625 | 1034.3 |
| 5 | 13,483,639,212 | 32,955,855 | 14,311,978,112 | 33,267,940 | 2,027,850 | 6,025,332,562 | 6,184,661 | 2,391,077 | 3,774,544 | 14,726,878,112 | 12,753,175 | 1,704,211 | 6,588,516,675 | 1,069,487,500 | 664.6 |
| 10 | 13,578,744,617 | 32,979,644 | 13,407,996,617 | 33,113,673 | 2,069,821 | 6,324,529,522 | 6,161,299 | 2,388,617 | 3,691,684 | 13,050,943,167 | 12,708,137 | 1,690,279 | 7,055,293,690 | 1,080,375,000 | 747.11 |
| 20 | 13,534,877,379 | 32,954,422 | 13,596,178,804 | 33,125,687 | 2,107,915 | 6,532,413,647 | 6,229,705 | 2,400,606 | 3,641,082 | 13,505,353,779 | 12,778,403 | 1,705,733 | 7,165,741,437 | 990,625,000 | 1952.98 |
| 30 | 13,672,466,157 | 32,923,901 | 13,992,991,990 | 32,970,357 | 2,122,544 | 6,609,279,470 | 6,356,957 | 2,400,035 | 3,977,237 | 14,193,024,007 | 12,636,652 | 1,692,422 | 7,553,665,014 | 929,081,250 | 3679.74 |
| 40 | 13,589,294,563 | 34,641,766 | 14,583,977,776 | 32,810,104 | 2,199,171 | 6,687,153,364 | 6,298,567 | 2,415,817 | 3,950,634 | 13,956,983,851 | 12,773,246 | 1,709,642 | 13,769,501,151 | 980,657,812 | 5040.11 |
| 50 | 13,776,535,151 | 32,789,001 | 14,081,979,751 | 33,072,709 | 2,285,782 | 6,764,527,654 | 6,185,901 | 2,697,848 | 5,259,693 | 16,254,930,461 | 14,017,706 | 1,818,455 | 7,465,078,177 | 1,019,361,250 | 5999.32 |
| 75 | 14,014,411,258 | 32,971,225 | 14,365,610,898 | 33,022,814 | 2,350,673 | 7,296,011,225 | 6,315,680 | 2,538,901 | 4,342,591 | 13,523,278,652 | 12,850,151 | 1,765,050 | 7,156,997,956 | 981,690,833 | 9295.41 |
| 100 | 19,351,911,321 | 33,052,763 | 17,326,255,431 | 33,028,469 | 2,226,692 | 7,246,144,383 | 6,034,886 | 2,500,958 | 4,125,927 | 16,513,323,191 | 12,870,466 | 2,041,085 | 8,713,369,479 | 960,275,000 | 12105.58 |
| 150 | 14,480,452,840 | 33,187,697 | 18,059,659,117 | 33,280,597 | 2,325,098 | 8,026,454,400 | 6,700,411 | 2,529,798 | 4,247,221 | 19,840,170,697 | 13,153,151 | 1,979,379 | 7,543,821,926 | 1,095,268,333 | 19273.83 |
| Current Costs | 200,000,000,000 | 30,790,845 | 200,000,000,000 | 30,952,533 | 5,611,005 | 200,000,000,000 | 70,482,867 | 31,018,539 | 32,117,307 | 200,000,000,000 | 38,220,384 | 11,531,556 | 110,000,000,000 | 2,280,000,000 |
This gives the following graphs:
In particular, it shows that while the results are mostly stable over a different number of iterations, the remaining instability seems to not be particularly lower when running more iterations. So the sweet spot will likely be running like ten 10-iters runs and doing data magic on them to decide what's good for us.
There is a slight ramp-up when increasing the iteration number, probably due to variance being better-detected and thus pushing the numbers up. This is likely not very relevant, as we should take a safety margin of more than this number.
So currently, our costs are set quite a bit higher than estimator results. If I understand Jakob's analysis correctly, this was done in order to derisk the flat storage launch: we wanted to be sure that our costs were not underestimated in the worst case.
AFAIU we're changing that policy nowadays to say that costs that just avoid missed chunks with our real-world load should be enough. Still, considering we're looking into various ways of getting state writes out of the critical path altogether, I'd suggest waiting until that happens to pursue gas costs reductions in writes: once flushing to disk is no longer a risk, we can safely ignore all the computations Jakob did in #8006 and just set values to the estimator-provided values.
Next steps
This project started around reevaluating the read costs with memtrie, so we should focus on that. But then we should likely continue with reducing the write costs too. My planned next steps are:
- Update the estimator to enable memtrie
- Confirm that the speedup order of magnitude is as expected
- Pick some prospective gas costs for reads (this won't need as much care as flat storage, because memtrie performance should be much more stable and much less dependent on filesystem caches etc.)
- Implement #11944
- Use it to validate the prospective gas costs
- If confirmed, perform the full protocol change and deploy the new gas costs in the next release
from nearcore.
akhi3030
commented on August 30, 2024
also CC: @pugachAG
from nearcore.
akhi3030
commented on August 30, 2024
this is very good work. I am happy to see that we are moving forward with these cost reductions. Excited for the next steps.
from nearcore.
pugachAG
commented on August 30, 2024
@Ekleog-NEAR we also need to consider how storage operations contribute to the chunk state witness size. My intuition says that with memtrie that would mostly determine gas costs, not the latency.
from nearcore.
akhi3030
commented on August 30, 2024
@pugachAG: not sure I understand your comment. Gas costs are related to chunk apply latency. If we reduce gas costs but the latency remains high, then we will have undercharging.
from nearcore.
akhi3030
commented on August 30, 2024
Just finished a call with @pugachAG to understand their concern above better.
When we access storage, there are two types of costs that we need to consider.
- the time to actually access the data and return it to the contract. This cost has gone down for reads due to memtrie as seen above.
- Storage accesses have to be added to the state witness. And there is an upper limit on how bit the state witness can be. Once, we hit this limit, we will no longer include additional transactions in the chunk.
The concern is that if we reduce gas costs based purely on 1. above, then we simply get limited by the state witness size limit. Hence, we should consider both the limits above to decide what the new gas values should be. I agree with this. I don't think it makes sense to reduce gas costs a lot when we will not see significant benefit of that in practice if we keep getting limited by the state witness size.
@Ekleog-NEAR: let's discuss in more detail when you are back from your PTO. I imagine that we should run a few more experiments to figure out how low we can make the gas costs before further reductions will not make sense.
Another point that we discussed in the call is that the current TTN for reading cost is too low. It was set incorrectly when we shipped flat state. So we should actually take this opportunity to see what the correct value should be and if check if it needs to be increased even.
from nearcore.
akhi3030
commented on August 30, 2024
@Ekleog-NEAR: I had a conversation with Bowen about this work. A question came up that it seems like we should expect memtrie to help with writes and not with reads. Reads were already optimised due to flat storage.
Finally, there are some other concerns with reducing the gas costs. We should look at what the current compte costs are and instead consider reducing them instead.
from nearcore.
bowenwang1996
commented on August 30, 2024
@Ekleog-NEAR when you run the experiments, did you include 0c23749 ? According to @pugachAG this significantly improves memtrie performance.
from nearcore.
Ekleog-NEAR
commented on August 30, 2024
The concern is that if we reduce gas costs based purely on 1. above, then we simply get limited by the state witness size limit. Hence, we should consider both the limits above to decide what the new gas values should be. I agree with this. I don't think it makes sense to reduce gas costs a lot when we will not see significant benefit of that in practice if we keep getting limited by the state witness size.
I don't think I agree with this concern. Compute costs and state witness size are two orthogonal limitations, and it just happens that storage reads consume the two. But we have a lot of other operations that consume only compute costs.
So for example, imagine that we have an original workload that does 1 storage read for 1 hash computation. There, it's useless to reduce compute costs by a lot, because we'll very quickly be limited by state witness size. But then a new workload comes in, that does 1 storage read followed by 1M hash computations. If we do not reduce the compute costs, then this workload will be heavily limited by compute costs.
As we don't know what future workloads will be coming on our blockchain, and there's no value in keeping compute costs higher than they should be, I do think that we should reduce them to the lowest value we feel comfortable handling — and increasing them again is not hard anyway.
Another point that we discussed in the call is that the current TTN for reading cost is too low. It was set incorrectly when we shipped flat state. So we should actually take this opportunity to see what the correct value should be and if check if it needs to be increased even.
@pugachAG My understanding was that TTN would only be dealing with values of less than 4k, so would literally never touch the disk trie. Is this understanding wrong?
(I also saw that TTN was set to too low a value when deploying flat storage, but if this understanding is correct then we no longer need to account for the variance there was in flat storage; and thus we can set it to almost the estimator-provided value. Said variance was the reason why TTN costs could at times have been underestimated)
@Ekleog-NEAR: I had a conversation with Bowen about this work. A question came up that it seems like we should expect memtrie to help with writes and not with reads. Reads were already optimised due to flat storage.
I probably don't understand this statement, because it seems incompatible with the discussion we had had with @tayfunelmas, summarized above.
Storage reads were already a bit better with flat storage, but memtrie allows us to bypass any disk read, which is a lot of improvement even compared to the only-one-disk-read of flat storage. And on the other hand storage writes still require us to write to all of memtrie, flat storage and on-disk trie, which means that writes are actually a bit slower with memtrie than without.
This understanding is confirmed by the numbers I posted above, so I probably don't understand this sentence.
Finally, there are some other concerns with reducing the gas costs. We should look at what the current compte costs are and instead consider reducing them instead.
The project that was given to me was to reduce gas costs. But as per the first message, I only looked at reducing compute costs for now, and all the "from" numbers given are for compute costs.
This being said, the numbers I suggested changing to are not only lower than our current compute costs, but also lower than our current gas costs. So after this lands and we confirm that it doesn't break anything (ie. after we're sure we won't need to revert the changes in a hurry), we will be able to consider also lowering the gas costs to the new value of the compute costs.
from nearcore.
Ekleog-NEAR
commented on August 30, 2024
@Ekleog-NEAR when you run the experiments, did you include 0c23749 ? According to @pugachAG this significantly improves memtrie performance.
I did not; IIRC it was still marked as draft when I ran the experiments. I'll run a new set of experiments and see how things evolve with it.
from nearcore.
Ekleog-NEAR
commented on August 30, 2024
Here are the results of the new set of experiments, run on fc87c85 so including the changes of 0c23749:
I'm actually not finding much difference between the results with 0c23749 and the results without it: the estimator results are almost always within one standard deviation of the previous results. My guess is that this is because the estimator does not actually exercise the prefetcher, and so the prefetcher is a no-op for it, while it originally brought slowdown to our real-world use case.
TL;DR: No changes with 0c23749, I still recommend the compute costs changes from #11925 (comment). The speedup from it is probably in parts that had never been measured by the estimator anyway, because it was too much contract-specific.
from nearcore.
Ekleog-NEAR
commented on August 30, 2024
Summary of the meeting with @akhi3030, @Longarithm and @pugachAG:
- We don't expect compute costs reductions for the write-specific costs, only for TTN which would then indirectly reduce write costs
- We should run an experiment: add metrics specifically for each cost we're looking to reduce as a histogram, and check that on a mainnet canary node the durations are as we would expect
- Yes TTN only ever touches memtrie, so we don't need to care about the 4k size split
- We currently don't charge TTN for reads because of flat storage, this is something we should fix now that we're significantly reducing compute costs, and because memtrie reintroduces actually touching trie nodes
Considering this zulip thread, my assumption on storage_has_key was also correct.
So to sum up the next steps:
- Split the costs that need to be different for values less than 4k into two costs
- Reduce the costs that we can simply reduce
- Add a new ReadTTN cost, with gas cost of 0 and compute cost of 6G, charged on TTN on reads (so that it doesn't charge more gas but only more compute costs)
- Introduce ReadTTN to the parameter estimator
- Add histogram metrics to all the costs we're considering changing
- Run for a while on canary mainnet, check that the metrics are as expected
- Assuming everything is fine, land
- (Interestingly, if everything goes fine, we'll have compute costs lower than gas costs here. This being said, we should wait before lowering gas costs, especially because of things like ReadTTN)
With this, this issue has become a bit of a hydra, with lots of new heads popping out. I'd expect the remaining time needed to do it all to be around 3-4 weeks. If we skip steps 5&6, it'd reduce by ~1 week (in exchange for relying on release testing to confirm it works), and probably the same if we skip step 3&4 (and instead charge arbitrarily more on read costs). This being said, this will be my first time adding a new cost, I'll have to hope the documentation we have is still up-to-date, even though it's more than a year old.
For now I'll proceed with doing all the steps, but feel free to let me know if you want me to skip some of them — I think for similar cost changes we usually would have skipped steps 5&6 in particular, because memtrie should have little variance.
from nearcore.
Related Issues (20)
- mainnet: synced node has regular Downloading blocks logs HOT 3
- [ReshardingV3] Receipts Reassignment HOT 1
- [EpochSync] Defend against malicious peers sending invalid epoch sync
- Difficulties Managing Gas Allocation for Chained Cross-Contract Calls HOT 2
- publishing: add a CI task to test packaging/building of all published crates individually HOT 1
- High cardinality metric HOT 4
- Downloading blocks stuck at 0.00% HOT 3
- Removing the validator key and then sending SIGHUP to the neard process does not appear to stop it validating
- [network] Expand STUN-based IP self-discovery to all nodes
- [network] Implement Tier3 connection pool
- [network] Implement p2p state part transfer
- [state sync] Update sync actor to use p2p state part transfer
- [network] Implement rate limiting for state part requests
- [Tracking Issue] P2P State Part Transfer
- Validate FT transfer benchmark performance HOT 3
- Print an error log when the CPU doesn't have the required features
- Spammy network messages in the logs HOT 1
- Locust issue in multinode mocknet/ft-bench: no txs send when `num-passive-users` is high
- There is a branch named 2.1.0 and a tag named 2.1.0 and they point at different commits HOT 4
- [ReshardingV3] State Witness
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from nearcore.