Connection Slot Exhaustion with Passive Nodes about go-ethereum HOT 19 OPEN

@weiihann Dear weiihann, the situation you described is partially correct, but there are two scenarios where Passive nodes will still exist in large numbers.

1. Nodes that are started honestly will try to select the neighbor node with the highest block height for block synchronization, which means that as long as malicious nodes do not actively declare their block height to be very high, they can still exist.
2. Nodes that are synchronizing with the network will not start with other honest nodes because, in the case of ETH 2.0, blocks are inserted into Geth by Prysm.

from go-ethereum.

I'm curious why hasn't this occur a long time ago, if the attack cost is relatively low.

This problem may be first proposed around 2014 or earlier. However, as mentioned in #29327 (comment) and #29327 (comment) and #29034 (comment) by karalabe, the early developer encountered a very severe error where each peer rejects each other due to similar "improvement". Since that, they try not to make changes and hold the idea it is acceptable.

from go-ethereum.

If an honest geth node connects to a malicious node and attempts to sync with it, it will be dropped as the malicious node couldn't provide the correct data, referring to this code section.

from go-ethereum.

I am also looking into this problem. see #29327

from go-ethereum.

I'm curious why hasn't this occur a long time ago, if the attack cost is relatively low.

Btw, just in case you missed it, I'd suggest to submit this to the bug bounty program.

from go-ethereum.

It was already submitted to the bug bounty program, and then got sent here.

from go-ethereum.

I'm curious why hasn't this occur a long time ago, if the attack cost is relatively low.

This problem may be first proposed around 2014 or earlier. However, as mentioned in #29327 (comment) and #29327 (comment) and #29034 (comment) by karalabe, the early developer encountered a very severe error where each peer rejects each other due to similar "improvement". Since that, they try not to make changes and hold the idea it is acceptable.

Thanks for sharing!

from go-ethereum.

The attack is mostly a theoretical one. However, it would still be nice to fix it. @lyciumlee if you have a fix in mind, I am open to discuss it!

from go-ethereum.

@weiihann Dear weiihann, I have already submitted the report to the Ethereum Bug Bounty Program before disclosing it to the community, and I have email correspondence records. Fredrik Svantes suggested that I post the report here to discuss the issue with everyone.

from go-ethereum.

@fjl Dear fjl, I am primarily dedicated to the research of blockchain network protocols and have a keen interest in this area. I am also very willing to participate in the activities of building the Ethereum community. To address this issue, a score-based Peer mechanism can be used. The reasons are as follows:

1. The score-based Peer mechanism is a passive system, the introduction of which can allow the entire network of nodes to upgrade gradually.
2. . The score-based Peer mechanism aligns with the behavior code of honest nodes.

For the first reason, even if some nodes forget to upgrade, this mechanism will not affect the non-upgraded nodes, nor will it cause the network to be segmented or forked. Because this mechanism is entirely a passive behavior marking mechanism, an honest node will transfer and broadcast many messages to the network through the ETH Wire Protocol and protocols based on the ETH Wire Protocol. Therefore, the upgraded nodes know that these non-upgraded nodes are honest nodes, which will not lead to the network being split.

For the second reason, Execution Layer Nodes primarily use the ETH Wire Protocol for broadcasting Transactions-related messages, meaning when a new Transaction enters the Mempool of an honest node, it notifies other nodes that are unaware of the transaction. Therefore, upon receiving the message, upgraded nodes under the score-based Peer mechanism will recognize that the node is an honest node. Nodes that only receive messages without broadcasting any are actually harmful to the network. Although they do nothing but accept all the network messages they receive, they occupy connection slots of other nodes. The score-based mechanism can identify those that haven't generated any beneficial ETH Wire Protocol messages for a long time. If these malicious nodes are forced to participate in the message forwarding process, we also achieve the purpose of preventing this malicious behavior.

from go-ethereum.

One problem with your solution suggestion is the existence of syncing nodes. While the node is syncing, it cannot relay transactions.

from go-ethereum.

@fjl Dear fjl, I understand the sync process you mentioned.
During the sync process, it appears that nodes do not send any protocols related to transactions. However, looking at it from another perspective, this process occurs because honest nodes are attempting to synchronize with the network. Moreover, interactions among these nodes also occur in pairs, such as the StorageRangesMsg and GetStorageRangesMsg in the Sync protocol. Synchronizing with the network is actually an intentional act regarding the network itself. Therefore, the score-based mechanism refers to marking all related messages as active. This is in contrast to the one-way network propagation mechanism that existed during the ETH 1.0 era with block propagation and continues today with transaction propagation, resulting in completely passive nodes.

from go-ethereum.

1. It's easy to build a naive peer scoring system.
2. It's easy to bypass ( send random "get storagerangemsg" to simulate syncing node, but still be passive)
3. It's easy to improve the peer scoring system (detect the faked storage ranges, improve the faker-detection)
4. It's easy to bypass (use a different set of messages to fake "active" peer)
5. and on, and on

I'm a bit scared that if we build a peer scoring system, we're entering a game which has no end. Every 6 months, a new whitepaper will be presented on how some researchers bypassed geth's peer scoring system. And at some point, a scoring-system will unintentionally cut off syncing nodes, or client X (besu / nethermind), or clients with limited tx pool capacity, or something else.

from go-ethereum.

@holiman Dear holiman . I understand the concerns of the community. It seems we need to find a balance between simplicity and security.

from go-ethereum.

Every 6 months, a new whitepaper will be presented on how some researchers bypassed geth's peer scoring system.

haha, such an interesting insight. Many companies have the motivation and incentive to bypass geth's peer scoring system. I might even secure a job with a higher salary, as they would be willing to pay for expertise in countering the scoring system in the future😃.

from go-ethereum.

I think we should just add a system where we disconnect a random peer every so often. It doesn't need any score/rules.

from go-ethereum.

I think we should just add a system where we disconnect a random peer every so often. It doesn't need any score/rules.

But the peer can/would just retry the connection immediately upon disconnection?

from go-ethereum.

The reason we haven't added a reputation system or an algorithmic disconnect is because they are too easy to game. Since Geth's code is public, it's trivial to see what the rules are and how to fake them. What you'll end up with is non-zero probability of unintended side effects dropping legitimate peers and close-zero probability of actual effect on malicious peers who just fake some traffic. As @holiman mentioned, it becomes a game of whack-a-mole.

Whilst I agree that your concern is legitimate, IMO it's very hard to find a solution to a non-existing problem (as in not-actively exploited), because we just don't know how the problem would look like and what the actual solution would be. Fixing every possible attack scenario anyone can ever dream up in the future is a questionable effort (whilst noble).

An alternative line of thought is what the probabilities and gains are for such an attack. Currently block propagation is handled by consensus nodes, so to filling connection slots doens't really block the network from functioning. Transaction propagation can be impacted, but there are many MEV private pools that could be injected into directly (which many do already), so it's not obvious what the gain would be to block on mechanism whilst the other is still going strong.

My 2 cents are that we need resilience more than fool-proof-ness. For example, for the discovery protocol, we have two mechanism: the DHT itself and the DNS discovery. Both could in theory be attacked, but doing it simultaneously is probably non-trivial and would be quickly detected. We've added 2 to have each be a backup/fallback in case the other has some issues. Sure, we want to make both as robust as meaningful, but neither needs to be absolute perfect resilient.

For the transactions, we again have the two mechanisms (txpools, mev pools) that act as one-another's backup. Of course they are not serving the same purpose, but they do provide resiliency.

The sync code at some point was quite agressive with dropping "useless" peers, so it should be kind of hard to eclipse syncing nodes off from the network - at least as much as EL is concerned.

IMO it is more valuable to have a robust monitoring to detect anomalous behavior and course correct (on top of the resilient mechanisms) rather then to cover all possible bases all the time, investing infinite resources.

from go-ethereum.

Thank you all for your opinions, and thanks to the Ethereum developers for their enthusiastic answers. I have benefited immensely from everyone's responses.

from go-ethereum.