Adding multiple SNMPv3-enabled switches in NAV causes spurious timeouts and `snmpAgentDown` alerts about nav HOT 3 OPEN

Immediately, it seems something is amiss with the SNMPv3 handshake procedure in the instances that cause timeout reports.

Although SNMP itself is normally a stateless protocol, SNMPv3 adds more complexity to this to achieve privacy and secrecy. Specifically, establishing SNMPv3 communications between two devices now entails a handshake process, in where the client entity must first send a packet to query for the remote entity's SNMP engine details. The remote responds with a report package that contains these details, and these details must be used by the client entity in subsequent communication with the remote entity.

The necessary SNMP engine details seem to be mostly these three:

1. msgAuthoritativeEngineID
2. msgAuthoritativeEngineBoots
3. msgAuthoritativeEngineTime

Sniffing packets off the wire, it seems that when these spurious timeouts occur, it's due to the fact that NAV sometimes sends requests with an incorrect msgAuthoritativeEngineTime value once the handshake is completed, and the remote entity is responding with a new report packet to message this fact (responding with a SNMP-USER-BASED-SM-MIB::usmStatsNotInTimeWindows value). NAV leaves all these details to the underlying SNMP library (pynetsnmp-2 and Net-SNMP), but instead of handling these reports in a transparent manner, they seem to be turned into timeout errors by the underlying libraries (i.e. it seems the reports are being ignored and the lack of a corresponding reply to the request is interpreted as a timeout).

The correct response to such a report is probably to re-synchronize with the remote entity and try the request again. I would tend to think these details should be handled by Net-SNMP, but it's yet unclear to me whether the problem is with how NAV sets up the sessions, how pynetsnmp-2 interfaces with Net-SNMP, or with Net-SNMP itself.

from nav.

Section RFC 2274 does mention there is a default time window of 150 seconds. A message whose msgAuthoritativeEngineTime value differs more than 150 seconds from the one established in the handshake should be dismissed as invalid. Still not sure why sometime the first NAV message is way off...

from nav.

At this point, it seems the issue may actually be misconfiguration of the switches in question: They all seem to report the same hardcoded SNMP Engine ID, which is BAD™. This is likely confusing Net-SNMP, which seems to keep track of session synchronization values based on the SNMP Engine ID. Will check with our network engineering team before proceeding.

from nav.