Comments (13)
robbat2
commented on June 11, 2024
1
I patched go-snmp to print the OID when it fails to decode.
1.3.6.1.4.1.1718.3.2.3.1.11.1.1.13 is the OID when it hits the bug, but it's WEIRD.
This bug is only happening when snmp_exporter is walking the OID tree.
It does NOT happen with other gosnmp examples.
Here's the byte dump from before, split up:
02 00 # ASN.1 int, length 0; trips the error the first time
30 13 06 0f 2b 06 01 04 01 8d 36 03 02 03 01 0b 01 01 0e 02 00 # 1.3.6.1.4.1.1718.3.2.3.1.11.1.1.14, ASN.1 int, asnlength 0
30 13 06 0f 2b 06 01 04 01 8d 36 03 02 03 01 0b 01 01 0f 02 00 # 1.3.6.1.4.1.1718.3.2.3.1.11.1.1.15, ASN.1 int, asnlength 0
30 13 06 0f 2b 06 01 04 01 8d 36 03 02 03 01 0b 01 01 10 02 00 # 1.3.6.1.4.1.1718.3.2.3.1.11.1.1.16, ASN.1 int, asnlength 0
30 13 06 0f 2b 06 01 04 01 8d 36 03 02 03 01 0b 02 01 01 02 00 # 1.3.6.1.4.1.1718.3.2.3.1.11.1.2.1, ASN.1 int, asnlength 0
30 13 06 0f 2b 06 01 04 01 8d 36 03 02 03 01 0b 02 01 02 02 00 # ...
30 13 06 0f 2b 06 01 04 01 8d 36 03 02 03 01 0b 02 01 03 02 00 # ...
30 13 06 0f 2b 06 01 04 01 8d 36 03 02 03 01 0b 02 01 04 02 00 # ...
30 13 06 0f 2b 06 01 04 01 8d 36 03 02 03 01 0b 02 01 05 02 00 # ...
30 13 06 0f 2b 06 01 04 01 8d 36 03 02 03 01 0b 02 01 06 02 00 # ...
30 13 06 0f 2b 06 01 04 01 8d 36 03 02 03 01 0b 02 01 07 02 00 # ...
30 13 06 0f 2b 06 01 04 01 8d 36 03 02 03 01 0b 02 01 08 02 00 # ...
30 13 06 0f 2b 06 01 04 01 8d 36 03 02 03 01 0b 02 01 09 02 00 # ...
30 13 06 0f 2b 06 01 04 01 8d 36 03 02 03 01 0b 02 01 0a 02 00 # ...
30 13 06 0f 2b 06 01 04 01 8d 36 03 02 03 01 0b 02 01 0b 02 00 # ...
30 13 06 0f 2b 06 01 04 01 8d 36 03 02 03 01 0b 02 01 0c 02 00 # ...
30 13 06 0f 2b 06 01 04 01 8d 36 03 02 03 01 0b 02 01 0d 02 00 # ...
30 13 06 0f 2b 06 01 04 01 8d 36 03 02 03 01 0b 02 01 0e 02 00 # ...
30 13 06 0f 2b 06 01 04 01 8d 36 03 02 03 01 0b 02 01 0f 02 00 # ...
30 13 06 0f 2b 06 01 04 01 8d 36 03 02 03 01 0b 02 01 10 02 00 # 1.3.6.1.4.1.1718.3.2.3.1.11.1.2.16, ASN.1 int, asnlength 0
In the debug output from snmp_exporter, we see an ASN.1 encoded integer supposedly as "02 00"
02 = integer
00 = asnlength
But the net-snmp output for it shows boring and correct ASN.1 (02 01 00)
net-snmp source should also error on a integer w/ asnlength==0
https://github.com/net-snmp/net-snmp/blob/master/snmplib/asn1.c#L604-L607
trace: snmp_pdu_parse(): snmp_api.c, 4724:
dumph_recv: VarBindList
trace: snmp_pdu_parse(): snmp_api.c, 4740:
dumph_recv: VarBind
trace: snmp_parse_var_op(): snmp.c, 165:
dumph_recv: Name
dumpx_recv: 06 0F 2B 06 01 04 01 8D 36 03 02 03 01 0B 01 01 0D
dumpv_recv: ObjID: SNMPv2-SMI::enterprises.1718.3.2.3.1.11.1.1.13
trace: snmp_pdu_parse(): snmp_api.c, 4749:
dumph_recv: Value
dumpx_recv: 02 01 00
dumpv_recv: Integer: 0 (0x00)
trace: _sess_process_packet_parse_pdu(): snmp_api.c, 5677:
gosnmp example2 (minor patches to pass OID & community) also WORKS:
gosnmp/examples/example2 $ GOSNMP_TARGET=10.0.1.18 GOSNMP_COMMUNITY=(redacted) go run . 1.3.6.1.4.1.1718.3.2.3.1.11.1.1.13
2024/01/13 17:58:14 OIDs: [1.3.6.1.4.1.1718.3.2.3.1.11.1.1.13]
SEND INIT
SENDING PACKET: Version:2c, MsgFlags:NoAuthNoPriv, SecurityModel:SnmpV3SecurityModel(0), SecurityParameters:, ContextEngineID:, ContextName:, Community:(redacted), PDUType:GetRequest, MsgID:0, RequestID:2068017646, MsgMaxSize:0, Error:NoError, ErrorIndex:0, NonRepeaters:0, MaxRepetitions:0, Variables:[{<nil> 1.3.6.1.4.1.1718.3.2.3.1.11.1.1.13 Null}]
WAITING RESPONSE...
2024/01/13 17:58:14 Query latency in seconds: 0.014885664
GET RESPONSE OK: [48 53 2 1 1 4 10 79 83 76 95 112 117 98 108 105 99 162 36 2 4 123 67 113 238 2 1 0 2 1 0 48 22 48 20 6 15 43 6 1 4 1 141 54 3 2 3 1 11 1 1 13 2 1 0]
Packet sanity verified, we got all the bytes (55)
parseRawField: version
Parsed version 1
parseRawField: community
Parsed community (redacted)
UnmarshalPayload Meet PDUType 0x476574526573706f6e7365. Offset 17
getResponseLength: 38
parseRawField: request id
requestID: 2068017646
parseRawField: error-status
errorStatus: 0
parseRawField: error index
error-index: 0
vblLength: 24
parseRawField: OID
OID: .1.3.6.1.4.1.1718.3.2.3.1.11.1.1.13
decodeValue: type is Integer
decodeValue: value is 0
0: oid: .1.3.6.1.4.1.1718.3.2.3.1.11.1.1.13 number: 0
next up, trying to a tcpdump of snmp_exporter and comparing to the tcpdump of gosnmp examples.
from snmp_exporter.
robbat2
commented on June 11, 2024
Git bisect shows the error was introduced with 4449004.
That commit included an update to github.com/gosnmp/gosnmp: v1.32.0 -> v1.34.0
git bisect start
# status: waiting for both good and bad commits
# good: [c33572b6c8c8e43a479fde0f9fa8ac86e15598bc] Merge pull request #612 from prometheus/richih/0.20.0
git bisect good c33572b6c8c8e43a479fde0f9fa8ac86e15598bc
# status: waiting for bad commit, 1 good commit known
# bad: [0d8c3527cac0c26f1d6005b84b74413d14264c37] Release 0.21.0 (#819)
git bisect bad 0d8c3527cac0c26f1d6005b84b74413d14264c37
# good: [15dd541b71cbf768135fb5f2a3c3f33bad61c886] Merge pull request #670 from Crabbey/main
git bisect good 15dd541b71cbf768135fb5f2a3c3f33bad61c886
# bad: [f0de4cf14f87760e911106a8733ad198eb85915b] Bump github.com/gosnmp/gosnmp from 1.34.0 to 1.35.0 (#765)
git bisect bad f0de4cf14f87760e911106a8733ad198eb85915b
# bad: [4449004cf8f07be51bec99070e9d96c43abe7f65] Update build (#697)
git bisect bad 4449004cf8f07be51bec99070e9d96c43abe7f65
# good: [3726d4612f55ae5b9cb497cfd7bda03011b582aa] Merge pull request #677 from candlerb/candlerb/vrrpstats
git bisect good 3726d4612f55ae5b9cb497cfd7bda03011b582aa
# good: [8a56b28af255377f9d52d823df906d7e246742f9] Hacky workaround for net-snmp version detection (#704)
git bisect good 8a56b28af255377f9d52d823df906d7e246742f9
# good: [83399c23888fc08b7c943a471ad39331e0ba3d96] Fix broken link (#705)
git bisect good 83399c23888fc08b7c943a471ad39331e0ba3d96
# first bad commit: [4449004cf8f07be51bec99070e9d96c43abe7f65] Update build (#697)
from snmp_exporter.
robbat2
commented on June 11, 2024
Revert to [email protected], with commenting out the LocalAddr feature, does make the excporter work again.
from snmp_exporter.
SuperQ
commented on June 11, 2024
It seems like this error message was introduced in gosnmp/gosnmp#373 to fix invalid SNMP packet data.
from snmp_exporter.
SuperQ
commented on June 11, 2024
Interestingly, I also have a servertech sentry3 device. I tested with v0.25.0 and don't run into this. I wonder if your device firmware has a bug. What version of the firmware are you running?
from snmp_exporter.
robbat2
commented on June 11, 2024
Sentry3-MIB::systemVersion.0 = STRING: Sentry Version 5.3q
from snmp_exporter.
SuperQ
commented on June 11, 2024
That version is from 2008. There have been major updates to the Sentry firmware since then. I'm guessing this is a firmware bug in the old SNMP library.
https://cdn10.servertech.com/assets/documents/documents/909/original/swcdu-history.txt
from snmp_exporter.
robbat2
commented on June 11, 2024
snmpbulkwalk does handle this case correctly, so I'm wondering if it's worthwhile to provide parity. I'll chase the people with admin to the device to update it anyway.
from snmp_exporter.
SuperQ
commented on June 11, 2024
Adding a workaround for this would probably require upstream changes in gosnmp. Opening an issue there, with some tcpdump packet captures of net-snmp snmpbulkwalk would be needed in order to add reproduction cases.
Especially helpful would be to reduce the walk to the specific OIDs that are returning the weird data.
from snmp_exporter.
robbat2
commented on June 11, 2024
Did the tcpdumps, found a reproduction case.
snmp_exporter-0.20.0 just silently returned a value of 0 for these fields instead of throwing an error.
Frame 33: 102 bytes on wire (816 bits), 102 bytes captured (816 bits)
Linux cooked capture v2
Internet Protocol Version 4, Src: (redacted:host-ip), Dst: (redacted:device-ip)
User Datagram Protocol, Src Port: (redacted:host-port), Dst Port: 161
Simple Network Management Protocol
version: v2c (1)
community: (redacted:community)
data: getBulkRequest (5)
getBulkRequest
request-id: 1366421655
non-repeaters: 0
max-repetitions: 25
variable-bindings: 1 item
1.3.6.1.4.1.1718.3.2.3.1.11.1.1.12: Value (Null)
Object Name: 1.3.6.1.4.1.1718.3.2.3.1.11.1.1.12 (iso.3.6.1.4.1.1718.3.2.3.1.11.1.1.12)
Value (Null)
Frame 34: 587 bytes on wire (4696 bits), 587 bytes captured (4696 bits)
Linux cooked capture v2
Internet Protocol Version 4, Src: (redacted:device-ip), Dst: (redacted:host-ip)
User Datagram Protocol, Src Port: 161, Dst Port: (redacted:host-port)
Simple Network Management Protocol
version: v2c (1)
community: (redacted:community)
data: get-response (2)
get-response
request-id: 1366421655
error-status: noError (0)
error-index: 24
variable-bindings: 25 items
1.3.6.1.4.1.1718.3.2.3.1.11.1.1.13:
Object Name: 1.3.6.1.4.1.1718.3.2.3.1.11.1.1.13 (iso.3.6.1.4.1.1718.3.2.3.1.11.1.1.13)
Integral value is zero-length
[Expert Info (Note/Undecoded): Integral value is zero-length]
[Integral value is zero-length]
[Severity level: Note]
[Group: Undecoded]
1.3.6.1.4.1.1718.3.2.3.1.11.1.1.14:
Object Name: 1.3.6.1.4.1.1718.3.2.3.1.11.1.1.14 (iso.3.6.1.4.1.1718.3.2.3.1.11.1.1.14)
Integral value is zero-length
[Expert Info (Note/Undecoded): Integral value is zero-length]
[Integral value is zero-length]
[Severity level: Note]
[Group: Undecoded]
...
from snmp_exporter.
robbat2
commented on June 11, 2024
Deeper debugging shows there's a subtle firmware bug in requesting the last OID values available, depending on the max repetitions.
PR forthcoming to change max repetitions for this device, via Generator
The device I have access to, exposes 349 OID values, when I do snmpbulkwalk.
- With max-repetitions from range 1-4 inclusive, no OIDs fail for BULKGET.
- With max-repetitions=5, the very last OID, 1.3.6.1.4.1.1718.3.2.3.1.11.2.1.16 fails.
- With max-repetitions=6, 2nd to last OID, 1.3.6.1.4.1.1718.3.2.3.1.11.2.1.15 fails.
... - With max-repetitions=350, the 355th to last OID (so the 4th OID), 1.3.6.1.4.1.1718.3.1.4.0
- With max-repetitions=353, the 1st OID fails! 1.3.6.1.4.1.1718.3.1.1.0
- With max-repetitions=354-359, no OIDs fail again!
- With max-repetitions=360+ the pattern starts again
from snmp_exporter.
robbat2
commented on June 11, 2024
I'm going to use max-repetitions=4 in the fix, because that's the highest value for which no OID probed will ever trigger the bug.
from snmp_exporter.
SuperQ
commented on June 11, 2024
Thanks for all the investigation work.
from snmp_exporter.
Related Issues (20)
- How to detect that private MIB files conform to this program
- Packets crawled using snmp_exporter are in {}; how to solve the problem
- Lookups returning empty strings on Checkpoint MIB HOT 13
- Inconsistent support of IPv6 literals
- Setting a non-default retries value for any module affects all modules
- can‘t find gcc-g++ pkg in rhel HOT 3
- curl: option --no-progress-meter: is unknown
- Exec format error
- how to update snmp.yaml and do not restart snmp_expoter
- Override help string
- collected metric was collected before with the same name and label values HOT 6
- No specific metrics are generated
- error collecting metric Desc{fqName: "snmp_error", help: "Error scraping target", constLabels: {module="okiprinter"}, variableLabels: {}}: error walking target ipv4: not in time window
- snmp_exporter does not parse InetAddressIPv4z correctly
- yaml/yq parser round-trip behavior: unmarshaling of integer keys in dicts HOT 2
- Go build error when trying to generate "generator" file HOT 6
- Handle connection failures on resolution failure HOT 2
- Sonicwall Firewall / snmp exporter / prometheus
- Generator - get without .0 + unwanted walk
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from snmp_exporter.