Udp listen about gosnmp HOT 48 CLOSED

@codedance poke :-)

from gosnmp.

Bump! Just realized this was already an Open Issue (the title is a bit misleading). Pasting my info here:

I ran into this issue when attempting to leverage Telegraf's SNMP Plugin, related issue here: influxdata/telegraf#3320

Use Case: Attempting to collect SNMP data from a remote Cisco UCS Manager that has a Cluster Virtual IP Address.

Issue: The outbound request sends data to the cluster IP but the response comes back from the originating server. This causes a change in the remote host address and the origin will not match the original destination. This is fine when handled at an OS level, tools like snmpget will handle this properly. Also interpreted languages like Ruby that leverage the OS network stack work as well. However there appears to be a low level implementation in the Golang networking stack that doesn't handle this. I'm trying to track down if its at this library or if we need to go deeper.

I have a packet capture that I can share (will send via email as it contains hostnames/IPs) and will be attempting to hack a bit at the code and see what else I can find.

Response from compiling an example with my host:
2017/10/11 08:54:48 Get() err: Request timeout (after 3 retries)

from gosnmp.

@robcowart I have little concerns about it either. As you said, this is done in other projects as well and kind of the expected behavior. Even snmpget in the console does it.

@SuperQ Maybe we can reopen this?

from gosnmp.

@SuperQ Now that this repo is community maintained, is there anything we can do about this?

#47 (comment) this approach actually helps with the issue

from gosnmp.

It appears that this does solve the issue. Thank you.

For future reference: Just enable this in your client and it will work.

client := &gosnmp.GoSNMP{
    UseUnconnectedUDPSocket: true,
    ...
}

from gosnmp.

Have you updated the library to the latest master?

What OS are you running on?
Can you ping the host? Tried scanning successfully with another tool?

from gosnmp.

Yes, I downloaded the library yesterday.

This test I run on FreeBSD. And:

# pkg info | grep go-
go-1.4.2,1                     Go programming language

Standard utility works without "problems":

snmpget -v2c -c ... 10.15.44.1 1.3.6.1.2.1.31.1.1.1.1.811
IF-MIB::ifName.811 = STRING: xe-4/1/0

10:28:02.704857 IP 10.12.2.2.29694 > 10.15.44.1.161:  C=totgfh0xre GetRequest(32)  .1.3.6.1.2.1.31.1.1.1.1.811
10:28:02.711669 IP 8.9.1.1.161 > 10.12.2.2.29694:  C=totgfh0xre GetResponse(40)  .1.3.6.1.2.1.31.1.1.1.1.811=[|snmp]

And my program, writing on perl, also do not have this "problem".
(In perl I uses this module: http://search.cpan.org/~dtown/Net-SNMP-v6.0.1/lib/Net/SNMP.pm)

I'm not sure it's really a problem. But before I had no problems with it.

My guess:
When I run the program and waiting for an answer I see:

# sockstat -4
...
root     status     99423 5  udp4   10.12.2.2:52482       10.15.44.1:161
...

Standard utility work:

# sockstat -4 | grep snmpwalk
root     snmpwalk   99753 3  udp4   *:30709               *:*

Answer it expects all on a particular port. I think the decision to use DialUDP. Listen port should specify request-response.

from gosnmp.

I'm not sure I fully understand the problem. Is it that the return packet comes from a different address?

There is a public value on GoSNMP called Conn

    // Conn is net connection to use, typically establised using GoSNMP.Connect()
    Conn net.Conn

You may be able to change the Conn to suite your requirements?

If we can understand the requirements we may be able to look at a library change (or config) as long as we can do so with no regressions/limitations, etc.

from gosnmp.

Thanks @benjamin-thomas and @codedance for responding on this, I've been busy with clients.

I've had some offline queries too. This project is starting to get more popular with end-users, I've been thinking of setting up two email lists on Google groups, gosnmp-discuss and gosnmp-announce. Thoughts? Just keep using issues?

from gosnmp.

It seems to me that the replacement net.Conn on net.PacketConn (respectively Read and Write to ReadFrom and WriteTo).

And it really helps. (many problem in parsing, but packet is received)
(This is only a test and not all changes, only the essence):

in Connect()

x.Conn, err = net.ListenUDP("udp", nil)

in dispatch(..)

tempRA, _ := net.ResolveUDPAddr("udp","10.15.44.1:161")
 _, err := c.WriteTo(outBuf,tempRA)
...
n, _, err := c.ReadFrom(resp)

and my program now "looks" like this (i use gorutines):

root     status     9450  5  udp4 6 *:50490               *:*
root     status     9450  10 udp4 6 *:23674               *:*
root     status     9450  11 udp4 6 *:11356               *:*
root     status     9450  12 udp4 6 *:15842               *:*
root     status     9450  13 udp4 6 *:14882               *:*
root     status     9450  14 udp4 6 *:64307               *:*
root     status     9450  15 udp4 6 *:40162               *:*
root     status     9450  16 udp4 6 *:39175               *:*
root     status     9450  17 udp4 6 *:59840               *:*
root     status     9450  18 udp4 6 *:25288               *:*
root     status     9450  19 udp4 6 *:19469               *:*

from gosnmp.

@soniah - Regarding issues vs. list. IMHO. I think stick with issues for the moment. There will be a tipping point though. Requests like this one I think belong here... It's easier to post code snippets or logs and/or reference it if we end up making code changes.

from gosnmp.

@abaluta OK. I think I understand. By using PacketConn rather than Conn, you're able to read on *:port rather than client-ip:port. I understand how this would address your problem and solve the issue where the return packet arrives from a different address.

I've done some digging around in an attempt to understand the Go net library socket code and it's implementation. In affect changing the code as you've suggested would end up calling the same base code functions anyway:

newUDPConn()
WriteToUDP()

I'm not an IP stack expert. I'd welcome any further comment from @soniah, @benjamin-thomas and co. (i.e. any issues with ephemeral port overuse?)

I might also quick mock up the code changes and do a bit of playing around. This would also make it easier for people to review and comment.

from gosnmp.

I believe Go supports SO_REUSEADDR, so this likely won't cause issues. (hopefully maybe)

from gosnmp.

@soniah I think github issues is fine IMO.

Definitely not an expert myself.

I haven't had the chance to look into this. I don't see why @abaluta would need to listen on 0.0.0.0 though.

Do you have 2 NICs? Also are you trying to share your gosnmp instance across your goroutines?

I think it'd be useful if you could provide a quick gist demonstrating the problem.

from gosnmp.

@benjamin-thomas I want to listen on *:port, because there is equipment that respond to SNMP requests from other addresses (the first local address). You can see this in tcpdump output in first message.
And that's normal. I did not find the document, which states that SNMP must answer from the same address. :(. But I now: In UDP need to get an answer to the same port.

from gosnmp.

+1 on @abaluta's suggestion of listening to *:port instead of client-ip:port. "Higher grade" routers with multiple physical interfaces and loopback interfaces could respond to SNMP requests using a different source address, such as loopback.

@abaluta meanwhile, could you send the SNMP request to 8.9.1.1?

from gosnmp.

@ctrlrsf No. This is real IP on Loopback interface on router. I communicate with the equipment in a separate vlan. Management network should be separated.

off topic: Do not hurt my router :)
from: http://www.juniper.net/documentation/en_US/junos13.1/topics/reference/general/snmp-junos-faq.html

The source IP address used in the response PDUs for SNMP requests is the IP address of the outgoing interface to reach the destination. The source IP address cannot be configured for responses. It can only be configured for traps.

from gosnmp.

@abaluta I incorporated some of the changes discussed above (ListenUDP vs DialTimeout) to my fork. I was wondering if this worked for you. I tried testing with some of my Juniper devices, but found they're all responding with same source that I'm targeting (QFXs, SRXs, and MXs).

https://github.com/ctrlrsf/gosnmp

from gosnmp.

@ctrlrsf Thanks in advance. Tomorrow, after the test, be sure to write.
off topic:
We use different routing instance. But SNMP can run only in inet.0...

from gosnmp.

I've taken a look at equivalent code in net-snmp and snmp4j. I think it's a valid change. The key is doing in an elegant way without busting the public API (Not that I think it's the best API anyway... another topic :-)

My thinking at the moment is abstract away the "transport" portion into a new (transport.go). It will be an implementation of the net.Conn interface (so API remains the same). Calling the Connect() method will create an instance of the "default transport" and set as gsnmp.Conn. The default transport will be "NonFilteringUDPConn". Not sure about the name NonFilteringUDPConn... best name I could come up with as it will receive on *:port and not filter to "client:port"

from gosnmp.

@abaluta It took me a while to understand your problem. I thought you were scanning multiple devices originally.

I just want to make sure I understand: you send a request to 10.15.44.1 and this same device responds with 8.9.1.1 which is another NIC available on the device itself?

Feels odd to me. From a security standpoint I would prefer never binding to 0.0.0.0 by default because that could open up other NICs from the client side or even the internet in some circumstances (ipv6).

from gosnmp.

@ctrlrsf That'S Perfect! everything works. I tested with all the equipment that I have. Everything works. Running in a separate gorutunes works too.

I tested this program on our monitoring server (3к snmp packets in 10 seconds) and don't have snmp transport problem.

@codedance Or go the way of developers package net (Not very logical, SNMP work over TCP,UDP):

// SNMPConn is the implementation of the Conn and PacketConn interfaces
// for SNMP network connections. 

DialSNMP // - Conn
ListenSNMP // - PacketConn
Write (GetRequest, GetBulk, ...) - // Conn interface (client:port)
WriteTo  //PacketConn
...

@benjamin-thomas I understand you. But Juniper (not a small company) went on this way:
One snmpd, listening on all interfaces, and security is based on the rules of firewall. (This is the biggest problem :)). We do not have the routers from other vendors, I can not say how it is implemented in other routers. (We have a couple of Juniper switches. They use the same address for a response. But they don't have different routing tables... But that's a topic for another discussion)

from gosnmp.

@benjamin-thomas I agree with your security point. SNMP over UDP is already "loaded" in terms of security :-) In some respect this change would open it up "a bit" but spoofing is pretty easy already with UDP (of course you'd also need to have a valid packet and also match the request ID but that's not security).

Neverless the change would open things up. This is one reason why I took a look at what snmp4J and net-snmp do. It's hard sifting through their code, but by my reading both project's client implementations bind and recv on 0.0.0.0/0. (e.g. any free port on open net.)

http://grepcode.com/file/repo1.maven.org/maven2/org.snmp4j/snmp4j/1.9.1f/org/snmp4j/transport/DefaultUdpTransportMapping.java

I wonder if it's worth reaching out to a lead on either of these projects and ask for their thoughts? I can't find anything in the RFC(s) on the topic.

@abaluta Regarding how to implement in the code (if we decide it's the right thing to do!): I think the net.Conn interface is fine for our purpose. There is no need to invent a new interface. It's just the implementation that (slightly) needs to change.

from gosnmp.

@codedance I so did not find an exact description of how it should be in the RFC. But the package net-snmp in *unix works that way.
Another thought: This change is a step towards realization snmp trap server. I think you need to remember about it.

from gosnmp.

@abaluta glad my changes worked for you.

@codedance https://gist.github.com/ctrlrsf/f5c9c17d4a06aa8fe2bf is a diff of what I changed. Note it changes GoSNMP.Conn from net.Conn to *net.UDPConn. If this is sufficient and doesn't clash with any future plans for project I can submit PR.

from gosnmp.

Guys @abaluta @codedance @ctrlrsf (and @wdreeveii) be aware that over the weekend I'll be looking at (and possibly merging) Whit's snmpv3 branch (https://github.com/soniah/gosnmp/tree/whit_master_rebase).

I haven't looked at Whit's latest code yet, I'd be interested in comments. I'll probably first start with a rebase to squash some of my small edits, etc. #50

from gosnmp.

Unfortunately I'm super busy so I haven't been able to lean on this. But I'd like to make a point

root@ubuntu-1404:~# ip addr show eth0 && ip addr show eth1
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
    link/ether 08:00:27:5d:6f:76 brd ff:ff:ff:ff:ff:ff
    inet 10.0.2.15/24 brd 10.0.2.255 scope global eth0
       valid_lft forever preferred_lft forever
    inet6 fe80::a00:27ff:fe5d:6f76/64 scope link 
       valid_lft forever preferred_lft forever
3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
    link/ether 08:00:27:7e:3b:96 brd ff:ff:ff:ff:ff:ff
    inet 172.16.8.3/24 brd 172.16.8.255 scope global eth1
       valid_lft forever preferred_lft forever
    inet6 fe80::a00:27ff:fe7e:3b96/64 scope link 
       valid_lft forever preferred_lft forever
root@ubuntu-1404:~# ip route show
default via 10.0.2.2 dev eth0 
10.0.2.0/24 dev eth0  proto kernel  scope link  src 10.0.2.15 
169.254.0.0/16 dev eth0  scope link  metric 1000 
172.16.8.0/24 dev eth1  proto kernel  scope link  src 172.16.8.3  metric 1 
172.17.0.0/16 dev docker0  proto kernel  scope link  src 172.17.42.1 
root@ubuntu-1404:~# netstat -unp                                                                                                       

Active Internet connections (w/o servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name
udp        0      0 10.0.2.15:55708         192.168.1.203:161       ESTABLISHED 13630/main      
udp        0      0 10.0.2.15:56458         192.168.1.201:161       ESTABLISHED 13630/main      
udp        0      0 10.0.2.15:32858         192.168.1.202:161       ESTABLISHED 13630/main      
udp        0      0 10.0.2.15:57731         192.168.1.200:161       ESTABLISHED 13630/main      

In this scenario I've got 2 NICs and my program uses one goroutine per scanned device (192.168.1.{200, 201, 202, 203})

The proper NIC/ip_addr is selected by the OS because my routing table is coherent. There is no need to expose my other NIC to this UDP traffic. What if that NIC was directly connected to the net? I wouldn't want anybody having the ability to poke at this remotely, even if I could setup my firewall accordingly.

@codedance regarding security I wasn't thinking in terms on the SNMP protocol, but more along the lines of the practice in security to first close everything, then open up accordingly and the fact that there could be an unforeseen consequences to this. What if there was a bug in the go runtime around the IP stack that would open the system to more thing that it should?

I still don't understand why listening on 0.0.0.0 solves the problem so, if not for my own ignorance, can anybody explain to me what's happening here?

from gosnmp.

@ctrlrsf No need for a PR quite yet. I still think there is a bit of debate and design to go. I also disagree with changing Conn to UDPConn for various API reasons and future vision around supporting "transport" types like other SNMP libs. Thanks for the POC code though. It's a good starting point for considering a design if we go down this way.

@benjamin-thomas I'm in the same position regarding time. I'd live to dive deep into the SNMPv3 code. Regarding how listening on 0.0.0.0 solves the problem... it would allow gosnmp to support devices where the response IP does not match the request IP. Other SNMP implementations seem to support this. I concur with your security view though. My thinking at the moment is that if we were to support this, it should be a non-default option.

from gosnmp.

@benjamin-thomas I think you might be misunderstanding what's going on here. It's not what interface we bind to, but what src IP the response packets have verses what IP we thought we were communicating with. The current implementation Dials to a specific IP, and if the response packets don't match that Dialed IP, then they don't get read causing a timeout. The POC changes I made basically use a function to listen on a port and does not discriminate what the source IP of the response UDP datagram is when reading from the socket.

A quick illustration is if you were to send and SNMP GET to 192.168.1.1, which lets say is physical interface ge-0/0/0 on a Juniper router. If that router does not respond to the packets using source address 192.168.1.1 the packets would get dropped by current implementation. As shown in @abaluta's case, some routers may respond to those packets from a loopback interface such as 1.1.1.1 depending on configuration (multiple routing instances, etc).

This issue orthogonal to what interface we bind to locally to listen for the packets on the client side. Whether the go library is bound to 0.0.0.0, or to your eth0 interface specifically, it can either Dial to a specific IP, or listen to all UDP datagrams returning to a specific port.

As to the security concerns, SNMPv2 isn't very secure and other popular SNMP libraries support this already.

Hope this helps.

from gosnmp.

Yeah sorry for the noise, I don't know what I was thinking in my last comment!

I wanted to simulate the situation myself for testing purposes. My first thought was to use iptables to rewrite the response source ip but this seems impossible (one can only overwrite the outgoing source ip).

Any suggestions?

from gosnmp.

@benjamin-thomas For to simulate, you must configure snmpd to listen on loopback interface.
(for *unix):
Add (alias) ip address to loopback interface and run snmpd with -x address. (or set "agentaddress" in snmpd.conf). Do not forget to setup the route to this loopback address.

from gosnmp.

I am reopening this issue because other people are reporting this problem. I do not believe security based objections to the proposed change have merit because the existing implementation is vulnerable to packet spoofing (in the V2 case). I would also like to bring the functionality in this library in line with other SNMP libraries.

from gosnmp.

👍

from gosnmp.

@wdreeveii Agree. Matching behaviors of other libraries that have already blazed the path should be a strong design consideration. I've used snmp4j and net-snmp as benchmarks for areas such as retry behavior and default timeouts, etc.

If we are short of hands up for this, I'd be happy to have a crack and do a pull request. My thinking is that the connection should be refactored into a "Transport" interface - again taking a leaf from a few other library's books. I think we can do this without modifying public interface too.

Thoughts?

from gosnmp.

@codedance by all means, go ahead and take a stab at it. Thanks!

Can you tell me a little bit more about the goals and objectives of the "Transport" interface?

from gosnmp.

👍

from gosnmp.

@codedance poke :-)

from gosnmp.

@derekmwright,

I have a similar problem. For a quick fix look at my post from Mar 3, 2015.

from gosnmp.

As illuminated by the tests, see the travis-ci log, this change is producing some issues.

from gosnmp.

Thanks @abaluta for prodding us on this issue.

When you've got a safe packet dump, would you mind uploading it to https://pastebin.com/ or email me at [email protected].

I'll be using that to write tests, so please choose captures that are safe for public viewing. In your capture, use some simple requests, like 2 or 3 Integer/Counter32/Gauge32 - so I can focus on the multiple ip stuff and not the unmarshalling. See https://github.com/soniah/gosnmp/blob/master/marshal_test.go#L356 to see what I'll be doing.

You could also write the tests, they're good for the soul ;-)

from gosnmp.

I worked with self snmp collector the last year. My collector use path from @wdreeveii.
I don't have any problem.
(My written English is not perfect, In advance I'm sorry).

The main problem is that gosnmp opens a "client" connection when sending a request:
root status 99423 5 udp4 10.12.2.2:52482 10.15.44.1:161
(Expect the package from 10.15.44.1:161 to 10.12.2.2:52482, and nothing else).

The "standard" snmpwalk ( net-snmp.sourceforge.net ) utility open a "server" connection when sending a request:
# sockstat -4 | grep snmpwalk
root snmpwalk 99753 3 udp4 *:30709 *:*
(Expect the package from : to *:30709)

I did not find a clear indication in the RFC, and I can not say which version is right.
But there is equipment (Juniper eg, if using HA (bug #3320 in telegraf)), who are responding from a different address.

from gosnmp.

I'm closing this issue as "Too Hard". Pull requests welcome :-)

from gosnmp.

This issue really needs to be reopened as it breaks how SNMP is intended to work. The examples provided are valid, that some devices (typically routers) may respond from a different IP than that on which they received the request. The "request-id" in the SNMP request and response headers, along with the UDP port, is what should be used to match responses to requests. Not the IP address. This flaw is a show-stopper in many environments.

from gosnmp.

@robcowart I fixed it in a fork for my use… I have no idea how good that fix is or if it‘s secure enough. It‘s about the same as the proposed change earlier in this issue.

https://github.com/infinytum/gosnmp

This works just fine. Why can‘t we do something like this in upstream?

from gosnmp.

I will take a look @Infinytum, thanks. I find the whole security conversation further up in the thread to be a bit of a red herring. It isn't that the potential security concern is invalid. It is just that it is well known, and understood if one is using SNMP. Even if the IP address is used, I can still spoof a response using the IP address, src port and request-id from an observed request. Remember SNMP (v1 and 2 anyway, and usually v3) is completely unencrypted on the wire. Pulling out the data from a request to spoof a response is no more difficult if the poller is matching on IP address or not.

from gosnmp.

@Infinytum works for me too, thanks.
You need to have a choice: security or work on your network.

from gosnmp.

I haven't re-read the whole thread, but doesn't this solve it: #277

from gosnmp.

I haven't re-read the whole thread, but doesn't this solve it: #277

I will test that and get back to you. Would be awesome if thats the case!

from gosnmp.