Table traversal about snimpy HOT 19 OPEN

Hey!

The main goal of Snimpy is to feel "pythonic". The use of iteritems() is "less" pythonic. But I agree this is less efficient. There is also caching that could help here.

As for iteritems, I don't remember why I implement that instead of doing all that in iter. I may need to dig a bit about that. Depending on that, adding a bulk parameter would be possible or not.

For (3), which version of PySNMP are you using?

from snimpy.

Hello Vincent,

I understand and agree you've chosen MIB "columns" to be accessible like Python dicts. I just proposed to document how to write the traversal more efficiently (the historical coincidence Python dicts iterate over keys by default, which causes Snimpy iterator to fetch again the same values it has just received -- I agree your current API is more pythonic, its right).

I believe you implemented .iteritems() just as a more efficient alternative to .__iter__() :-) I noticed this behavior (the additional SNMP get(s) when traversing with default iterator as in the readthedocs) by an accident (wasn't looking for efficiency): I have a buggy agent (SNMP card for a UPS) which returns tabular data only to 'get next' (is not able to send a table item in response to specific 'get').

I discovered (3) at first on my Debian stable: Python 2.7.9 + Snimpy 0.8.3. I've just tried with Debian testing as well: Python 3.4.3 + Snimpy 0.8.8. But it behaves the same: walks (SNMP bulk get next) all the values:

#!/usr/bin/python3
import snimpy.manager
snimpy.manager.load('IF-MIB')
m = snimpy.manager.Manager(..., version = 2)
for index, if_descr in manager.ifDescr.iteritems():
    break

from snimpy.

For (3), I would have believed that I should have yield each bulk get. I'll check. I know this is not the case anymore with PySNMP 4.3 which turns a GETBULK request into a complete walk, but if you are using Debian packages, PySNMP is not at this version yet.

For (2), I believe that this could be done with contexts instead through the use of with. This way, we wouldn't need to rely on the fact that iteritems() is not the standard Python function.

For (1), my solution would have been to use caching. I have updated the documentation in respect of that.

from snimpy.

For (3), you are right. While I yield values in the higher levels, in the lower layers, I am using the simple command generator in PySNMP and it doesn't seem to provide an iterator on the results with versions < 4.3.0. I'll either upgrade to 4.3 at some point or rollback to using NetSNMP (for independent reasons). I'll fix that at this moment.

from snimpy.

Thank you. Not only for this small change, but for the (idea of) Snimpy in general: I haven't found any "ORM" SNMP analogy for any scripting langugage yet, your library is my favorite as being nearest. Not as a toy to retrieve few statistical SNMP variables, but for a larger project to autom8 configuration of a carrier L2/L3 device: my friend ended up with entering tens of OIDs in a messy PHP script to achieve the same... Btw, do you have any idea/plan to GET multiple variables in a single request (SET is already possible in with: context)?

from snimpy.

The difficulty for that is to find a pythonic way to do that. Until now, I didn't find anything. This is one of the main feature of Snimpy that I want to keep "true".

from snimpy.

So, about iteritems(), there is still no "abstract" way to do that in Python (I mean, no __iteritems__() in the different abstract classes). So iteritems() is here to stay. I'll add contextual bulk parameters shortly.

from snimpy.

Sorry for touching this again: I found an unexpected behavior just now. When trying to traverse an empty column, at least with SNMPv1 (as tested), the Session._op() raises SNMPEndOfMibView(), which is not handled in the iterator: instead of returning no items (throwin StopIteration), it leaks the SNMPEndOfMibView. Tested with 0.8.3 (the mentioned Debian setup) and skimming through master version here didn't reveal any patch for this.

from snimpy.

Could you show the exception? I think I see where the problem is but can't look at it right now. The exception would help.

from snimpy.

Sure. Please note it is 0.8.3 so are the line numbers. The ups_snmp in my script is a Manager instance, SNMPv1, with standard UPS-MIB loaded.

Traceback (most recent call last):
  File "/usr/local/bin/eatonbtest", line 193, in <module>
    for idx, upsAlarmDescr in ups_snmp.upsAlarmDescr.iteritems():
  File "/usr/lib/python2.7/dist-packages/snimpy/manager.py", line 379, in iteritems
    for noid, result in self.session.walk(oid):
  File "/usr/lib/python2.7/dist-packages/snimpy/snmp.py", line 268, in walk
    return self._op(self._cmdgen.nextCmd, *oids)
  File "/usr/lib/python2.7/dist-packages/snimpy/snmp.py", line 246, in _op
    raise SNMPEndOfMibView("no more stuff after this OID")  # nopep8
SNMPEndOfMibView: no more stuff after this OID

tcpdump: 1x get-next-request (OID=upsAlarmDescr), 1x get-response (OID>upsAlarmDescr)

from snimpy.

OK, that should be easy to fix.

from snimpy.

I have pushed a8551d7 for that.

from snimpy.

Hi Vincent,

I'm right now confronted with the efficiency issue (thousands of devices, and many OID's per device), that's probably what I should call "success" :-)

Question: any idea when you will have contextual bulk parameters implemented ?

Thx,
Charles

from snimpy.

I am currently swamped, so I can't give you any timeline on this.

from snimpy.

ok no problem, I can wait, thanks for your work Vincent !

from snimpy.

Hi,

As I understand ghost's request and Vincent's answer on 1.12.2015, there is currently no efficient / getbulk way of getting multiple "columns" for a table using a single iteritem() loop ?

My goal would be to have one loop over ifTable (1.3.6.1.2.1.2.2) and get a few values for each index, e.g. the following code produces a bulk-get (quick) and then loops over the ifMtu column, very "slowly".

for index, desc in m.ifDescr.iteritems():
    interface = {}
    interface['index'] = index
    interface['ifDescr'] = desc
    interface['ifMtu'] = m.ifMtu[index]

As I understand, the only way would be to to multiple distinct loops, one for each column I need, and then merge the results together. Or any other suggestion ?

Using pysnmp==4.3.2 and snimpy==0.8.10

Thx !
Charles

from snimpy.

@cbueche We could create a proxy object where we register several OID:

for index, desc, mtu in m.ifDescr.and.ifMtu.iteritems():
    ...

This is a bit more Ruby than Python.

from snimpy.

with the 7-8 columns I need, the line length might be indecent :-)
Anyway, thx, I will implement multiple iteritems() and merge the results together.

from snimpy.

Well, you could do that in several times:

proxy = m.ifDescr
proxy &= m.ifMtu
proxy &= ...

More pythonic to just overload &.

from snimpy.