toni-moreno / snmpcollector Goto Github PK

A full featured Generic SNMP data collector with Web Administration Interface for InfluxDB

License: MIT License

Go 51.52% TypeScript 30.03% HTML 15.81% CSS 1.29% JavaScript 0.16% Shell 1.07% Dockerfile 0.12%

influxdb snmp collector metrics performance performance-monitoring time-series monitoring cacti opennms

snmpcollector's Introduction

SnmpCollector

SnmpCollector is a full featured Generic SNMP data collector with Web Administration Interface Open Source tool which has as main goal simplify the configuration for getting data from any device which snmp protocol support and send resulting data to an influxdb backend.

For complete information on installation from binary package and configuration you could read the snmpcollector wiki.

If you wish to compile from source code you can follow the next steps

Run from master

If you want to build a package yourself, or contribute. Here is a guide for how to do that.

Dependencies

Go 1.5 for snmpcollector < 0.8
Go 1.11 for snmpcollector >= 0.8
NodeJS >=6.2.1

Get Code and setup example config

git clone https://github.com/toni-moreno/snmpcollector.git
cd snmpcollector
cp conf/sample.config.toml conf/config.toml

Building the backend

go run build.go build

Building frontend and backend in production mode

npm install
PATH=$(npm bin):$PATH            # or export PATH=$(npm bin):$PATH depending on your shell
npm run build:prod               # will build fronted and backend

Creating minimal package tar.gz

After building frontend and backend you wil do

npm run postbuild #will build fronted and backend

Creating rpm and deb packages

you will need previously installed the fpm/rpm and deb packaging tools. After building frontend and backend you will do.

go run build.go latest

Running first time

To execute without any configuration you need a minimal config.toml file on the conf directory.

cp conf/sample.config.toml conf/config.toml
./bin/snmpcollector

This will create a default user with username adm1 and password adm1pass (don't forget to change them!).

Recompile backend on source change (only for developers)

go get github.com/unknwon/bra
npm start

will init a change autodetect webserver with angular-cli (ng serve) and also a autodetect and recompile process with bra for the backend

Online config

Now you wil be able to configure metrics/measuremnets and devices from the builting web server at http://localhost:8090 or http://localhost:4200 if working in development mode (npm start)

Offline configuration.

You will be able also insert data directly on the sqlite db that snmpcollector has been created at first execution on config/snmpcollector.db examples on example_config.sql

cat conf/example_config.sql |sqlite3 conf/snmpcollector.db

snmpcollector's People

Contributors

Stargazers

Watchers

snmpcollector's Issues

Error on measurement validation

It seem like the new metric inserted in db is not found when asigned to a measurment.

DEBU[2016-12-19 16:30:18] &{ID:myMeas Name:my_meas_name GetMode:value IndexOID: IndexTag: IndexAsValue:false Fields:[{ID:myNnewMetric Report:true}] fieldMetric:[] evalMetric:[] Description:}
DEBU[2016-12-19 16:30:18] looking for measurement my_meas_name : fields: myNnewMetric : Report true
WARN[2016-12-19 16:30:18] measurement field  myNnewMetric NOT FOUND in Metrics Database !
WARN[2016-12-19 16:30:18] Error on insert Measurement myMeas  , affected : 0 , error: No metrics found with names;myNnewMetric in measurement Config myMeas
[Macaron] 2016-12-19 16:30:18: Completed /measurement 404 Not Found in 270.53µs

Error selecting measurement groups

It seems that some devices are taking measurement groups that doesn't belong to them. It happens when 2 measurement groups share a bit of letters ... Trying to fix..

[Feature Request] Self - Monitoning data in only one measurement

Now self Monitoring is sent as one field x measurement ..

# <prefix>.goroutines
# <prefix>.memory.allocated
# <prefix>.memory.mallocs
# <prefix>.memory.frees
# <prefix>.memory.gc.total_pause
# <prefix>.memory.heap
# <prefix>.memory.gc.pause_per_second
# <prefix>.memory.gc.pause_per_interval
# <prefix>.memory.gc.gc_per_second

but can be easily sent as one big measurement with 9 fields.

Devices Deactivated after changed the "influxdb" output

After a change in the influxdb name output ID ( name) all devices have become "disabled"

[Feature request] Support for OctetString Indexes

As in this link is exposed (http://stackoverflow.com/questions/17650430/snmp-table-index-using-octet-string) there are devices indexing tables with strings rather than numéric indexes.

[Feature request] Config update without restart

First initialisation of snmp devices take large time for big envirenment.
Need a feature to update snmp device list without collector restarting.

npm install installs [email protected] which breaks the UI

What:

npm install installs [email protected] which breaks the UI

Version

SNMPCollector - 0.5.3 - ff26b4b (Nov 7, 2016)

Steps to reproduce:

Git clone and follow all build Steps
Validate NPM installed [email protected]

npm ls ng2-bootstrap
[email protected] /Users/jboswell/go/src/snmpcollector
└── [email protected]

./bin/snmpcollector
Browse to UI and you will see the following errors:

GET http://localhost:8090/node_modules/ng2-bootstrap/bundles/ng2-bootstrap.js 
GET http://localhost:8090/node_modules/ng2-bootstrap/bundles/ng2-bootstrap.js 404 (Not Found)
GET http://localhost:8090/node_modules/ng2-bootstrap/ng2-bootstrap.js/ng2-bootstrap 404 (Not Found)
Error: (SystemJS) Error: XHR error (404 Not Found) loading http://localhost:8090/node_modules/ng2-bootstrap/ng2-bootstrap.js/ng2-bootstrap(…)
GET http://localhost:8090/node_modules/ng2-bootstrap/ng2-bootstrap.js/components/modal/modal.component 404 (Not Found)

Fix:

Force NPM to install [email protected]

rm -fR node_modules/ng2-bootstrap/
npm install [email protected]

[Feature Request] More standar namming for measurement types

Now there is two type of measurements.

value
indexed

When looking for SNMP info these two type of snmp data are commonly referred as

scalar data
tabular ( o table ) data.

So could be interesting change these values:

value will be scalar
indexed will be table

What do you think about?

PANIC: send on closed channel

[Macaron] PANIC: send on closed channel
/usr/local/go/src/runtime/panic.go:443 (0x4972f9)
/usr/local/go/src/runtime/chan.go:167 (0x46e456)
/usr/local/go/src/runtime/chan.go:100 (0x46e2a3)
/home/developer/src/gospace/src/github.com/toni-moreno/snmpcollector/pkg/main.go:339 (0x417b54)
/home/developer/src/gospace/src/github.com/toni-moreno/snmpcollector/pkg/webserver.go:228 (0x4444ff)
/usr/local/go/src/runtime/asm_amd64.s:472 (0x4c6c5e)
/usr/local/go/src/reflect/value.go:435 (0x76a9cd)
/usr/local/go/src/reflect/value.go:303 (0x769691)
```

Seem like we forgot unset initialized flag to false after Remove

[Bug] Hidden fields on edit/new mode with v3 SNMPDevice

It seems that some needed fields are hidden when v3 is selected caused by some incorrect ngIf.

NoAuthNoPriv should show V3AuthUser
NoAuthPriv should show V3AuthUser, V3AuthPass, V3AuthProt
AuthPriv should show V3AuthUser, V3AuthPass, V3AuthProt, V3PrivPass, V3PrivProt

[Feature Request] Add metric as counter for conditions

Could be interesnting to evaluate a condicion over an Indexed OID , by example we would like to check how many ports are in "up" state. We will count how many interfaces has errors...

collector takes several minutes to reboot when a lot of devices without connectivity

Device Initializatión takes timeout seconds to end on connection failure

This is a problem when we have N devices where N > 100 devices , with any kind of incidence , on each initialization time to begin to really gather data is N*Timeout.

We should skip gather system info until connection could be ready

[Feature Request] Add Metric Parser for some Hardware specific "strange" values

These are values got from some hardware ....

1.3.6.1.4.1.7779.3.1.1.2.1.10.1.3.39 = CPU_TEMP: +37.00
1.3.6.1.4.1.7779.3.1.1.2.1.10.1.3.41 = SYS_TEMP: +38.00 C
1.3.6.1.4.1.7779.3.1.1.2.1.10.1.3.36 = Power1 (AC) OK
1.3.6.1.4.1.7779.3.1.1.2.1.10.1.3.37 = Power2 (AC) OK
1.3.6.1.4.1.7779.3.1.1.2.1.10.1.3.28 = FAN 1: 8725 RPM
1.3.6.1.4.1.7779.3.1.1.2.1.10.1.3.29 = FAN 2: 8355 RPM
1.3.6.1.4.1.7779.3.1.1.2.1.10.1.3.30 = FAN 3: 8725 RPM
1.3.6.1.4.1.7779.3.1.1.2.1.10.1.3.31 = FAN 4: 9280 RPM
1.3.6.1.4.1.7779.3.1.1.2.1.10.1.3.32 = FAN 5: 8355 RPM

[Feature Request] Add Option to force disable "bulk" queries in snmp v2/v3

There is some IBM storage devices with problems with any snmpbulkwalk data query. It seems to work ok ( but really slow ) with snmpwalk

Could be interesting add a flag for each device to disable this option as a workaround to the "bulk" option.

[RFC] Request for a new API discussion.

After adding logs of new functions it seems that a API refactor is needed..

I propose this one ( old -> new )

******* Login methods **********
(no changes on these )
/login
/logout

*******Config related methods *****

/metric                     /api/cfg/metric
/measurement                /api/cfg/measurement
/measgroups                 /api/cfg/measgroup
/measfilters                /api/cfg/measfilters
/influxservers              /api/cfg/influxservers
/snmpdevice                 /api/cfg/snmpdevice

*********runtime related methods ****

/runtime/agent/reloadconf/  /api/rt/agent/reload
/runtime/snmpping/          /api/rt/agent/snmpping
/runtime/version/           /api/rt/agent/info/version
/runtime/info/              /api/rt/device/info
/runtime/activatedev/       /api/rt/device/activate
/runtime/deactivatedev/     /api/rt/device/deactivate
/runtime/actsnmpdbg/        /api/rt/device/actsnmpdebug
/runtime/deactsnmpdbg/      /api/rt/device/deactsnmpdbg
/runtime/setloglevel/       /api/rt/device/setloglevel
/runtime/getdevicelog       /api/rt/device/getdevicelog
/runtime/forcefltupdate     /api/rt/device/forcefltupdate

Error on build with last version

[root@grafanalab01 snmpcollector]# go run build.go build
Version: 0.2.0, Linux Version: 0.2.0, Package Iteration: 
rm -r bin
rm -r Godeps/_workspace/pkg
rm -r Godeps/_workspace/bin
rm -r dist
rm -r tmp
rm -r /root/grafana-lab/pkg/linux_amd64/github.com/toni-moreno/snmpcollector
rm -r ./bin/snmpcollector
rm -r ./bin/snmpcollector.md5
GOPATH=/root/grafana-lab/src/github.com/toni-moreno/snmpcollector/Godeps/_workspace:/root/grafana-lab
go version
go version go1.7.1 linux/amd64
go build -ldflags -w -X main.version=0.2.0 -X main.commit=913983f -X main.buildstamp=1475275941 -o ./bin/snmpcollector ./pkg/
# github.com/toni-moreno/snmpcollector/pkg
pkg/model.go:163: cannot call non-function dbc.x.ShowSQL (type bool)
pkg/model.go:171: cannot call non-function dbc.x.Logger (type core.ILogger)
exit status 2
exit status 1

This is becaouse of we forgot to add xorm as dependency. :) we will fix soon

Error on set alternate log and config directories

I've created a /opt/snmpcollector/san_conf/config.toml file with this entries in the general log.

[general]
 logdir = "./san_log"
 logLevel = "debug"

When executing with -config option config dir and log dir is chosen as below:

#./bin/snmpcollector -config ./san_conf/config.toml
INFO[2016-10-19 11:57:33] set Default directories :
   - Exec: /opt/snmpcollector/
   - Config: /opt/snmpcollector/conf
   -Logs: /opt/snmpcollector/log

Expected values should be

Exec: /opt/snmpcollector/
Config /opt/snmpconllector/san_conf/
Logs: /opt/snmpcollector/san_log/ ( specified in config.toml)

[Feature Request] Automatic device Log rotate on each restart/time period

Now we have lots of log files and after some days we have filled the logs filesystem.

[Bug Report] SNMPDevice Table wrong info on "ExtraTags" column

The data shown on "ExtraTags" column is showing DeviceTagValue field instead of the extra tags.

Indexed mode tag issue

Can't set ifIndex (OID=1.3.6.1.2.1.2.2.1.1) and ifNumber (OID=1.3.6.1.2.1.2.1) as IndexTAG.
Everything is fine for ifDescr (OID=1.3.6.1.2.1.2.2.1.2) and ifName (OID=1.3.6.1.2.1.31.1.1.1.1).
Snmpbulkwalk for these OIDs gives fine result, but snmpcollector don't want to write these OIDs to InfluxDB.
Example of ifIndex response:

IF-MIB::ifIndex.1 = INTEGER: 1
IF-MIB::ifIndex.40 = INTEGER: 40
IF-MIB::ifIndex.10001 = INTEGER: 10001
IF-MIB::ifIndex.10002 = INTEGER: 10002
IF-MIB::ifIndex.10003 = INTEGER: 10003
IF-MIB::ifIndex.10004 = INTEGER: 10004
IF-MIB::ifIndex.10005 = INTEGER: 10005
IF-MIB::ifIndex.10006 = INTEGER: 10006
IF-MIB::ifIndex.10007 = INTEGER: 10007
IF-MIB::ifIndex.10008 = INTEGER: 10008
IF-MIB::ifIndex.10009 = INTEGER: 10009
IF-MIB::ifIndex.10010 = INTEGER: 10010
IF-MIB::ifIndex.10011 = INTEGER: 10011
IF-MIB::ifIndex.10012 = INTEGER: 10012
IF-MIB::ifIndex.10013 = INTEGER: 10013
IF-MIB::ifIndex.10014 = INTEGER: 10014
IF-MIB::ifIndex.10015 = INTEGER: 10015
IF-MIB::ifIndex.10016 = INTEGER: 10016
IF-MIB::ifIndex.10017 = INTEGER: 10017
IF-MIB::ifIndex.10018 = INTEGER: 10018
IF-MIB::ifIndex.10019 = INTEGER: 10019
IF-MIB::ifIndex.10020 = INTEGER: 10020
IF-MIB::ifIndex.10021 = INTEGER: 10021
IF-MIB::ifIndex.10022 = INTEGER: 10022
IF-MIB::ifIndex.10023 = INTEGER: 10023
IF-MIB::ifIndex.10024 = INTEGER: 10024
IF-MIB::ifIndex.10101 = INTEGER: 10101
IF-MIB::ifIndex.10102 = INTEGER: 10102
IF-MIB::ifIndex.10501 = INTEGER: 10501

Error on Change: not modified filter cross reference id on change measurement id

After change the ID for the SAN_CISCO_ISL_Thoughput the ID has not changed on the Measurement Filter

[Feature Request] Add more metric types according to OID data types

On adding/modifying a metric some data types are missing on the selector. This can create confusion to the user according to the result of a get/walk.

Integer - Signed 32bit Integer (values between -2147483648 and 2147483647).
Integer32 - Same as Integer.
UInteger32- Unsigned 32bit Integer (values between 0 and 4294967295).
Octet String - Arbitrary binary or textual data, typically limited to 255 characters in length.
Object Identifier - An OID.
Bit String -Represents an enumeration of named bits. This is an unsigned datatype.
IpAddress - An IP address.
Counter32 - Represents a non-negative integer which monotonically increases until it reaches a maximum value of 32bits-1 (4294967295 dec), when it wraps around and starts increasing again from zero.
Counter64 - Same as Counter32 but has a maximum value of 64bits-1.
Gauge32 - Represents an unsigned integer, which may increase or decrease, but shall never exceed a maximum value.
TimeTicks - Represents an unsigned integer which represents the time, modulo 232 (4294967296 dec), in hundredths of a second between two epochs.
Opaque - Provided solely for backward-compatibility, its no longer used.
NsapAddress - Represents an OSI address as a variable-length OCTET STRING.

404s in Frontend

Hi @toni-moreno I just cloned your repo, as I wanted to check it out. Sadly the front end seems to have several 404 path issues

node version : v6.9.1 (Boron LTS)

cd /tmp
git clone https://github.com/toni-moreno/snmpcollector/
cd snmpcollector
npm install
npm start

> [email protected] lite /private/tmp/snmpcollector
[1] > lite-server
[1]
[1] ** browser-sync config **
[1] { injectChanges: false,
[1]   files: [ './public/**/*.{html,htm,css,js}' ],
[1]   watchOptions: { ignored: 'node_modules' },
[1]   server: { baseDir: './public', middleware: [ [Function], [Function] ] } }
[1] [BS] Access URLs:
[1]  --------------------------------------
[1]        Local: http://localhost:3002
[1]     External: http://192.168.0.104:3002
[1]  --------------------------------------
[1]           UI: http://localhost:3003
[1]  UI External: http://192.168.0.104:3003
[1]  --------------------------------------
[1] [BS] Serving files from: ./public
[1] [BS] Watching files...
[1] 16.11.05 17:39:46 200 GET /index.html
[1] 16.11.05 17:39:46 404 GET /public/css/simple-sidebar.css
[1] 16.11.05 17:39:46 404 GET /node_modules/bootstrap/dist/css/bootstrap.css
[1] 16.11.05 17:39:46 404 GET /node_modules/core-js/client/shim.min.js
[1] 16.11.05 17:39:46 404 GET /node_modules/zone.js/dist/zone.js
[1] 16.11.05 17:39:46 404 GET /node_modules/reflect-metadata/Reflect.js
[1] 16.11.05 17:39:46 404 GET /node_modules/systemjs/dist/system.src.js
[1] 16.11.05 17:39:46 200 GET /systemjs.config.js
[1] 16.11.05 17:39:46 404 GET /node_modules/ng2-bootstrap/bundles/ng2-bootstrap.js
[1] 16.11.05 17:39:46 404 GET /node_modules/jquery/dist/jquery.js
[1] 16.11.05 17:39:46 404 GET /node_modules/bootstrap/dist/js/bootstrap.js
[1] [BS] File changed: public/home/home.js
[1] [BS] File changed: public/common/headers.js
[1] [BS] File changed: public/login/login.js
[1] [BS] File changed: public/common/auth.guard.js
[1] [BS] File changed: public/app/app.routes.js
[1] [BS] File changed: public/app/app.js
[1] [BS] File changed: public/common/validation.service.js
[1] [BS] File changed: public/common/control-messages.component.js
[1] [BS] File changed: public/common/multiselect-dropdown.js
[1] [BS] File changed: public/home/index.js
[1] [BS] File changed: public/influxmeas/influxmeascfg.service.js
[1] [BS] File changed: public/snmpmetric/snmpmetriccfg.service.js
[1] [BS] File changed: public/influxmeas/influxmeascfg.component.js
[1] [BS] File changed: public/influxserver/influxservercfg.service.js
[1] [BS] File changed: public/influxserver/influxservercfg.component.js
[0] 5:39:49 PM - Compilation complete. Watching for file changes.
[1] [BS] File changed: public/login/index.js
[1] 16.11.05 17:39:49 304 GET /index.html
[1] 16.11.05 17:39:49 200 GET /index.html
[1] 16.11.05 17:39:49 404 GET /public/css/simple-sidebar.css
[1] 16.11.05 17:39:49 404 GET /node_modules/bootstrap/dist/css/bootstrap.css
[1] 16.11.05 17:39:49 404 GET /node_modules/core-js/client/shim.min.js
[1] 16.11.05 17:39:49 404 GET /node_modules/zone.js/dist/zone.js
[1] 16.11.05 17:39:49 404 GET /node_modules/reflect-metadata/Reflect.js
[1] 16.11.05 17:39:49 304 GET /systemjs.config.js
[1] 16.11.05 17:39:49 404 GET /node_modules/ng2-bootstrap/bundles/ng2-bootstrap.js
[1] 16.11.05 17:39:49 404 GET /node_modules/systemjs/dist/system.src.js
[1] 16.11.05 17:39:49 404 GET /node_modules/jquery/dist/jquery.js
[1] 16.11.05 17:39:49 404 GET /node_modules/bootstrap/dist/js/bootstrap.js
[1] 16.11.05 17:39:49 404 GET /node_modules/ng2-bootstrap/bundles/ng2-bootstrap.js
[1] 16.11.05 17:39:49 404 GET /node_modules/jquery/dist/jquery.js
[1] [BS] File changed: public/measfilter/measfiltercfg.component.js
[1] [BS] File changed: public/main.module.js
[1] [BS] File changed: public/main.js
[1] 16.11.05 17:39:49 404 GET /node_modules/bootstrap/dist/js/bootstrap.js
[1] 16.11.05 17:39:56 404 GET /public/css/simple-sidebar.css
[1] 16.11.05 17:39:56 404 GET /node_modules/bootstrap/dist/css/bootstrap.css

SQL log goes always to $CWD/sql.log although modified in config.toml

In the conf

[general]
 logdir = "./logs/san"
 logLevel = "debug"

but goes to

./log/sql.log

Out of order failure

If I add some specific devices to configuration, snmpcollector stops working with such error:

FATA[0856] SNMP bulkwalk error Out of order response

To avoid such error in net-snmp commands can be used key "-Cc":
http://www.net-snmp.org/docs/man/snmpbulkwalk.html

Panic on selfmon

INFO[2016-12-07 19:49:58] beggining Influx Sender thread: [default]    
panic: runtime error: invalid memory address or nil pointer dereference
[signal 0xb code=0x1 addr=0x0 pc=0x434064]

goroutine 26 [running]:
panic(0xc29860, 0xc82000a110)
	/usr/local/go/src/runtime/panic.go:481 +0x3e6
main.(*SelfMonConfig).AddDeviceMetrics(0x1316ef0, 0xc82020c6e0, 0xb, 0x3fd3ca557e8ca2e6)
	/home/developer/src/gospace/src/github.com/toni-moreno/snmpcollector/pkg/selfmon.go:112 +0x4c4
main.(*SnmpDevice).startGatherGo(0xc8201b5500, 0x13348a0)
	/home/developer/src/gospace/src/github.com/toni-moreno/snmpcollector/pkg/snmpdevice.go:508 +0x1eb2
created by main.(*SnmpDevice).StartGather
	/home/developer/src/gospace/src/github.com/toni-moreno/snmpcollector/pkg/snmpdevice.go:418 +0x3f

It seems related to the self monitoring object when is disabled

[Feature Request] Import/Export Configutation Templates

When working with more than one agent we will need share configuration items. In this scenario user might want import/export.

Items could be:

metric
measurement
measurement groups
measurement filters
influxdb servers
devices (full config)
device templates ( not full configured) ??

[Feature Request] send only on value changes

Could be interesting implement a new metric type which only sends value changes as sugested in influxdata/influxdb#7739.

[Feature Request] Add SNMP Interactive Query Console on the device config UI.

In the same way we are already done snmp connection tests , could be a big improvement add a little console to send arbitrary OID/Metrics though the already configured device.

UI Web Server error on runtime/snmpdev/:id request

Server is returning error when trying to request the runtime info of a single snmpdev:

GET http://myurl/runtime/info/mySNMPDevice

body:"json: unsupported type: func() (int64, int64, error)"
headers:Headers
ok:false
status:500
statusText:"Internal Server Error"

[Feature Request] Add a download log file button to the Runtime WebUI viewer

I did a mock-up.

InfluxDB write error

In 0.5.1 release have such errors for several devices:

INFO[2016-10-26 12:34:07] beggining Influx Emmiter thread: 5577006791947779410 
INFO[2016-10-26 12:34:07] Server is running on localhost:8090...       
ERRO[2016-10-26 12:34:07] influxdb write error:  {"error":"partial write:\nunable to parse 'Counters,ifIndex=11439,router=sw127x142_N69_SKD ifHCInBroadcastPkts=,ifHCInMulticastPkts=,ifHCInOctets=,ifHCInUcastPkts=,ifHCOutBroadcastPkts=,ifHCOutMulticastPkts=,ifHCOutOctets=,ifHCOutUcastPkts=': missing field value\nunable to parse 'Counters,ifIndex=10088,router=sw127x142_N69_SKD ifHCInBroadcastPkts=,ifHCInMulticastPkts=,ifHCInOctets=,ifHCInUcastPkts=,ifHCOutBroadcastPkts=,ifHCOutMulticastPkts=,ifHCOutOctets=,ifHCOutUcastPkts=': missing field value\nunable to parse 'Counters,ifIndex=10001,router=sw127x142_N69_SKD ifHCInBroadcastPkts=,ifHCInMulticastPkts=,ifHCInOctets=,ifHCInUcastPkts=,ifHCOutBroadcastPkts=,ifHCOutMulticastPkts=,ifHCOutOctets=,ifHCOutUcastPkts=': missing field value\nunable to parse 'Counters,ifIndex=8193,router=sw127x142_N69_SKD ifHCInBroadcastPkts=,ifHCInMulticastPkts=,ifHCInOctets=,ifHCInUcastPkts=,ifHCOutBroadcastPkts=,ifHCOutMulticastPkts=,ifHCOutOctets=,ifHCOutUcastPkts=': missing field value"}
ERRO[2016-10-26 12:34:37] influxdb write error:  {"error":"partial write:\nunable to parse 'Counters,ifIndex=11439,router=sw127x142_N69_SKD ifHCInBroadcastPkts=,ifHCInMulticastPkts=,ifHCInOctets=,ifHCInUcastPkts=,ifHCOutBroadcastPkts=,ifHCOutMulticastPkts=,ifHCOutOctets=,ifHCOutUcastPkts=': missing field value\nunable to parse 'Counters,ifIndex=10088,router=sw127x142_N69_SKD ifHCInBroadcastPkts=,ifHCInMulticastPkts=,ifHCInOctets=,ifHCInUcastPkts=,ifHCOutBroadcastPkts=,ifHCOutMulticastPkts=,ifHCOutOctets=,ifHCOutUcastPkts=': missing field value\nunable to parse 'Counters,ifIndex=10001,router=sw127x142_N69_SKD ifHCInBroadcastPkts=,ifHCInMulticastPkts=,ifHCInOctets=,ifHCInUcastPkts=,ifHCOutBroadcastPkts=,ifHCOutMulticastPkts=,ifHCOutOctets=,ifHCOutUcastPkts=': missing field value\nunable to parse 'Counters,ifIndex=8193,router=sw127x142_N69_SKD ifHCInBroadcastPkts=,ifHCInMulticastPkts=,ifHCInOctets=,ifHCInUcastPkts=,ifHCOutBroadcastPkts=,ifHCOutMulticastPkts=,ifHCOutOctets=,ifHCOutUcastPkts=': missing field value"}

It seem that errors appear only for VLAN interfaces:

IF-MIB::ifName.8193 = STRING: ifc8193 (Slot: 11 Port: 2)
IF-MIB::ifName.10001 = STRING: ifc10001 VLAN #1
IF-MIB::ifName.10088 = STRING: ifc10088 VLAN #88
IF-MIB::ifName.11439 = STRING: ifc11439 VLAN #1439

We didn't have such errors in previous snmpcollector releases.

TypeError in ValidationService.OIDValidator, UI lockup

What:

Reset form action in SNMP metrics form throws a TypeError. Second reset of form locks up the entire UI. All buttons and left side menu become unresponsive.

Version

SNMPCollector - 0.5.3 - ff26b4b (Nov 7, 2016)

Steps to reproduce:

Login
Select SNMP metrics
Select Add metric
Select DataSrcType Integer (any of them really)
Click Reset (at this point you will get error below)
Fill in the form again
Click Reset again (now the entire UI will lockup including left-side menu)

Angular 2 is running in the development mode. Call enableProdMode() to enable the production mode.
id_token: OK
check canActivate:
creating home!! id_token:OK
Task Service created. Http {_backend: XHRBackend, _defaultOptions: BaseRequestOptions}
Task Service created. Http {_backend: XHRBackend, _defaultOptions: BaseRequestOptions}
Task Service created. Http {_backend: XHRBackend, _defaultOptions: BaseRequestOptions}
Task Service created. Http {_backend: XHRBackend, _defaultOptions: BaseRequestOptions}
MAP SERVICE null
DONE
Task Service created. Http {_backend: XHRBackend, _defaultOptions: BaseRequestOptions}
MAP SERVICE [Object]
FOREACH LOOP Object {ID: "1", FieldName: "ifHCInOctets", Description: "ifHCInOctets", BaseOID: ".1.3.6.1.2.1.31.1.1.1.6", DataSrcType: "COUNTER64"…} 1
DONE
EXCEPTION: Error in public/snmpmetric/snmpmetriceditor.html:41:4 caused by: Cannot read property 'match' of null
ORIGINAL EXCEPTION: Cannot read property 'match' of null
ORIGINAL STACKTRACE:
TypeError: Cannot read property 'match' of null
    at ValidationService.OIDValidator (http://localhost:8090/public/common/validation.service.js:52:38)
    at eval (http://localhost:8090/node_modules/@angular/forms/bundles/forms.umd.js:658:53)
    at Array.map (native)
    at _executeValidators (http://localhost:8090/node_modules/@angular/forms/bundles/forms.umd.js:658:27)
    at eval (http://localhost:8090/node_modules/@angular/forms/bundles/forms.umd.js:638:37)
    at eval (http://localhost:8090/node_modules/@angular/forms/bundles/forms.umd.js:658:53)
    at Array.map (native)
    at _executeValidators (http://localhost:8090/node_modules/@angular/forms/bundles/forms.umd.js:658:27)
    at FormControl.eval [as validator] (http://localhost:8090/node_modules/@angular/forms/bundles/forms.umd.js:638:37)
    at FormControl.AbstractControl._runValidator (http://localhost:8090/node_modules/@angular/forms/bundles/forms.umd.js:2203:53)
ERROR CONTEXT:
DebugContext {_view: _View_SnmpMetricCfgComponent5, _nodeIndex: 1, _tplRow: 41, _tplCol: 4}
Uncaught ViewWrappedError {_nativeError: Error: Error in public/snmpmetric/snmpmetriceditor.html:41:4 caused by: Cannot read property 'match'…, originalError: TypeError: Cannot read property 'match' of null
    at ValidationService.OIDValidator (http://localh…, context: DebugContext}
```

Influx database support

Which influx databases are supported? I currently use 0.13.0. But I don't see any metrics in the database.

Kind regards,
Theo

JavaScript errors in Frontend

Chrome console is reporting the following JS errors. Same install and environment as #27

Uncaught ReferenceError: System is not defined
    at http://localhost:3002/systemjs.config.js:6:3
    at http://localhost:3002/systemjs.config.js:56:3

Uncaught ReferenceError: System is not defined
    at http://localhost:3002/:23:9

All for System not being defined

[Feature Request] Add option to initialize only the web ui frontend and operate snmp agent as needed

Could be good if not sure if our config is good working with configurations in the webui and the background agent stopped until needed.

In the same way the webUI will need a start/stop/restart button and the REST API will have to provide these new features.

[Feature Request ] Self Statistics 2 suggested improvements

Related to #25

Now we have process time for all measurements for each device, but we have only 2 tags.
instance and device.

Improvement A

Could be interesting to add all device ExtraTAGS to each device selfmon metric. With this tags we can group devices as the customer group for its Metrics.

Improvement B

We are currently seeing errors on runtime view but could be interesting if we have total_metrics vs metric_errors in the sefl Statistics measurement

So current selfmon_rt should be renamed to selfmon_device_stats

with following fields

metric_total
metric_error
process_t

Feature-request: Distributed infrastructure

We would like to be able to configure multiple collectors from a single point/interface. Also data should be stored either centrally or the metadata should be ordered in such a way that graphical interfaces like grafana can be used to generate templates to display the collected data.

Kind regards

Theo

[ Feature Request ] send gather snmp time as Self Stats monitoring

We are computing and logging for each device the time that the engine has took to get all data, could be interesting also send to the influx monitoring DB.

INFO[2016-10-28 05:54:58] snmpdevice [hostsnmpv3b] snmp pooling took [476.647899ms]  
INFO[2016-10-28 05:55:28] snmpdevice [hostsnmpv3b] snmp pooling took [488.502427ms]  
INFO[2016-10-28 05:55:58] snmpdevice [hostsnmpv3b] snmp pooling took [526.179549ms]  
INFO[2016-10-28 05:56:28] snmpdevice [hostsnmpv3b] snmp pooling took [579.377993ms]  
INFO[2016-10-28 05:56:58] snmpdevice [hostsnmpv3b] snmp pooling took [473.428373ms]  
INFO[2016-10-28 05:57:28] snmpdevice [hostsnmpv3b] snmp pooling took [527.389091ms]  
INFO[2016-10-28 05:57:58] snmpdevice [hostsnmpv3b] snmp pooling took [523.378573ms]  
INFO[2016-10-28 05:58:28] snmpdevice [hostsnmpv3b] snmp pooling took [510.147521ms]  
INFO[2016-10-28 05:58:58] snmpdevice [hostsnmpv3b] snmp pooling took [488.666555ms]  
INFO[2016-10-28 14:28:50] snmpdevice [hostsnmpv3b] snmp pooling took [334.849698ms]

Making daemon issue

Trying to make demon for snmpcollector with such script:

#! /bin/sh
# /etc/init.d/snmpcollector
#

# Some things that run always
#touch /var/lock/blah

# Carry out specific functions when asked to by the system
DAEMON=/opt/snmpcollector/snmpcollector
DAEMONARGS='-config /opt/snmpcollector/conf/config.toml -logs /opt/snmpcollector/log/'
case "$1" in
  start)
    echo "Starting snmpcollector "
    start-stop-daemon --start --make-pidfile --pidfile /var/run/snmpcollector.pid --exec $DAEMON -- $DAEMONARGS
    ;;
  stop)
    echo "Stopping script blah"
    start-stop-daemon --stop --pidfile /var/run/snmpcollector.pid
    ;;
  *)
    echo "Usage: /etc/init.d/blah {start|stop}"
    exit 1
    ;;
esac

exit 0

I get error: "Fail to sync database InfluxCfg: unable to open database file"
Can you help solving it?

Full daemon log:

Oct 20 15:00:04 debian snmpcollector[6854]: time="2016-10-20 15:00:04" level=info msg="set Default directories : \n   - Exec: /\n   - Config: /conf\n   -Logs: /opt/snmpcollector/log/\n"
Oct 20 15:00:04 debian snmpcollector[6854]: time="2016-10-20 15:00:04" level=debug msg="{General:{LogDir:./log LogLevel:debug} Database:{Type:sqlite3 Host: Name:snmpcollector User: Pass: SQLLogFile:sql.log Debug:1 x:<nil>} Selfmon:{Enabled:true Freq:60 Prefix:gvm ExtraTags:[instance=snmpcollector01] Influx:<nil> runtimeStatsRunning:false TagMap:map[] bps:<nil>} Metrics:map[] Measurements:map[] MFilters:map[] GetGroups:map[] SnmpDevice:map[] Influxdb:map[] HTTP:{Port:8090 AdminUser:adm1 AdminPassword:adm1pass}}"
Oct 20 15:00:04 debian snmpcollector[6854]: time="2016-10-20 15:00:04" level=debug msg="Database config: &{Type:sqlite3 Host: Name:snmpcollector User: Pass: SQLLogFile:sql.log Debug:1 x:<nil>}"
Oct 20 15:00:04 debian snmpcollector[6854]: time="2016-10-20 15:00:04" level=fatal msg="Fail to sync database InfluxCfg: unable to open database file\n"

InfluxDB write error

Can't find collected data in influxdb.
Collector files:
https://drive.google.com/open?id=0B0rkzfzNglIwaTlFaG05OVowM2s
Logs says that data was sent, but influx database is empty.
Tried to change remote influxdb to local, creation of new database. Nothing helped.

Standart output:

#./snmpcollector
INFO[2016-10-14 12:48:44] set Default directories : 
   - Exec: /opt/snmpcollector
   - Config: /opt/snmpcollector/conf
   -Logs: /opt/snmpcollector/log

DEBU[2016-10-14 12:48:44] {General:{LogDir:./log LogLevel:debug} Database:{Type:sqlite3 Host: Name:snmpcollector User: Pass: SQLLogFile:sql.log Debug:1 x:<nil>} Selfmon:{Enabled:true Freq:60 Prefix:gvm ExtraTags:[instance=snmpcollector01] Influx:<nil> runtimeStatsRunning:false TagMap:map[] bps:<nil>} Metrics:map[] Measurements:map[] MFilters:map[] GetGroups:map[] SnmpDevice:map[] Influxdb:map[] HTTP:{Port:8090 AdminUser:adm1 AdminPassword:adm1pass}} 
DEBU[2016-10-14 12:48:44] Database config: &{Type:sqlite3 Host: Name:snmpcollector User: Pass: SQLLogFile:sql.log Debug:1 x:<nil>} 
DEBU[2016-10-14 12:48:44] {ID:default Host:172.26.40.253 Port:8086 DB:test_db User: Password: Retention:default} 
DEBU[2016-10-14 12:48:44] {ID:Linux_user_CPU_percent FieldName:user Description:percentage of user CPU time BaseOID:.1.3.6.1.4.1.2021.11.9.0 DataSrcType:INTEGER GetRate:false Scale:0 Shift:0} 
DEBU[2016-10-14 12:48:44] {ID:Linux_system_CPU_percent FieldName:system Description:percentage of system CPU time BaseOID:.1.3.6.1.4.1.2021.11.10.0 DataSrcType:INTEGER GetRate:false Scale:0 Shift:0} 
DEBU[2016-10-14 12:48:44] {ID:Linux_idle_CPU_percent FieldName:idle Description:percentage of Idle CPU time BaseOID:.1.3.6.1.4.1.2021.11.11.0 DataSrcType:INTEGER GetRate:false Scale:0 Shift:0} 
DEBU[2016-10-14 12:48:44] {ID:ifHCInOctets FieldName:In.bytes Description:Bytes In - 64-bit Counters BaseOID:.1.3.6.1.2.1.31.1.1.1.6 DataSrcType:COUNTER64 GetRate:false Scale:0 Shift:0} 
DEBU[2016-10-14 12:48:44] {ID:ifHCOutOctets FieldName:Out.bytes Description:Bytes Out - 64-bit Counters BaseOID:.1.3.6.1.2.1.31.1.1.1.10 DataSrcType:COUNTER64 GetRate:false Scale:0 Shift:0} 
DEBU[2016-10-14 12:48:44] {ID:ifHCInUcastPkts FieldName:In.packets Description:Packets In - 64-bit Counters BaseOID:.1.3.6.1.2.1.31.1.1.1.7 DataSrcType:COUNTER64 GetRate:false Scale:0 Shift:0} 
DEBU[2016-10-14 12:48:44] {ID:ifHCOutUcastPkts FieldName:Out.packets Description:Packets Out - 64-bit Counters BaseOID:.1.3.6.1.2.1.31.1.1.1.11 DataSrcType:COUNTER64 GetRate:false Scale:0 Shift:0} 
DEBU[2016-10-14 12:48:44] {ID:ifInOctets FieldName:In.bytes Description:Bytes In - 32-bit Counters BaseOID:.1.3.6.1.2.1.2.2.1.10 DataSrcType:COUNTER32 GetRate:false Scale:0 Shift:0} 
DEBU[2016-10-14 12:48:44] {ID:ifOutOctets FieldName:Out.bytes Description:Bytes Out - 32-bit Counters BaseOID:.1.3.6.1.2.1.2.2.1.16 DataSrcType:COUNTER32 GetRate:false Scale:0 Shift:0} 
DEBU[2016-10-14 12:48:44] {ID:linux_cpu Name:linux.cpu GetMode:value IndexOID: IndexTag: Fields:[Linux_user_CPU_percent Linux_system_CPU_percent Linux_idle_CPU_percent] fieldMetric:[]} 
DEBU[2016-10-14 12:48:44] {ID:linux_ports Name:linux.ports GetMode:indexed IndexOID:.1.3.6.1.2.1.31.1.1.1.1 IndexTag:portName Fields:[ifHCInOctets ifHCOutOctets ifHCInUcastPkts ifHCOutUcastPkts] fieldMetric:[]} 
DEBU[2016-10-14 12:48:44] {ID:linux_ports_by_index Name:linux.ports GetMode:indexed IndexOID:.1.3.6.1.2.1.2.2.1.1 IndexTag:ifindex Fields:[ifHCInOctets ifHCOutOctets ifHCInUcastPkts ifHCOutUcastPkts] fieldMetric:[]} 
DEBU[2016-10-14 12:48:44] {ID:network_32bits Name:32bits-ports GetMode:indexed IndexOID:.1.3.6.1.2.1.31.1.1.1.1 IndexTag:portName Fields:[ifInOctets ifOutOctets] fieldMetric:[]} 
DEBU[2016-10-14 12:48:44] {ID:filter_ports_file_a IDMeasurementCfg:linux_ports FType:file FileName:lp_file_filter_a.txt EnableAlias:true OIDCond: CondType: CondValue:} 
DEBU[2016-10-14 12:48:44] {ID:filter_ports_file_b IDMeasurementCfg:linux_ports FType:file FileName:lp_file_filter_b.txt EnableAlias:true OIDCond: CondType: CondValue:} 
DEBU[2016-10-14 12:48:44] {ID:filter_port_if_status_up IDMeasurementCfg:linux_ports FType:OIDCondition FileName: EnableAlias:false OIDCond:.1.3.6.1.2.1.2.2.1.8 CondType:neq CondValue:1} 
DEBU[2016-10-14 12:48:44] {ID:filter_port_if_name_match_eth IDMeasurementCfg:linux_ports FType:OIDCondition FileName: EnableAlias:false OIDCond:.1.3.6.1.2.1.31.1.1.1.1 CondType:match CondValue:eth.*} 
DEBU[2016-10-14 12:48:44] {ID:Linux Measurements:[linux_cpu linux_ports]} 
DEBU[2016-10-14 12:48:44] {ID:Issue_index Measurements:[linux_cpu linux_ports_by_index]} 
DEBU[2016-10-14 12:48:44] {ID:switch Host:172.26.40.5 Port:161 Retries:2 Timeout:60 Repeat:5 SnmpVersion:2c Community:public V3SecLevel: V3AuthUser: V3AuthPass: V3AuthProt: V3PrivPass: V3PrivProt: Freq:60 OutDB: LogLevel:debug LogFile: SnmpDebug:true DeviceTagName:router DeviceTagValue:id ExtraTags:[tagA=4 tagB=5 tagC=6] MetricGroups:[] MeasFilters:[]} 
DEBU[2016-10-14 12:48:44] {General:{LogDir:./log LogLevel:debug} Database:{Type:sqlite3 Host: Name:snmpcollector User: Pass: SQLLogFile:sql.log Debug:1 x:0xc82014e960} Selfmon:{Enabled:true Freq:60 Prefix:gvm ExtraTags:[instance=snmpcollector01] Influx:<nil> runtimeStatsRunning:false TagMap:map[] bps:<nil>} Metrics:map[ifOutOctets:0xc820185dc0 Linux_user_CPU_percent:0xc820185a40 Linux_idle_CPU_percent:0xc820185b20 ifHCInUcastPkts:0xc820185c70 ifHCOutUcastPkts:0xc820185ce0 ifInOctets:0xc820185d50 Linux_system_CPU_percent:0xc820185ab0 ifHCInOctets:0xc820185b90 ifHCOutOctets:0xc820185c00] Measurements:map[network_32bits:0xc8201a5580 linux_cpu:0xc8201a5400 linux_ports:0xc8201a5480 linux_ports_by_index:0xc8201a5500] MFilters:map[filter_ports_file_a:0xc8201a5c00 filter_ports_file_b:0xc8201a5d00 filter_port_if_status_up:0xc8201a5e00 filter_port_if_name_match_eth:0xc8201a5f00] GetGroups:map[Linux:0xc8201b9ec0 Issue_index:0xc8201b9ef0] SnmpDevice:map[switch:0xc8201cc600] Influxdb:map[default:0xc8201855e0] HTTP:{Port:8090 AdminUser:adm1 AdminPassword:adm1pass}} 
DEBU[2016-10-14 12:48:44] --------------------Initializing Config metrics------------------- 
DEBU[2016-10-14 12:48:44] Initializing SNMPMetricconfig...             
DEBU[2016-10-14 12:48:44] Initializing MEASSUREMENTSconfig...          
INFO[2016-10-14 12:48:44] processing measurement key: linux_cpu        
DEBU[2016-10-14 12:48:44] %+v&{linux_cpu linux.cpu value   [Linux_user_CPU_percent Linux_system_CPU_percent Linux_idle_CPU_percent] []} 
DEBU[2016-10-14 12:48:44] looking for measure linux.cpu fields: Linux_user_CPU_percent 
DEBU[2016-10-14 12:48:44] Found ok!                                    
DEBU[2016-10-14 12:48:44] looking for measure linux.cpu fields: Linux_system_CPU_percent 
DEBU[2016-10-14 12:48:44] Found ok!                                    
DEBU[2016-10-14 12:48:44] looking for measure linux.cpu fields: Linux_idle_CPU_percent 
DEBU[2016-10-14 12:48:44] Found ok!                                    
DEBU[2016-10-14 12:48:44] FIELDMETRICS: [0xc820185a40 0xc820185ab0 0xc820185b20] 
INFO[2016-10-14 12:48:44] processing measurement key: linux_ports      
DEBU[2016-10-14 12:48:44] %+v&{linux_ports linux.ports indexed .1.3.6.1.2.1.31.1.1.1.1 portName [ifHCInOctets ifHCOutOctets ifHCInUcastPkts ifHCOutUcastPkts] []} 
DEBU[2016-10-14 12:48:44] looking for measure linux.ports fields: ifHCInOctets 
DEBU[2016-10-14 12:48:44] Found ok!                                    
DEBU[2016-10-14 12:48:44] looking for measure linux.ports fields: ifHCOutOctets 
DEBU[2016-10-14 12:48:44] Found ok!                                    
DEBU[2016-10-14 12:48:44] looking for measure linux.ports fields: ifHCInUcastPkts 
DEBU[2016-10-14 12:48:44] Found ok!                                    
DEBU[2016-10-14 12:48:44] looking for measure linux.ports fields: ifHCOutUcastPkts 
DEBU[2016-10-14 12:48:44] Found ok!                                    
DEBU[2016-10-14 12:48:44] FIELDMETRICS: [0xc820185b90 0xc820185c00 0xc820185c70 0xc820185ce0] 
INFO[2016-10-14 12:48:44] processing measurement key: linux_ports_by_index 
DEBU[2016-10-14 12:48:44] %+v&{linux_ports_by_index linux.ports indexed .1.3.6.1.2.1.2.2.1.1 ifindex [ifHCInOctets ifHCOutOctets ifHCInUcastPkts ifHCOutUcastPkts] []} 
DEBU[2016-10-14 12:48:44] looking for measure linux.ports fields: ifHCInOctets 
DEBU[2016-10-14 12:48:44] Found ok!                                    
DEBU[2016-10-14 12:48:44] looking for measure linux.ports fields: ifHCOutOctets 
DEBU[2016-10-14 12:48:44] Found ok!                                    
DEBU[2016-10-14 12:48:44] looking for measure linux.ports fields: ifHCInUcastPkts 
DEBU[2016-10-14 12:48:44] Found ok!                                    
DEBU[2016-10-14 12:48:44] looking for measure linux.ports fields: ifHCOutUcastPkts 
DEBU[2016-10-14 12:48:44] Found ok!                                    
DEBU[2016-10-14 12:48:44] FIELDMETRICS: [0xc820185b90 0xc820185c00 0xc820185c70 0xc820185ce0] 
INFO[2016-10-14 12:48:44] processing measurement key: network_32bits   
DEBU[2016-10-14 12:48:44] %+v&{network_32bits 32bits-ports indexed .1.3.6.1.2.1.31.1.1.1.1 portName [ifInOctets ifOutOctets] []} 
DEBU[2016-10-14 12:48:44] looking for measure 32bits-ports fields: ifInOctets 
DEBU[2016-10-14 12:48:44] Found ok!                                    
DEBU[2016-10-14 12:48:44] looking for measure 32bits-ports fields: ifOutOctets 
DEBU[2016-10-14 12:48:44] Found ok!                                    
DEBU[2016-10-14 12:48:44] FIELDMETRICS: [0xc820185d50 0xc820185dc0]    
DEBU[2016-10-14 12:48:44] -----------------------END Config metrics---------------------- 
DEBU[2016-10-14 12:48:44] {General:{LogDir:./log LogLevel:debug} Database:{Type:sqlite3 Host: Name:snmpcollector User: Pass: SQLLogFile:sql.log Debug:1 x:0xc82014e960} Selfmon:{Enabled:true Freq:60 Prefix:gvm ExtraTags:[instance=snmpcollector01] Influx:<nil> runtimeStatsRunning:false TagMap:map[] bps:<nil>} Metrics:map[ifInOctets:0xc820185d50 ifOutOctets:0xc820185dc0 Linux_user_CPU_percent:0xc820185a40 Linux_idle_CPU_percent:0xc820185b20 ifHCInUcastPkts:0xc820185c70 ifHCOutUcastPkts:0xc820185ce0 Linux_system_CPU_percent:0xc820185ab0 ifHCInOctets:0xc820185b90 ifHCOutOctets:0xc820185c00] Measurements:map[linux_cpu:0xc8201a5400 linux_ports:0xc8201a5480 linux_ports_by_index:0xc8201a5500 network_32bits:0xc8201a5580] MFilters:map[filter_ports_file_a:0xc8201a5c00 filter_ports_file_b:0xc8201a5d00 filter_port_if_status_up:0xc8201a5e00 filter_port_if_name_match_eth:0xc8201a5f00] GetGroups:map[Linux:0xc8201b9ec0 Issue_index:0xc8201b9ef0] SnmpDevice:map[switch:0xc8201cc600] Influxdb:map[default:0xc8201855e0] HTTP:{Port:8090 AdminUser:adm1 AdminPassword:adm1pass}} 
INFO[2016-10-14 12:48:44] Initializing device switch

INFO[2016-10-14 12:48:44] Initializing influxdb with id = default      
INFO[2016-10-14 12:48:44] Server is running on localhost:8090...       
INFO[2016-10-14 12:48:44] beggining Influx Emmiter thread: 5577006791947779410 
DEBU[2016-10-14 12:48:44] sending data from Emmiter 5577006791947779410 
DEBU[2016-10-14 12:49:44] sending data from Emmiter 5577006791947779410 
DEBU[2016-10-14 12:50:44] sending data from Emmiter 5577006791947779410 
DEBU[2016-10-14 12:51:44] sending data from Emmiter 5577006791947779410 
DEBU[2016-10-14 12:52:44] sending data from Emmiter 5577006791947779410

Switch log:

�[34mINFO�[0m[0000] First SNMP connection to host  172.26.40.5 stablished 
�[37mDEBU�[0m[0000] DEBUG pdu:{Name:.1.3.6.1.2.1.1.1.0 Type:4 Value:[67 105 115 99 111 32 73 79 83 32 83 111 102 116 119 97 114 101 44 32 67 50 57 54 48 32 83 111 102 116 119 97 114 101 32 40 67 50 57 54 48 45 76 65 78 66 65 83 69 45 77 41 44 32 86 101 114 115 105 111 110 32 49 50 46 50 40 50 53 41 83 69 69 49 44 32 82 69 76 69 65 83 69 32 83 79 70 84 87 65 82 69 32 40 102 99 49 41 13 10 67 111 112 121 114 105 103 104 116 32 40 99 41 32 49 57 56 54 45 50 48 48 54 32 98 121 32 67 105 115 99 111 32 83 121 115 116 101 109 115 44 32 73 110 99 46 13 10 67 111 109 112 105 108 101 100 32 83 117 110 32 50 49 45 77 97 121 45 48 54 32 50 49 58 51 51 32 98 121 32 121 101 110 97 110 104]} 
�[37mDEBU�[0m[0000] DEBUG pdu:{Name:.1.3.6.1.2.1.1.3.0 Type:67 Value:4196630943} 
�[37mDEBU�[0m[0000] DEBUG pdu:{Name:.1.3.6.1.2.1.1.4.0 Type:4 Value:[]} 
�[37mDEBU�[0m[0000] DEBUG pdu:{Name:.1.3.6.1.2.1.1.5.0 Type:4 Value:[83 119 105 116 99 104]} 
�[37mDEBU�[0m[0000] DEBUG pdu:{Name:.1.3.6.1.2.1.1.6.0 Type:4 Value:[0]} 
�[34mINFO�[0m[0000] Got basic system info main.SysInfo{sysDescr:"\x00", sysUptime:41966309000000000, sysContact:"", sysName:"Switch", sysLocation:""}  
�[37mDEBU�[0m[0000] -----------------Init device 172.26.40.5------------------ 
�[34mINFO�[0m[0000] Beginning gather process for device switch (172.26.40.5) 
�[34mINFO�[0m[0000] snmpdevice [switch] snmp poolling took [255ns]  
�[34mINFO�[0m[0000] snmpdevice [switch] influx send took [1.153µs] 
�[34mINFO�[0m[0060] snmpdevice [switch] snmp poolling took [737ns]  
�[34mINFO�[0m[0060] snmpdevice [switch] influx send took [16.326µs]

[Feature Request] Add subIndexOID on measurements go get index descriptors

To explain this feature let me show you an example.

On Cisco Switches, the CPU Base OID (table) is: .1.3.6.1.4.1.9.9.109.1.1.1.1.7 giving as ouptut the different CPU usage according to the physical devices.

CPU Query

SNMPv2-SMI::enterprises.9.9.109.1.1.1.1.7.22 = Gauge32: 2
SNMPv2-SMI::enterprises.9.9.109.1.1.1.1.7.23 = Gauge32: 1
SNMPv2-SMI::enterprises.9.9.109.1.1.1.1.7.24 = Gauge32: 18
SNMPv2-SMI::enterprises.9.9.109.1.1.1.1.7.25 = Gauge32: 23

The table indexes are: 22,23,24,25.
According to Cisco docs, the indexes are defined on physicalIndex table: .1.3.6.1.2.1.47.1.1.1.1.7, giving as output all indexes:

PhysicalIndex Query

SNMPv2-SMI::enterprises.9.9.109.1.1.1.1.2.22 = INTEGER: 22
SNMPv2-SMI::enterprises.9.9.109.1.1.1.1.2.23 = INTEGER: 23
SNMPv2-SMI::enterprises.9.9.109.1.1.1.1.2.24 = INTEGER: 24
SNMPv2-SMI::enterprises.9.9.109.1.1.1.1.2.25 = INTEGER: 25

As you can see this is not returning related names of the physical devices so indexName bound with the device would be:

key	value
cpuphysicalIndex	22
cpuphysicalIndex	23
cpuphysicalIndex	24
cpuphysicalIndex	25

To get the real name (descriptor) assigned to each physDevice the following OID must be used to index: .1.3.6.1.2.1.47.1.1.1.1.7
PhysicalDescriptors Query

SNMPv2-SMI::mib-2.47.1.1.1.1.7.10 = STRING: "Fabric Stack Root"
SNMPv2-SMI::mib-2.47.1.1.1.1.7.22 = STRING: "Supervisor Module-2"
SNMPv2-SMI::mib-2.47.1.1.1.1.7.23 = STRING: "Supervisor Module-2"
SNMPv2-SMI::mib-2.47.1.1.1.1.7.24 = STRING: "10 Gbps Ethernet Module"
SNMPv2-SMI::mib-2.47.1.1.1.1.7.25 = STRING: "1/10 Gbps Ethernet Module"
SNMPv2-SMI::mib-2.47.1.1.1.1.7.149 = STRING: "Nexus7000 C7004 (4 Slot) Chassis"
SNMPv2-SMI::mib-2.47.1.1.1.1.7.214 = STRING: "SupSlot-1"

So, in this case, the correct way to do it would be index by PhysicalDescriptors... but... the index 22 and 23 has the same name, so on our measurement only 1 of both would be generated!!

key	value
cpuphysicalIndex	Supervisor Module-2
cpuphysicalIndex	10 Gbps Ethernet Module
cpuphysicalIndex	1/10 Gbps Ethernet Module

To solve that would be great if we could make a second OID indexer to get the name querying only the indexes got by indexOID query. Following with the example above:

Base OID: .1.3.6.1.4.1.9.9.109.1.1.1.1.7 {22,23,24,25}
IndexOID: .1.3.6.1.2.1.47.1.1.1.1.7 {22,23,24,25}
Name Index:

SNMPv2-SMI::mib-2.47.1.1.1.1.7.22 = STRING: "Supervisor Module-2"
SNMPv2-SMI::mib-2.47.1.1.1.1.7.23 = STRING: "Supervisor Module-2"
SNMPv2-SMI::mib-2.47.1.1.1.1.7.24 = STRING: "10 Gbps Ethernet Module"
SNMPv2-SMI::mib-2.47.1.1.1.1.7.25 = STRING: "1/10 Gbps Ethernet Module"

And compose the indexName with the indexOID and NameIndex:

key	value
cpuphysicalIndex	22 - Supervisor Module-2
cpuphysicalIndex	23 - Supervisor Module-2
cpuphysicalIndex	24 - 10 Gbps Ethernet Module
cpuphysicalIndex	25 - 1/10 Gbps Ethernet Module

[Feature Request] add Not Match OIDCondition Option

Will be useful to be also a negative option to the string match condition.

[Feature Request] Add ability to add Tag to measurements on Filtering time

Could be interesting that measurement filters could add a new Tag in addition to filter.

Use case

We have some cisco switches with a lot of ports. We would like only get data from the portchannel ports , In this case We will filter by "ifAlias" looking only for "PortChannel X" string .
On runtime snmpcollector will send metrics ( In/out banwidh by example) with PortName as Tag "fc1/1" by example. But nowhere have been registered that this port belongs to PortChannel N ...

Desired

Measurement filters should have 2 more parameters.

"useAsTag" : if true , in addition to filter a new tag will be added to this Indexed value with the value of the index that has been filtered. ( PortChannel X by example)
"TagName" the name of the new tag ( "PortChannel" by example)

Panic on reload Indexes

goroutine 91 [running]:
panic(0xca2180, 0xc8200120f0)
        /usr/local/go/src/runtime/panic.go:481 +0x3e6
main.(*InfluxMeasurement).loadIndexedLabels.func1(0xc821c25a80, 0x17, 0x41, 0xb7bc80, 0xc82197c838, 0x7f29e16272f8, 0xc820488140, 0x0, 0x0)
        /home/developer/src/gospace/src/github.com/toni-moreno/snmpcollector/pkg/influxmeasurement.go:604 +0xaaf
github.com/toni-moreno/snmpcollector/vendor/github.com/soniah/gosnmp.(*GoSNMP).getToWalk(0xc8200e42a0, 0xc8201e4440, 0x1e, 0xc82172a800, 0x0, 0x0)
        /home/developer/src/gospace/src/github.com/toni-moreno/snmpcollector/vendor/github.com/soniah/gosnmp/walk.go:106 +0x20e
github.com/toni-moreno/snmpcollector/vendor/github.com/soniah/gosnmp.(*GoSNMP).walk(0xc8200e42a0, 0xc821d198a5, 0xc8201e4440, 0x1e, 0xc82172a800, 0x0, 0x0)
        /home/developer/src/gospace/src/github.com/toni-moreno/snmpcollector/vendor/github.com/soniah/gosnmp/walk.go:69 +0x74f
github.com/toni-moreno/snmpcollector/vendor/github.com/soniah/gosnmp.(*GoSNMP).BulkWalk(0xc8200e42a0, 0xc8201e4440, 0x1e, 0xc82172a800, 0x0, 0x0)
        /home/developer/src/gospace/src/github.com/toni-moreno/snmpcollector/vendor/github.com/soniah/gosnmp/gosnmp.go:355 +0x50
main.(*InfluxMeasurement).loadIndexedLabels(0xc820e66200, 0xe25ea0, 0x0, 0x0)
        /home/developer/src/gospace/src/github.com/toni-moreno/snmpcollector/pkg/influxmeasurement.go:632 +0x3a6
main.(*InfluxMeasurement).UpdateFilter(0xc820e66200, 0xc82ae53e4f, 0x0, 0x0)
        /home/developer/src/gospace/src/github.com/toni-moreno/snmpcollector/pkg/influxmeasurement.go:274 +0x316
main.(*SnmpDevice).startGatherGo(0xc82047b130, 0x13ce8f0)
        /home/developer/src/gospace/src/github.com/toni-moreno/snmpcollector/pkg/snmpdevice.go:398 +0x20f7
created by main.(*SnmpDevice).StartGather
        /home/developer/src/gospace/src/github.com/toni-moreno/snmpcollector/pkg/snmpdevice.go:348 +0x3f

[Bug] Panic when trying to reload filters on devices with non filter assigned

It seems to be a bug on reload filter function.
It crashes when they the service try to reload filters on devices with non filters assigned.

Log:

panic: runtime error: invalid memory address or nil pointer dereference
[signal 0xb code=0x1 addr=0x0 pc=0x40c287]

goroutine 56 [running]:
panic(0xc9b800, 0xc820012130)
        /usr/local/go/src/runtime/panic.go:481 +0x3e6
main.(*InfluxMeasurement).UpdateFilter(0xc8201982a0, 0x5, 0x0, 0x0)
        /home/developer/src/gospace/src/github.com/toni-moreno/snmpcollector/pkg/influxmeasurement.go:281 +0x13c7
main.(*SnmpDevice).Gather(0xc8201cfb20, 0xc820308aa0)
        /home/developer/src/gospace/src/github.com/toni-moreno/snmpcollector/pkg/snmpdevice.go:426 +0x1ed3
created by main.main
        /home/developer/src/gospace/src/github.com/toni-moreno/snmpcollector/pkg/main.go:278 +0x12a

Feature-request: REST Interface

We are looking for tools which can be part of a process chain. That means it should be able to be provisioned by other tools. So it has to have something like a rest interface for example, to add, update and delete measured objects.

Kind regards

Theo

[Feature Request] Add math expressions on measurement as fields

Could be interesting to add field type "expression" .

"suppose measurement with 3 fields a, b, c" , we would like to add another 3 fields for % of each one, we will do in this mode

d= a*100/(a+b+c)
e= b*100/(a+b+c)
f= c*100/(a+b+c)

we can use a evaluation library as https://github.com/Knetic/govaluate

Can not use de WeUI on more than one snmpcollector instances at once.

It has to be with shared cookie /toke name .. Pending to fix.

[Feature Request] Add a Collection Jitter parameter to distribute load on time

As Telegraf does, could be interesting add a collection Jitter to avoid all devices querying at once.

https://github.com/influxdata/telegraf/blob/master/etc/telegraf.conf#L48

toni-moreno / snmpcollector Goto Github PK

snmpcollector's Introduction

SnmpCollector

Run from master

Dependencies

Get Code and setup example config

Building the backend

Building frontend and backend in production mode

Creating minimal package tar.gz

Creating rpm and deb packages

Running first time

Recompile backend on source change (only for developers)

Online config

Offline configuration.

snmpcollector's People

Contributors

Stargazers

Watchers

Forkers

snmpcollector's Issues

What:

Version

Steps to reproduce:

Fix:

What:

Version

Steps to reproduce:

Improvement A

Improvement B

Use case

Desired

Recommend Projects

Recommend Topics

Recommend Org