shirou / mqforward Goto Github PK

MQTT to influxdb forwarder

License: MIT License

Go 98.10% Python 1.90%

mqforward's Introduction

mqforward

mqforward is forwarder from MQTT to Influxdb. This subscribes a topic with wildcard and forward a payload to Influxdb. The value should be JSON or msgpack.

for example,

Publish {"a": 1, "b": 2} to "mqforward/a/b",
Subscribe "mqforward/#",

in the influxdb,

value a=1 and b=2 into the a.b series. so you can SELECT a, b FROM "a.b"

If udp is true in the config, send series over UDP to InfluxDB. otherwise, use HTTP.

usage

install

Just go get (it takes for a while)

$ go get github.com/shirou/mqforward

then, type

$ go install github.com/shirou/mqforward

config

Config example is below. If you put config to ~/.mqforward.ini, it will be loaded automatically.

[mqforward-mqtt]
hostname= localhost
port = 1883
username= ""
password= ""
topic = mqforward/#

[mqforward-influxdb]
# url = https://127.0.01/subpath
hostname = 127.0.0.1
scheme = http # or https
port = 4444
db = test
username = root
password = root
caCerts = ~/.certs/cacert.pem # can be empty
insecure = true # if certificates are not checked
series = weather
topicMap = mqforward/{location}/{sensor} # creates tags 'location' and 'sensor' from topic path

run

Set path to $GOPATH/bin,

$ mqforward run

mqforward run -c someconfig.ini

license

MIT

mqforward's People

Contributors

Stargazers

Watchers

mqforward's Issues

SSL connection

Can you add possibility for adding cert files as you did it in this package https://github.com/shirou/mqttcli ?

{
  "caCert": "path/to/root-CA.crt",
  "clientCert": "path/to/2a338xx2xxf-certificate.pem.crt",
  "privateKey": "path/to/aad380efffx-private.pem.key"
}

json error

I don't know if this is an error with mqforward, influxdb or an error in my data. This is the output when I run mqforward -

./go/bin/mqforward run
INFO[0000] Broker URI: tcp://localhost:1883
INFO[0000] connecting...
INFO[0000] influxdb host: http://127.0.0.1:8086
INFO[0000] mqtt connected
INFO[0000] influxdb connected.
INFO[0000] subscribed: map[#:0]
INFO[0009] topic:ESP_b5dad/adc/internal_temp/c
INFO[0009] topic:ESP_b5dad/temp/c
INFO[0009] topic:ESP_b5dad/external/temp_sensor/c
INFO[0009] topic:ESP_b5dad/pressure_mbar
INFO[0009] topic:ESP_b5dad/relative_humidity
INFO[0009] topic:ESP_b5dad/adc/soil_1
INFO[0009] topic:ESP_b5dad/adc/soil_2
INFO[0009] topic:ESP_b5dad/adc/uv_sensor
INFO[0009] topic:ESP_b5dad/adc/input_voltage
INFO[0009] topic:ESP_b5dad/chip/vcc
INFO[0009] topic:ESP_b5dad/pir
INFO[0009] topic:ESP_b5dad/chip/free_heap
WARN[0010] json: cannot unmarshal number into Go value of type map[string]interface {}
WARN[0010] json: cannot unmarshal number into Go value of type map[string]interface {}
WARN[0010] json: cannot unmarshal number into Go value of type map[string]interface {}
WARN[0010] json: cannot unmarshal number into Go value of type map[string]interface {}
WARN[0011] json: cannot unmarshal number into Go value of type map[string]interface {}
WARN[0011] json: cannot unmarshal number into Go value of type map[string]interface {}
WARN[0011] json: cannot unmarshal number into Go value of type map[string]interface {}
WARN[0011] json: cannot unmarshal number into Go value of type map[string]interface {}
INFO[0024] topic:ESP_b5dad/adc/internal_temp/c
INFO[0024] topic:ESP_b5dad/temp/c

mqtt from The Things Network can not be decoded

The Things Network (TTN) serves the following message to mqforward.

{
  "analog_in_4": 0.94,
  "barometric_pressure_1": 0.5,
  "relative_humidity_3": 127.5,
  "temperature_2": -1
}

mqforward seems to have trouble to decode this JSON messages:

INFO[0000] Broker URI: tcp://eu.thethings.network:1883  
INFO[0000] connecting...                                
INFO[0000] influxdb host: https://iot.bla.bla:8086   
INFO[0000] mqtt connected                               
INFO[0000] subscribed: map[miromicrotest/devices/#:0]   
INFO[0000] influxdb connected.                          
INFO[0258] topic:miromicrotest/devices/multisensorhub1/up 
INFO[0258] topic:miromicrotest/devices/multisensorhub1/up/barometric_pressure_1 
INFO[0258] topic:miromicrotest/devices/multisensorhub1/up/temperature_2 
INFO[0258] topic:miromicrotest/devices/multisensorhub1/up/relative_humidity_3 
INFO[0258] topic:miromicrotest/devices/multisensorhub1/up/analog_in_4 
WARN[0259] json: cannot unmarshal number into Go value of type map[string]interface {} 
WARN[0259] json: cannot unmarshal number into Go value of type map[string]interface {} 
WARN[0259] json: cannot unmarshal number into Go value of type map[string]interface {} 
WARN[0260] json: cannot unmarshal number into Go value of type map[string]interface {}

The measurement multisensorhub1.up exists in the target influxdb.

Do you have any experience with TTN? Might this be a configuration issue or shall I dive into the GO-code?

thanks.

Any plans to support TimescaleDB

Hi,
I guess this is a feature request. I am starting to look at TimescaleDB instead of InfluxDB. Any plans for this? I am very much a GO noob, but if there is interest I would be happy to at least help where I can.
Cheers,
Leo

Fails to build due to MQTT issues

mqforward currently fails to build for me, possibly due to changes in the Paho MQTT library:

# github.com/shirou/mqforward
../.gvm/pkgsets/system/global/src/github.com/shirou/mqforward/mqtt.go:81: cannot use ret.SubscribeOnConnect (type func(*mqtt.Client)) as type mqtt.OnConnectHandler in argument to opts.SetOnConnectHandler
../.gvm/pkgsets/system/global/src/github.com/shirou/mqforward/mqtt.go:82: cannot use ret.ConnectionLost (type func(*mqtt.Client, error)) as type mqtt.ConnectionLostHandler in argument to opts.SetConnectionLostHandler
../.gvm/pkgsets/system/global/src/github.com/shirou/mqforward/mqtt.go:97: cannot use mqtt.NewClient(m.Opts) (type mqtt.Client) as type *mqtt.Client in assignment:
    *mqtt.Client is pointer to interface, not interface
../.gvm/pkgsets/system/global/src/github.com/shirou/mqforward/mqtt.go:101: m.Client.Connect undefined (type *mqtt.Client has no field or method Connect)
../.gvm/pkgsets/system/global/src/github.com/shirou/mqforward/mqtt.go:123: client.SubscribeMultiple undefined (type *mqtt.Client has no field or method SubscribeMultiple)
../.gvm/pkgsets/system/global/src/github.com/shirou/mqforward/mqtt.go:135: m.Client.IsConnected undefined (type *mqtt.Client has no field or method IsConnected)
../.gvm/pkgsets/system/global/src/github.com/shirou/mqforward/mqtt.go:136: m.Client.Disconnect undefined (type *mqtt.Client has no field or method Disconnect)

Package bufio not found

Hello !
I'm new with golang and i'm trying to "go get" your project but i always have the following error on Rpi3 :
fatal: repository 'https://github.com/vmihailenco/bufio/' not found

Indeed this repository doesnt exist...
How can i fix the problem ?

go get github.com/shirou/mqforward doesn't get

go get github.com/shirou/mqforward
package github.com/influxdb/influxdb/client: code in directory /home/robyn/go/src/github.com/influxdb/influxdb/client expects import "github.com/influxdata/influxdb/client"

example failes - ERR: retention policy not found: a

Hi,

Ive installed InfluxDB 0.9.6 and mqforward. Then Ive followed the usage document, but got error:

SELECT a, b FROM a.b
ERR: retention policy not found: a

show MEASUREMENTS

name: measurements

name
a.b

[iot@localhost ~]$ mqforward run
INFO[0000] Broker URI: tcp://85.119.83.194:1883
INFO[0000] connecting...
INFO[0000] influxdb host: http://127.0.0.1:8086
INFO[0000] mqtt connected
INFO[0000] subscribed: map[mqforward/#:0]
INFO[0000] influxdb connected.
INFO[0062] topic:mqforward/a/b
ERRO[0077] client disconnected: EOF
INFO[0077] mqtt connected
INFO[0077] subscribed: map[mqforward/#:0]

Regards,

Sergey

Problems when inserting data

I tried mqforward on two different machines and ran into two issues:

Machine #1: data would be added to influxdb, but with a wrong timestamp (Time.now() being interpreted wrongly as nanoseconds)
Machine #2: data would not even end up in influxDB (v 1.0.1)

In the process of debugging this, I found that both issues disappear when upgrading to the v2 client interface, so I'm providing the code via pull request #6

Fails to build - undefined: client.ClientConfig

Home-MacBook:mqforward david$ go install .
github.com/shirou/mqforward
./influxdb.go:42: undefined: client.ClientConfig
./influxdb.go:92: ifc.Client.WriteSeriesOverUDP undefined (type *client.Client has no field or method WriteSeriesOverUDP)
./influxdb.go:96: ifc.Client.WriteSeries undefined (type *client.Client has no field or method WriteSeries)
./influxdb.go:140: undefined: client.Series
./influxdb.go:141: undefined: client.Series
./influxdb.go:156: undefined: client.Series
./influxdb.go:166: undefined: client.Series