SyncFlux is an Open Source InfluxDB Data syncronization and replication tool with HTTP API Interface which has as main goal recover lost data from any handmade HA influxDB 1.X cluster ( made as any simple relay https://github.com/influxdata/influxdb-relay or our Smart Relay http://github.com/toni-moreno/influxdb-srelay )
| Debian | RedHat | Docker |
|---|---|---|
| deb - signature | rpm - signature | docker run -d --name=syncflux_instance00 -p 4090:4090 -v /mylocal/conf:/opt/syncflux/conf -v /mylocal/log:/opt/syncflux/log tonimoreno/syncflux |
All releases here.
If you want to build a package yourself, or contribute. Here is a guide for how to do that.
- Go 1.11
go get -d github.com/toni-moreno/syncflux/...cd $GOPATH/src/github.com/toni-moreno/syncflux
go run build.go build After building frontend and backend you will do
go run build.go pkg-min-taryou will need previously installed the fpm/rpm and deb packaging tools. After building frontend and backend you will do.
go run build.go latestTo execute without any configuration you need a minimal config.toml file on the conf directory.
cp conf/sample.syncflux.toml conf/syncflux.toml
./bin/syncflux [options]make -f Makefile.docker
docker run tonimoreno/syncflux:latest -version
docker run tonimoreno/syncflux:latest -h
docker run -p 4090:4090 -v /mylocal/conf:/opt/syncflux/conf -v /mylocal/log:/opt/syncflux/log tonimoreno/syncflux:latest [options]To rebuild on source change (requires that you executed godep restore)
go get github.com/Unknwon/bra
bra run will init a change autodetect webserver with angular-cli (ng serve) and also a autodetect and recompile process with bra for the backend
Usage of ./bin/syncflux:
-action: hamonitor(default),copy,fullcopy,replicaschema
-chunk: set RW chuck periods as in the data-chuck-duration config param
-config: config file
-copyorder: backward (most to least recent, default), forward (least to most recent)
-db: set the db where to play
-end: set the endtime do action (no valid in hamonitor) default now
-full: copy full database or now()- max-retention-interval if greater retention policy
-logmode: log mode [console/file] default console
-logs: log directory (only apply if action=hamonitor and logmode=file)
-master: choose master ID from all those in the config file where to get data (override the master-db parameter in the config file)
-meas: set the meas where to play
-newdb: set the db to work on
-newrp: set the rp to work on
-pidfile: path to pid file
-rp: set the rp where to play
-slave: choose master ID from all those in the config file where to write data (override the slave-db parameter in the config file)
-start: set the starttime to do action (no valid in hamonitor) default now-24h
-v: set log level to Info
-version: display the version
-vv: set log level to Debug
-vvv: set log level to Trace
# -*- toml -*-
# -------GENERAL SECTION ---------
# syncflux could work in several ways,
# not all General config parameters works on all modes.
# modes
# "hamonitor" => enables syncflux as a daemon to sync
# 2 Influx 1.X OSS db and sync data between them
# when needed (does active monitoring )
# "copy" => executes syncflux as a new process to copy data
# between master and slave databases
# "replicashema" => executes syncflux as a new process to create
# the database/s and all its related retention policies
# "fullcopy" => does database/rp replication and after does a data copy
[General]
# ------------------------
# logdir ( only valid on hamonitor action)
# the directory where to place logs
# will place the main log "
#
logdir = "./log"
# ------------------------
# loglevel ( valid for all actions )
# set the log level , valid values are:
# fatal,error,warn,info,debug,trace
loglevel = "debug"
# -----------------------------
# sync-mode (only valid on hamonitor action)
# NOTE: rigth now only "onlyslave" (one way sync ) is valied
# (planned sync in two ways in the future)
sync-mode = "onlyslave"
# ---------------------------
# master-db choose one of the configured InfluxDB as a SlaveDB
# this parameter will be override by the command line -master parameter
master-db = "influxdb01"
# ---------------------------
# slave-db choose one of the configured InfluxDB as a SlaveDB
# this parameter will be override by the command line -slave parameter
slave-db = "influxdb02"
# ------------------------------
# check-interval
# the inteval for health cheking for both master and slave databases
check-interval = "10s"
# ------------------------------
# min-sync-interval
# the inteval in which HA monitor will check both are ok and change
# the state of the cluster if not, making all needed recovery actions
min-sync-interval = "20s"
# ---------------------------------------------
# initial-replication
# tells syncflux if needed some type of replication
# on slave database from master database on initialize
# (only valid on hamonitor action)
#
# none: no replication
# schema: database and retention policies will be recreated on the slave database
# data: data for all retention policies will be replicated
# be carefull: this full data copy could take hours,days.
# both: will replicate first the schema and them the full data
initial-replication = "none"
#
# monitor-retry-durtion
#
# syncflux only can begin work when master and slave database are both up,
# if some of them is down syncflux will retry infinitely each monitor-retry-duration to work.
monitor-retry-interval = "1m"
#
# data-chuck-duration
#
# duration for each small, read from master -> write to slave, chuck of data
# smaller chunks of data will use less memory on the syncflux process
# and also less resources on both master and slave databases
# greater chunks of data will improve sync speed
data-chuck-duration = "60m"
#
# max-retention-interval
#
# for infinite ( or bigger ) retention policies full replication should begin somewhere in the time
# this parameter set the max retention.
max-retention-interval = "8760h" # 1 year
# ---- HTTP API SECTION (Only valid on hamonitor action)
# Enables an HTTP API endpoint to check the cluster health
[http]
name = "example-http-influxdb"
bind-addr = "127.0.0.1:4090"
admin-user = "admin"
admin-passwd = "admin"
cookie-id = "mysupercokie"
# ---- INFLUXDB SECTION
# Sets a list of available DB's that can be used
Available actions:
- Replicate Schema
- Copy data
- Full copy (replicate schema + copy data)
Allows the user to copy DB schemas from DB1 to DB2. DB schema are DBs and RPs.
Syntax
./bin/syncflux -action replicaschema [-master <master_id>] [-slave <slave_id>] [-db <db_regex_selector>] [-newdb <newdb_name>] [-rp <rp_regex_selector>] [-newrp <newrp_name>] [-meas <meas_regex_selector>]
Description of syntax
If no master or slave are provided it takes the default from config file. The db selector allows to filter with regex expression on all dbs.
If the slave schema must be different than the master, the new schema can be set using newdb and newrp flags
Limitations
- Only the default RP can be renamed
Important Notes
When copying big databases, there is a few things you shoult take care, to ensure data is corretly copied.
Syncflux tool copy data by doing "select * from XXXXX where time > [INIT_CHUNK] AND time > [END_CHUNK]" for each one of the existing measurements in the choosen database, It does num queries concurrently Depending on the measurement cardinality these queries could take long time (be carefull with timeouts) and also need for resources (memory mainly ) in both databases , but also in for the syncflux process itself.
We recomends increase/disable all query timeouts:
- InfluxDB query-timeout=0
- Syncflux InfluxDB Section timeouts
- Smallest chunk sizes (1h, 30m, 10m) will spend less memory but will spend more time to finish the complete copy
- Run syncflux tool process ouside the influxdb servers.
Examples
Example 1: Copy schema from Influx01 to Influx02
Influx01 schema
----------------
|-- db1
|-- rp1*
|-- rp2
|-- db2
|-- rp1*
|-- rp2./bin/syncflux -action "replicaschema" -master "influx01" -slave "influx02"The result will be that the schema of Influx01 will be replicated on Influx02
Influx02 schema
----------------
|-- db1
|-- rp1*
|-- rp2
|-- db2
|-- rp1*
|-- rp2Example 2: Copy schema from Influx01-DB1 to Influx02
Influx01 schema
----------------
|-- db1
|-- rp1*
|-- rp2
|-- db2
|-- rp1*
|-- rp2./bin/syncflux -action "replicaschema" -master "influx01" -slave "influx02" -db "^db1$"The result will be that the schema of Influx01 will be replicated on Influx02
Influx02 schema
----------------
|-- db1
|-- rp1*
|-- rp2Example 3: Copy schema from Influx01-DB1 to Influx02-DB3 (new db called DB3) and only from rp1
Influx01 schema
----------------
|-- db1
|-- rp1*
|-- rp2
|-- db2
|-- rp1*
|-- rp2
./bin/syncflux -action "replicaschema" -master "influx01" -slave "influx02" -db "^db1$" -newdb "db3" -rp "^rp1$"The result will be that the schema of Influx01 will be replicated on Influx02
Influx02 schema
----------------
|-- db3
|-- rp1*Example 4: Copy schema from Influx01-DB1 to Influx02-DB3 (new db called DB3) and set the defaultrp to rp3
Influx01 schema
----------------
|-- db1
|-- rp1*
|-- rp2
|-- db2
|-- rp1*
|-- rp2./bin/syncflux -action "replicaschema" -master "influx01" -slave "influx02" -db "^db1$" -newdb "db3" -newrp "rp3"The result will be that the schema of Influx01 will be replicated on Influx02
Influx02 schema
----------------
|-- db3
|-- rp3*
|-- rp2Example 5: Copy data and schema from Influx01-DB1 to Influx02-DB3 (new db called DB3) and only from meas "cpu.*"
Influx01 schema
----------------
|-- db1
|-- rp1*
|-- cpu
|-- mem
|-- swap
|-- ...
|-- rp2
|-- db2
|-- rp1*
|-- rp2Allows the user to copy DB data from master to slave. DB schema are DBs and RPs.
Syntax
./bin/syncflux -action copy [-master <master_id>] [-slave <slave_id>] [-db <db_regex_selector>] [-newdb <newdb_name>] [-rp <rp_regex_selector>] [-newrp <newrp_name>] [-meas <meas_regex_selector>] { [-start <start_time>] [-endtime <end_time>] , [-full] }
Description of syntax
If no master or slave are provided it takes the default from config file. The db selector allows to filter with regex expression on all dbs.
If the slave schema must be different than the master, the new schema can be set using newdb and newrp flags
The start end end allow to define a time window to copy data. If full is passed, the data will be copied from now to max-retention-interval
Remember that with this action schema is not replicated so if the DB or RP on slave doesn't exists it will be skipped
Limitations
- ...
Examples
Example 1: Copy all data from Influx01 to Influx02
Influx01 schema
----------------
|-- db1
|-- rp1*
|-- rp2
|-- db2
|-- rp1*
|-- rp2./bin/syncflux -action "coy" -master "influx01" -slave "influx02"The command above will copy data from all dbs from Influxdb01 into Influx02
Influx02 schema
----------------
|-- db1
|-- rp1*
|-- rp2
|-- db2
|-- rp1*
|-- rp2Example 2: Copy data from Influx01-DB1 to Influx02 on a time window and only from rp1
Influx01 schema
----------------
|-- db1
|-- rp1*
|-- rp2
|-- db2
|-- rp1*
|-- rp2./bin/syncflux -action "copy" -master "influx01" -slave "influx02" -db "^db1$" -rp "^rp1$" -start -10h end -5hThe command above will repicate all data from Influx01 to InfluxDB but only from db1.rp1 and with a time window from -10h to -5h
Influx02 schema
----------------
|-- db1
|-- rp1*
|-- rp2Example 3: Copy data from Influx01-DB1 to Influx02-DB3 (existing db called DB3)
Influx01 schema
----------------
|-- db1
|-- rp1*
|-- rp2
|-- db2
|-- rp1*
|-- rp2
./bin/syncflux -action "copy" -master "influx01" -slave "influx02" -db "^db1$" -newdb "db3"The command above will replicate all data from Influx01-db1 to InfluxDB on a new DB called 'db3'
Influx02 schema
----------------
|-- db3
|-- rp1*
|-- rp2Example 4: Copy data from Influx01-DB1 to Influx02-DB3 (existing db called DB3) and set the defaultrp to existing rp3
Influx01 schema
----------------
|-- db1
|-- rp1*
|-- rp2
|-- db2
|-- rp1*
|-- rp2./bin/syncflux -action "copy" -master "influx01" -slave "influx02" -db "^db1$" -newdb "db3"The command above will replicate all data from Influx01-db1 to InfluxDB on a new DB called 'db3' and a new defaultrp called rp3
Influx02 schema
----------------
|-- db3
|-- rp3*
|-- rp2Example 5: Copy data from Influx01-DB1 to Influx02-DB3 (new db called DB3) and only from meas "cpu.*"
Influx01 schema
----------------
|-- db1
|-- rp1*
|-- cpu
|-- mem
|-- swap
|-- ...
|-- rp2
|-- db2
|-- rp1*
|-- rp2./bin/syncflux -action "copy" -master "influx01" -slave "influx02" -db "^db1$" -newdb "db3" -mes "cpu.*"The command above will replicate all data from Influx01-db1 to InfluxDB on a new DB called 'db3' and a new defaultrp called rp3
Influx02 schema
----------------
|-- db3
|-- rp3*
|-- cpu
|-- rp2Allows the user to copy DB data from master to slave. DB schema are DBs and RPs.
Syntax
./bin/syncflux -action fullcopy [-master <master_id>] [-slave <slave_id>] [-db <db_regex_selector>] [-newdb <newdb_name>] [-rp <rp_regex_selector>] [-newrp <newrp_name>] [-meas <meas_regex_selector>] { [-start <start_time>] [-endtime <end_time>] , [-full] }
Description of syntax
If no master or slave are provided it takes the default from config file. The db selector allows to filter with regex expression on all dbs.
If the slave schema must be different than the master, the new schema can be set using newdb and newrp flags
The start end end allow to define a time window to copy data. If full is passed, the data will be copied from now to max-retention-interval
Remember that with this action schema is not replicated so if the DB or RP on slave doesn't exists it will be skipped
Limitations
- Only the default RP can be renamed
Examples
Example 1: Copy all data and schema from Influx01 to Influx02
Influx01 schema
----------------
|-- db1
|-- rp1*
|-- rp2
|-- db2
|-- rp1*
|-- rp2./bin/syncflux -action "coy" -master "influx01" -slave "influx02"The command above will create the schema and will copy data from all dbs from Influxdb01 into Influx02
Influx02 schema
----------------
|-- db1
|-- rp1*
|-- rp2
|-- db2
|-- rp1*
|-- rp2Example 2: Copy data and schema from Influx01-DB1 to Influx02 on a time window
Influx01 schema
----------------
|-- db1
|-- rp1*
|-- rp2
|-- db2
|-- rp1*
|-- rp2./bin/syncflux -action "copy" -master "influx01" -slave "influx02" -db "^db1$" -start -10h end -5hThe command above will create the schema and will repicate all data from Influx01 to InfluxDB but only from db1 and with a time window from -10h to -5h
Influx02 schema
----------------
|-- db1
|-- rp1*
|-- rp2Example 3: Copy data from Influx01-DB1 to Influx02-DB3 (new db called DB3)
Influx01 schema
----------------
|-- db1
|-- rp1*
|-- rp2
|-- db2
|-- rp1*
|-- rp2
./bin/syncflux -action "copy" -master "influx01" -slave "influx02" -db "^db1$" -newdb "db3"The command above will create the schema and will replicate all data from Influx01-db1 to InfluxDB on a new DB called 'db3'
Influx02 schema
----------------
|-- db3
|-- rp1*
|-- rp2Example 4: Copy data and schema from Influx01-DB1 to Influx02-DB3 (new db called DB3) and set the defaultrp to rp3
Influx01 schema
----------------
|-- db1
|-- rp1*
|-- rp2
|-- db2
|-- rp1*
|-- rp2./bin/syncflux -action "copy" -master "influx01" -slave "influx02" -db "^db1$" -newdb "db3"The command above will create the schema and will replicate all data from Influx01-db1 to InfluxDB on a new DB called 'db3' and a new defaultrp called rp3
Influx02 schema
----------------
|-- db3
|-- rp3*
|-- rp2Example 5: Copy data and schema from Influx01-DB1 to Influx02-DB3 (new db called DB3) and only from meas "cpu.*"
Influx01 schema
----------------
|-- db1
|-- rp1*
|-- cpu
|-- mem
|-- swap
|-- ...
|-- rp2
|-- db2
|-- rp1*
|-- rp2./bin/syncflux -action "copy" -master "influx01" -slave "influx02" -db "^db1$" -newdb "db3" -mes "cpu.*"The command above will create the schema and will replicate all data from Influx01-db1 to InfluxDB on a new DB called 'db3' and a new defaultrp called rp3
Influx02 schema
----------------
|-- db3
|-- rp3*
|-- cpu
|-- rp2./bin/syncflux -config ./conf/syncflux.conf -action hamonitor syncflux by default search a file syncflux.conf in the CWD/conf/ and syncflux has hamonitor action by default so this last is equivalent to this one
./bin/syncflux you can check the cluster state with any HTTP client, posibles values are:
- OK: both nodes are ok
- CHECK_SLAVE_DOWN: current slave is down
- RECOVERING: both databases are working but slave leaks some data and syncflux is recovering them
% curl http://localhost:4090/api/health
{
"ClusterState": "CHECK_SLAVE_DOWN",
"ClusterNumRecovers": 0,
"ClusterLastRecoverDuration": 0,
"MasterState": true,
"MasterLastOK": "2019-04-06T09:45:05.461897766+02:00",
"SlaveState": false,
"SlaveLastOK": "2019-04-06T09:44:55.465393243+02:00"
}
% curl http://localhost:4090/api/health
{
"ClusterState": "RECOVERING",
"ClusterNumRecovers": 0,
"ClusterLastRecoverDuration": 0,
"MasterState": true,
"MasterLastOK": "2019-04-06T10:28:25.459701432+02:00",
"SlaveState": true,
"SlaveLastOK": "2019-04-06T10:28:25.55500823+02:00"
}
% curl http://localhost:4090/api/health
{
"ClusterState": "OK",
"ClusterNumRecovers": 1,
"ClusterLastRecoverDuration": 2473620691,
"MasterState": true,
"MasterLastOK": "2019-04-06T10:28:25.459701432+02:00",
"SlaveState": true,
"SlaveLastOK": "2019-04-06T10:28:25.55500823+02:00"
}![dependabot[bot] avatar](https://avatars.githubusercontent.com/in/29110?v=4 "dependabot[bot]")
React
Typescript
Django
Laravel
Facebook
Microsoft
Google
Alibaba
D3
Tencent