breser / git2consul Goto Github PK

Currently git2consul pushes it's configuration (if one is passed) to the Consul KV before pulling and using it. However, this doesn't really buy us anything and can potentially introduce race conditions.

One such case is with multiple instances of git2consul running. If git2consul is running as a service on multiple machines (or containers) and has a config-file passed in as well as halt_on_change enabled, it will go into a perpetual restart cycle. Every time the service starts, it will push the config to the KV and trigger halt_on_change on the other running git2consul service(s), which in turn will restart, push the config, and trigger restarts on its sister services.

Another example is with the logger configuration. When git2consul first starts and there is no config seeded yet, the process doesn't use the logging config provided by the user until after it's seeded and read #L79. This means that logging output before config_reader.read()'s callback (e.g. config_reader.read() itself and config_seeder.set()) is done with the default logger configuration.

Moving the config to be read locally removes complexity, eliminates potential race conditions, and makes it clearer what configuration a running git2consul service is using (though this can be addressed in today's version with halt_on_change but introduces other issues).

Dynamic configuration (i.e. updating configuration without re-deploying) can still be done with tools like consul-template. The decision of using the KV for dynamic git2consul configuration was done before such tool became available.

Test fix

The URL for the consul download in the Vagrantfile seems to be wrong now. I've got the following pull request that should resolve the issue: #101

Switch to silas/node-consul and add support for ACLs in requests.

git2consul has steadily been gaining users since its early stages, and for that we thank you. This project would not have been nearly as successful without the involvement of the community, opening issues, contributing code, and just running the application.

As the project has evolved and increased in functionality as well as complexity, we wanted to take this chance to re-visit git2consul as a whole. We would like to take this opportunity make some announcements for the future and direction of git2consul.

We have invested some time and resources to develop a proof-of-concept port of git2consul in Go. Note that this version is highly-experimental, non-feature-complete, and not recommended for production. On a similar note, we would like to stress the fact that [node-]git2consul project and repository will not be immediately abandoned, and it will continue to be the recommended solution.

There are several reasons for this experiment. Primarily, it lets us leverage the ecosystem and native Go libraries that the folks at Hashicorp provide. Additionally, also allows us to re-examine the entire code base and give us the chance to make architectural changes to improve git2consul. Some of these changes, which will be part of the initial version of go-git2consul, include moving the configuration to the local system and removing runtime dependencies such as node and git (the latter done with the libgit2 library).

We tried to keep user-experience as similar as possible to the existing git2consul, so that our users would not need to familiarize with the tool once again. However, there might be some features that will be different, such as the webhook settings. go-git2consul is still under development and architectural decisions have not exactly been set in stone. You are more than welcome to contribute to the project if you think there is a better solution to a specific problem! The repository for go-git2consul can be found here.

This initiative was sparked by the community, more specifically by the request for file expansion on different extensions. As such, we would love to hear feedback from you about this decision.

An option to convert file names in more generic way by applying conversion functions to convert file names on consul side

I have a test repo to illustrate the problem here. You can see the individual commits.

My config looks like this:

{
  "version": "1.0",
  "repos": [
    {
      "name": "config",
      "url": "https://github.com/harnash/git2consul_test.git",
      "source_root": "config/",
      "branches": [
        "master"
      ],
      "include_branch_name" : false,
      "hooks": [
        {
          "type": "polling",
          "interval": "1"
        }
      ]
    },
    {
      "name": "registry",
      "url": "https://github.com/harnash/git2consul_test.git",
      "source_root": "registry/",
      "branches": [
        "master"
      ],
      "include_branch_name" : false,
      "hooks": [
        {
          "type": "polling",
          "interval": "1"
        }
      ]
    }
  ],
  "logger": {
    "name": "git2consul",
    "streams": [
      {
        "level": "debug",
        "stream": "process.stdout"
      }
    ]
  }
}

So after a while the synchronisation stopped for no good reason. I've noticed this on my dev instances too. After a while it just stops and even restarting doesn't help, only removing whole folder on Consul KV forces it to restore whole tree. I'm suspecting that it might be due that my two repos in git2consul points to the same repo and branch on Github.

Here is the dump from my logs:

{"name":"git2consul","hostname":"1e8ebe2b6d54","pid":1,"level":30,"msg":"git2consul is running","time":"2016-02-05T12:23:22.488Z","v":0}
{"name":"git2consul","hostname":"1e8ebe2b6d54","pid":1,"level":30,"msg":"Initting repo config","time":"2016-02-05T12:23:22.492Z","v":0}
{"name":"git2consul","hostname":"1e8ebe2b6d54","pid":1,"level":30,"msg":"Initting branch /tmp/config /tmp/config/master","time":"2016-02-05T12:23:22.493Z","v":0}
{"name":"git2consul","hostname":"1e8ebe2b6d54","pid":1,"level":30,"msg":"Initting repo registry","time":"2016-02-05T12:23:22.494Z","v":0}
{"name":"git2consul","hostname":"1e8ebe2b6d54","pid":1,"level":30,"msg":"Initting branch /tmp/registry /tmp/registry/master","time":"2016-02-05T12:23:22.494Z","v":0}
{"name":"git2consul","hostname":"1e8ebe2b6d54","pid":1,"level":40,"msg":"Purging branch cache /tmp/config/master for branch master in repo config","time":"2016-02-05T12:23:22.495Z","v":0}
{"name":"git2consul","hostname":"1e8ebe2b6d54","pid":1,"level":40,"msg":"Purging branch cache /tmp/registry/master for branch master in repo registry","time":"2016-02-05T12:23:22.496Z","v":0}
{"name":"git2consul","hostname":"1e8ebe2b6d54","pid":1,"level":30,"msg":"Initialized branch master from registry","time":"2016-02-05T12:23:23.354Z","v":0}
{"name":"git2consul","hostname":"1e8ebe2b6d54","pid":1,"level":20,"msg":"Branches initialized","time":"2016-02-05T12:23:23.355Z","v":0}
{"name":"git2consul","hostname":"1e8ebe2b6d54","pid":1,"level":20,"msg":"Polling hook initialized with 1 minute intervals","time":"2016-02-05T12:23:23.358Z","v":0}
{"name":"git2consul","hostname":"1e8ebe2b6d54","pid":1,"level":30,"msg":"Loaded repo registry","time":"2016-02-05T12:23:23.359Z","v":0}
{"name":"git2consul","hostname":"1e8ebe2b6d54","pid":1,"level":30,"msg":"Initialized branch master from config","time":"2016-02-05T12:23:23.368Z","v":0}
{"name":"git2consul","hostname":"1e8ebe2b6d54","pid":1,"level":20,"msg":"Branches initialized","time":"2016-02-05T12:23:23.369Z","v":0}
{"name":"git2consul","hostname":"1e8ebe2b6d54","pid":1,"level":20,"msg":"Polling hook initialized with 1 minute intervals","time":"2016-02-05T12:23:23.370Z","v":0}
{"name":"git2consul","hostname":"1e8ebe2b6d54","pid":1,"level":30,"msg":"Loaded repo config","time":"2016-02-05T12:23:23.370Z","v":0}
{"name":"git2consul","hostname":"1e8ebe2b6d54","pid":1,"level":20,"msg":"Updates in branch master complete","time":"2016-02-05T12:24:53.256Z","v":0}
{"name":"git2consul","hostname":"1e8ebe2b6d54","pid":1,"level":20,"msg":"Updates in branch master complete","time":"2016-02-05T12:25:53.118Z","v":0}
{"name":"git2consul","hostname":"1e8ebe2b6d54","pid":1,"level":20,"msg":"Updates in branch master complete","time":"2016-02-05T12:26:53.126Z","v":0}

After that it just stays like that. After restart it does pick up the new files but it does not remove the old one (which were deleted on repo).

Log after restart:

{"name":"git2consul","hostname":"fe8de965417b","pid":1,"level":30,"msg":"git2consul is running","time":"2016-02-05T12:47:06.516Z","v":0}
{"name":"git2consul","hostname":"fe8de965417b","pid":1,"level":30,"msg":"Initting repo config","time":"2016-02-05T12:47:06.520Z","v":0}
{"name":"git2consul","hostname":"fe8de965417b","pid":1,"level":30,"msg":"Initting branch /tmp/config /tmp/config/master","time":"2016-02-05T12:47:06.521Z","v":0}
{"name":"git2consul","hostname":"fe8de965417b","pid":1,"level":30,"msg":"Initting repo registry","time":"2016-02-05T12:47:06.522Z","v":0}
{"name":"git2consul","hostname":"fe8de965417b","pid":1,"level":30,"msg":"Initting branch /tmp/registry /tmp/registry/master","time":"2016-02-05T12:47:06.522Z","v":0}
{"name":"git2consul","hostname":"fe8de965417b","pid":1,"level":40,"msg":"Purging branch cache /tmp/config/master for branch master in repo config","time":"2016-02-05T12:47:06.524Z","v":0}
{"name":"git2consul","hostname":"fe8de965417b","pid":1,"level":40,"msg":"Purging branch cache /tmp/registry/master for branch master in repo registry","time":"2016-02-05T12:47:06.525Z","v":0}
{"name":"git2consul","hostname":"fe8de965417b","pid":1,"level":30,"msg":"Initialized branch master from registry","time":"2016-02-05T12:47:08.526Z","v":0}
{"name":"git2consul","hostname":"fe8de965417b","pid":1,"level":20,"msg":"Branches initialized","time":"2016-02-05T12:47:08.527Z","v":0}
{"name":"git2consul","hostname":"fe8de965417b","pid":1,"level":20,"msg":"Polling hook initialized with 1 minute intervals","time":"2016-02-05T12:47:08.528Z","v":0}
{"name":"git2consul","hostname":"fe8de965417b","pid":1,"level":30,"msg":"Loaded repo registry","time":"2016-02-05T12:47:08.529Z","v":0}
{"name":"git2consul","hostname":"fe8de965417b","pid":1,"level":30,"msg":"Initialized branch master from config","time":"2016-02-05T12:47:08.535Z","v":0}
{"name":"git2consul","hostname":"fe8de965417b","pid":1,"level":20,"msg":"Branches initialized","time":"2016-02-05T12:47:08.536Z","v":0}
{"name":"git2consul","hostname":"fe8de965417b","pid":1,"level":20,"msg":"Polling hook initialized with 1 minute intervals","time":"2016-02-05T12:47:08.537Z","v":0}
{"name":"git2consul","hostname":"fe8de965417b","pid":1,"level":30,"msg":"Loaded repo config","time":"2016-02-05T12:47:08.538Z","v":0}

I am polling a bitbucket repo but git2consul prompts me for a password each time it polls. It would help if there were documentation on how to get around this.

As a user, I would like to be able to set and ENV var called something like BRANCHES that would allow me to dynamically set to config branches setting when I run git2consul in certain environments where I only want a single branch to be injected into the Consul cluster.

hi,

i am getting this issue in git2consul. I don't know why leadership is getting lost for consul

Our team has a requirement to duplicate the contents of a repo into Consul to be read by another process. This other process has a strict dependency on the Consul structure. Additionally, we run git2consul on-demand rather than having it running as a polling process. Here's what our command looks like:

git2consul --endpoint <CONSUL_URL> --port 8500 --config-file <JSON CONFIG> -n --config_key <CONSUL_KEY_PATH>

After using git2consul to push our keys up, an additional file is found in the root of the path where the keys are located. This file is named 'develop.ref', after the branch we're using from our git repo. The contents of the file have the SHA-1 from our latest commit on that branch that we're pushing. Looking through the documentation I could not find a mention of this file.

Is it possible that a flag can be added to make the creation of this file optional, or if git2consul is dependent on this key, can the flag be configured to remove this key as a last step? I've uploaded a screenshot for our environment to show the file in question. Thanks.

This seems to be a recent problem. When I run git2consul, regardless of my endpoint config, it's trying to hit 127.0.0.1:8500 for some reason:

"err": {"message":"connect ECONNREFUSED 127.0.0.1:8500", ...}

I'm running latest. Verified global.endpoint is correct by logging it.

As a user, I would like to be able to import a .yml file and have git2consul split each key/value into k/v in Consul similar to how the json expand_keys works. This would allow me to standardize my configurations to yml.

Is it currently possible to just set up a post-receive hook for the change notification? Then I could just push directly to the repository being monitored by git2consul and have the changes be detected.

Or just the changed ones?

Our Consul cluster goes into leadership election/transition almost every git2consul push - just trying to narrow down potential sources of the problem.

hi,
i am using git2consul in docker and ,it working fine but for bulk data push it throws error maxbuffer size exceeded, any suggestions

We should allow users to select the KV in which git2consul config is stored.

We now have support for treating Java properties files as a special case, but I'm generally uneasy with this: I would rather keep g2c thin and general purpose. We should investigate moving such strategies into a plugins layer.

It seems that adding source_root parameter causes folders to be truncated by one char.

Setup:

1. I have set poz-dev/ as source_root in my git2consul config
2. In my repo I have path: poz-dev/service/settings

Problems

1. After running git2consul I get ervice/settings in Consul's KV
2. Entry which holds info about the revision instead of master.ref is named just f

A CHANGELOG.md file would be awesome. Specially for breaking/important changes since checking commit logs one by one is not efficient if you are coming from a few version back.

Thanks

If you do a git push --force (which is, of course, evil) git2consul gets hosed in such a way that you need to manually delete the branch.ref from Consul's KV and then bounce git2consul. We should come up with a way to self-heal in this case.

I loaded the sample git2consul configuration file into git.

{
  "version": "1.0",
  "repos" : [{
    "name" : "sample_configuration",
    "url" : "https://github.com/ryanbreen/git2consul_data.git",
    "branches" : ["dev"],
    "hooks": [{
      "type" : "polling",
      "interval" : "1"
    }]
  }]
}

I configured git2consul to monitor this file and the end result in Consul is only version makes it into Consul:

$ curl "consul.service.consul:8500/v1/kv/poc/master/config/git2consul.json/?recurse&pretty&token=$CONSUL_TOKEN"
[
    {
        "LockIndex": 0,
        "Key": "poc/master/config/git2consul.json/version",
        "Flags": 0,
        "Value": "MS4w",
        "CreateIndex": 1297420,
        "ModifyIndex": 1297420
    }
]

{
  "version": "1.0",
  "local_store": "/tmp/git2consul_cache",
  "halt_on_change": true,
  "logger" : {
    "name" : "git2consul",
    "streams" : [
      {
        "level":  "trace",
        "stream": "process.stdout"
      },
      {
        "level": "debug",
        "type": "rotating-file",
        "path": "/tmp/git2consul.log"
      }
    ]
  },
  "repos": [{
    "name": "poc",
    "url":  "ssh://[email protected]/git2consul.git",
    "expand_keys": true,
    "source_root": "config",
    "branches": ["master"],
    "hooks": [{
      "type": "stash",
      "port": "5051",
      "url": "/gitpoke"
    },
    {
      "type": "polling",
      "interval": "1"
    }]
  }]
}

It would be nice if git2consul could handle arrays. Thanks!

Hi guys ,

I am currently testing git2consul and would like to implement at our company . I am finding issues to load example config to our K/V store at consul .

I am following the readme provided . Is there something aim doing wrong ?

git2consul --config-file /tmp/git2consul.json

{"name":"git2consul","hostname":"consul-20032495","pid":25691,"level":30,"msg":"Adding /tmp/git2consul.json to KV git2consul/config as: \n{\n \"version\": \"1.0\",\n \"repos\" : [{\n \"name\" : \"sample_configuration\",\n \"url\" : \"https://github.com/ryanbreen/git2consul_data.git\",\n \"branches\" : [\"dev\"],\n \"hooks\": [{\n \"type\" : \"polling\",\n \"interval\" : \"1\"\n }]\n }]\n}\n","time":"2016-08-06T01:41:34.225Z","v":0}

{"name":"git2consul","hostname":"consul-20032495","pid":25691,"level":50,"err":{"message":"connect ECONNREFUSED","name":"Error","stack":"Error: connect ECONNREFUSED\n at errnoException (net.js:901:11)\n at Object.afterConnect [as oncomplete] (net.js:892:19)","code":"ECONNREFUSED"},"msg":"connect ECONNREFUSED","time":"2016-08-06T01:41:34.233Z","v":0}

p.s Our consul cluster is https endpoint.

same result with git2consul --endpoint consul.example.com --port 80 --config-file /tmp/git2consul.json

Hello,

Thank you for this greate project!
Not an issue but strange question, we try to use KV as source for our nginx configuration, we use git2consul with expand_keys, all works like a charm, we use also custom script with long polling to consul KV to determine changes in KV branch (managed by git2consul) the problem is that when git2consul is starting to change KV after commit our script triggers on very first change and we get invalid nginx configuration, however after some seconds if I run it manually again - all is ok, because git2-consul has updated all needed keys. So the question is could I determine the moment when all changes done to query new proper configuration. I'm thinking of using .ref key that is created by git2consul in needed branch, but I wanted to clarify if it is updated after all changes are done?

Thanks!
Roman

I'd like to use git2consul for my current project but my org mandates using vsts for git.

https://www.visualstudio.com/en-us/docs/integrate/get-started/service-hooks/events#git.push

If I get some time I will try to implement this and send a pr...

In the docs it says git2consul will if "files have been added, updated, or deleted and replicates those changes to the KV" but this does not see to apply to "folders":

Here is our scenario:

1. Running on "master" branch with folders named "x", "y".
2. Switch to branch "feature" which added a folder named "z".
3. Switch back to branch "master"
4. Now we have folders "x", "y", and "z" in the K/V.

I would expect the k/v to mirror the current branch, but maybe I am missing something.

I have git2consul running inside of a docker container and I'm trying to dig into an issues in which it appears that the polling seems to stop. When I look at the container logs some last actions I see many times is that is detected a change and sync'd the KV store and then there is no more polling after that.

I'm currently using 0.12.1

Is there anything in particular that might cause this behavior? Would intermittent issues of connectivity with the git repository cause any failures and for it to stop working?

Even without any changes to the repos, git2consul is updating the KV on each polling interval. This is unnecessary churn.

When the polling hook runs and finds changes, every subsequent run of the hook will find "changes", leading to unnecessary git operations.

I'm just trying to run the example from your quick start guide. So I'm pointing at your test repository and my instance of Consul.

$ cat /tmp/git2consul.json
{
  "version": "1.0",
  "repos": [{
    "name": "sample_configuration",
    "url":  "https://github.com/ryanbreen/git2consul_data.git",
    "branches": ["dev"],
    "hooks": [{
      "type": "polling",
      "interval": "1"
    }]
  }]
}
$ node . --endpoint consul.service.consul --port 8500 --token $MASTER_TOKEN --config-file /tmp/git2consul.json

I get:

{
  "name":"git2consul",
  "hostname":"host.example.com",
  "pid":16987,
  "level":30,
  "msg":"Adding /tmp/git2consul.json to KV git2consul/config as
: \n{\n  \"version\": \"1.0\",\n  \"repos\": [{\n    \"name\": \"poc\",\n    \"url\":  \"https://github.com/ryanbreen/git2consul_data.git\",\n\    "branches\": [\"dev\"],\n    \"hooks\": [{\n      \"type\": \"polling\",\n      \"interval\": \"1\"\n    }]\n  }]\n}\n",
  "time":"2016-03-24T20:15:29.262Z",
  "v":0
}  
{
  "name":"git2consul",
  "hostname":"host.example.com",
  "pid":16987,
  "level":30,
  "msg":"item:  undefined",
  "time":"2016-03-24T20:15:29.359Z",
  "v":0
}  
Config value is not valid JSON: undefined

I look in Consul and the config for git2consul is there but I do not see any of the keys for sample_configuration. I've validated the JSON in /tmp/git2consul.json and it's valid.

• OS: CentOS Linux release 7.1.1503 (Core)
• Consul: v0.6.3
• node: v5.9.1
• git2consul: v0.12.8

Before I try different versions of node or git2consul or reverse engineering, do you have any suggestions? Thanks!

@ryanbreen : How we can use --entrypoint (-e) for config_seeder.js? I found https://github.com/Cimpress-MCP/git2consul/blob/master/utils/config_seeder.js#L29 So looks like it is not possible atm.

My problem is that I'm running git2consul in a container with BRIDGE mode and Consul service is on different host instead 127.0.0.1. So I wish there is something node utils/config_seeder.js /tmp/git2consul.json --entrypoint 10.0.2.15 OR CONSUL_URL=10.0.2.15:8500 node utils/config_seeder.js /tmp/git2consul.json

Some good options would be the possibility in the configuration to:

• Remove file extension (.json, .yaml, .properties) so the file:

file.json
{
    "key1" : "value1",
    "key2" : "value2"
}

Will end up in

--/file/key1
--/file/key2

and not

--/file.json/key1
--/file.json/key2

• Remove filename so the file:

file.json
{
    "key1" : "value1",
    "key2" : "value2"
}

Will end up in

--/key1
--/key2

and not

--/file.json/key1
--/file.json/key2

Thanks!

It would be nice to make a config switch for the expanded file extension.

I agree that it's very clear what happens if you see keys like /myprops.json/bla but it also forces me to make a clear choice up front how to organise my repo(s).

If at some point I decide the json file gets to big and I'd like to split it up in smaller pieces, I can never replicate the original keys that where generated, where I might have already various applications depending on a certain structure. Allowing the extension to be optional gives a bit more flexibility.

i.e.

{
    "obj1" : {
        "key1" : "value1"
    },
    "obj2" : {
        "key2" : "value2"
    }
}

Will end up in {

myfile.json/
--/obj1/key1
--/obj2/key2

If my obj1 grows to big, with more nested objects, it might make sense to split it, bit then I end up with:

myfile.json/
--/obj1.json/key1
--/somenewobject.json/anotherkey
--/obj2/key2

Basically having to change all application that depend on the obj1 key.
So it would be nice if I can force the output to be:

myfile/
--/obj1/key1
--/somenewobject/anotherkey
--/obj2/key2

Also with the upcoming features for .properties files it would be nice to be able to switch, without an effect on the generated keys.

With regards to the .properties support, will keys be exported as is, or will there be an option or created folders per . separation in the key?

Right now, git2consul has no listener for uncaughtException. Fix this.

The gitter room has been pretty quiet, so I will ask this here.

Does anybody have advice on using branches for different environments (or datacenters) vs constructing the repo with paths for each environments (or datacenters) and configuring source_root??

I started building out a "configuration" repo using branches for "dev", "test", etc, and a top level path for each datacenter ("us", "eu", etc). I haven't spent much time with that arrangement yet, but it seems like it might be a pain to do diffs or pull requests across branches and even to keep changes that should be in sync across environments and/or datacenters.

We already use a "git flow" like workflow with our apps and Ansible projects with feature branches built from a mainline branch, reviewed, and accepted via PR.

I am currently experimenting with using a branch for each stage of the workflow ("dev", "master"), namespacing the data in the repo with "{datacenter}/{environment}", and using the same "{datacenter}/{environment}" as my source_root value.

This is more familiar, but it does mean that I have to pick which environments should track the "dev" branch and which should track the "master" branch.

Please release v0.12.11 on npm to match the current git master branch and documentation.

Since we have use of using different data centers for various reasons, we'd like to be able to point different configurations at a specific data center. Nominally, this should be simple to do, as it is very much like the token property; it's something which is added on all consul calls; in fact, I've done this already in a fork (and would be happy to share)

My question is whether or not there is a particular reason this hasn't been done already; is it just something which never came up / was never needed, or is there some insidious bug lurking below the surface?

I initially had git2consul configured to only monitor 1 repo. I expand the configuration to 2 repos. When I restart git2consil with the --config-file option, it reads in the new config file and saves the new configuration to Consul. But once it starts running, only the first repo is being monitored. If I stop git2consul and restart it without the --config-file option, then I see the 2nd repo being watched.

Hi there

We're looking at using this; it sounds like exactly how we'd want to manage configuration information.

Could you please enhance the docs to describe how to run it, though? Sorry if I've missed something obvious - not so familiar with node.

I'm assuming something like

• set up a repo to hold configuration
• put a configuration file into KV:/git2consul/config (per https://github.com/Cimpress-MCP/git2consul#configuration)
• ... run something!

I read the documentation but did not find if during the loading of the KV store from the git repo, it was possible for the KV store to be in an incorrect state vis-a-vis the state of the git repo.

The reason I ask is I'm considering using Consul Template, and wondering if it might be triggered multiple times during the update, if we update several keys.

Support for the option to parse the source config files from the watched git repo and recreate the structure of the config's properties as a tree of kv pairs in consul

I should be able to

npm install git2consul

Then use git2consul from terminal instead of having to check out the repo.

I have enabled support tag by **"support_tags" : true,**in config file and then i added tag to repo. but still nothing is reflected on consul. How actually it works?? can you please explain??

Hello.

I would love to see option to allow synchronising a given subfolder (path) of the repo. The problem is that we have multiple data centres with specific configuration for each. With current solution requires me to store configuration on different branches (one branch per DT) or store configuration in a subfolders (on folder per DT).

I we had option to allow synchronise specific repo we could divide our configuration like this:

/datacentre1
/datacentre2
....

and then in config (on a specific datacentre):

{
     "repos": [
         {
             "source_folder": "/datacentre1"
         },
     ]
}

Hi,

It would be nice if git tags were supported and could be used as "a branch that doesn't change". It could allow teams to version their property changes.

It currently already works if we specify a tag in "branches" field when configuring git2consul, but that's a bit hacky and ideally git2consul should pull all the tags by itself without having to manually add a tag.

We could introduce a new option field like "support_tags":true.

I've tweaked the code a bit locally to load all tags as branches and it works. I'm able to create this kind of hierarchy :

-sample-config
-----v1 ->first version
-----v2 -> second version
-----v3 -> third version
-----master -> master branch aka latest version

It still a hack because git2consul thinks it's a branch and tries to pull changes on those tags (resulting in exceptions), so we probably need to do something a bit more cleaver than just "loading tags as branches", but the concept works.

What do you think ?

I run the command 'git move filename1 filename2' after git2consul picks up changes both filename1 and filename2 are in the KV.

We're looking at using this in a kubernetes cluster and it would be super nice to support interpolation of the config file, making it very easy to provide key details via secrets.

Namely, to be able to create a public docker image with the following config bundled in:

{
  "version": "1.0",
  "repos" : [{
    "name" : "$REPO_NAME",
    "url" : "$REPO_URL",
    "source_root": "$SOURCE_ROOT",
    "mountpoint": "$MOUNTPOINT",
    "branches" : ["$BRANCH"],
    "include_branch_name": $INCLUDE_BRANCH_NAME,
    "hooks": [{
      "type" : "polling",
      "interval" : "1"
    }]
  }]
}

Otherwise we have to create and host our own image, which isn't the end of the world but would be nice if we didn't need to.

This issue was first brought up by @jsullivan3 internally.

When git2consul identifies that a Git repository has a conflicting change, it currently removes the local copy of the repository and then re-clones the remote repository. This process can take a non-trivial amount of time for a large repository.

A better approach would be to fetch refs from the remote and then perform a "git reset --hard" to update the local ref to match the remote ref. This approach will work provided no changes are made to the local repository, because any local changes will be lost as a result of the reset.

An empty repo will currently cause git2consul to fail to initialize. Fix this, with a test case.

We run a multi-tenant system. It would be really handy to be able to "mount" repositories at particular points within the K/V storage, so that one tenant is isolated from another. We plan to have several repositories within each tenant for logically distinct segments of configuration (eg, application-settings, infra-settings, secrets), to end up with something like

/kv
  /environment
    /shared-app1
    /shared-app2
  /tenant1
    /application
      /app1
      /app2
    /infrastructure
    /secrets

Driven by git2consul configuration that might look similar to this:

{
  "version": "1.0",
  "repos" : [
    {
      "name" : "shared-app1",
      "mount_path": "tenant1/application",
      "url" : "https://my-settings.com/shared-app1",
      "branches" : ["dev"],
      "hooks": [{
        "type" : "polling",
        "interval" : "1"
      }]
    }
  ]
}