Make Cronicle a bit more automation-friendly about cronicle HOT 1 CLOSED

Hey @barryw,

Yup, I totally understand the need. My own company has started using Cronicle, and we have a fully automated installation and configuration process. The trick is to supply your own customized /opt/cronicle/conf/setup.json file which can contain your own administrator users, plugins, API keys, and/or server group regexps in it, and then you simply wrap up the entire /opt/cronicle directory in a tarball, RPM, DEB, Docker container, or whatever you use to deploy software. Then, when you extract it and run the bin/control.sh setup command on a fresh environment, it will use your own setup.json file, and create all the necessary data you need. Here is how we did ours:

We have a build system which generates software packages to install onto our staging and production servers. On the build server, we actually run the initial Cronicle auto-install command, which downloads and installs the base software and all the npm dependencies:

curl -s https://raw.githubusercontent.com/jhuckaby/Cronicle/master/bin/install.js | node

But the build script stops there, and doesn't run any additional setup commands. It then copies in our own files from our private Git repo, replacing things where appropriate. This is where our own /opt/cronicle/conf/setup.json file gets swapped into place, replacing the stock Cronicle one. Our copy of the file contains a special administrator user with a known password (hashed with bcrypt -- see below for explanation), some of our own custom Plugins, our own server group definitions (server regexps), and of course some API keys. Here is our file, with some sensitive bits stripped off, and explanations below:

https://www.dropbox.com/s/lx39yl5p82mo2y7/cronicle-custom-setup.json?dl=0

So, the setup.json file has two main top-level sections, storage and build. You can completely ignore the build section, as it is simply used during installation to compact files for distribution, and stuff like that. In fact, that entire section will be moving to its own file in the future, so you don't have to worry about your copy getting out of date. All you need to focus on is the storage section.

The storage section of the file is a JSON array of actions to perform, such as creating records, creating lists, or appending items onto lists. These are all simply run in sequence, and you can use this mechanism to create any data records you need. For example, to add an administrator user with username admin2, include an item in the storage array like this:

[ "put", "users/admin2", {
	"username": "admin2",
	"password": "********************************",
	"full_name": "Administrator 2",
	"email": "[email protected]",
	"active": 1,
	"modified": 1434125333,
	"created": 1434125333,
	"salt": "salty",
	"privileges": {
		"admin": 1
	}
} ],

Each "command" is an sub-array of exactly 3 elements. The command name (e.g. put), the storage path (e.g. users/admin2) and the JSON object to insert.

A few things to note here. The storage path for users is users/USERNAME and the USERNAME must be lower-case, and it must match the username in the username property. The password is a Bcrypt hash of the password plus the salt value (which should be some unique string). The best way to do this is actually to export a user that already exists in a separate install of Cronicle, and has his/her password and salt already created. To extract a user, run this command:

/opt/cronicle/bin/storage-cli.js get users/USERNAME

Replace USERNAME with the username of the user you wish to extract (lower-case).

Then you can just copy & paste that user's JSON right into your setup.json file, but remember that the command to put a storage record is actually an array that contains 3 elements, the command itself (put), then path (users/USERNAME) then the user record as a JSON object. It might be easier to just copy the password and salt string values in, as those are the only tricky ones to deal with.

Note that users not only need a record created with the put command, but they also need to be added to the master user list (so they show up on the "Users" tab in the UI). This is accomplished by these actions here:

[ "listCreate", "global/users", { "page_size": 100 } ],
[ "listPush", "global/users", { "username": "admin" } ],
[ "listPush", "global/users", { "username": "admin2" } ],

Here we are creating the master users list via the listCreate command with path global/users, then we are adding two users to that list, the standard admin, and our own custom admin2.

So, taking a look at adding a server group definition (regexp), the process is very similar, but these guys live inside a list only, and don't require the two step dance that users do. All you need here is a listPush after the global/server_groups list is created:

[ "listCreate", "global/server_groups", {} ],
[ "listPush", "global/server_groups", {
	"id": "allgrp",
	"title": "All Servers",
	"regexp": ".+",
	"master": 0
} ],
[ "listPush", "global/server_groups", {
	"id": "mtxgrp",
	"title": "MTX Group",
	"regexp": "(dev01|devrelease|mtx\\d+)\\.",
	"master": 1
} ],

So the listCreate and first listPush are in the default setup.json file. But we've added our own addition here, with that second listPush operation:

[ "listPush", "global/server_groups", {
	"id": "mtxgrp",
	"title": "MTX Group",
	"regexp": "(dev01|devrelease|mtx\\d+)\\.",
	"master": 1
} ],

This is a custom group we added, and you'll note that the master property is set to 1. This makes these servers that match the specified hostname regexp eligible to become Cronicle master servers.

Two things to keep in mind here. First, the id property needs to be some kind of unique alphanumeric identifier. Put whatever you want here, but make sure it is unique. And second, remember to escape your backslashes in the regexp, as this is a JSON string.

If you want to dump out (extract) your existing server groups in JSON format (like, those you added from inside the UI for example) from a running Cronicle installation, type this command on the server:

/opt/cronicle/bin/storage-cli.js list_get global/server_groups

Finally, let's look at adding your own custom API key in setup.json. This is very similar to adding a server group, as API keys simply live in a list. Add your own listPush command after the listCreate for the global/api_keys list. Example here:

[ "listCreate", "global/api_keys", {} ],
[ "listPush", "global/api_keys", {
	"active": "1",
	"title": "Custom Roll Script",
	"description": "This is an API Key for the Roll Script to disable events and abort jobs during rolls.",
	"id": "customkey1234",
	"key": "****************************************",
	"username": "admin",
	"modified": 1434125333,
	"created": 1434125333,
	"privileges": {
		"admin": 0,
		"create_events": 1,
		"edit_events": 1,
		"delete_events": 1,
		"run_events": 1,
		"abort_events": 1,
		"state_update": 1
	}
} ]

So here you can see the JSON format for an API key record, and how to add in your own. The id needs to be a unique alphanumeric identifier for the key (this is not the API key itself -- it is just used internally to identify the API key record). The key is the actual API key itself, i.e. the thing you put into your HTTP REST requests via the X-API-Key header to authenticate.

The active property should be set to 1, which specifies that the API key is enabled. The title and description are used in the UI. The username is the user who created the API key (you can just put admin here). The created and modified are Unix Epoch timestamps. The privileges control what the API key is allowed to do, and these correspond to the checkboxes in the UI when creating API keys the normal way.

Please note that setting the admin privilege to 1 is not supported. I am not sure if it would even work. API keys were not designed to have administrator privileges, so the behavior is undefined.

Note that the listCreate command for the global/api_keys list is already in the stock setup.json file. All we did in the example here is add our own custom API key just under it.

If you want to dump out (extract) your existing API keys in JSON format (like, those you added from inside the UI for example) from a running Cronicle installation, type this command on the server:

/opt/cronicle/bin/storage-cli.js list_get global/api_keys

So that's all you need to know for creating your own custom setup.json file. Just make sure your automated installation process moves this file into place, replacing the existing stock file here: /opt/cronicle/conf/setup.json. Then, after you replace the file with your copy, run the standard Cronicle setup command, and it will execute your custom commands. Note that you only need to run the setup command once per environment, not per server.

Allow us to create API keys via the API (yes, I know this is a turtles-all-the-way down scenario). Basically, we need a programmatic way of generating API keys so that we can stuff them into Consul so that our services can talk to Cronicle.

Alas, there's no way for an API key to execute any administrator-level actions, like creating more API keys. But I did show you how to insert your own API keys into the setup.json file, so they can be available on all your standard installations. Please let me know if this won't suffice, and you need something else.

Allow us to at least change the administrator password via the API. We don't need a full user API, just something to secure the installation. Can this be done in setup.json?

Well, I showed you how to insert your own administrator users with known passwords into the setup.json file, for standard installs. But you can also create or "recover" an administrator account on the command-line, by using this command:

/opt/cronicle/bin/control.sh admin USERNAME PASSWORD

But please note this is really for emergency recovery purposes, like if you are locked out. This is more for creating a temporary administrator account.

Explained in the docs here: https://github.com/jhuckaby/Cronicle#recover-admin-access

I hope all this helps!

- Joe

from cronicle.