• Use Vault Agent for AWS Auth
• Test out AWS secrets engine configuration with the changes in the AWS secrets engine changes (hashicorp/terraform-provider-vault#194)

Note that the user_data.sh script in the nomad-clients module is a symlink to the User data in Core module.

Implement our own poor man's alternative to Vault Enterprise's auto unseal.

https://www.consul.io/docs/agent/encryption.html

Use TLS-1-2-2017-01 instead.

See https://docs.aws.amazon.com/elasticloadbalancing/latest/application/create-https-listener.html

Generic question if someone wants to create their infra based on this repo, how to do that?

The current packer commands use ansible 2.7 syntax, however, we should upgrade to ansible 2.9 which is officially compatible with python 3.8, to avoid issues like this:

ansible/ansible#63973

https://www.nomadproject.io/docs/agent/encryption.html

Add check for missing submodules like roles/ansible-ca-store and roles/ansible-docker-ubuntu.

Otherwise it can be quite troublesome to only realise this when the new AMI is built and run without the right certificates.

Breaking changes since 0.1.2: https://github.com/hashicorp/terraform-aws-consul/releases

Currently, when Traefik is being upgraded in a rolling manner, there will be a period of time while ELB updates its target groups when end users will get a "bad gateway error".

This is because the Traefik job can potentially be running on less than the maximum number of Nomad clients.

There are two ways to solve this:

• Run Traefik as a system job. But we must be careful to write a constraint such that only those that are in the ASG that is used as the target group for Traefik's ELB has this scheduled.
• Run some kind of service that reacts to consul watch changes in the Traefik service and then react accordingly.

• Vault TLS Private Key
• Nomad's Vault configuration file containing the token

As a post processor

We will need to parse the manifest JSON using JQ. Then we can write it to a file for Terraform to use.

• Nomad
• Consul

We still have to configure Elasticsearch via the HTTP API, see https://docs.aws.amazon.com/elasticsearch-service/latest/developerguide/es-createupdatedomains.html#es-createdomain-configure-slow-logs.

We need to configure:

• Slow index logs
• Slow search logs
• Rolling indices

# https://www.elastic.co/guide/en/elasticsearch/reference/5.5/index-modules-slowlog.html
# This is PER INDEX
# We should probably do an index template...?
# curl -XPUT  \
#  https://vpc-tf-l-xxx.ap-southeast-1.es.amazonaws.com/syslog-*/_settings \
#  -H 'Content-Type: application/json' \
#  --data '{ "index.indexing.slowlog.threshold.index.warn": "10s", "index.indexing.slowlog.threshold.index.info": "5s", "index.indexing.slowlog.threshold.index.debug": "2s", "index.indexing.slowlog.threshold.index.trace": "500ms", "index.indexing.slowlog.level": "info", "index.indexing.slowlog.source": "1000"}'

See https://docs.aws.amazon.com/elasticsearch-service/latest/developerguide/cloudwatch-alarms.html

This refers to Vault's stdout and stderr.

Currently configured by supervisord to log to files. No way to ask Vault to log to syslog.

Might be related to hashicorp/terraform-aws-vault#73

https://github.com/metacloud/molecule

Consider tox for testing. Example: https://github.com/cloudalchemy/ansible-prometheus/blob/master/tox.ini

For end to end TLS for Vault

See https://www.vaultproject.io/guides/operations/production.html

Make this be "optional"

Currently, the run-telegraf script configures itself with a statsd server and output to Elasticsearch. At the same time, it also generates the telemetry stanza for Consul, Nomad and Vault.

This presents us with a few problems:

• Not easily extensible to "future" server types
• Does not allow for ease of adding additional Telemetry options. For example, for Nomad clients, you would want to enable allocation and node metrics.

Let's refactor the logic for generating the configuration for Nomad/Consul/Vault by putting them into additional or existing "configure" scripts for Nomad/Consul/Vault. Nomad already has such a script. We will have to create one for Consul and Vault.

Instead of calling Vault directly. Free retry and error handling from Consul Template.

This is wrt #201

We currently repeat the actions five times to support five relabel actions. Write some Ansible Dictionary merging loop to allow us to simply define the number of relabel actions to support with a variable.

Including:

• IP Address in octet format (currently, the host name is something obscure like "ip-10-123-123-123" which is not nice to work with
• EC2 Instance ID

Retrieve it from the instance metadata.

Packer image can be provisioned with the aid of https://github.com/cloudalchemy/ansible-prometheus

• Each node should run their own "Prometheus server" via Telegraf, and then advertise it as such on Consul. We can then use Prometheus' built in service discover with Consul to do the scraping.~
• Integrate this into the Telegraf module by updating it to generate the configuration file for Telegraf to run the Prometheus server and to advertise its presence via Consul for Prometheus to discover.

Integrations:

• AWS Auth
• Curator
• td-agent
• Traefik
• Vault SSH

We run pretty elaborate scripts in the user_data portions of the EC2 instances.

We need some way to detect if these scripts have failed.

Probabilities:

• https://www.uvd.co.uk/blog/create-health-check-aws-user-data-script
• https://stackoverflow.com/questions/11245356/how-to-check-user-data-status-while-launching-the-instance-in-aws

Idea:

• Define the existence of a user_data completion marker file as a service health check in Consul
• Forward logs via td-agent from User_data

TODOs:

• Consul Server
• Consul Agents (#131)
• Forward user_data logs via td-agent

#95 tried to reduce the frequency in which the template reading from Vault's auth/token/lookup-self is too frequent. But I failed to realise that this basically also affected the time the initial template is also rendered. #101 reverts that.

We should change this behaviour. The Vault token should never change anyway, so instead of having a template write this out, we should just write it out to ~/.vault-token as part of the run-consul-template.sh script.

Register Health Check in Consul by opening a connection to its socket_listener.

This won't check that the agent is actually collecting and sending stats. We probably need some other kind of "absence" monitoring.

Since 2.7

    ubuntu-1804-xxx-ami: [DEPRECATION WARNING]: Invoking "apt" only once while using a loop via
    ubuntu-1804-xxx-ami: squash_actions is deprecated. Instead of using a loop to supply multiple items
    ubuntu-1804-xxx-ami: and specifying `name: {{ item }}`, please use `name: ['apt-transport-https',
    ubuntu-1804-xxx-ami: 'ca-certificates', 'curl', 'software-properties-common']` and remove the loop.
    ubuntu-1804-xxx-ami: This feature will be removed in version 2.11. Deprecation warnings can be
    ubuntu-1804-xxx-ami: disabled by setting deprecation_warnings=False in ansible.cfg.

Set to 169.254.1.1 (DNSMasq)

Some of them can be run as Nomad Jobs.

• Consul service metrics
• Elasticsearch metrics. Circular dependency?
• Fluentd
• Docker on Nomad Clients

To Consider

• Cloudwatch to do alerting.
• "Grokking" logs for alerts instead of using Cloudwatch for: CloudTrail, VPC Flow Logs, application errors etc.

There are several current (and future) integration in the user_data scripts that require reading secrets from Vault. We might want to consider refactoring this to better support Vault operations. In particular, many applications do not know the existence of Vault and may fail to renew the lease on their secrets.

This is usually not a problem for secrets passed to Nomad (except for the future Consul ACL token) because Nomad will update the secrets as needed.

Steps the user_data script should perform:

1. Check with Consul if Vault is up and the KV store to see if AWS authentication is enabled for the type of servers.
2. Get a Vault Token (via vault login with AWS auth) and save it to the usual spot at ~/.vault_token. This will be done as the Root user so /root/.vault_token.
3. Change existing Nomad Vault integration to derive a Token from this Token to pass to Nomad instead of using the vault login token.
4. Get additional secrets from Vault if needed (e,g, a future Consul ACL token)
5. Set up a cron job to renew the lease for the login token and any other secrets. Currently, only the token passed to Nomad for Vault integration will automatically get renewed by Nomad.

We might want to consider using consul_template to render and manage the renewal of these tokens.

Open Questions:

• What if we were to enable ACL in Consul and then have Vault provide a Consul secrets backend to provide a Consul ACL. Won't this cause a chicken and egg problem when we try to discover Vault? Possibilities: anonymous tokens or agent specific default token

• Nomad Client
• Nomad Server
• Consul Server
• Vault

https://www.terraform.io/docs/providers/vault/r/ssh_secret_backend_ca.html

https://github.com/koalaman/shellcheck#travis-ci

There might be some false positives on "unassigned variables" in some of the templated scripts.

To 0.3.3

• New module for consul SG rules
• Built in scripting support for TLS and encryption

https://github.com/hashicorp/terraform-aws-nomad/tree/master/modules/run-nomad#how-do-you-handle-encryption

Use terraform-docs to standardize all input and output descriptions for all Terraform modules.

Command to use is terraform-docs md .

This is a regression.

Error: Error running plan: 6 error(s) occurred:

* module.core.module.vault_consul_gossip.aws_security_group_rule.allow_serf_lan_udp_inbound_from_security_group_ids: aws_security_group_rule.allow_serf_lan_udp_inbound_from_security_group_ids: value of 'count' cannot be computed
* module.core.module.nomad_clients.module.nomad_client_consul_gossip.aws_security_group_rule.allow_serf_lan_tcp_inbound_from_security_group_ids: aws_security_group_rule.allow_serf_lan_tcp_inbound_from_security_group_ids: value of 'count' cannot be computed
* module.core.module.vault_consul_gossip.aws_security_group_rule.allow_serf_lan_tcp_inbound_from_security_group_ids: aws_security_group_rule.allow_serf_lan_tcp_inbound_from_security_group_ids: value of 'count' cannot be computed
* module.core.module.nomad_server_consul_gossip.aws_security_group_rule.allow_serf_lan_tcp_inbound_from_security_group_ids: aws_security_group_rule.allow_serf_lan_tcp_inbound_from_security_group_ids: value of 'count' cannot be computed
* module.core.module.nomad_server_consul_gossip.aws_security_group_rule.allow_serf_lan_udp_inbound_from_security_group_ids: aws_security_group_rule.allow_serf_lan_udp_inbound_from_security_group_ids: value of 'count' cannot be computed
* module.core.module.nomad_clients.module.nomad_client_consul_gossip.aws_security_group_rule.allow_serf_lan_udp_inbound_from_security_group_ids: aws_security_group_rule.allow_serf_lan_udp_inbound_from_security_group_ids: value of 'count' cannot be computed

Maybe Terraform 0.12 will fix this, but for now, this doesn't work.

Or hashicorp/terraform#4149 will allow this.

Many times when running packer build for Nomad servers fail (likely the same for the other three services), because it is unable to install some package (obviously should be there) such as gcc, even though Ansible has stated that apt has already updated the cache and upgraded the state to the latest.

The log looks something like this:

    ubuntu-1604-nomad-server-ami:
    ubuntu-1604-nomad-server-ami: PLAY [Provision AMI] ***********************************************************
    ubuntu-1604-nomad-server-ami:
    ubuntu-1604-nomad-server-ami: TASK [Gathering Facts] *********************************************************
    ubuntu-1604-nomad-server-ami: ok: [default]
    ubuntu-1604-nomad-server-ami:
    ubuntu-1604-nomad-server-ami: TASK [Upgrade all packages to the latest version] ******************************
    ubuntu-1604-nomad-server-ami:  [WARNING]: Could not find aptitude. Using apt-get instead.
    ubuntu-1604-nomad-server-ami: changed: [default]

...

    ubuntu-1604-nomad-server-ami: TASK [xxx/terraform-modules/modules/core/packer/nomad_servers/../../../../roles/td-agent : Install dependencies for td-agent] ***
    ubuntu-1604-nomad-server-ami: failed: [default] (item=['gcc', 'make']) => {"changed": false, "item": ["gcc", "make"], "msg": "No package matching 'gcc' is available"}
    ubuntu-1604-nomad-server-ami:       to retry, use: --limit @xxx/terraform-modules/modules/core/packer/nomad_servers/site.retry
    ubuntu-1604-nomad-server-ami:
    ubuntu-1604-nomad-server-ami: PLAY RECAP *********************************************************************
    ubuntu-1604-nomad-server-ami: default                    : ok=5    changed=3    unreachable=0    failed=1

https://docs.fluentd.org/v1.0/articles/in_forward#how-to-enable-tls-mutual-authentication

Use Vault as a CA.

• Use the tail input: https://docs.fluentd.org/v1.0/articles/in_tail
• Set read_from_head = true
• Log is at /var/log/user-data.log
• Tag with the server_type
• Maybe ship it to system.user_data index

Find some mechanism to register a health check with Consul.

This will not check that logs are being delivered. We need some form of "absence" monitoring.

With Curator: https://github.com/elastic/curator

Run this as a Nomad Job.

Specifically, add tags using

• snapshot_tags
• tags

Include the following:

• Name
• Base name
• Date time created
• A packer indication

0.6.0

In relation to #32 and to better support Vault 0.10.0