💡 Summary

Gophish instances (which contain a Dockerized postfix container) should fetch the TLS certificate for their specified domain from our S3 certificates bucket (if available) and place it in the appropriate location for postfix to use it.

Motivation and context

This will allow postfix to run with a valid and appropriate certificate for the instance. It is also one of the acceptance criteria specified in https://github.com/cisagov/cool-system/issues/90.

Implementation notes

This issue does NOT include figuring out how the certificate is created and put in our S3 bucket. Initially, that will be a manual process that we want to automate in the future.

Acceptance criteria

• When a Gophish instance is deployed, the fullchain.pem and privkey.pem files for the appropriate TLS certificate (i.e. the one that corresponds to the email_sending_domain Terraform variable) are deployed to /var/pca/pca-gophish-composition/secrets/postfix/ before postfix starts up.
• A mail client (e.g. Thunderbird) is able to securely connect to postfix without encountering any security exceptions (assuming that the TLS certificate is valid and unexpired).
• If a certificate for the specified domain is NOT found in S3, a warning is logged and the default certificate data (from cisagov/postfix-docker) is untouched.

Integrate the assessment VPC with the Transit Gateway in our Shared Services account.

💡 Summary

Enable an assessment to have more than one email-sending domain.

Motivation and context

Some types of assessments require multiple instances (e.g. Teamservers), where each instance sends email from a different domain. Currently, our email_sending_domain variable only supports a single domain.

Implementation notes

Convert the email_sending_domain variable from a string to a list.

Acceptance criteria

• Terraform and cloud-init function correctly with no email-sending domains, a single Teamserver, and a single Gophish instance
• Terraform and cloud-init function correctly with a single email-sending domain, a single Teamserver, and a single Gophish instance
• Terraform and cloud-init function correctly with two email-sending domains, two Teamservers, and two Gophish instances
• Documentation explains how list of email-sending domains is applied to instances that send mail (teamserver, gophish).

💡 Summary

Allow Teamserver instances to talk to port 587 on Gophish instances.

Motivation and context

This change will enable mail to be sent from Cobalt Strike (on the Teamservers) via the SMTP mail-submission port published by the Postfix Docker container (587) on the Gophish instances.

Implementation notes

None

Acceptance criteria

• Any Teamserver in an assessment environment can send mail via Postfix running on any Gophish instance in the same environment.

Add Terraform to spin up a Kali instance in the VPC.

@dav3r commented on Mon Mar 02 2020

Add Terraform to spin up a Guacamole instance in the VPC.

@bjb28 commented on Thu May 14 2020

🚀 Feature Proposal

Request to have an Elastic IP assigned to the PCA Teamserver to prevent us from needing to change DNS records during an assessment.

Motivation

Limit the chances that an IP change happens which would require us to update DNS records. If this were to happen we would need to update the customer with the new IP and change DNS records.

Example

Same IP through entire assessment.

Pitch

An elastic IP will help to avoid additional work and possible missed clicks during an assessment.

🐛 Summary

The cisagov/assessor-portal-packer repo was renamed to cisagov/assessor-workbench-packer, and the name of the resulting AMI was changed in cisagov/assessor-workbench-packer#31 from assessor-portal-hvm-${local.timestamp}-x86_64-ebs to assessor-workbench-hvm-${local.timestamp}-x86_64-ebs. The new name more closely matches what folks are calling the tool. The AMI name was unfortunately not updated in this repository, so the refreshed AMIs with the newer name are not being used.

The AMI name should be updated, and all resource names, etc. that reference the old "Assessor Portal" language should be updated to instead reference "Assessor Workbench". Note that the name of the instance type corresponding to the Assessor Workbench would need to be updated in the inbound_ports_allowed and operations_instance_counts input variables, so this is a breaking change.

💡 Summary

Instead of having the Guacamole instance and the Terraformer instances in the same subnets, we should split the private subnets into separate sets of Guacamole and Terraformer subnets.

Motivation and context

The Guacamole server is touched by traffic that comes from outside the assessment environment, so the subnet in which it resides should not contain any instances that do not touch such traffic. The Guacamole and Terraformer use cases are also quite different, so it is difficult for them to share NACL rules without making them overly broad for both types of instances.

In addition, I found that I can't ssh via SSM to a Terraformer instance unless I put it in the first private subnet alongside Guacamole. I believe this has something to do with the NACLs that are in place for that subnet specifically. This renders the second private subnet useless for our purposes.

Acceptance criteria

• The Terraformer instance(s) are placed in a subnet (or set of subnets) separate from the operations subnet or the subnet(s) where Guacamole resides.
• Dev team members are able to ssh into the Terraformer instances via AWS SSM.
• The Terraformer instance(s) are also reachable via Guacamole.
• The Terraformer instance(s) can still be used to create, destroy, and modify all appropriate AWS resources.

Currently, when the Guacamole instance is deployed, it has default credentials for the admin user. Modify this so that a new password is applied (probably should be pulled from the SSM Parameter Store).

🐛 Summary

The indexing created here and used here works, but it causes problems if ports are added or removed from the inbound_ports_allowed input variable. This is for the same reason that it is preferable to use a map instead of a list when creating multiple resources via a single resource block: if you add a single entry in the middle then multiple resources must be destroyed because the indexing changes.

💡 Summary

It is possible to assign policies to the VPC endpoints. These policies restrict what can queries can be sent via the VPC endpoint.

Motivation and context

This would provide yet another layer of security, albeit at the cost of more maintenance.

Acceptance criteria

• Policies are created and assigned to the VPC endpoints.
• The assessment environment continues to function as it did before.

💡 Summary

We should add EFS access points to control file-system access to the EFS share.

Motivation and context

Currently there is no mechanism to enforce file-system permissions on access to the EFS share. This means that files and directories with uids/gids different from those owning the EFS share mount point can be created. This can result in difficulties interacting with EFS share contents on systems that cannot sudo (like Windows instances).

Implementation notes

There should be an access point for each EFS mount target created (there is only one currently). The efs-mount cloud-config will need to be updated to support this functionality as well.

Acceptance criteria

How do we know when this work is done?

• EFS share file-system permissions are enforced

It looks like the postgres service (in the guacamole Docker composition) ran out of memory today, after being up for less than 24 hours:

postgres              | 2020-04-02 22:10:20.341 UTC [1] LOG:  database system is ready to accept connections
postgres              | 2020-04-03 20:23:05.774 UTC [1] LOG:  could not fork autovacuum worker process: Cannot allocate memory
postgres              | 2020-04-03 20:23:12.364 UTC [1] LOG:  could not fork autovacuum worker process: Cannot allocate memory

I ssh'd to the instance and restarted the service with:
systemctl restart guacamole-composition.service

However, we probably want to move our Guacamole instance to something with a bit more memory to avoid this in the future.

🐛 Summary

Guacamole was temporarily unavailable in env22-production yesterday because the logs associated with the Docker composition filled up the root disk on the Guacamole instance. Giving the Guacamole instance a larger root disk will help alleviate this.

See also cisagov/guacamole-composition#54.

To reproduce

Steps to reproduce the behavior:

1. Spin up an RTA environment (many instances accessible via guacamole).
2. Wait a few months.
3. Observe that the root disk on the Guacamole instance fills up.

Expected behavior

There should be no interruption in availability of the Guacamole service.

Any helpful log output or screenshots

root@guac:/usr/bin# df -h
Filesystem       Size  Used Avail Use% Mounted on
udev             1.9G     0  1.9G   0% /dev
tmpfs            388M  756K  387M   1% /run
/dev/nvme0n1p1   7.7G  6.4G  936M  88% /
tmpfs            1.9G     0  1.9G   0% /dev/shm
tmpfs            5.0M     0  5.0M   0% /run/lock
/dev/nvme0n1p15  124M  5.9M  118M   5% /boot/efi
overlay          7.7G  6.4G  936M  88% /var/lib/docker/overlay2/5453e28a36f9ce890a760c86e1f2d5e94f8c6659ad3d229c7c964e370d5ea2a3/merged
shm               64M     0   64M   0% /var/lib/docker/containers/3f5bd5504fc149f9fe74c3abe28c85f3f6e8bfcd7d8bded026d3125efe005060/mounts/shm
overlay          7.7G  6.4G  936M  88% /var/lib/docker/overlay2/570012723f1dbb24eaa5504a75c3d2a346437e6d271d99dca3cfc1f7b91031a8/merged
shm               64M     0   64M   0% /var/lib/docker/containers/16cca99e59f0c08e0bdaef4d79ec6785c2322b9a916c6c79fb62a31929778775/mounts/shm
overlay          7.7G  6.4G  936M  88% /var/lib/docker/overlay2/4f4cba0ebff605b2300ad541baec5879cd00f892a08d9527299d736a31b8b3d6/merged
shm               64M     0   64M   0% /var/lib/docker/containers/9538eabf5828278a1c24c420a700a658b580b0542213a10977e1ef680221781d/mounts/shm
overlay          7.7G  6.4G  936M  88% /var/lib/docker/overlay2/f650455c4ce1035e78ad84f61c6373e68ac123b600067bbbf538788f52f425ce/merged
shm               64M   16K   64M   1% /var/lib/docker/containers/176e7686b05da94828c46c0419398dd7df2b3699bda952447cc312b52f571c1d/mounts/shm
tmpfs            388M     0  388M   0% /run/user/0

💡 Summary

Now that I know that the number of security groups that can be applied to an ENI is not a hard limit, I'd like to create a single cloudwatch_endpoint_client_sg security group, for example. That will allow me to simplify the corresponding security group rules by dropping the new CIDR-based rule added in #143 and getting rid of all the other SG-based rules. I will do something similar for the other VPC endpoint security groups. Instances spun up via the Terraformer instance can then simply add themselves to those security groups as necessary to gain access to the necessary VPC endpoints.

This change will also allow me to get rid of the existing instance type SGs, except for those that actually require an explicit mention. For example, the Debian desktop SG will probably be rendered unnecessary as it isn't mentioned explicitly outside of these endpoint SGs, but the Teamserver instance will still be becessary because certain other need to be instance types are allowed ingress to the Teamservers.

Motivation and context

"Simplify, simplify, simplify!" - Henry David Thoreau

Not sure how many are live, but at .04/hr vs .16/hr for a t3.medium vs t3.xlarge changing even one instance saves ~$1000/yr, and the cpu need for these things is probably pretty minimal.

Update the Guacamole cloud-init scripts so that they do the following:

1. Create the guacuser (non-admin) Guacamole account.
2. Allow guacuser to access every configured Guacamole connection.

🐛 Bug Report

When an assessment environment is created without a Kali instance and with no operations_subnet_inbound_tcp_ports_allowed and no operations_subnet_inbound_udp_ports_allowed, the resulting Guacamole instance starts up in a messed-up state and is not accessible via SSH.

To Reproduce

Steps to reproduce the behavior:

• Create a terraform variables file that includes:

operations_instance_counts = {
  "debiandesktop" : 1,
  "kali" : 0,
}
operations_subnet_inbound_tcp_ports_allowed = []
operations_subnet_inbound_udp_ports_allowed = []

• Run terraform apply

Expected behavior

The Guacamole instance should start up correctly and be accessible by SSH (via SSM).

Any helpful log output

Here is the section of the log that shows that the Docker bridge fails to come up, which prevents the Guacamole composition from starting successfully:

[   73.135967] cloud-init[579]: Cloud-init v. 20.2 running 'modules:config' at Tue, 03 Nov 2020 21:34:11 +0000. Up 71.06 seconds.
[   79.777026] bridge: filtering via arp/ip/ip6tables is no longer available by default. Update your scripts to load br_netfilter if you need this.
[   79.809832] Bridge firewalling registered
[   80.781409] Initializing XFRM netlink socket
[   80.872609] IPv6: ADDRCONF(NETDEV_UP): docker0: link is not ready
[   83.996722] IPv6: ADDRCONF(NETDEV_UP): br-4516774645af: link is not ready
[  100.326197] IPv6: ADDRCONF(NETDEV_UP): br-f26879dd3117: link is not ready
[  116.666879] IPv6: ADDRCONF(NETDEV_UP): br-fee1ac643615: link is not ready
[  132.850902] IPv6: ADDRCONF(NETDEV_UP): br-1375b6d8322d: link is not ready
[  149.434985] IPv6: ADDRCONF(NETDEV_UP): br-c351ac01eb7e: link is not ready
[  165.578743] IPv6: ADDRCONF(NETDEV_UP): br-724ec8dcc51f: link is not ready
[  182.214864] IPv6: ADDRCONF(NETDEV_UP): br-16a24fb24d6a: link is not ready
[  198.574740] IPv6: ADDRCONF(NETDEV_UP): br-a70d667a1eb2: link is not ready
[  214.986790] IPv6: ADDRCONF(NETDEV_UP): br-e21ddb74d51b: link is not ready
[  231.342529] IPv6: ADDRCONF(NETDEV_UP): br-f1cc3b89432f: link is not ready
[  247.735274] IPv6: ADDRCONF(NETDEV_UP): br-a95cdf69980f: link is not ready

It is unclear why this would happen based on the lack of a Kali instance or allowing inbound ports. Our current suspicion is that it has something to do with the cloud-init scripts that generate the Guacamole connections:

• write-guac-connection-sql-template.tpl.yml
• render-guac-connection-sql-template.py

We think that those scripts may be failing in such a way that cloud-init gets hung up and cannot continue, but more research is needed.

It's also worth noting that this message in the log (bridge: filtering via arp/ip/ip6tables is no longer available by default. Update your scripts to load br_netfilter if you need this.) sounds bad, but we have confirmed that it shows up on instances where Guacamole starts up successfully, so we don't believe it is related to this issue.

🚀 Feature Proposal

Right now there is some manual AWS CLI foo that takes place to associate the assessment VPC with the private hosted zone of the Shared Services VPC. Terraform does not yet support this kind of cross-account association. We should make use of a module, such as this one so that association need not take place manually.

Motivation

We should eschew manual steps, since

• It is difficult to automate a process that has manual steps.
• Manual steps are error-prone.

💡 Summary

Consider updating the EFS to use elastic throughput mode.

Motivation and context

I heard from an RTA entity that the EFS is very slow in their assessment environment. I also recently received an email from AWS that stated that one or more EFS shares had been throttled because the current mode (bursting throughput mode) is insufficient to handle the traffic. The email suggested switching to elastic throughput mode. (The EFS can also apparently switch between the two modes on demand.)

Contents of the email from AWS:

Hello,

You are receiving this message because, at least once in the past 30 days, one or more Amazon EFS file systems in your account has been constrained by the performance limits of EFS’s Bursting throughput mode. Specifically, these file systems have exhausted the Bursting mode’s available throughput and/or IOPS performance, which could result in a degraded experience for your application or end users.

Your file system is currently configured to use EFS's Bursting Throughput mode. Bursting file systems' throughput capacity is determined by a credit model (“burst credits”) and IOPS capacity is up to 35,000 read IOPS and up to 7,000 write IOPS [1]. When your file system exhausts its “burst credits” or reaches its IOPS limit, throttling of additional I/O can result in unexpected slowness, timeouts, or other application failures for your end users or workloads.

We have identified the following impacted Amazon EFS file system(s) that have been throttled in the past 30 days and require more throughput or IOPS than what Bursting Throughput provides.

A list of your affected resource(s) can be found in the 'Affected resources' tab in the AWS Health Dashboard.

#Action Recommended:
To help your end users and applications to achieve their required throughput and IOPS performance, we recommend that you update your file system configuration to Elastic Throughput mode [2]. When in Elastic Throughput mode, your file system can achieve up to 10 GiB/s of read throughput or 3 GiB/s of write throughput — depending on the AWS Region [3] — and up to 55,000 read IOPS and 25,000 write IOPS. Elastic Throughput charges a pay-as-you-go rate for data transfer to/from your file system, which means you only pay for what you use [4]. You can update your file system configuration to switch between Elastic and Bursting throughput modes on demand.

If you have already updated your file system configuration to one of EFS’s enhanced throughput modes - Elastic or Provisioned Throughput - you can disregard this message.

If you have questions or concerns, please contact AWS Support [5].

[1] https://urldefense.us/v3/__https://docs.aws.amazon.com/efs/latest/ug/performance.html*performance-overview__;Iw!!BClRuOV5cvtbuNI!EogXGg2V2HigA96RB39oMQc433wJ7T0KegTJXaXvs7PGLdiHUkFlAVIMAxzCJ-jhRN8h35x-sfds1IBoiSEYoa6H12FwpyVgFhNv5TjW07-ahyg$
[2] https://urldefense.us/v3/__https://docs.aws.amazon.com/efs/latest/ug/performance.html*throughput-modes__;Iw!!BClRuOV5cvtbuNI!EogXGg2V2HigA96RB39oMQc433wJ7T0KegTJXaXvs7PGLdiHUkFlAVIMAxzCJ-jhRN8h35x-sfds1IBoiSEYoa6H12FwpyVgFhNv5TjWygC6JEY$
[3] https://urldefense.us/v3/__https://docs.aws.amazon.com/efs/latest/ug/limits.html__;!!BClRuOV5cvtbuNI!EogXGg2V2HigA96RB39oMQc433wJ7T0KegTJXaXvs7PGLdiHUkFlAVIMAxzCJ-jhRN8h35x-sfds1IBoiSEYoa6H12FwpyVgFhNv5TjWNjU9s3U$
[4] https://urldefense.us/v3/__https://aws.amazon.com/efs/pricing/__;!!BClRuOV5cvtbuNI!EogXGg2V2HigA96RB39oMQc433wJ7T0KegTJXaXvs7PGLdiHUkFlAVIMAxzCJ-jhRN8h35x-sfds1IBoiSEYoa6H12FwpyVgFhNv5TjWXjBwRUY$
[5] https://urldefense.us/v3/__https://aws.amazon.com/support__;!!BClRuOV5cvtbuNI!EogXGg2V2HigA96RB39oMQc433wJ7T0KegTJXaXvs7PGLdiHUkFlAVIMAxzCJ-jhRN8h35x-sfds1IBoiSEYoa6H12FwpyVgFhNv5TjW_TGYkJ0$

Sincerely,
Amazon Web Services

Amazon Web Services, Inc. is a subsidiary of Amazon.com, Inc. Amazon.com is a registered trademark of Amazon.com, Inc. This message was produced and distributed by Amazon Web Services Inc., 410 Terry Ave. North, Seattle, WA 98109-5210

Implementation notes

None.

Acceptance criteria

How do we know when this work is done?

• The EFS is switched to use elastic throughput mode.
• The assessors verify increased performance of the EFS.

💡 Summary

In order to leverage proxychains between our CommandoVM and TeamServer we need to have port range 5000-5999 TCP open between these instances.

Motivation and context

This will allow us to tunnel traffic from the CommandoVM into the target networks. Some tools leveraged on our engagements are run from Windows and in our current design we are unable to leverage those tools during the External week.

Implementation notes

• CommandoVM would allow inbound traffic from both TeamServer instances over ports 5000-5999 TCP
• TeamServer would allow inbound traffic from any deployed CommandoVMs over ports 5000-5999 TCP

Acceptance criteria

How do we know when this work is done?

• CommandoVM would allow inbound traffic from both TeamServer instances over ports 5000-5999 TCP
• TeamServer would allow inbound traffic from any deployed CommandoVMs over ports 5000-5999 TCP

💡 Summary

We should give each operations instance type its own security group instead of lumping several instance types into the operations security group. We actually already do this with some instance types, but not others.

Motivation and context

The Nessus instances are assigned to the operations security group, but they and only they require access to the AWS STS VPC endpoint. By splitting the nessus instances out into their own security group we could avoid needlessly giving other operations instance types permissions they do not need.

It might also make sense to have an all_egress security group for the Kali and Nessus instance types, then also create Kali- and Nessus-specific security groups that give only the extra (VPC endpoint) access they need.

Acceptance criteria

• Nessus instances are placed in their own security group that assigns them permission to use the STS VPC endpoint.
• Nessus instances can still perform their scanning function as before.

💡 Summary

Make an input variable for the domain where emails will be sent from and configure the Postfix container on the Gophish instance to correctly use that domain.

Motivation and context

Many of our assessments have the need to send emails from a specific domain, which currently requires several manual configuration steps. This would automate part of that process.

This is part of https://github.com/cisagov/cool-system/issues/90.

Implementation notes

Ensure that the PRIMARY_DOMAIN is set for Postfix before the container starts up. As long as this is done by cloud-init, I think we should be fine, since the pca-gophish-composition SystemD service doesn't start up until after cloud-final.service.

Acceptance criteria

When a fresh assessment environment containing a Gophish instance is deployed:

• The Postfix container's primary domain is correctly set based on the value of the Terraform input variable for the email-sending domain
• The Postfix container and the pca-gophish-composition SystemD service on the Gophish instance start up correctly
• Email can be successfully sent via Postfix from the email-sending domain

🚀 Feature Proposal

Automatically disassociate the default Transit Gateway route table association before Terraform attempts to perform the association for the assessment environment.

Motivation

We want to avoid this Terraform error (which we currently get):

Error: error associating EC2 Transit Gateway Route Table (tgw-rtb-04f86cffc92794d4b) association (tgw-attach-0e1e8577d46e6cd93): Resource.AlreadyAssociated: Transit Gateway Attachment tgw-attach-0e1e8577d46e6cd93 is already associated to a route table.
        status code: 400, request id: 35096b88-77b9-46af-8507-0254bce8a190

When we get this error now, we have to manually disassociate the default Transit Gateway route table association, then re-run the terraform apply.

Pitch

We want to remove as many manual steps as possible from our Terraforming process.

💡 Summary

Customize the Gophish instance so that Thunderbird will automatically configure email accounts for the specified assessment email-sending domain.

Motivation and context

In https://github.com/cisagov/cool-system/issues/90, the RVA team has requested an email client that requires minimal configuration to use during assessments.

Implementation notes

To meet this need, two things must be done:

• Deploy a Thunderbird autoconfiguration file to the Gophish instance (at /usr/lib/thunderbird/isp/myemailsendingdomain.gov.xml) containing the following (customized for the appropriate domain):

<?xml version="1.0"?>
<clientConfig version="1.1">
  <emailProvider id="myemailsendingdomain.gov">
    <domain>myemailsendingdomain.gov</domain>
    <displayName>myemailsendingdomain.gov</displayName>
    <displayShortName>myemailsendingdomain.gov</displayShortName>
    <incomingServer type="imap">
      <hostname>myemailsendingdomain.gov</hostname>
      <port>993</port>
      <socketType>SSL</socketType>
      <authentication>password-cleartext</authentication>
      <username>%EMAILLOCALPART%</username>
    </incomingServer>
    <outgoingServer type="smtp">
      <hostname>myemailsendingdomain.gov</hostname>
      <port>587</port>
      <socketType>STARTTLS</socketType>
      <authentication>password-cleartext</authentication>
      <username>%EMAILLOCALPART%</username>
    </outgoingServer>
  </emailProvider>
</clientConfig>

• Add a line to /etc/hosts for the email sending domain, e.g.:
  127.0.0.1 myemailsendingdomain.gov

These two things (combined with a successfully-deployed TLS certificate) will enable a simple, seamless setup of email accounts for the specified domain in Thunderbird.

Acceptance criteria

• /usr/lib/thunderbird/isp/myemailsendingdomain.gov.xml is deployed at instance startup
• /etc/hosts contains an entry like 127.0.0.1 myemailsendingdomain.gov
• When using the Thunderbird "new email account" wizard to create accounts for the assessment email-sending domain, new accounts can be added without any errors and with the correct SMTP/IMAP configuration
• Emails can be sent using the new account in Thunderbird
• Emails sent from the new account can be viewed in Thunderbird via IMAPS

💡 Summary

Use a narrower role (or roles) than the Shared Services "provision account" role.

Motivation and context

We currently use a provider based on the very powerful "provision account" role in the Shared Services account to provision assessment environments. To reduce risk, a narrower, more tailored role (or roles) should be created with only the permissions necessary to accomplish the goal.

Implementation notes

Note that we currently have two providers based on the same Shared Services role- this duplication should be eliminated:

• dns_sharedservices
• provisionsharedservices

Acceptance criteria

• A less-powerful role (or roles) is used to manipulate resources in the Shared Services account.
• Assessment environments can still be successfully created, updated, and destroyed.

Currently, all infrastructure in this project is deployed to a single AWS availability zone, as defined by the aws_region and aws_availability_zone variables. To improve our fault-tolerance, we should come up with a reasonable way to spread assessment resources across multiple availability zones so that assessment operations can continue if a particular availability zone becomes unavailable.

Our solution should also consider the usage of multiple AWS regions as well.

💡 Summary

Move the persistent Gophish data currently stored on the EFS volume over to the new EBS volume created in #119.

Motivation and context

Since Gophish data lives within Docker, it can be automatically persisted on the EBS volume similar to how our Dockerized postfix data is stored (see ). Now that this EBS volume exists, there is no good reason to store the Gophish data on the EFS volume.

Implementation notes

I think this change will pretty much be deleting this file and the code that calls it.

Note that this issue is dependent on cisagov/pca-gophish-composition-packer#48 being completed first.

Acceptance criteria

Gophish data persists in all of the following cases:

• After a Gophish instance is destroyed and redeployed
• After a Gophish instance is stopped and started
• After the PCA-Gophish composition is restarted

Modify the Nessus installation so that it listens on port 443, rather than the default of 8834. Even better would be to make the port a Terraform variable with a default of 8834.

See this page to see how to easily change the xmlrpc_listen_port via a nessuscli command.

Add the ability to spin up one or more GoPhish instances based on pca-gophish-composition-packer.

🐛 Summary

A nameless assessor reports that he needs to store some Docker images on the Terraformer instance's root disk, but it is not large enough to do so. I suggested storing them on the EFS share, but he states that "EFS is very slow". He further states that a Terraformer root disk of 128GB would suffice.

🚀 Feature Proposal

For each instance that currently gets a VNC connection, create an additional Guacamole SSH-only connection.

Motivation

If an assessment user does not require GUI access to an instance, they can use the SSH-only connection, which should (in theory) have less latency and be more responsive to keystrokes, thus improving the overall user experience.

Pitch

If you don't need all the bells and whistles of a VNC connection to an instance, this gives our users a more lightweight and potentially more responsive option.

💡 Summary

#233 should be reverted when and if that becomes possible.

Motivation and context

canonical/cloud-init#4764 suggests that cloud-init allow for the user to specify Netplan DHCP options. If implemented, this change would mean that we would no longer need to fix the Netplan configuration generated by cloud-init.

See also cisagov/skeleton-packer#300.

Terraform is currently unable to automate the process of associating a VPC in one AWS account with a private DNS zone in a different AWS account. This feature request details the issue.

This process can be done via the AWS CLI, so that is our current manual workaround:

aws route53 create-vpc-association-authorization \
--hosted-zone-id COOLPRIVATEZONEID \
--vpc VPCRegion=us-east-1,VPCId=vpc-MY_ENV_VPC_ID \
--profile cool-sharedservices-provisionaccount

aws route53 associate-vpc-with-hosted-zone \
--hosted-zone-id COOLPRIVATEZONEID \
--vpc VPCRegion=us-east-1,VPCId=vpc-MY_ENV_VPC_ID \
--profile cool-MYENV-provisionaccount

It may be possible to use a Terraform module (such as this one) to accomplish this automatically until Terraform is able to natively perform this action.

🚀 Feature Proposal

A Fed lead requests that the PentestPortal instances be labeled via 1-based indexing instead of 0-based indexing.

Motivation

This will make the PentestPortal instance indexing agree with some internal document they use that lists their customers.

💡 Summary

The Guacamole server in the private subnet of an assessment environment requires permission to send outgoing HTTPS requests in order to:

1. Obtain temporary credentials for its instance role via STS, and
2. Download its TLS certificate from an S3 bucket

It should be possible to do away with the external internet access and instead access STS and S3 via a VPC interface endpoint and a VPC gateway endpoint, respectively.

We should really be using these sorts of endpoints everywhere we can, since it keeps some of our AWS traffic from ever leaving AWS.

Motivation and context

This is a good idea because:

• It will keep some of our AWS traffic from ever leaving AWS, and therefore increase security
• Once we do that we will no longer require certain NAT gateway resources
• By virtue of the previous bullet, it will simplify our OmniGraffle network diagrams for the COOL. (@cisagov/team-ois - does anyone have a link to these diagrams in Google Docs?)

Acceptance criteria

• The STS interface endpoint is created
• The S3 gateway endpoint is created
• Outgoing access to the internet via HTTPS is removed, and the Guacamole server is still able to perform its functions.
• All cloud-init scripts are able to complete successfully.
• The staging environment is verified to function just as it did before.

Add the ability to include one or more Nessus instances (see nessus-packer) in an assessment environment.

💡 Summary

Teamserver instances should fetch the TLS certificate for their specified domain from our S3 certificates bucket (if available) and then appropriately modify the amazon.profile and ocsp.profile files to include a valid https-certificate section.

Motivation and context

In https://github.com/cisagov/cool-system/issues/90, we are working to remove the need to use ServerSetup.sh. One of the acceptance criteria for this is to replicate the functionality of the httpsc2doneright() function from ServerSetup.sh.

Implementation notes

This issue does NOT include figuring out how the certificate is created and put in our S3 bucket. Initially, that will be a manual process that we want to automate in the future.

Acceptance criteria

• When a Teamserver instance is deployed, the amazon.profile and ocsp.profile files each contain a valid https-certificate section.
• Cobalt Strike is able to successfully utilize the updated profiles for HTTPS (probably have to work with an assessment operator to validate this).
• If a certificate for the specified domain is NOT found in S3, a warning is logged and the Cobalt Strike profiles are not touched.

🐛 Summary

In #219 we added a band-aid to replace the authorized_keys file so that a valid SSH public key is configured for access on Windows instances. This should be removed as soon as available Windows AMIs are built with the correct SSH key configured for access.

🐛 Summary

If any items in the email_sending_domains Terraform variable contain uppercase letters, their corresponding certificates will not be found in the S3 bucket because they are always lowercase in the bucket.

To reproduce

Steps to reproduce the behavior:

1. Create a certificate for xyz.com and store it in the certificates bucket
2. Define email_sending_domains = ["XYZ.com"] in a .tfvars` file
3. Run a terraform apply with that .tfvars file

Note that the certificate cannot be pulled via our install-certificates.py cloud-init - it will fail with an error like this:

Error fetching 'mycertificatesbucket/live/XYZ.com/fullchain.pem' from S3: An error occurred (AccessDenied) when calling the GetObject operation: Access Denied
Exiting script!

Expected behavior

The uppercase domain XYZ.com should be converted to lowercase before we attempt to fetch the certificate from the S3 bucket, so that install-certificates.py runs successfully.

CC: @m1j09830

🐛 Summary

What's wrong? Please be specific.

When running the terraform_apply.sh script, errors are generated relating to the sleep command.

Steps to reproduce the behavior:

From a VM Mac already configured to run the COOL Terraform builds, updated to Monterey OS...

1. Run terraform_apply.sh with the given options

• AWS_SHARED_CREDENTIALS_FILE=~/.aws/credentials AWS_DEFAULT_REGION=us-east-1 ./terraform_apply.sh -auto-approve -var-file=envX-production.tfvars

2. The following errors are generated:

Error: local-exec provisioner error
│
│   with null_resource.break_association_with_default_route_table,
│   on transitgateway.tf line 41, in resource "null_resource" "break_association_with_default_route_table":
│   41:   provisioner "local-exec" {
│
│ Error running command 'aws --profile cool-sharedservices-provisionaccount --region us-east-1 ec2
│ disassociate-transit-gateway-route-table --transit-gateway-route-table-id tgw-rtb-05fb5dc2866fe06e5
│ --transit-gateway-attachment-id tgw-attach-0969c580aa7eee9b3 && while aws --profile cool-sharedservices-provisionaccount
│ --region us-east-1 ec2 get-transit-gateway-route-table-associations --transit-gateway-route-table-id
│ tgw-rtb-05fb5dc2866fe06e5 | grep --quiet vpc-0251fecd0fb37eb96; do sleep 5s; done': exit status 1. Output: {
│     "Association": {
│         "TransitGatewayRouteTableId": "tgw-rtb-05fb5dc2866fe06e5",
│         "TransitGatewayAttachmentId": "tgw-attach-0969c580aa7eee9b3",
│         "ResourceId": "vpc-0251fecd0fb37eb96",
│         "ResourceType": "vpc",
│         "State": "disassociating"
│     }
│ }
│ usage: sleep seconds
│ usage: sleep seconds
│ usage: sleep seconds
│ usage: sleep seconds
│ usage: sleep seconds
│ usage: sleep seconds
│ usage: sleep seconds
│ usage: sleep seconds
│ usage: sleep seconds
│ usage: sleep seconds
│ usage: sleep seconds
│ usage: sleep seconds
│ usage: sleep seconds
│ usage: sleep seconds
│ usage: sleep seconds
│ usage: sleep seconds
│ usage: sleep seconds
│ usage: sleep seconds
│ usage: sleep seconds
│ usage: sleep seconds
│ usage: sleep seconds
│ usage: sleep seconds
│ usage: sleep seconds
│ usage: sleep seconds
│ usage: sleep seconds
│ usage: sleep seconds

Expected behavior

Successful completion of the terraform_apply.sh script

Any helpful log output or screenshots

After discussion with @dav3r, it was determined to come from a change in the 'sleep' command resulting from upgrading to the Monterey version of MacOS. The command sleep Xs does not apply anymore and instead sleep X is used. After modifying this command under the directory, the script completed as expected.

Paste the results here:

This is another error that will appear on subsequent attempts:

│ Error: local-exec provisioner error
│
│   with null_resource.break_association_with_default_route_table,
│   on transitgateway.tf line 41, in resource "null_resource" "break_association_with_default_route_table":
│   41:   provisioner "local-exec" {
│
│ Error running command 'aws --profile cool-sharedservices-provisionaccount --region us-east-1 ec2
│ disassociate-transit-gateway-route-table --transit-gateway-route-table-id tgw-rtb-05fb5dc2866fe06e5
│ --transit-gateway-attachment-id tgw-attach-0969c580aa7eee9b3 && while aws --profile cool-sharedservices-provisionaccount
│ --region us-east-1 ec2 get-transit-gateway-route-table-associations --transit-gateway-route-table-id
│ tgw-rtb-05fb5dc2866fe06e5 | grep --quiet vpc-0251fecd0fb37eb96; do sleep 5s; done': exit status 254. Output:
│ An error occurred (InvalidAssociation.NotFound) when calling the DisassociateTransitGatewayRouteTable operation:
│ Association tgw-attach-0969c580aa7eee9b3 does not exists in Transit Gateway Route Table tgw-rtb-05fb5dc2866fe06e5.

Add any screenshots of the problem here.

💡 Summary

In #118 we add an https-certificate block to the Amazon and OCSP Cobalt Strike C2 profiles from rsmudge/Malleable-C2-Profiles. It would be a nice improvement to instead allow the user to provide a list of C2 profiles to which an https-certificate block should be added; alternatively, the user could provide as an input one or more directories containing C2 profiles and we could add an https-certificate block to each *.profile files in those directories.

It also makes sense to allow the user to specify the location of the Java keystore.

Motivation and context

@dav3r mentioned in #118 that this would be a nice improvement to this repository. It would support a more general use case beyond just the Amazon and OCSP Cobalt Strike C2 profiles.

Acceptance criteria

• The Terraform template add-https-certificate-block-to-cs-profiles.tpl.sh takes as an input a list of Cobalt Strike C2 profile files (or, alternatively, a list of directories containing such profiles) to which an https-certificate block should be added.
• add-https-certificate-block-to-cs-profiles.tpl.sh adds an https-certificate block to each of the Cobalt Strike C2 profiles in the previous list item.
• The Terraform template add-https-certificate-block-to-cs-profiles.tpl.sh takes as an input the location of the Java keystore to be created.
• add-https-certificate-block-to-cs-profiles.tpl.sh uses the Java keystore location when populating the https-certificate blocks in the Cobalt Strike C2 profiles.

Currently, only one Guacamole connection to a single Kali instance (kali0) is automatically created. Modify the cloud-init scripts so that a Guacamole connection is created for every instance specified in var.operations_instance_counts.

💡 Summary

This is related to cisagov/skeleton-generic#172. After having run checkov scans locally against cool-assessment-terraform the scan shows: passed checks: 873, Failed checks: 176, Skipped checks: 0.

These checks need to be addressed in a systematic way as we are planning to integrate checkov into the pre-commit linting jobs. Since this ticket is made from just cool-assessment-terraform there might be other checks that fail on other Terraform repositories in https://github.com/cisagov, but this will be a good start.

NOTE: Each failed check might not necessarily need to be fixed. There could be some cases of false flags in which we don't want to adhere to the policies that checkov enforces. In these cases we can setup configurations to bypass these checks but they will need to be approved before bypassing.

Here is the file from a full scan of cool-assessment-terraform: checkov_results.txt

Each check will have a guide linked to it for applying a fix. The checks that failed for cool-assessment-terraform are as follows:

• CKV_AWS_1: Ensure IAM policies that allow full "-" administrative privileges are not created.
• CKV_AWS_8: Ensure all data stored in the EBS is securely encrypted.
• CKV_AWS_23: Ensure every security groups rule has a description.
• CKV_AWS_49: Ensure no IAM policies documents allow "*" as a statement's actions.
• CKV_AWS_88: EC2 instance should not have a public IP.
• CKV_AWS_111: Ensure IAM policies do not allow write access without constraints.
• CKV_AWS_126: Ensure that detailed monitoring is enabled for EC2 instances.
• CKV_AWS_135: Ensure that EC2 is EBS optimized.
• CKV_AWS_184: Ensure resource is encrypted by KMS using a customer managed Key (CMK).
• CKV_AWS_231: Ensure no NACL allow ingress from 0.0.0.0:0 to port 3389.
• CKV_AWS_277: Ensure no security groups allow ingress from 0.0.0.0:0 to port -1.
• CKV_AWS_352: Ensure NACL ingress does not allow all Ports.
• CKV_AWS_356: Ensure no IAM policies documents allow "*" as a statement's resource for restrictable actions.
• CKV2_AWS_11: Ensure VPC flow logging is enabled in all VPCs.
• CKV2_AWS_12: Ensure the default security group of every VPC restricts all traffic.
• CKV2_AWS_19: Ensure that all EIP addresses allocated to a VPC are attached to EC2 instances.
• CKV2_AWS_23: Route53 A Record has Attached Resource.
• CKV2_AWS_38: Ensure DNSSEC signing is enabled for Amazon Route 53 public hosted zones.
• CKV2_AWS_39: Ensure DNS query logging is enabled for Amazon Route 53 hosted zones.
• CKV_TF_1: Ensure Terraform module sources use a commit hash.
• CKV2_GHA_1: Ensure top-level permissions are not set to write-all.

Write the Terraform for the basic assessment VPC.

Make var.private_domain optional, then set up a local variable for private_domain:

• If var.private_domain is not empty: local.private_domain is set to the value of var.private_domain
• If var.private_domain is empty: local.private_domain is set to the value of var.assessment_account_name

🐛 Bug Report

When PentestPortal EC2 instances are spun up, the EFS mount sometimes fails.

To Reproduce

Steps to reproduce the behavior:

• Spin up several PentestPortal EC2 instances
• Check to see if any have an attempted EFS mount that failed

Expected behavior

The EFS mounts should always be available on PentestPortal instances.

We currently use a profile name to make the AWS CLI command in the null_resource resource assume the correct role. This means that the success of this Terraform code relies on the user having a local setup with cisagov/aws-profile-sync using the expected credentials files. It would be better to pluck the desired role ARN out of remote state and simply assume the role for the duration of the command. (Best would be if we could specify a provider for a null_resource, but this is not something currently supported by Terraform and/or the AWS provider.)

As great as this sounds, it is actually a fair amount of work to make it happen. I have found a couple of Terraform modules that do almost what we want, but not quite:

• opetch/terraform-aws-cli-resource
• digitickets/terraform-aws-cli - (also on Terraform Registry)

I think what we actually need is two resources. The first resource would be defined from the Terraform external provider. It would take a few parameters as JSON input and output JSON containing the assumed role credentials. This could be as simple filling in the JSON generated by aws sts assume-role --generate-cli-skeleton, passing it to the AWS CLI via aws sts assume-role --cli-input-json, and returning the output.

The second resource is also defined from the Terraform external provider. It would take the output of the resource described in the previous paragraph as input and then use those credentials to execute the desired AWS CLI commands. It would not export the relevant AWS environment variables, since that clobbers any environment variables we may have set in order to run terraform.

The first resource could be a Terraform module, except that Terraform 0.12 does not allow a module to appear in a depends_on list. (This restriction has been lifted in Terraform 0.13.) The second resource would have to be created where needed, since it would be customized each time it is used.

I will put a comment in the code linking to the issue, and we can revisit this once we have taken the plunge into Terraform 0.13.

Originally posted by @jsf9k in #72 (comment)

💡 Summary

The COOL Operations Team requests modification of the COOL Archival Script (archive-artifact-data-to-bucket.sh) to enable it to run on terraformer instances.

Motivation and context

Recent discussions with the FAST Team Lead (Mike Rocha) indicate that the team will require a lone terraformer instance for future environment builds. The FAST Team Lead (this week 8/28/23) requested zero kali instances be provisioned, by the Ops Team, as they have developed their own IAC from which to deploy the kalis.

To run the archival script, current processes require the Ops Team to manually deploy a kali instance from which to run it in assessment environments without existing kali instances (RTAs and now FASTs). Since FASTs comprise a significant portion of assessments in the COOL, this additional step of provisioning a kali adds time and complexity to the archiving and data transfer process and is inefficient. Modifying the archive-artifact-data-to-bucket.sh script to run on terraformer instances (in addition to kali instances) bypasses the need for provisioning a new instance simply to perform the archival and transfer of data from the COOL to the AE.

Implementation notes

Please provide details for implementation, such as:

Ability to run the archive-artifact-data-to-bucket.sh script on terraformer instances within RTA and FAST environments using the existing runbook (https://confluence.ceil.cyber.dhs.gov/pages/viewpage.action?pageId=124977211).

Acceptance criteria

When the Ops Team can successfuly run the archive-artifact-data-to-bucket.sh script on the terraformer instances within RTA and FAST environments and verifies that the data transferred to the AE. Ideally, acceptance testing would encompass both one FAST and one RTA at a minimum.