Giter VIP home page Giter VIP logo

amazon-ecs-exec-checker's Introduction

Amazon ECS Exec Checker

The check-ecs-exec.sh script allows you to check and validate both your CLI environment and ECS cluster/task are ready for ECS Exec, by calling various AWS APIs on behalf of you. You can learn more about ECS Exec on the containers blog post and the official developer guide.

Prerequisites

The check-ecs-exec.sh requires the following commands.

  • jq
  • AWS CLI v1.19.28/v2.1.30 or later

Usage

$ ./check-ecs-exec.sh <YOUR_ECS_CLUSTER_NAME> <YOUR_ECS_TASK_ID>

Example 1 - Run without cloning Git repo

The check-ecs-exec.sh will use your default AWS CLI profile and the AWS region.

$ bash <( curl -Ls https://raw.githubusercontent.com/aws-containers/amazon-ecs-exec-checker/main/check-ecs-exec.sh ) <YOUR_ECS_CLUSTER_NAME> <YOUR_ECS_TASK_ID>

Example 2 - With AWS_* variables

The check-ecs-exec.sh will use the myprofile AWS CLI profile and us-west-2 AWS region.

$ export AWS_PROFILE=myprofile
$ export AWS_REGION=us-west-2

$ bash <( curl -Ls https://raw.githubusercontent.com/aws-containers/amazon-ecs-exec-checker/main/check-ecs-exec.sh ) <YOUR_ECS_CLUSTER_NAME> <YOUR_ECS_TASK_ID>

Example 3 - With MFA

The check-ecs-exec.sh automatically detects your MFA configuration for the AWS CLI. But you can also explicitly specify which MFA device to use by setting the ARN of the MFA device to AWS_MFA_SERIAL environment variable.

Example 4 - Switch AWS CLI binaries

If you have multiple AWS CLI installations in your environment, both AWS CLI v1 and v2 for example, you can choose which AWS CLI binary to use by passing the AWS_CLI_BIN env variable.

$ AWS_CLI_BIN=aws-v1 ./check-ecs-exec.sh <YOUR_ECS_CLUSTER_NAME> <YOUR_ECS_TASK_ID>

Checks

The check-ecs-exec.sh shows the results with three text colors, ๐ŸŸข(Green), ๐ŸŸก(Yellow), and ๐Ÿ”ด(Red). Each color tells how you'll handle the results.

  1. ๐ŸŸข(Green) - The configuration or the status is okay.
  2. ๐ŸŸก(Yellow) - The configuration or the status should or would be recommended to fix, but you can use ECS Exec without fixing them.
  3. ๐Ÿ”ด(Red) - You need to fix those results before using ECS Exec.

In the following screenshot for instance, we need to install the Session Manager plugin and give required permissions to the task role at least, but we can ignore the audit-logging configuration.

Note that it shows "SSM PrivateLink" at the bottom as a ๐ŸŸก(yellow) result, but it can be a ๐Ÿ”ด(red) result if your ECS task doesn't have proper outbound internet connectivity. In this case, you will need to configure an SSM PrivateLink in your VPC.

example-result

Reference - How to handle ๐Ÿ”ด(Red) and ๐ŸŸก(Yellow) items

  1. ๐Ÿ”ด Pre-flight check failed: jq command is missing
    Install the jq command. See the official documentation for the details and how to install.

  2. ๐Ÿ”ด Pre-flight check failed: aws command is missing
    Install the latest AWS CLI. See the official documentation for the AWS CLI v2 or the official documentation for the AWS CLI v1 for the details and how to install.

  3. ๐Ÿ”ด Pre-flight check failed: ECS Exec requires the AWS CLI v1.19.28/v2.1.30 or later
    Upgrade to the latest AWS CLI. See the official documentation for the AWS CLI v2 or the official documentation for the AWS CLI v1 for the details and how to upgrade.

  4. ๐Ÿ”ด Session Manager Plugin | Missing
    Install the Session Manager plugin. See the official documentation for the details and how to install.

  5. ๐ŸŸก Cluster Configuration | Audit Logging Not Configured / Disabled This check item won't block you to use ECS Exec, but we recommend you to enable logging and auditing for your ECS cluster from the security perspective. See the official documentation for the details and how to enable them.

  6. ๐Ÿ”ด Can I ExecuteCommand? | ecs:ExecuteCommand: implicitDeny
    The IAM user/role you used for the check-ecs-exec.sh are not allowed to use the ecs:ExecuteCommand API. See the "Using IAM policies to limit access to ECS Exec section in the official documentation to add the required permission to the IAM user/role.
    Note that the Condition element of the IAM policy is not currently supported to evaluate by check-ecs-exec.sh.

  7. ๐Ÿ”ด Can I ExecuteCommand? | kms:GenerateDataKey: implicitDeny
    The IAM user/role you used for the check-ecs-exec.sh are not allowed to use the kms:GenerateDataKey API with the given KMS Key ID which you're using for the logging and auditing configuration for ECS exec. See the "IAM permissions required for encryption using your own KMS customer master key (CMK) section under the "Logging and Auditing using ECS Exec" section in the official documentation to add the required permission to the IAM user/role.
    Note that the Condition element of the IAM policy is not currently supported to evaluate by check-ecs-exec.sh.

  8. ๐ŸŸก Can I ExecuteCommand? | ssm:StartSession denied?: allowed
    The result means your IAM user/role is allowed to do ssm:StartSession action to the ECS task. This check item won't block you to use ECS Exec, but we recommend you to limit access to the ssm:StartSession API, from the security and the principle of least privilege perspectives. See the ECS official documentation for further details.
    Note that the Condition element of the IAM policy is not currently supported to evaluate by check-ecs-exec.sh.

  9. ๐Ÿ”ด Task Status | DEACTIVATING or STOPPING or DEPROVISIONING or STOPPED
    Your ECS task has already stopped, or is shutting down. ECS Exec requires the task is in the RUNNING state. Restart your ECS task if it's a standalone task, or wait for another task if it's a part of an ECS service. See also the Task lifecycle in the ECS documentation for more details.

  10. ๐ŸŸก Task Status | PROVISIONING or ACTIVATING or PENDING
    Your ECS task is in the middle of its starting process. ECS Exec requires the task is in the RUNNING state. Wait few more seconds for the task to be ready. See also the Task lifecycle in the ECS documentation for more details.

  11. ๐Ÿ”ด Platform Version | 1.3.0 (Required: >= 1.4.0)
    On AWS Fargate, ECS Exec requires the Platform version 1.4.0 or higher (Linux) or 1.0.0 (Windows). If your ECS task is part of an ECS service, then you can update the platform version by specifying the PlatformVersion parameter for the UpdateService API. If your ECS task is a standalone task, then you need to re-run the ECS task with the PlatformVersion parameter specified for the RunTask API. See also the migration guide from the previous PVs.

  12. ๐Ÿ”ด ECS Agent Version | x.y.z (Required: >= 1.50.2)
    You need to update the version of the ECS Container Agent for your EC2 instance where your ECS task runs. See the ECS official documentation for the details and how to update.

  13. ๐Ÿ”ด Exec Enabled for Task | NO
    You need to enable the ECS Exec feature for your ECS service or your ECS standalone task. If your ECS task is part of an ECS service, then you can update the ECS by specifying the EnableExecuteCommand parameter for the UpdateService API. If your ECS task is a standalone task, then you need to re-run the ECS task with the EnableExecuteCommand parameter specified for the RunTask API.

  14. ๐Ÿ”ด Managed Agent Status | STOPPED (Reason: stopped-reason-here)
    The managed agent for a container in your Task has stopped for some reasons. If you see this error again and again even after re-running your ECS task, then make sure you have other results from check-ecs-exec.sh are all green.

  15. ๐ŸŸก Init Process Enabled | Disabled
    This check item won't block you to use ECS Exec, but we recommend you to add the initProcessEnabled flag to your ECS task definition for each container to avoid having orphaned and zombie processes. See the "Considerations for using ECS Exec" in the ECS official documentation for more details.

  16. ๐Ÿ”ด Read-Only Root Filesystem | ReadOnly
    ECS Exec uses the SSM agent as its managed agent, and the agents requires that the container file system is able to be written in order to create the required directories and files. Therefore, you need to set the readonlyRootFilesystem flag as false in your task definition to exec into the container using ECS Exec. See the "Considerations for using ECS Exec" in the ECS official documentation for more details.

  17. ๐Ÿ”ด EC2 or Task Role | Not Configured" or {serviceName}:{ActionName}: implicitDeny
    Your ECS task needs a task role or an instance role of the underlying EC2 instance with some permissions for using SSM Session Manager at least. See the IAM permissions required for ECS Exec section and the Enabling logging and auditing in your tasks and services section in the official documentation for the details.
    Note that the Condition element of the IAM policy is not currently supported to evaluate by check-ecs-exec.sh.

  18. ๐ŸŸก SSM PrivateLink "com.amazonaws.(region).ssmmessages" not found
    The check-ecs-exec.sh found one or more VPC endpoints configured in the VPC for your task, so you may want to add an additional SSM PrivateLink for your VPC. Make sure your ECS task has proper outbound internet connectivity, and if it doesn't, then you need to configure an additional SSM PrivateLink for your VPC.

  19. ๐Ÿ”ด VPC Endpoints | CHECK FAILED
    The check-ecs-exec.sh doesn't support checking this item for shared VPC subnets using AWS Resouce Access Manager (AWS RAM). In short, this may not an issue to use ECS Exec if your ECS task VPC doesn't have any VPC endpoint and the task has proper outbound internet connectivity. Make sure to consult your administrator with the official ECS Exec documentation](https://docs.aws.amazon.com/AmazonECS/latest/developerguide/ecs-exec.html#ecs-exec-considerations) to find if your VPC need to have an additional VPC endpoint.

  20. ๐ŸŸก Environment Variables : defined
    SSM uses the AWS SDK which uses the default chain when determining authentication. This means if AWS_ACCESS_KEY, AWS_ACCESS_KEY_ID, and AWS_SECRET_ACCESS_KEY are defined in the environment variables and the permissions there do not provide the required permissions for SSM to work, then the execute-command will fail. It is recomended not to define these environment variables.

Security

See CONTRIBUTING for more information.

License

Licensed under the MIT-0 License. See the LICENSE file.

amazon-ecs-exec-checker's People

Contributors

adamjkeller avatar anorlondo448 avatar erjanmx avatar fujiwara avatar grimm26 avatar hi1280 avatar inbarrose avatar marek-knappe avatar masayoshi644 avatar nathanpeck avatar rectalogic avatar sindrig avatar sugikeitter avatar thelateperseus avatar toricls avatar ttres avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

amazon-ecs-exec-checker's Issues

Platform Version Check Incorrect For Windows Fargate

When running the script against a Windows Fargate task, I see:
Platform Version | 1.0.0 (Required: >= 1.4.0)

This is incorrect for Windows containers per the documentation: "If you're using AWS Fargate, you must use platform version 1.4.0 or higher (Linux) or 1.0.0 (Windows)."

Everything else seems to be working correctly, although I haven't tried initProcessEnabled (I'm unclear if it uses the now incorrectly named "linuxParameters" configuration).

script fails on policy simulations using KMS Id

policy simluations involving the KMS key supply its Id as an arg to --resource-arns and this throws an error:

An error occurred (InvalidInput) when calling the SimulatePrincipalPolicy operation: ResourceNames are not in a valid ARN format: 14a050d4-021e-4efe-84bb-0871bf6d0fda.

The arn should be looked up and used instead. #34 fixes this

Support for MFA

I'm getting error while the tool tries to DescribeTask but not because I'm missing the permission, I'm just missing the MFA requirement of the Role.

ECS execute-command failed due to an internal error.

Hi there, I'm trying to run execute-command to open an interactive shell against my ECS Fargate task. I'm using this checker to validate my configuration:

$ bash <( curl -Ls https://raw.githubusercontent.com/aws-containers/amazon-ecs-exec-checker/main/check-ecs-exec.sh ) clusterName cf41c924968e426c9be535f3f47545be
-------------------------------------------------------------
Prerequisites for check-ecs-exec.sh v0.7
-------------------------------------------------------------
  jq      | OK (/usr/bin/jq)
  AWS CLI | OK (/usr/local/bin/aws)

-------------------------------------------------------------
Prerequisites for the AWS CLI to use ECS Exec
-------------------------------------------------------------
  AWS CLI Version        | OK (aws-cli/2.4.0 Python/3.8.8 Linux/5.11.0-40-generic exe/x86_64.ubuntu.20 prompt/off)
  Session Manager Plugin | OK (1.2.205.0)

-------------------------------------------------------------
Checks on ECS task and other resources
-------------------------------------------------------------
Region : eu-south-1
Cluster: clusterName
Task   : cf41c924968e426c9be535f3f47545be
-------------------------------------------------------------
  Cluster Configuration  | Audit Logging Not Configured
  Can I ExecuteCommand?  | arn:aws:iam::ACCOUNT_ID:role/ADMIN_ROLE_NAME
     ecs:ExecuteCommand: allowed
     ssm:StartSession denied?: allowed
  Task Status            | RUNNING
  Launch Type            | Fargate
  Platform Version       | 1.4.0
  Exec Enabled for Task  | OK
  Container-Level Checks | 
    ----------
      Managed Agent Status
    ----------
         1. RUNNING for "taskName"
    ----------
      Init Process Enabled (taskName:1)
    ----------
         1. Enabled - "taskName"
    ----------
      Read-Only Root Filesystem (taskName:1)
    ----------
         1. Disabled - "taskName"
  Task Role Permissions  | arn:aws:iam::ACCOUNT_ID:role/taskName-ecs-task
     ssmmessages:CreateControlChannel: allowed
     ssmmessages:CreateDataChannel: allowed
     ssmmessages:OpenControlChannel: allowed
     ssmmessages:OpenDataChannel: allowed
  VPC Endpoints          | 
    Found existing endpoints for vpc-ID:
      - com.amazonaws.eu-south-1.ssm
      - com.amazonaws.eu-south-1.ec2messages
      - com.amazonaws.eu-south-1.ssmmessages

However, I'm getting TargetNotConnectedException. I've also opened an issue here.

Am I missing something..?

"aws ecs execute-command" reports timeout despite all-green report from the checker

Adding a summary here by @toricls for people who came here through search engines.

Issue description

Cannot exec into container with an error Encountered error while initiating handshake. Handshake timed out. Please ensure that you have the latest version of the session manager plugin. under the following conditions, despite the checker script reports all green.

Steps to reproduce the issue

This issue happens when using 1) AWS SSO temporary credentials and 2) KMS encryption enabled for ECS Exec. Here is the steps to reproduce the error.

  1. Configure ECS cluster's ExecuteCommandConfiguration with any KMS key ID.
  2. Run aws ecs execute-command with AWS credentials obtained by aws sso login command

Root cause

The session-manager-plugin doesn't support AWS SSO temporary credentials today (See aws/session-manager-plugin#4)

Workaround

See the comment below.

UPDATE Aug. 23rd, 2021

The Session Manager plugin now supports AWS SSO temporary credentials with the version 1.2.245.0 or later.


I'm pretty stumped by the failure mode we're seeing, and I imagine so is the exec-checker, as it reports an all-green status (with a warning for sts:StartSession, which we will tune in due time):

$ env AWS_PROFILE=mz-core-admin ./check-ecs-exec.sh apps-cluster-255a2cf d63f19bb0b434ae489612b6d9215e5f8
-------------------------------------------------------------
Prerequisites for check-ecs-exec.sh v0.7
-------------------------------------------------------------
  jq      | OK (/run/current-system/sw/bin/jq)
  AWS CLI | OK (/run/current-system/sw/bin/aws)

-------------------------------------------------------------
Prerequisites for the AWS CLI to use ECS Exec
-------------------------------------------------------------
  AWS CLI Version        | OK (aws-cli/2.2.14 Python/3.9.5 Darwin/20.5.0 source/arm64 prompt/off)
  Session Manager Plugin | OK (1.2.234.0)

-------------------------------------------------------------
Checks on ECS task and other resources
-------------------------------------------------------------
Region : us-east-1
Cluster: apps-cluster-255a2cf
Task   : d63f19bb0b434ae489612b6d9215e5f8
-------------------------------------------------------------
  Cluster Configuration  |
     KMS Key       : arn:aws:kms:us-east-1:834237029485:key/5513178e-9387-4969-be9a-67dee8883899
     Audit Logging : OVERRIDE
     S3 Bucket Name: Not Configured
     CW Log Group  : /ecs/apps-cluster, Encryption Enabled: true
  Can I ExecuteCommand?  | arn:aws:iam::834237029485:role/aws-reserved/sso.amazonaws.com/AWSReservedSSO_Administrator_3c1b1a3b260fa528
     ecs:ExecuteCommand: allowed
     kms:GenerateDataKey: allowed
     ssm:StartSession denied?: allowed
  Task Status            | RUNNING
  Launch Type            | Fargate
  Platform Version       | 1.4.0
  Exec Enabled for Task  | OK
  Container-Level Checks |
    ----------
      Managed Agent Status
    ----------
         1. RUNNING for "bors"
    ----------
      Init Process Enabled (apps-bors:6)
    ----------
         1. Enabled - "bors"
    ----------
      Read-Only Root Filesystem (apps-bors:6)
    ----------
         1. Disabled - "bors"
  Task Role Permissions  | arn:aws:iam::834237029485:role/bors-task-b7530f5
     ssmmessages:CreateControlChannel: allowed
     ssmmessages:CreateDataChannel: allowed
     ssmmessages:OpenControlChannel: allowed
     ssmmessages:OpenDataChannel: allowed
     -----
     kms:Decrypt: allowed
     -----
     logs:DescribeLogGroups: allowed
     logs:CreateLogStream: allowed
     logs:DescribeLogStreams: allowed
     logs:PutLogEvents: allowed
  VPC Endpoints          | SKIPPED (vpc-0693f607384d76722 - No additional VPC endpoints required)

So it sounds like our task/cluster gets a clean bill of health from the checker, but trying to execute a command fails with a timeout:

$ aws ecs execute-command --profile mz-core-admin --region us-east-1 --cluster apps-cluster-255a2cf  --task d63f19bb0b434ae489612b6d9215e5f8 --container bors --command '/bin/sh' --interactive

The Session Manager plugin was installed successfully. Use the AWS CLI to start a session.


Starting session with SessionId: ecs-execute-command-02212f04c8f7d94e3
----------ERROR-------
Encountered error while initiating handshake. Handshake timed out. Please ensure that you have the latest version of the session manager plugin.

I'm sure there's something wrong in our configuration of ECS exec, but if there is, I hope the checker can be extended to pick up whatever error is lurking in this config.

need to check the environment variables for AWS_ACCESS_KEY and AWS_SECRET_ACCESS_KEY

We had an annoying issue where the checker was showing everything was configured okay. and we could connect to some of our containers but not all of them. we had to get AWS support involved.

the error we got was

An error occurred (TargetNotConnectedException) when calling the ExecuteCommand operation: The execute command failed due to an internal error. Try again later.

it turns out, that the SSM uses the AWS SDK which by default uses the AWS_ACCESS_KEY and AWS_SECRET_ACCESS_KEY environment variables if they are present. we had those configured in our container but with very limited access, and this caused the SSM to fail, and so we couldn't do execute command.

I suggest to add a check when we "describe task" that will see if there are AWS_ACCESS_KEY and AWS_SECRET_ACCESS_KEY defined in a container, that this could cause an issue. and ideally it should not be used, but if it is used then it should have SSM permissions.

ref: https://docs.aws.amazon.com/sdk-for-java/v1/developer-guide/credentials.html#credentials-default

All critical checkpoint passed, but still get error to execute the command.

Got everything passed but still cannot run the command due to error, is there any other checkpoint need to be added to the script?

% aws ecs execute-command
--region $AWS_REGION
--cluster rd-xxx-xxxx-ecs-cluster
--task 9496e2baead84a7d8116e989c2bca473
--container xxx-xxxxx
--command /bin/bash
--interactive
--profile dev

The Session Manager plugin was installed successfully. Use the AWS CLI to start a session.
An error occurred (TargetNotConnectedException) when calling the ExecuteCommand operation: The execute command failed due to an internal error. Try again later.

The script result is as below

% bash <( curl -Ls https://raw.githubusercontent.com/aws-containers/amazon-ecs-exec-checker/main/check-ecs-exec.sh ) rd-dlas-op1service-ecs-cluster 9496e2baead84a7d9116e903c2bca473

Prerequisites for check-ecs-exec.sh v0.7

jq | OK (/usr/local/bin/jq)
AWS CLI | OK (/usr/local/bin/aws)


Prerequisites for the AWS CLI to use ECS Exec

AWS CLI Version | OK (aws-cli/2.2.32 Python/3.8.8 Darwin/20.6.0 exe/x86_64 prompt/off)
Session Manager Plugin | OK (1.2.245.0)


Checks on ECS task and other resources

Region : us-east-1
Cluster: rd-xxx-xxxx-ecs-cluster
Task : 9496e2baead84a7d8916e903c2bca473

Cluster Configuration | Audit Logging Not Configured
Can I ExecuteCommand? | arn:aws:iam::0000000:role/US-_AWS000000_Developer_2
ecs:ExecuteCommand: allowed
ssm:StartSession denied?: allowed
Task Status | RUNNING
Launch Type | Fargate
Platform Version | 1.4.0
Exec Enabled for Task | OK
Container-Level Checks |
----------
Managed Agent Status
----------
1. STOPPED (Reason: null) for "xxx-xxxproxy" - LastStartedAt: null
2. RUNNING for "xxx-xxxservice"
----------
Init Process Enabled (rd-dlas-op1service:72)
----------
1. Enabled - "xxx-xxxservice"
2. Enabled - "xxx-xxxproxy"
----------
Read-Only Root Filesystem (rd-dlas-op1service:72)
----------
1. Disabled - "xxx-xxxservice"
2. Disabled - "xxx-xxxproxy"
Task Role Permissions | arn:aws:iam::xxxxxxxx:role/ADD-rd-xxx-xxx-ECSTask
ssmmessages:CreateControlChannel: allowed
ssmmessages:CreateDataChannel: allowed
ssmmessages:OpenControlChannel: allowed
ssmmessages:OpenDataChannel: allowed
VPC Endpoints |
Found existing endpoints for vpc-039e987a:
- com.amazonaws.us-east-1.s3
- com.amazonaws.us-east-1.execute-api
- com.amazonaws.us-east-1.ecr.api
- com.amazonaws.us-east-1.ecr.dkr
- com.amazonaws.us-east-1.cloudformation
- com.amazonaws.us-east-1.ecs
- com.amazonaws.us-east-1.ecs-agent
- com.amazonaws.us-east-1.ecs-telemetry
- com.amazonaws.us-east-1.logs
- com.amazonaws.us-east-1.sqs
- com.amazonaws.vpce.us-east-1.vpce-svc-xxxx
- com.amazonaws.vpce.us-east-1.vpce-svc-xxxxx
- com.amazonaws.us-east-1.dynamodb
- com.amazonaws.us-east-1.monitoring
- com.amazonaws.us-east-1.ssm
- com.amazonaws.us-east-1.ec2
- com.amazonaws.us-east-1.ec2messages
- com.amazonaws.us-east-1.ssm
- com.amazonaws.us-east-1.ec2messages
- com.amazonaws.us-east-1.ssmmessages
- com.amazonaws.us-east-1.kms
- com.amazonaws.us-east-1.sns
- com.amazonaws.us-east-1.glue
- com.amazonaws.us-east-1.execute-api
- com.amazonaws.us-east-1.secretsmanager
- com.amazonaws.vpce.us-east-1.vpce-svc-xxxxxx
- com.amazonaws.us-east-1.kinesis-firehose
- com.amazonaws.us-east-1.textract
- com.amazonaws.us-east-1.states
- com.amazonaws.us-east-1.autoscaling
- com.amazonaws.us-east-1.email-smtp

Task Role Permissions check fails if policies use aws:RequestedRegion key

There are probably lots of other keys that this applies to, but this key in particular is probably very common in Permissions Boundaries, and unlike lots of other keys, we always know what value it will be set to.

To reproduce:

  • Create a, ECS Task Role that allows the 4 required ssmmessages: permissions
  • Create a Permissions Boundary Policy (attached to that ECS Task Role) that includes something like the block below (i.e. requiring the aws:RequestedRegion key to match a given region, and allow anything else, or at least the above 4 permissions)
  • Run the checker script
{
  "Condition": {
    "StringEquals": {
      "aws:RequestedRegion": [
        "us-east-1"
      ]
    }
  },
  "Resource": "*",
  "Effect": "Allow",
  "Action": "*"
}

It will say that it the permissions failed due to "implicitDeny".

You can validate this using the CLI's simulate-principal-policy directly:

# This will fail
aws iam simulate-principal-policy --policy-source-arn <policy-arn> --action-names "ssmmessages:CreateControlChannel"
{
  "EvaluationResults": [
    ...
    "EvalDecision": "implicitDeny",
    ...
    "PermissionsBoundaryDecisionDetail": {
      "AllowedByPermissionsBoundary": false
    }
  ]
}

# This will succeed:
aws iam simulate-principal-policy --policy-source-arn <policy-arn> --action-names "ssmmessages:CreateControlChannel" --context-entries ContextKeyName=aws:RequestedRegion,ContextKeyValues=us-east-1,ContextKeyType=string

Could you set other context keys and/or allow a user-specified way of specifying other context keys? Sure. But this one is "easy" and common enough, that I think this script should always set it.

Checker IAM evaluation is not correct

IAM evaluation relies on aws iam simulate-principal-policy but I have noticed that this can give surprising/false results.

For instance I'm using a role with AdministratorAccess managed policy attached to it, and the simulator returns implicitDeny!!

> aws iam simulate-principal-policy --policy-source-arn arn:aws:iam::<redacted>:role/<redacted> --action-names ecs:ExecuteCommand --resource-arns arn:aws:ecs:eu-west-3:<redacted>:task/<redacted> --profile <redacted>
{
    "EvaluationResults": [
        {
            "EvalActionName": "ecs:ExecuteCommand",
            "EvalResourceName": "arn:aws:ecs:eu-west-3:<redacted>:task/<redacted>",
            "EvalDecision": "implicitDeny",
            "MatchedStatements": [],
            "MissingContextValues": [],
            "OrganizationsDecisionDetail": {
                "AllowedByOrganizations": false
            }
        }
    ]
}

Check "ssm:start-session" is denied in principal's IAM policy

What we're going to add

We're going to add an additional check in the Can I ExecuteCommand? check, to make sure the ssm:StartSession action is explicitly denied for the caller identity.

Currently the Can I ExecuteCommand? check only checks the following items:

  • is ecs:ExecuteCommand allowed?
  • is kms:GenerateDataKey allowed? (Only when the executeCommandConfiguration has configured with a KMS key ID)

but it also should check on:

  • is ssm:StartSession denied?

Background

Today we can start SSM sessions on a container in task outside of ECS Exec, and this could potentially result in the sessions not being logged.

As per we described in the "Limiting access to the Start Session action" section in the ECS documentation, the check-ecs-exec.sh should check and show if the caller principal is explicitly denied to use ssm:StartSession by its IAM policy from the principle of least privilege perspective.

Item type

The item type (yellow or red) will be determined later.

Note: Considering the check-ecs-exec.sh's original goal to show if users can use ECS exec, this check item would a yellow item as a warning. But it also should be a red item from the security perspective.

VPC Endpoints shown as "CHECK FAILED" in a RAM-shared VPC/subnets context.. but actually unnecessary

I'm facing the following result:

VPC Endpoints          | CHECK FAILED
     Amazon ECS Exec Checker doesn't support VPC endpoint validation for AWS RAM shared VPC/subnets.

I don't have VPC Endpoints and ECS Exec still works like a charm since I put my tasks in AWSVPC mode and my VPC has a NAT Gateway (and my subnet route table route 0.0.0.0/0 to this NAT Gateway).

Maybe additional checks can be performed and/or error message mention this alternative to VPC Endpoints?

Script passes but error with CloudWatch encrpytion

We ran the script and everything came out green or yellow. (SSM PrivateLink was yellow). When we run the aws ecs execute-command we get an error message

SessionId: xxx: We couldn't start the session because encryption is not set up on the selected CloudWatch Logs log group. Either encrypt the log group or choose an option to enable logging without encryption.

Does the script check for whether the log group is encrypted?

Support for Profile?

While trying to use the tool, I get an error that suggests that I use aws configure. However I have several accounts/credentials and none of them are default to my environment. Using profile would be ideal.

Feature: Add check for SSM Session Manager connect to EC2

I recently ran into a case where I had all green checks using this script, but still could not docker exec into my container because the EC2 instance it was running on did not have egress connectivity for the SSM agent. Thus, I couldn't use session manager to log in to it. There should be more checks added to the EC2 case to see if it is running an AMI compatible for SSM and if it is, attempt to connect to it using the session manager plugin.

Non STS session explicitDeny overlaps with STS Allow

I have Deny for all permissions if its not a STS session with MFA, except getSessionToken, listMFADevices, setMFADevice and few more.

{ "Sid": "DenyAllExceptListedIfNoMFA", "Effect": "Deny", "NotAction": [ "iam:CreateVirtualMFADevice", "iam:EnableMFADevice", "iam:GetUser", "iam:ListMFADevices", "iam:ListVirtualMFADevices", "iam:ResyncMFADevice", "sts:GetSessionToken" ], "Resource": "*", "Condition": { "BoolIfExists": { "aws:MultiFactorAuthPresent": "false" } } }
This checker said I have an explicitDeny on ecs:ExecuteCommand and ssm:StartSession even with a valid STS session and permission, because of explicitDeny without MFA Session. So, I had to add the two permissions to my list

{ "Sid": "DenyAllExceptListedIfNoMFA", "Effect": "Deny", "NotAction": [ "iam:CreateVirtualMFADevice", "iam:EnableMFADevice", "iam:GetUser", "iam:ListMFADevices", "iam:ListVirtualMFADevices", "iam:ResyncMFADevice", "sts:GetSessionToken", "ecs:ExecuteCommand", "ssm:StartSession" ], "Resource": "*", "Condition": { "BoolIfExists": { "aws:MultiFactorAuthPresent": "false" } } }

which means, I can execCommand on my cluster without a MFA STS session.

Version number printed at wrong place with AWS CLI v1

Since the AWS CLI v1 outputs its version number to the standard error, the version number value is not stored in the variable.

Behavior

...
-------------------------------------------------------------
Prerequisites for the AWS CLI to use ECS Exec
-------------------------------------------------------------
aws-cli/1.19.32 Python/2.7.16 Darwin/20.3.0 botocore/1.20.32
  AWS CLI Version        | OK ()
...

Expected

...
-------------------------------------------------------------
Prerequisites for the AWS CLI to use ECS Exec
-------------------------------------------------------------
  AWS CLI Version        | OK (aws-cli/1.19.32 Python/2.7.16 Darwin/20.3.0 botocore/1.20.32)
...

Issue ?

ubuntu@DESKTOP-I7DBFAI:~$ aws ecs describe-tasks --profile=dma --region=eu-west-1 --cluster=mps-corporate-web --tasks=2fb8c764dcaf4b819f51d2c2b6fbcf70
{
    "tasks": [
        {
            "attachments": [
                {
                    "id": "af9341d2-fa6c-4af2-9964-1e40007caeed",
                    "type": "ElasticNetworkInterface",
                    "status": "ATTACHED",
                    "details": [
                        {
                            "name": "subnetId",
                            "value": "subnet-0a7ce6575e6d811ff"
                        },
... etc

so that command is working, but:

ubuntu@DESKTOP-I7DBFAI:~$ export AWS_PROFILE=dma
ubuntu@DESKTOP-I7DBFAI:~$ export AWS_REGION=eu-west-1
ubuntu@DESKTOP-I7DBFAI:~$ check-ecs-exec.sh mps-corporate-web 2fb8c764dcaf4b819f51d2c2b6fbcf70
-------------------------------------------------------------
Prerequisites for check-ecs-exec.sh v0.7
-------------------------------------------------------------
  jq      | OK (/usr/bin/jq)
  AWS CLI | OK (/usr/local/bin/aws)

-------------------------------------------------------------
Prerequisites for the AWS CLI to use ECS Exec
-------------------------------------------------------------

An error occurred (ClusterNotFoundException) when calling the DescribeTasks operation: Cluster not found.
ubuntu@DESKTOP-I7DBFAI:~$

Am I doing something wrong?

We use several aws accounts: the subaccount (in which the cluster is) is accessed with a role as can be seen in the aws config

ubuntu@DESKTOP-I7DBFAI:~/.aws$ cat config
[default]
[profile dma]
role_arn = arn:aws:iam::059416150415:role/DMAAccessRole
source_profile = default
ubuntu@DESKTOP-I7DBFAI:~/.aws$

check-ecs-exec.sh works if I test a cluster in our main aws account in the same region - basically ignoring export AWS_PROFILE=dma.

If BSD sed is used, the check with ap-northeast-1 fails.

The following error occurred when running ECS Exec Checker on Mac.

Could not connect to the endpoint URL: "https://sts.ap-notheast-1.amazonaws.com/"

Verified that the region setting for profile is correct.

$ aws configure get region
ap-northeast-1

When using BSD sed, it seems that โ€œrโ€ is removed from ap-northeast-1 in the following places.

REGION=$(${AWS_CLI_BIN} configure get region | sed -e 's/\r//g' || echo "")

โ€œrโ€ was not removed when gnu-sed was used.

$ aws configure get region | sed -e "s/\r//g"
ap-notheast-1

$ brew install gnu-sed

$ aws configure get region | gsed -e "s/\r//g"
ap-northeast-1

I think it would be better to describe it in the prerequisites.

jq: error (at <stdin>:173): Cannot iterate over null (null)

When running the checker, I get the following jq error:

-------------------------------------------------------------
Prerequisites for check-ecs-exec.sh v0.7
-------------------------------------------------------------
  jq      | OK (/opt/homebrew/bin/jq)
  AWS CLI | OK (/opt/homebrew/bin/aws)

-------------------------------------------------------------
Prerequisites for the AWS CLI to use ECS Exec
-------------------------------------------------------------
  AWS CLI Version        | OK (aws-cli/2.15.17 Python/3.11.7 Darwin/23.0.0 source/arm64 prompt/off)
  Session Manager Plugin | OK (1.2.553.0)

-------------------------------------------------------------
Checks on ECS task and other resources
-------------------------------------------------------------
Region : eu-central-1
Cluster: REDACTED
Task   : REDACTED
-------------------------------------------------------------
  Cluster Configuration  | Audit Logging Not Configured
  Can I ExecuteCommand?  | arn:aws:iam::xxxxxxxxxxxxx:user/[email protected]
     ecs:ExecuteCommand: allowed
     ssm:StartSession denied?: allowed
  Task Status            | RUNNING
  Launch Type            | Fargate
  Platform Version       | 1.4.0
  Exec Enabled for Task  | OK
  Container-Level Checks |
    ----------
      Managed Agent Status
    ----------
jq: error (at <stdin>:173): Cannot iterate over null (null)

I found out that not all containers have a managedAgent property. I was able to fix it by changing line 422 to

agentsStatus=$(echo "${describedTaskJson}" | jq -r ".tasks[0].containers[] | (.managedAgents // [])[].lastStatus // \"FallbackValue\"")

This is of course only a quick fix. The underlying issue is that we have AWS GuardDuty enabled. GuardDuty injects a container into each task but those GuardDuty containers do not have a managedAgent.

This is how the container comes back after describing it:

{
  "containerArn": "arn:aws:ecs:eu-central-1:xxxxxxxxx:container/xxx-cluster-xxx/xxxxx/9efcbebb-1204-4212-84fa-1471bcadbf8c",
  "taskArn": "arn:aws:ecs:eu-central-1:xxxxxxxxx:task/xxxx-cluster-xxx/xxx",
  "name": "aws-guardduty-agent-GAhgQ",
  "imageDigest": "sha256:9f8cd438fb66f62d09bfc641286439f7ed5177988a314a6021ef4ff880642e68",
  "runtimeId": "c9103216b805432497d68c0190237d44-4043820195",
  "lastStatus": "RUNNING",
  "networkBindings": [
    
  ],
  "networkInterfaces": [
    {
      "attachmentId": "c88ab07b-c263-419c-ba64-adea5c51eb07",
      "privateIpv4Address": "10.10.4.210"
    }
  ],
  "healthStatus": "UNKNOWN"
},

Check task status as Red item

At this moment the script doesn't check the task lifecycle status, more specifically the task status is required to be "RUNNING" for exec into containers.

We should check it as a "Red" item because users obviously cannot exec into any stopped ECS tasks even if all checks are green.

Include "managedAgents.lastStartedAt/name" field value in the output

At this moment we only use the lastStatus and the reason fields under the managedAgents[] field in the output here.

Given that the managed SSM agent could restart eventually, the lastStartedAt field and the name field (which are just ignored with the current implementation) are worth to be included in the output to make troubleshooting a little easier.

Script fails with IAM role name with nested multiple paths

This issue is a follow-up for the PR #10 - "Fix script execution when used by assumed roles from SSO".

The PR above implemented a support for assumed IAM roles without slash in the role name part (e.g. arn:aws:sts::123456789012:assumed-role/Role-Name-Without-Slash/JaneDoe), the script still fails with role names with slash(es) (e.g. arn:aws:sts::123456789012:assumed-role/This-Role-Has/Multiple-Paths/In-Role-Name/JaneDoe).

This issue aims to implement a support for IAM roles which have nested multiple paths in their role name. See the following as typical examples.

parse error: Invalid escape at line 107, column 25

[Console outputs]

with "enableExecuteCommand": false
โฏ check-ecs-exec.sh "staging-xxxxxxx-ecs-cluster-artisan" "c1aad09cc7ae435e8c635a11b134078d"
-------------------------------------------------------------
Prerequisites for check-ecs-exec.sh
-------------------------------------------------------------
  jq      | OK (/usr/local/bin/jq)
  AWS CLI | OK (/usr/local/bin/aws)

-------------------------------------------------------------
Prerequisites for the AWS CLI to use ECS Exec
-------------------------------------------------------------
  AWS CLI Version        | OK (aws-cli/2.1.31 Python/3.8.8 Darwin/19.6.0 exe/x86_64 prompt/off)
  Session Manager Plugin | OK (1.2.54.0)

-------------------------------------------------------------
Configurations for ECS task and other resources
-------------------------------------------------------------
Cluster: staging-xxxxxxx-ecs-cluster-artisan
Task   : c1aad09cc7ae435e8c635a11b134078d
-------------------------------------------------------------
  Cluster Configuration  | Audit Logging Not Configured
  Can I ExecuteCommand?  | arn:aws:iam::xxxxxxx:user/xxxxxxx
     ecs:ExecuteCommand: allowed
  Launch Type            | Fargate
  Platform Version       | 1.4.0
  Exec Enabled for Task  | NO
  Managed Agent Status   | SKIPPED
parse error: Invalid escape at line 107, column 25
with "enableExecuteCommand": true
โฏ check-ecs-exec.sh "staging-xxxxxxx-ecs-cluster-artisan" "c9c52abe108343bd8dcb3841f395aeb9"
-------------------------------------------------------------
Prerequisites for check-ecs-exec.sh
-------------------------------------------------------------
  jq      | OK (/usr/local/bin/jq)
  AWS CLI | OK (/usr/local/bin/aws)

-------------------------------------------------------------
Prerequisites for the AWS CLI to use ECS Exec
-------------------------------------------------------------
  AWS CLI Version        | OK (aws-cli/2.1.31 Python/3.8.8 Darwin/19.6.0 exe/x86_64 prompt/off)
  Session Manager Plugin | OK (1.2.54.0)

-------------------------------------------------------------
Configurations for ECS task and other resources
-------------------------------------------------------------
Cluster: staging-xxxxxxx-ecs-cluster-artisan
Task   : c9c52abe108343bd8dcb3841f395aeb9
-------------------------------------------------------------
  Cluster Configuration  | Audit Logging Not Configured
  Can I ExecuteCommand?  | arn:aws:iam:: xxxxxxx:user/xxxxxxx
     ecs:ExecuteCommand: allowed
  Launch Type            | Fargate
  Platform Version       | 1.4.0
  Exec Enabled for Task  | OK
  Managed Agent Status   |
     1. RUNNING for "app-back" container
parse error: Invalid escape at line 107, column 25

[Environment info]

  • Mac Mini (Late 2012)
  • macOS Catalina 10.15.7
  • jq --version => jq-1.6
  • AWS region = ap-southeast-1

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.