Bug: "Timed out while attempting to establish a connection to the container" error after fresh install of Docker and AWS SAM about aws-sam-cli HOT 5 OPEN

Thanks for reporting this, can you paste the contents of sam --info into a code block here?

I see that you mention it gets stuck when mounting, if you run sam local invoke again, and while it is waiting to mount could you do a quick docker ps to see if any node 20 containers were created before the command exists?

from aws-sam-cli.

While it was mounting, I ran sudo docker ps and got:

(base) mnicol@COECE-057000D:~/Documents/samtest$ sudo docker ps
[sudo] password for mnicol: 
CONTAINER ID   IMAGE                                          COMMAND                  CREATED          STATUS          PORTS                                                  NAMES
558462bdc785   public.ecr.aws/lambda/nodejs:20-rapid-x86_64   "/var/rapid/aws-lamb…"   52 seconds ago   Up 50 seconds   127.0.0.1:6816->8080/tcp                               cool_panini
0b5f96d489e2   mysql:latest                                   "docker-entrypoint.s…"   33 minutes ago   Up 33 minutes   0.0.0.0:3306->3306/tcp, :::3306->3306/tcp, 33060/tcp   mysqlkeycloak

After it timed out, I checked again, and the container was no longer there.

Output from sam --info

(base) mnicol@COECE-057000D:~/Documents/samtest$ sam --info
{
  "version": "1.116.0",
  "system": {
    "python": "3.11.8",
    "os": "Linux-5.4.0-182-generic-x86_64-with-glibc2.31"
  },
  "additional_dependencies": {
    "docker_engine": "24.0.5",
    "aws_cdk": "Not available",
    "terraform": "Not available"
  },
  "available_beta_feature_env_vars": [
    "SAM_CLI_BETA_FEATURES",
    "SAM_CLI_BETA_BUILD_PERFORMANCE",
    "SAM_CLI_BETA_TERRAFORM_SUPPORT",
    "SAM_CLI_BETA_RUST_CARGO_LAMBDA"
  ]
}

from aws-sam-cli.

Thanks for getting back with the additional info, we'll have to investigate this further, it looks like its failing to create a raw socket connection to the port the container spins up as part of the validation we do before invoking. In the meantime, can you try bumping the timeout before it fails using SAM_CLI_CONTAINER_CONNECTION_TIMEOUT? By default it is set to 20 seconds, running something like

SAM_CLI_CONTAINER_CONNECTION_TIMEOUT=60 sam local invoke --debug

will force the logic to wait a minute instead.

from aws-sam-cli.

Facing the same issue, thought it was awscliv1. So, I upgraded to v2, did a fresh installation of both aws and sam and made sure docker is enabled.

uname -srvmpio : Linux 6.5.0-35-generic #35~22.04.1-Ubuntu SMP PREEMPT_DYNAMIC Tue May 7 09:00:52 UTC 2 x86_64 x86_64 x86_64 GNU/Linux

docker --version : Docker version 26.1.3, build b72abbb

sam --version : SAM CLI, version 1.117.0

aws --version : aws-cli/2.15.58 Python/3.11.8 Linux/6.5.0-35-generic exe/x86_64.ubuntu.22

Even after trying @lucashuy's suggestion, still getting an immediate timeout:
$ SAM_CLI_CONTAINER_CONNECTION_TIMEOUT=60 sam local invoke --debug
2024-05-28 19:30:49,684 | Config file location: /home/kaoka/Documents/ht/project/project-source/source/samconfig.toml
2024-05-28 19:30:49,695 | Loading configuration values from [default.['local', 'invoke'].parameters] (env.command_name.section) in config file at '/home/kaoka/Documents/ht/project/project-source/source/samconfig.toml'...
2024-05-28 19:30:49,699 | Configuration values successfully loaded.
2024-05-28 19:30:49,701 | Configuration values are: {'stack_name': 'source', 'region': 'us-west-2', 'profile': 'access'}
2024-05-28 19:30:49,727 | Using SAM Template at /home/kaoka/Documents/ht/project/project-source/source/.aws-sam/build/template.yaml
2024-05-28 19:30:49,859 | Using config file: samconfig.toml, config environment: default
2024-05-28 19:30:49,861 | Expand command line arguments to:
2024-05-28 19:30:49,864 | --template_file=/home/kaoka/Documents/ht/project/project-source/source/.aws-sam/build/template.yaml --no_event --layer_cache_basedir=/home/kaoka/.aws-sam/layers-pkg --container_host=localhost --container_host_interface=127.0.0.1
2024-05-28 19:30:49,868 | local invoke command is called
2024-05-28 19:30:49,894 | No Parameters detected in the template
2024-05-28 19:30:49,974 | Sam customer defined id is more priority than other IDs. Customer defined id for resource projectsourceConnFunction is projectsourceConnFunction
2024-05-28

19:30:49,977 | There is no customer defined id or cdk path defined for resource projectsourceConnFunctionsourceTriggerPermissions, so we will use the resource logical id as the resource id
2024-05-28 19:30:49,980 | There is no customer defined id or cdk path defined for resource projectProcessBackupCompletionNotificationRoleForsource, so we will use the resource logical id as the resource id
2024-05-28 19:30:49,986 | 0 stacks found in the template
2024-05-28 19:30:49,988 | No Parameters detected in the template
2024-05-28 19:30:50,064 | Sam customer defined id is more priority than other IDs. Customer defined id for resource projectsourceConnFunction is projectsourceConnFunction
2024-05-28 19:30:50,067 | There is no customer defined id or cdk path defined for resource projectsourceConnFunctionsourceTriggerPermissions, so we will use the resource logical id as the resource id
2024-05-28 19:30:50,069 | There is no customer defined id or cdk path defined for resource projectProcessBackupCompletionNotificationRoleForsource, so we will use the resource logical id as the resource id
2024-05-28 19:30:50,074 | 3 resources found in the stack
2024-05-28 19:30:50,077 | Found Serverless function with name='projectsourceConnFunction' and ImageUri='projectsourceconnfunction:v1.0.0'
2024-05-28 19:30:50,080 | --base-dir is not presented, adjusting uri /home/kaoka/Documents/ht/project/project-source/source/source relative to /home/kaoka/Documents/ht/project/project-source/source/.aws-sam/build/template.yaml
2024-05-28 19:30:50,083 | --base-dir is not presented, adjusting uri . relative to /home/kaoka/Documents/ht/project/project-source/source/.aws-sam/build/template.yaml
2024-05-28 19:30:50,102 | Found one Lambda function with name 'projectsourceConnFunction'
2024-05-28 19:30:50,105 | Invoking Container created from projectsourceconnfunction:v1.0.0
2024-05-28 19:30:50,108 | Loading AWS credentials from session with profile 'access'
2024-05-28 19:30:54,599 | Code None is not a zip/jar file
2024-05-28 19:30:54,607 | Local image was not found.
2024-05-28 19:30:54,609 | Removing rapid images for repo projectsourceconnfunction
Building image...............
2024-05-28 19:30:55,875 | Failed to build Docker Image
NoneType: None
2024-05-28 19:30:55,881 | Cleaning all decompressed code dirs
2024-05-28 19:30:55,884 | Telemetry endpoint configured to be https://aws-serverless-tools-telemetry.us-west-2.amazonaws.com/metrics
2024-05-28 19:30:55,934 | Telemetry endpoint configured to be https://aws-serverless-tools-telemetry.us-west-2.amazonaws.com/metrics
2024-05-28 19:30:55,937 | Unable to find Click Context for getting session_id.
2024-05-28 19:30:55,943 | Sending Telemetry: {'metrics': [{'commandRun': {'requestId': '91a73f08-cfe9-42b8-a6a7-626486445662', 'installationId': 'd664b28c-9a6b-4c07-b9d6-3d8e72a49673', 'sessionId': 'cc023f96-f7cb-4389-910e-85c864028c42', 'executionEnvironment': 'CLI', 'ci': False, 'pyversion': '3.11.8', 'samcliVersion': '1.117.0', 'awsProfileProvided': True, 'debugFlagProvided': True, 'region': 'us-west-2', 'commandName': 'sam local invoke', 'metricSpecificAttributes': {'projectType': 'CFN', 'gitOrigin': None, 'projectName': '6ffe9bce28cdff262b30c342183ba88015e3cd094cb35d105067c518430534cb','initialCommit': None}, 'duration': 6024, 'exitReason': 'ImageBuildException', 'exitCode': 1}}]} 2024-05-28 19:30:55,951 | Sending Telemetry: {'metrics': [{'events': {'requestId': '1d8e2aba-f074-461f-9e7c-c4435e0e5079', 'installationId': 'd664b28c-9a6b-4c07-b9d6-3d8e72a49673', 'sessionId': 'cc023f96-f7cb-4389-910e-85c864028c42', 'executionEnvironment': 'CLI', 'ci': False, 'pyversion': '3.11.8', 'samcliVersion': '1.117.0', 'commandName': 'sam local invoke', 'metricSpecificAttributes': {'events': [{'event_name': 'SamConfigFileExtension', 'event_value': '.toml', 'thread_id': 'ba0066cbc4c642aaa0f7220abc9661ee', 'time_stamp': '2024-05-28 16:30:49.683', 'exception_name': None}, {'event_name': 'SamConfigFileExtension', 'event_value': '.toml', 'thread_id': '422868ebff7e47578a98b17140b389d3', 'time_stamp': '2024-05-28 16:30:49.858', 'exception_name': None}]}}}]}
2024-05-28 19:30:57,113 | HTTPSConnectionPool(host='aws-serverless-tools-telemetry.us-west-2.amazonaws.com', port=443): Read timed out. (read timeout=0.1) Error: Error building docker image: The command '/bin/sh -c mv /var/rapid/aws-lambda-rie-x86_64 /var/rapid/aws-lambda-rie && chmod +x /var/rapid/aws-lambda-rie' returned a non-zero code: 1
2024-05-28 19:30:57,118 | HTTPSConnectionPool(host='aws-serverless-tools-telemetry.us-west-2.amazonaws.com', port=443): Read timed out. (read timeout=0.1)

Forgive the indent lol

Btw tried:
$ export SAM_CLI_CONTAINER_CONNECTION_TIMEOUT=60 && sam local invoke --debug
And still getting same outcome

from aws-sam-cli.