VSCode disconnects after credentials refresh. about sagemaker-ssh-helper HOT 6 OPEN

Hi, @harish-kamath , very nice that you like this library! I have few questions:
1/ When you say InternalServerError: We encountered an internal error. Please try again - which log file exactly do you see this message at? Is it in CloudWatch?
2/ Do you know which process generates this message, e.g., is there any prefix in front of this line?
3/ What SageMaker component you are connecting to, e.g., SageMaker Training or Studio, or Inference?

from sagemaker-ssh-helper.

@ivan-khvostishkov

Upon further digging - I'm not sure if it is actually the credentials (only).

I noticed that the ssm agent will first pull AWS credentials from the environment variables - so I tried including explicit AWS access key and secret key in my training job. The logs still show that the credentials are being refreshed, but the machine doesn't actually crash ~1m after the credentials are refreshed anymore. However, now it just crashes otherwise (even if there is nothing running, so no chance that it's a resource issue).

1. This error message is on the Sagemaker console, under Training Jobs. Here's an example:
   

And here is the last cloudwatch log:

(Note that in this case, I did not do sm-wait stop, but it doesn't actually matter for this error. It occurs even if I do that)

2. There's no prefix or process unfortunately. Since it just crashes the machine, and there's no persistent storage, I'm not sure how I can actually debug after a crash either.

3. Sagemaker Training Jobs

from sagemaker-ssh-helper.

On the bright side, it no longer crashes always after 30 mins of being connected. However, it is still crashing within an hour.

from sagemaker-ssh-helper.

Never mind, just got another crash in <30 minutes.

I'm pretty sure it is still this package, because connecting over plain SSH is still fine and never causes a crash.

from sagemaker-ssh-helper.

Hi, @harish-kamath , apologies for delay, this indeed seems to me very strange. I will have to investigate it further since in the short-running tests it never crashed like that before. In a meantime, is it possible for you to raise the support case from AWS Console? Please, add the link to this issue and mention my name:

https://docs.aws.amazon.com/awssupport/latest/user/case-management.html

from sagemaker-ssh-helper.

@harish-kamath I am using the following manual test to try to reproduce the issue. Without connection from VS Code the job successfully stops in 3 hours without any "Internal Server Error". I have few more asks and questions:

1/ What instance types did you try?

2/ Could you please run the test on ml.g4dn.xlarge:

3/ Don't connect with VS Code and wait for some time until credentials will refresh automatically for one or two times. The job should not crash. Then try to connect with VS Code.

So far for me it seems that the issue has nothing to do with Credential refresh, because they are refreshed all the time automatically and this is expected.

4/ How likely is that the VS Code runs some heavy process inside and the instance is running out of RAM?

Could you please check the utilization on the job page in AWS Console? The successful run looks like this:

Make the note at which exact time the credentials were refreshed and what time you've connected with VS Code.

I hope the above steps will help you to localize and isolate the issue down to some process that VS Code starts inside the container.

from sagemaker-ssh-helper.