Jobs stuck in Runnable about raster-vision HOT 9 CLOSED

As a workaround, you can run sudo start ecs on the instance to get it to progress out of the runnable state.

from raster-vision.

Tagging @azavea/operations in case it is of interest.

from raster-vision.

Hm. It is possible that the root cause is buried in another log (possibly the Docker agent log given this agent's inability to connect).

There have also been a flurry of updates to the ECS agent. Are we on a current version (one that supports Docker 17)?

from raster-vision.

   __|  __|  __|
   _|  (   \__ \   Amazon ECS-Optimized Amazon Linux AMI 2017.03.g
 ____|\___|____/

[ec2-user@ip-172-31-38-99 ~]$ docker --version
Docker version 1.12.6, build 7392c3b/1.12.6

from raster-vision.

This is one the problems of running off of a pre-baked AMI, instead of the official ECS AMI with cloud init installing everything else. If we could do the latter, we could simply bump the ECS AMI ID to the latest version; what we should do now is re-bake our custom AMI off of the latest ECS AMI.

from raster-vision.

We can probably take a stab and putting something more reproducible together if you can point us to the current steps.

from raster-vision.

The AMI we are using for Batch instances was generated as follows:

• Start an instance with a 128 GB root volume and 22 GB data volume using the ECS-optimized AMI with id ami-275ffe1 (see http://docs.aws.amazon.com/AmazonECS/latest/developerguide/ecs-optimized_AMI_launch_latest.html). There needs to be a 22 GB data volume or Docker will not work for reasons I don't understand. Having the root volume be 128 GB seems wasteful, but it was the easiest way of increasing the disk space, since Batch doesn't provide a way of explicitly setting the disk space for a compute environment.
• Install NVIDIA drivers 375.51 using instructions in http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/accelerated-computing-instances.html
• Run sudo stop ecs and then sudo rm /var/lib/ecs/data/ecs_agent_data.json
• Create the AMI

from raster-vision.

The above process was based on the recommendations in http://docs.aws.amazon.com/batch/latest/userguide/batch-gpu-ami.html, although we are not using their recipe for exposing the GPU to the container. Instead, we are using the recipe in https://blog.cloudsight.ai/deep-learning-image-recognition-using-gpus-in-amazon-ecs-docker-containers-5bdb1956f30e. We might want to go with the officially recommended way.

from raster-vision.

I created #74 to capture the task. I agree that the recommended way makes sense to chase. Please comment in that issue's thread with any additional tweaks we may need.

from raster-vision.