continuumio / anaconda-ec2 Goto Github PK

AWS has (finally) made it very easy to copy AMIs from one region to another. Unfortunately, they will not let me copy it (even though it's public) because of an "AMI ownership mismatch". Would you mind copying it to us-west-2 (Oregon)? Thank you.

I'm not sure what's going on here (or if it is even an issue with anaconda per se) but the pkginstaller plugin doesn't work for me any more. I don't remember having a problem with it in the past. I get

!!! ERROR - command 'DEBIAN_FRONTEND='noninteractive' apt-get -o Dpkg::Options::='--force-confnew' -y --force-yes install g++ gfortran git-core subversion python-all-dev python-virtualenv mongodb libblas-dev liblapack-dev libatlas-dev r-base-core cython python-dateutil python-requests python-pymongo python-rpy2 python-nltk r-base-core mysql-client build-essential python-setuptools python-dev python-numpy python-matplotlib prover9 python-scipy python-tz python-pandas python-mysqldb libsnappy-dev python-xlwt msgpack-python lzop libpq-dev python-psycopg2 python-pygresql' failed with status 100

I tried reducing the number of packages, in case one of them was the culprit, and then rerunning starcluster runplugin pkginstaller but it didn't seem to make a difference.

I thought conda might help, but I couldn't figure out how to get it to install across the whole cluster.

From IPython.parallel when I try to import Client('.ipython/profile_default/security/ipcontroller-client.json'), I get a value error: No JSON object could be decoded.

However, if I try json.load(open('.ipython/profile_default/security/ipcontroller-client.json')) it returns the JSON object I expect. This seems like a pretty odd error. It might be beyond the scope of the anaconda-ec2 project, but I haven't had this issue with vanilla StarCluster so it makes me wonder if something is wrong with the way ipcluster plugin works. I can't tell if this is related to #3

I'm running into this error when trying to start up a basic instance:
ERROR - InvalidBlockDeviceMapping: Invalid device name '/dev/xvdal'

Initialization fails and I'm unable to get the cluster running.

I'm using 32-bit anaconda on a windows machine, with all starcluster dependencies freshly installed with pip.

This is the only hint of a similar error message I could find on a Google search:
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/684875

While other AMI images do work properly, it seems that the Anaconda AMIs break on install even without the fancy scripts included here. Perhaps something changed and they need to be updated?

Error log:

C:\Users\sjoberge>starcluster start -s 1 ancluster
StarCluster - (http://star.mit.edu/cluster) (v. 0.94)
Software Tools for Academics and Researchers (STAR)
Please submit bug reports to [email protected]

>>> Using default cluster template: smallcluster
>>> Validating cluster template settings...
>>> Cluster template settings are valid
>>> Starting cluster...
>>> Launching a 1-node cluster...
>>> Creating security group @sc-ancluster...
>>> Opening tcp port range 8989-8989 for CIDR 0.0.0.0/0
!!! ERROR - InvalidBlockDeviceMapping: Invalid device name '/dev/xvdal'
Traceback (most recent call last):
  File "C:\Anaconda\lib\site-packages\starcluster\cli.py", line 274, in main
    sc.execute(args)
  File "C:\Anaconda\lib\site-packages\starcluster\commands\start.py", line 215,
in execute
    validate_running=validate_running)
  File "C:\Anaconda\lib\site-packages\starcluster\cluster.py", line 1471, in sta
rt
    return self._start(create=create, create_only=create_only)
  File "<string>", line 2, in _start
  File "C:\Anaconda\lib\site-packages\starcluster\utils.py", line 111, in wrap_f

    res = func(*arg, **kargs)
  File "C:\Anaconda\lib\site-packages\starcluster\cluster.py", line 1486, in _st
art
    self.create_cluster()
  File "C:\Anaconda\lib\site-packages\starcluster\cluster.py", line 1012, in cre
ate_cluster
    self._create_flat_rate_cluster()
  File "C:\Anaconda\lib\site-packages\starcluster\cluster.py", line 1036, in _cr
eate_flat_rate_cluster
    force_flat=True)[0]
  File "C:\Anaconda\lib\site-packages\starcluster\cluster.py", line 844, in crea
te_nodes
    resvs.append(self.ec2.request_instances(image_id, **kwargs))
  File "C:\Anaconda\lib\site-packages\starcluster\awsutils.py", line 443, in req
uest_instances
    block_device_map=block_device_map)
  File "C:\Anaconda\lib\site-packages\starcluster\awsutils.py", line 471, in run
_instances
    block_device_map=block_device_map)
  File "C:\Anaconda\lib\site-packages\boto\ec2\connection.py", line 725, in run_
instances
    verb='POST')
  File "C:\Anaconda\lib\site-packages\boto\connection.py", line 1096, in get_obj
ect
    raise self.ResponseError(response.status, response.reason, body)
EC2ResponseError: EC2ResponseError: 400 Bad Request
<?xml version="1.0" encoding="UTF-8"?>
<Response><Errors><Error><Code>InvalidBlockDeviceMapping</Code><Message>Invalid
device name '/dev/xvdal'</Message></Error></Errors><RequestID>8e33446d-de4f-4e9e
-bcfa-2bad7035630b</RequestID></Response>

Disco should work, out of the box, right? I'm new to Disco, so this could well be user error, but I'm trying to go through the tutorial at http://discoproject.org/doc/disco/start/tutorial.html When I try the first tutorial (prepare input data) I get an error saying:
Unable to access resource (http://master:8989/ddfs/new_blob/bigfile_txt-0?): couldn't connect to host
When I try the second tutorial (write job functions), I get a similar error:
disco.error.CommError: Unable to access resource (http://master:8989/disco/job/new): couldn't connect to host (is disco master running at http://master:8989?)
It looks like disco is not configured correctly on my cluster. I can't tell if that's because of something wrong in the AMI, or in the anaconda_plugin or in my own config file or something else entirely. Please help. Thank you.

Thank you for porting the AMI to us-west-2.
Can you do the same for us-west-1?

(Or is there a way I can do it myself?)

When I try to stop an anaconda cluster, I get:

!!! ERROR - Cluster 'anaconda_cluster' does not contain any 'stoppable'
!!! ERROR - nodes and can only be terminated. Please use the 'terminate'
!!! ERROR - command instead to destroy the cluster.

This is extremely annoying because I don't like wasting money keeping the cluster running when I'm not at work but starting a new one takes hours because of all the packages I install.