continuumio / anaconda-ec2 Goto Github PK
View Code? Open in Web Editor NEWAnaconda plugin for StarCluster
Anaconda plugin for StarCluster
From IPython.parallel when I try to import Client('.ipython/profile_default/security/ipcontroller-client.json'), I get a value error: No JSON object could be decoded.
However, if I try json.load(open('.ipython/profile_default/security/ipcontroller-client.json')) it returns the JSON object I expect. This seems like a pretty odd error. It might be beyond the scope of the anaconda-ec2 project, but I haven't had this issue with vanilla StarCluster so it makes me wonder if something is wrong with the way ipcluster plugin works. I can't tell if this is related to #3
Thank you for porting the AMI to us-west-2.
Can you do the same for us-west-1?
(Or is there a way I can do it myself?)
When I try to stop an anaconda cluster, I get:
!!! ERROR - Cluster 'anaconda_cluster' does not contain any 'stoppable'
!!! ERROR - nodes and can only be terminated. Please use the 'terminate'
!!! ERROR - command instead to destroy the cluster.
This is extremely annoying because I don't like wasting money keeping the cluster running when I'm not at work but starting a new one takes hours because of all the packages I install.
AWS has (finally) made it very easy to copy AMIs from one region to another. Unfortunately, they will not let me copy it (even though it's public) because of an "AMI ownership mismatch". Would you mind copying it to us-west-2 (Oregon)? Thank you.
I'm not sure what's going on here (or if it is even an issue with anaconda per se) but the pkginstaller plugin doesn't work for me any more. I don't remember having a problem with it in the past. I get
!!! ERROR - command 'DEBIAN_FRONTEND='noninteractive' apt-get -o Dpkg::Options::='--force-confnew' -y --force-yes install g++ gfortran git-core subversion python-all-dev python-virtualenv mongodb libblas-dev liblapack-dev libatlas-dev r-base-core cython python-dateutil python-requests python-pymongo python-rpy2 python-nltk r-base-core mysql-client build-essential python-setuptools python-dev python-numpy python-matplotlib prover9 python-scipy python-tz python-pandas python-mysqldb libsnappy-dev python-xlwt msgpack-python lzop libpq-dev python-psycopg2 python-pygresql' failed with status 100
I tried reducing the number of packages, in case one of them was the culprit, and then rerunning starcluster runplugin pkginstaller
but it didn't seem to make a difference.
I thought conda might help, but I couldn't figure out how to get it to install across the whole cluster.
I'm running into this error when trying to start up a basic instance:
ERROR - InvalidBlockDeviceMapping: Invalid device name '/dev/xvdal'
Initialization fails and I'm unable to get the cluster running.
I'm using 32-bit anaconda on a windows machine, with all starcluster dependencies freshly installed with pip.
This is the only hint of a similar error message I could find on a Google search:
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/684875
While other AMI images do work properly, it seems that the Anaconda AMIs break on install even without the fancy scripts included here. Perhaps something changed and they need to be updated?
Error log:
C:\Users\sjoberge>starcluster start -s 1 ancluster
StarCluster - (http://star.mit.edu/cluster) (v. 0.94)
Software Tools for Academics and Researchers (STAR)
Please submit bug reports to [email protected]
>>> Using default cluster template: smallcluster
>>> Validating cluster template settings...
>>> Cluster template settings are valid
>>> Starting cluster...
>>> Launching a 1-node cluster...
>>> Creating security group @sc-ancluster...
>>> Opening tcp port range 8989-8989 for CIDR 0.0.0.0/0
!!! ERROR - InvalidBlockDeviceMapping: Invalid device name '/dev/xvdal'
Traceback (most recent call last):
File "C:\Anaconda\lib\site-packages\starcluster\cli.py", line 274, in main
sc.execute(args)
File "C:\Anaconda\lib\site-packages\starcluster\commands\start.py", line 215,
in execute
validate_running=validate_running)
File "C:\Anaconda\lib\site-packages\starcluster\cluster.py", line 1471, in sta
rt
return self._start(create=create, create_only=create_only)
File "<string>", line 2, in _start
File "C:\Anaconda\lib\site-packages\starcluster\utils.py", line 111, in wrap_f
res = func(*arg, **kargs)
File "C:\Anaconda\lib\site-packages\starcluster\cluster.py", line 1486, in _st
art
self.create_cluster()
File "C:\Anaconda\lib\site-packages\starcluster\cluster.py", line 1012, in cre
ate_cluster
self._create_flat_rate_cluster()
File "C:\Anaconda\lib\site-packages\starcluster\cluster.py", line 1036, in _cr
eate_flat_rate_cluster
force_flat=True)[0]
File "C:\Anaconda\lib\site-packages\starcluster\cluster.py", line 844, in crea
te_nodes
resvs.append(self.ec2.request_instances(image_id, **kwargs))
File "C:\Anaconda\lib\site-packages\starcluster\awsutils.py", line 443, in req
uest_instances
block_device_map=block_device_map)
File "C:\Anaconda\lib\site-packages\starcluster\awsutils.py", line 471, in run
_instances
block_device_map=block_device_map)
File "C:\Anaconda\lib\site-packages\boto\ec2\connection.py", line 725, in run_
instances
verb='POST')
File "C:\Anaconda\lib\site-packages\boto\connection.py", line 1096, in get_obj
ect
raise self.ResponseError(response.status, response.reason, body)
EC2ResponseError: EC2ResponseError: 400 Bad Request
<?xml version="1.0" encoding="UTF-8"?>
<Response><Errors><Error><Code>InvalidBlockDeviceMapping</Code><Message>Invalid
device name '/dev/xvdal'</Message></Error></Errors><RequestID>8e33446d-de4f-4e9e
-bcfa-2bad7035630b</RequestID></Response>
Disco should work, out of the box, right? I'm new to Disco, so this could well be user error, but I'm trying to go through the tutorial at http://discoproject.org/doc/disco/start/tutorial.html When I try the first tutorial (prepare input data) I get an error saying:
Unable to access resource (http://master:8989/ddfs/new_blob/bigfile_txt-0?): couldn't connect to host
When I try the second tutorial (write job functions), I get a similar error:
disco.error.CommError: Unable to access resource (http://master:8989/disco/job/new): couldn't connect to host (is disco master running at http://master:8989?)
It looks like disco is not configured correctly on my cluster. I can't tell if that's because of something wrong in the AMI, or in the anaconda_plugin or in my own config file or something else entirely. Please help. Thank you.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.