randerzander / jupyter-service Goto Github PK

Ambari Service definition for an Jupyter (IPython3) Notebook service

Python 96.24% CSS 0.09% JavaScript 1.80% Shell 1.87%

jupyter-service's Introduction

An Ambari service for Jupyter (IPython3) Notebooks. It comes with Python2.7 and many packages for working with data in Hadoop (pydoop, scikit-learn, pandas, numpy, scipy, etc.).

Warning: The Jupyter master service must be co-located on a node with the Spark Client (Jupyter Notebook depends on Spark Client libs in /usr/hdp/current/spark-client).

If you want to use the R kernel, install the r-service first.

This stack deploys Python 2.7.9 as an altinstall. Make sure you backup any libraries already installed in /usr/local/lib/python2.7/site-packages.

To deploy, copy the entire directory into your Ambari stacks folder and restart Ambari:

Note: If using the HDP 2.2 Sandbox, stop the ambari service

sudo service ambari stop

and add a port forwarding rule for port 9999:

Stack Install Directions:

git clone https://github.com/randerzander/jupyter-service
sudo cp -r jupyter-service /var/lib/ambari-server/resources/stacks/HDP/2.2/services/
sudo service ambari-server restart
sudo service ambari-agent restart

Then you can click on 'Add Service' from the 'Actions' dropdown menu in the bottom left of the Ambari dashboard. When you've completed the install process, IPython Notebook will be available at your_server_host:9999

If you want to use your own Spark build, change the spark.home setting via Ambari to a directory accessible to jupyter.user (Default is 'jupyter'). Make sure your SPARK_HOME/conf/spark-defaults.conf includes your HDP version. For example:

sudo su jupyter
cd /home/jupyter
wget http://d3kbcqa49mib13.cloudfront.net/spark-1.3.1-bin-hadoop2.6.tgz
tar -xzvf spark-1.3.1-bin-hadoop2.6.tgz
echo "spark.driver.extraJavaOptions -Dhdp.version=2.2.0.0-2041" >> spark-1.3.1-bin-hadoop2.6/conf/spark-defaults.conf
echo "spark.yarn.am.extraJavaOptions -Dhdp.version=2.2.0.0-2041" >> spark-1.3.1-bin-hadoop2.6/conf/spark-defaults.conf
cp /etc/hive/conf/hive-site.xml spark-1.3.1-bin-hadoop2.6/conf/
# Disable ATS hooks & Tez
sed -i "s/org\.apache\.hadoop\.hive\.ql\.hooks\.ATSHook//g" spark-1.3.1-bin-hadoop2.6/conf/hive-site.xml
sed -i "s/<value>tez/<value>mr/g" spark-1.3.1-bin-hadoop2.6/conf/hive-site.xml
sed -ri 's ([0-9]+)s \1 g' spark-1.3.1-bin-hadoop2.6/conf/hive-site.xml

A 'remove.sh' script is provided in the project root for convenience. It'll remove the service package from Ambari's resources dir, remove /home/jupyter, and Python2.7 bits. Please edit remove.sh to set your Ambari login and cluster name details.

Special thanks to Ali Bajwa and Ofer Mendelevitch for the help with setup and build processes. Thanks also to ddkaiser and sakserv for help with regexes.

jupyter-service's People

Contributors

Stargazers

Watchers

Forkers

harsha2010 fdion rnalakurthi nsabharwal asears alexmc6 sainib carlessanagustin matt2cv teamsoo fonylew crazynuxer satchel9 yrchen zz22394 johanwasserman davenewham esaenz7

jupyter-service's Issues

Status detection broken

When the service first starts, the Ambari Dashboard shows a green status indicator. After a minute or so, it goes red, despite the fact that the service is still live.

Is this Kerberos Enabled?

Will this service work with a kerberos secure cluster?
Is this jupyter ipython notebook multiuser now? Or do I have to use https://github.com/jupyter/jupyterhub instead of this service?

Thanks

test on Centos6.6

seems there is a python 2.7.9 conflict with the installer in Centos/RHEL 6.6

Running localhost:9999, Jupyter stops in Ambari

Whenever i run localhost:9999 on my host server, Jupyter stops in Ambari, any work around for this?

Error installing on HDP 2.3 on node running RHEL 6.5

I'm getting this error when trying to install the service:

Traceback (most recent call last):
File "/var/lib/ambari-agent/cache/stacks/HDP/2.3/services/jupyter-service/package/scripts/ambari/master.py", line 41, in
Master().execute()
File "/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py", line 218, in execute
method(env)
File "/var/lib/ambari-agent/cache/stacks/HDP/2.3/services/jupyter-service/package/scripts/ambari/master.py", line 9, in install
helpers.add_repos()
File "/var/lib/ambari-agent/cache/stacks/HDP/2.3/services/jupyter-service/package/scripts/ambari/ambari_helpers.py", line 13, in add_repos
for repo in os.listdir(repo_dir):
UnboundLocalError: local variable 'repo_dir' referenced before assignment

Break Python2.7 installer into a separate Ambari service

Kerberos issue

I have a cluster with HDP 2.4.2 and Ambari 2.2.2 with Kerberos authentication on CentOS 7.

After solved the issue in file ambari_helper.py (see issue #6) I try to reinstall Jupyter but I get this message error

resource_management.core.exceptions.Fail: Execution of 'hadoop fs -mkdir -p /user/jupyter' returned 1. 16/07/26 16:08:30 WARN ipc.Client: Exception encountered while connecting to the server : 
javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]

("Create Principals" and "Create Keytabs" have finished "OK".)

Thanks in advance.

Cluster issue

Hi,

I installed this service on a cluster of 5 host (Jupyther server on host-4 and client on the other ones). Each hosts have 8 cores.

When I see on the tab "Cluster" of Jupiter Web UI" it tells that only 8 cores are available: I expect that should be all host-clients in this tab!

Any suggestions?

Ambari's "Restart All" feature makes notebook service fail to start

sc not defined

Installed on Ambari 2.0.1, HDP 2.2. HDFS/YARN/Tez/Zookeeper/Spark/Metrics

words = sc.textFile("file:///tmp/hello.txt")
---------------------------------------------------------------------------
NameError                                 Traceback (most recent call last)
<ipython-input-1-3ddcf4589bfc> in <module>()
----> 1 words = sc.textFile("file:///tmp/hello.txt")

NameError: name 'sc' is not defined

Add NLTK package

It would be great to add NLTK to the default set of packages installed.