Comments (11)
@tcfuji make sure that you have example files available fro docker environment.
from sparkling-water.
@mmalohlava Do you mean I need to download the examples from github?
Right now there is only:
ls examples/scripts/
Meetup20141217.md dlDemo.script
from sparkling-water.
@tcfuji which version of Sparkling Water are you using?
from sparkling-water.
I cloned it from github about 6 hours ago. I'm using the Dockerfile that
was included in that clone.
On Tue, Oct 20, 2015 at 6:12 PM Michal Malohlava [email protected]
wrote:
@tcfuji https://github.com/tcfuji which version of Sparkling Water are
you using?—
Reply to this email directly or view it on GitHub
#19 (comment)
.
from sparkling-water.
ahh,my fault, it is using old Sparkling Water version:
RUN curl -s http://d3kbcqa49mib13.cloudfront.net/spark-1.2.0-bin-cdh4.tgz | tar -xz -C /opt && \
ln -s /opt/spark-1.2.0-bin-cdh4 /opt/spark && \
mkdir /opt/spark/work && \
chmod 0777 /opt/spark/work
# Install Sparkling water latest version
RUN curl -s http://h2o-release.s3.amazonaws.com/sparkling-water/master/75/sparkling-water-0.2.10-75.zip --output sw.zip && \
unzip sw.zip -d /opt/ && \
ln -s /opt/sparkling-water-0.2.10-75 /opt/sparkling-water && \
If you edit it and replace spark by, for example:
http://d3kbcqa49mib13.cloudfront.net/spark-1.5.1-bin-hadoop2.4.tgz
and sparkling water by:
http://h2o-release.s3.amazonaws.com/sparkling-water/rel-1.5/2/sparkling-water-1.5.2.zip
Please free to make PR.
from sparkling-water.
Updating the versions seems to have introduced memory allocation problems.
When running docker run -i -t --rm sparkling-test-base bin/run-example.sh
:
---------
Using example : org.apache.spark.examples.h2o.AirlinesWithWeatherDemo2
Using master (MASTER) : local-cluster[3,2,1024]
Deploy mode (DEPLOY_MODE) : client
Driver memory (DRIVER_MEMORY): 1G
H2O JVM options (H2O_SYS_OPS) :
---------
Spark Command: /usr/lib/jvm/java-7-oracle/jre/bin/java -cp /opt/spark/conf/:/opt/spark/lib/spark-assembly-1.5.1-hadoop2.4.0.jar:/opt/spark/lib/datanucleus-core-3.2.10.jar:/opt/spark/lib/datanucleus-rdbms-3.2.9.jar:/opt/spark/lib/datanucleus-api-jdo-3.2.6.jar -Xms1G -Xmx1G org.apache.spark.deploy.SparkSubmit --master local-cluster[3,2,1024] --deploy-mode client --conf spark.driver.memory=1G --conf spark.driver.extraJavaOptions= --class org.apache.spark.examples.h2o.AirlinesWithWeatherDemo2 /opt/sparkling-water-1.5.2/assembly/build/libs/sparkling-water-assembly-1.5.2-all.jar
========================================
15/10/21 02:35:39 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
15/10/21 02:35:44 WARN MetricsSystem: Using default name DAGScheduler for source because spark.app.id is not set.
15/10/21 02:35:45 WARN H2OContext: Increasing 'spark.locality.wait' to value 30000
Java HotSpot(TM) 64-Bit Server VM warning: INFO: os::commit_memory(0x00007fa71956d000, 12288, 0) failed; error='Cannot allocate memory' (errno=12)
#
# There is insufficient memory for the Java Runtime Environment to continue.
# Native memory allocation (mmap) failed to map 12288 bytes for committing reserved memory.
When running docker run -i -t --rm sparkling-test-base bin/sparkling-shell
:
-----
Spark master (MASTER) : local-cluster[3,2,2048]
Spark home (SPARK_HOME) : /opt/spark
H2O build version : 3.2.0.5 (slater)
Spark build version : 1.5.0
----
Java HotSpot(TM) 64-Bit Server VM warning: INFO: os::commit_memory(0x0000000740000000, 2147483648, 0) failed; error='Cannot allocate memory' (errno=12)
#
# There is insufficient memory for the Java Runtime Environment to continue.
# Native memory allocation (mmap) failed to map 2147483648 bytes for committing reserved memory.
from sparkling-water.
@mmalohlava Should I make a PR anyway?
from sparkling-water.
Yes please 👍
from sparkling-water.
yes, please make a PR.
my guess is that the native mem oom might mean new flags to manage the
offheap DirectByteBuffers that might be needed in 1.5
thanks, sri
On Fri, Oct 23, 2015 at 1:00 PM, Ted [email protected] wrote:
@mmalohlava https://github.com/mmalohlava Should I make a PR anyway?
—
Reply to this email directly or view it on GitHub
#19 (comment)
.
culture . code . customer
c o m m u n i t y_, _h2o*.ai, *
+1-408.316.8192
from sparkling-water.
Done
from sparkling-water.
Hi tcfuji,
thanks for creating PR and sorry for so late response. We have actually created gradle task which creates docker file and determines latest version of sparkling-water automatically. It's gonna be in master and hopefully in a new sparkling-water release soon.
Kuba
from sparkling-water.
Related Issues (20)
- Install proper setuptools
- Scala 2.13 support - part 1 - investigation
- Scala 2.13 support - part 2 - implementation
- Use newer Ubuntu in test docker image
- Upgrade H2O to 3.44.0.3
- Can't install pysparkling after updating setuptools >= 69.0.0 HOT 2
- Quiet and Embedded arguments are not working in the last version 3.44.0.3 HOT 1
- libxgboost.so getting filled in /tmp HOT 8
- Error - Spark parameters on H2O Sparkling water SIG
- describe an h2oframe HOT 2
- describe an h2oframe
- RestApiCommunicationException: H2O node http://10.159.20.11:54321 responded with HOT 1
- Upgrade H2O to 3.46.0.1
- docs: out of date Spark version listings HOT 1
- AIC/Loglikelihood metrics generation problems
- when will sparkling-water 3.46.0.1 be released? HOT 1
- expose uuid for dai mojo
- Two users getting the same H2O Notebook
- Upgrade H2O to 3.46.0.2
- Upgrade H2O to 3.46.0.3
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from sparkling-water.