Comments (6)
Hi Yang,
if you would like to point to some data store on HDFS/S3, please point URI to them:
val hf = new H2OFrame(new java.net.URI("hdfs//mynamenode/mydirectory/myfile.csv"))
The same for S3.
If you would like to parse a local file, it has to be distributed to each node in the cluster:
val hf = new H2OFrame(new java.io.File("/my/cluster/datastore/file.csv"))
Right now, we our API does not provide a shortcut for upload of local file (open issue:
https://0xdata.atlassian.net/browse/SW-56).
Thank you
Michal
On 12/3/15 11:59 AM, Yang Lei wrote:
I can run sample using spark-submit against a --master=local[2]. However when I target it to my
mesos cluster, I got NPE: by class water.parser.ParseSetup$GuessSetupTsk; class
java.lang.NullPointerException: null
at water.parser.ParseSetup$GuessSetupTsk.map(ParseSetup.java:269)
at water.MRTask.compute2(MRTask.java:624)
at water.H2O$H2OCountedCompleter.compute(H2O.java:1017)If the issue is related to how the sample is loading the data, I wonder if we should be creating
some sample that will work with remote clusters, e.g. hosting the data on S3..Thank you .
Yang.
—
Reply to this email directly or view it on GitHub
#23.
from sparkling-water.
Thanks Michal,
So basically we are saying the samples only work for local mode. That is the reason I asked if we should host the sample data somewhere like s3 and so it can run out of box.
I will lose the issue now. Thank you.
Yang
from sparkling-water.
Another thought is if the sample can read the "SPARKLING_WATER_HOME" to construct the full path of where the file is. So that as long as the target slaves also having the Sparking Water installed, it will be able to load the file.
Thanks. Yang.
from sparkling-water.
verified the sample works after changing the file location to be downloadable.
from sparkling-water.
How to connect sparklingwater to DCOS Mesos Spark Cluster ?
from sparkling-water.
Right now, we do not provide any explicit support for DC/OS. However, any feedback, recommendations, or requirements are welcomed.
from sparkling-water.
Related Issues (20)
- Sparkling Water not properly configuring RAM on Databricks HOT 1
- R docker build failing again
- h2o-pysparkling-3.x does not support pep517 builds HOT 4
- Install proper setuptools
- Scala 2.13 support - part 1 - investigation
- Scala 2.13 support - part 2 - implementation
- Use newer Ubuntu in test docker image
- Upgrade H2O to 3.44.0.3
- Can't install pysparkling after updating setuptools >= 69.0.0 HOT 2
- Quiet and Embedded arguments are not working in the last version 3.44.0.3 HOT 1
- libxgboost.so getting filled in /tmp HOT 8
- Error - Spark parameters on H2O Sparkling water SIG
- describe an h2oframe HOT 2
- describe an h2oframe
- RestApiCommunicationException: H2O node http://10.159.20.11:54321 responded with HOT 1
- Upgrade H2O to 3.46.0.1
- docs: out of date Spark version listings HOT 1
- AIC/Loglikelihood metrics generation problems
- when will sparkling-water 3.46.0.1 be released? HOT 1
- expose uuid for dai mojo
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from sparkling-water.