Comments (4)
Hey @felixloesing !
Thanks for bringing up this discussion and contributing to Sparkler.
Elasticsearch, Kibana, dockers all sound good.
Does the three-container setup make sense for the project or do you prefer a different structure?
I think three container setup is fine in theory, but it may make setups a bit more complicated.
My suggestion: let's make all these work in a single container. that's step one.
Then after proper testing, move the services to different containers. that's step two.
If you are confident enough you could skip step one and jump to two, otherwise let's go step-by-step. We definitely need one container setup for testing and debugging.
Should we reuse the existing Sparkler container and always pull in Solr (also in Elasticsearch environments)?
No. let's build a separate one for elasticsearch env that does not have solr. Let's make a copy of the current docker file that has solr, and modify it with elasticsearch setup.
Would the Sparkler + Elasticsearch network be started with a new script made by us (which starts the three containers and configures the network) or should an existing script be updated to support an optional command line argument to select Elasticsearch?
Please make a new script for elastic.
We can obviously remove/retire old scripts and images as we make progress.
from sparkler.
Thank you for your detailed feedback, @thammegowda!
We will incorporate your suggestions here into the development process and I will use this issue to keep you posted about potential problems or changes to our approach, as necessary.
from sparkler.
We definitely need one container setup for testing and debugging.
Can you expand on this? docker-compose makes it easy to bring up a service with multiple containers and communicate between each. I don't think testing / debugging will be more difficult with a multi-container approach, you'd just need to ssh into each container as necessary. It'd be helpful to understand the use case for more context though.
Our main concern with a single image is the size and the frequency of needing to rebuild (a change in any service requires a rebuild of the entire image).
from sparkler.
@nhandyal I don't have much experience with docker-compose. If you think its easy, then good, we shall use multiple containers and docker-compose for testing as well. Thanks.
from sparkler.
Related Issues (20)
- Update CI so users can download built Sparkler package
- Investigate pipeline frameworks HOT 1
- Sparkler cannot be executed on Databricks because sparkContext not pulled from sparkSession
- Sparkler Elasticsearch storage engine HOT 10
- Elasticsearch for Sparkler - Command Line Configuration HOT 2
- Elasticsearch for Sparkler - Maven Profiles
- Elasticsearch for Sparkler - Factory Design Pattern HOT 1
- Broken run script HOT 4
- Fix sparkler CI build
- Writing Data to Elasticsearch Storage Engine HOT 1
- Failed to create thread
- Unit Tests for Sparkler and Elasticsearch
- Debugging Elasticsearch Connection
- Sparkler not distributing work over nodes HOT 1
- Exclude net.jpountz.lz4 lz4 from kafka-clients dependency in sparkler-app/pom.xml HOT 2
- Caught Server refused connection at: http://localhost:8983/solr/crawldb HOT 19
- Build fails: could not find com.browserup:browserup-proxy-core:jar:3.0.0-SNAPSHOT
- Error from server at http://localhost:8983/solr/crawldb: ERROR: [doc=<>] unknown field 'contenthash' HOT 3
- warning: usage of JAVA_HOME is deprecated, use ES_JAVA_HOME HOT 3
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from sparkler.