Giter VIP home page Giter VIP logo

Comments (8)

swoehrl-mw avatar swoehrl-mw commented on May 23, 2024

Hi @superivaj,

I suspect that one of the loadbalancing and proxy components does not deal correctly with websocket connections and as zeppelin relies on them it does not work without them.
How do you expose zeppelin? Via marathon-lb? Dou you have a (hardware) loadbalancer in front of your public agents? Is it running in layer-4 or layer-7 mode?
I know that AWS Classic ELBs in layer-7 mode (http/https) do not support websocket connections.

Best,
Sebastian

from dcos-zeppelin.

superivaj avatar superivaj commented on May 23, 2024

Hi Sebastian,

Thanks for your quick reply. I have deployed DC/OS on a few machines hosted in-house, we are not using a cloud service. These machines are running CoreOS. All firewall/switch ports are open for communication between the masters and the nodes as far as I know.

I have not deployed Marathon_LB either because my intention was to use zeppelin from http:///services/zeppelin. I understood that Marathon_LB was optional. I do not have public agent nodes either.

Is there any log that I can look at to dig into the problem? Maybe a CoreOS configuration? The rest of applications in DC/OS work fine.

from dcos-zeppelin.

swoehrl-mw avatar swoehrl-mw commented on May 23, 2024

Hi @superivaj,

you are correct that using zeppelin via the adminrouter (/services/zeppelin) should work (I tried it myself on a 1.11.1 cluster), marathon-lb is optional.
You can check the zeppelin logs (since the app is running in docker mode you have to ssh into the host and use docker exec), you find them in the container under /zeppelin/logs. You can also check the system logs on your master(s) using journalctl for logmessages from the adminrouter, though I am unsure what level of detail they contain.
What you can also try is access zeppelin directly via its host (you can find out the random port assigned by mesos to the app by checking the zeppelin logs, that port should be reachable from other hosts as the task is started in host network mode) to rule out a problem with the adminrouter.

Best,
Sebastian

from dcos-zeppelin.

superivaj avatar superivaj commented on May 23, 2024

Hi Sebastian,

Thanks for the guidance.
It is working when accessed directly by the node+port but it is not working when accessed by the DC/OS gui.

The Zeppelin logs look fine to me. I added some comments to indicate when I connected from the DC/OS UI and when I did it from the node+port.

 INFO [2018-06-12 08:56:28,737] ({main} ZeppelinConfiguration.java[create]:101) - Load configuration from file:/zeppelin/conf/zeppelin-site.xml
 INFO [2018-06-12 08:56:28,850] ({main} ZeppelinConfiguration.java[create]:109) - Server Host: 0.0.0.0
 INFO [2018-06-12 08:56:28,850] ({main} ZeppelinConfiguration.java[create]:111) - Server Port: 13810
 INFO [2018-06-12 08:56:28,850] ({main} ZeppelinConfiguration.java[create]:115) - Context Path: /
 INFO [2018-06-12 08:56:28,856] ({main} ZeppelinConfiguration.java[create]:116) - Zeppelin Version: 0.7.3
 INFO [2018-06-12 08:56:28,879] ({main} Log.java[initialized]:186) - Logging initialized @525ms
 INFO [2018-06-12 08:56:28,982] ({main} ZeppelinServer.java[setupWebAppContext]:346) - ZeppelinServer Webapp path: /zeppelin/webapps
 INFO [2018-06-12 08:56:29,439] ({main} ZeppelinServer.java[main]:187) - Starting zeppelin server
 INFO [2018-06-12 08:56:29,443] ({main} Server.java[doStart]:327) - jetty-9.2.15.v20160210
 INFO [2018-06-12 08:56:32,217] ({main} StandardDescriptorProcessor.java[visitServlet]:297) - NO JSP Support for /, did not find org.eclipse.jetty.jsp.JettyJspServlet
 WARN [2018-06-12 08:56:32,405] ({main} Helium.java[loadConf]:101) - /zeppelin/conf/helium.json does not exists
 WARN [2018-06-12 08:56:32,723] ({main} Interpreter.java[register]:406) - Static initialization is deprecated for interpreter sql, You should change it to use interpreter-setting.json in your jar or interpreter/{interpreter}/interpreter-setting.json
 INFO [2018-06-12 08:56:32,724] ({main} InterpreterSettingManager.java[init]:305) - Interpreter psql.sql found. class=org.apache.zeppelin.postgresql.PostgreSqlInterpreter
 INFO [2018-06-12 08:56:32,994] ({main} InterpreterSettingManager.java[init]:337) - InterpreterSettingRef name ignite
 INFO [2018-06-12 08:56:32,994] ({main} InterpreterSettingManager.java[init]:337) - InterpreterSettingRef name python
 INFO [2018-06-12 08:56:32,995] ({main} InterpreterSettingManager.java[init]:337) - InterpreterSettingRef name jdbc
 INFO [2018-06-12 08:56:32,996] ({main} InterpreterSettingManager.java[init]:337) - InterpreterSettingRef name psql
 INFO [2018-06-12 08:56:32,996] ({main} InterpreterSettingManager.java[init]:337) - InterpreterSettingRef name lens
 INFO [2018-06-12 08:56:32,996] ({main} InterpreterSettingManager.java[init]:337) - InterpreterSettingRef name pig
 INFO [2018-06-12 08:56:32,997] ({main} InterpreterSettingManager.java[init]:337) - InterpreterSettingRef name flink
 INFO [2018-06-12 08:56:32,997] ({main} InterpreterSettingManager.java[init]:337) - InterpreterSettingRef name angular
 INFO [2018-06-12 08:56:32,997] ({main} InterpreterSettingManager.java[init]:337) - InterpreterSettingRef name livy
 INFO [2018-06-12 08:56:32,997] ({main} InterpreterSettingManager.java[init]:337) - InterpreterSettingRef name file
 INFO [2018-06-12 08:56:32,997] ({main} InterpreterSettingManager.java[init]:337) - InterpreterSettingRef name elasticsearch
 INFO [2018-06-12 08:56:32,997] ({main} InterpreterSettingManager.java[init]:337) - InterpreterSettingRef name sh
 INFO [2018-06-12 08:56:32,997] ({main} InterpreterSettingManager.java[init]:337) - InterpreterSettingRef name cassandra
 INFO [2018-06-12 08:56:32,997] ({main} InterpreterSettingManager.java[init]:337) - InterpreterSettingRef name spark
 INFO [2018-06-12 08:56:32,997] ({main} InterpreterSettingManager.java[init]:337) - InterpreterSettingRef name md
 INFO [2018-06-12 08:56:32,998] ({main} InterpreterSettingManager.java[init]:337) - InterpreterSettingRef name alluxio
 INFO [2018-06-12 08:56:32,998] ({main} InterpreterSettingManager.java[init]:337) - InterpreterSettingRef name bigquery
 INFO [2018-06-12 08:56:32,998] ({main} InterpreterSettingManager.java[init]:337) - InterpreterSettingRef name hbase
 INFO [2018-06-12 08:56:32,998] ({main} InterpreterSettingManager.java[init]:337) - InterpreterSettingRef name kylin
 INFO [2018-06-12 08:56:33,020] ({main} InterpreterSettingManager.java[init]:366) - InterpreterSetting group angular : id=2DJ1A6PPT, name=angular
 INFO [2018-06-12 08:56:33,020] ({main} InterpreterSettingManager.java[init]:366) - InterpreterSetting group sh : id=2DHJ4PKAY, name=sh
 INFO [2018-06-12 08:56:33,020] ({main} InterpreterSettingManager.java[init]:366) - InterpreterSetting group kylin : id=2DFNS4XUZ, name=kylin
 INFO [2018-06-12 08:56:33,020] ({main} InterpreterSettingManager.java[init]:366) - InterpreterSetting group psql : id=2DG3RYZBF, name=psql
 INFO [2018-06-12 08:56:33,020] ({main} InterpreterSettingManager.java[init]:366) - InterpreterSetting group lens : id=2DF1SNKAA, name=lens
 INFO [2018-06-12 08:56:33,020] ({main} InterpreterSettingManager.java[init]:366) - InterpreterSetting group hbase : id=2DHA8944Z, name=hbase
 INFO [2018-06-12 08:56:33,021] ({main} InterpreterSettingManager.java[init]:366) - InterpreterSetting group bigquery : id=2DHVHJMBW, name=bigquery
 INFO [2018-06-12 08:56:33,021] ({main} InterpreterSettingManager.java[init]:366) - InterpreterSetting group spark : id=2DF64M2G6, name=spark
 INFO [2018-06-12 08:56:33,021] ({main} InterpreterSettingManager.java[init]:366) - InterpreterSetting group ignite : id=2DGF8ERFP, name=ignite
 INFO [2018-06-12 08:56:33,021] ({main} InterpreterSettingManager.java[init]:366) - InterpreterSetting group alluxio : id=2DEY9JYGX, name=alluxio
 INFO [2018-06-12 08:56:33,021] ({main} InterpreterSettingManager.java[init]:366) - InterpreterSetting group flink : id=2DF4XFBKW, name=flink
 INFO [2018-06-12 08:56:33,021] ({main} InterpreterSettingManager.java[init]:366) - InterpreterSetting group pig : id=2DGSQYGEB, name=pig
 INFO [2018-06-12 08:56:33,021] ({main} InterpreterSettingManager.java[init]:366) - InterpreterSetting group md : id=2DFCJYT2F, name=md
 INFO [2018-06-12 08:56:33,021] ({main} InterpreterSettingManager.java[init]:366) - InterpreterSetting group livy : id=2DHBS7U9W, name=livy
 INFO [2018-06-12 08:56:33,022] ({main} InterpreterSettingManager.java[init]:366) - InterpreterSetting group elasticsearch : id=2DGM4CD6K, name=elasticsearch
 INFO [2018-06-12 08:56:33,022] ({main} InterpreterSettingManager.java[init]:366) - InterpreterSetting group python : id=2DHFF3AAF, name=python
 INFO [2018-06-12 08:56:33,022] ({main} InterpreterSettingManager.java[init]:366) - InterpreterSetting group jdbc : id=2DFZN8CWJ, name=jdbc
 INFO [2018-06-12 08:56:33,022] ({main} InterpreterSettingManager.java[init]:366) - InterpreterSetting group file : id=2DJN58EZ3, name=file
 INFO [2018-06-12 08:56:33,022] ({main} InterpreterSettingManager.java[init]:366) - InterpreterSetting group cassandra : id=2DF9CFKSX, name=cassandra
 INFO [2018-06-12 08:56:33,028] ({main} InterpreterFactory.java[<init>]:130) - shiroEnabled: false
 INFO [2018-06-12 08:56:33,143] ({main} VfsLog.java[info]:138) - Using "/tmp/vfs_cache" as temporary files store.
 INFO [2018-06-12 08:56:33,251] ({main} GitNotebookRepo.java[<init>]:63) - Opening a git repo at '/zeppelin/notebook'
 INFO [2018-06-12 08:56:33,419] ({main} GitNotebookRepo.java[<init>]:66) - Git repo /zeppelin/notebook/.git does not exist, creating a new one
 INFO [2018-06-12 08:56:33,712] ({main} NotebookAuthorization.java[loadFromFile]:96) - /zeppelin/conf/notebook-authorization.json
 INFO [2018-06-12 08:56:33,713] ({main} Credentials.java[loadFromFile]:102) - /zeppelin/conf/credentials.json
 INFO [2018-06-12 08:56:33,743] ({main} StdSchedulerFactory.java[instantiate]:1184) - Using default implementation for ThreadExecutor
 INFO [2018-06-12 08:56:33,746] ({main} SimpleThreadPool.java[initialize]:268) - Job execution threads will use class loader of thread: main
 INFO [2018-06-12 08:56:33,758] ({main} SchedulerSignalerImpl.java[<init>]:61) - Initialized Scheduler Signaller of type: class org.quartz.core.SchedulerSignalerImpl
 INFO [2018-06-12 08:56:33,759] ({main} QuartzScheduler.java[<init>]:240) - Quartz Scheduler v.2.2.1 created.
 INFO [2018-06-12 08:56:33,760] ({main} RAMJobStore.java[initialize]:155) - RAMJobStore initialized.
 INFO [2018-06-12 08:56:33,760] ({main} QuartzScheduler.java[initialize]:305) - Scheduler meta-data: Quartz Scheduler (v2.2.1) 'DefaultQuartzScheduler' with instanceId 'NON_CLUSTERED'
  Scheduler class: 'org.quartz.core.QuartzScheduler' - running locally.
  NOT STARTED.
  Currently in standby mode.
  Number of jobs executed: 0
  Using thread pool 'org.quartz.simpl.SimpleThreadPool' - with 10 threads.
  Using job-store 'org.quartz.simpl.RAMJobStore' - which does not support persistence. and is not clustered.

 INFO [2018-06-12 08:56:33,761] ({main} StdSchedulerFactory.java[instantiate]:1339) - Quartz scheduler 'DefaultQuartzScheduler' initialized from default resource file in Quartz package: 'quartz.properties'
 INFO [2018-06-12 08:56:33,761] ({main} StdSchedulerFactory.java[instantiate]:1343) - Quartz scheduler version: 2.2.1
 INFO [2018-06-12 08:56:33,761] ({main} QuartzScheduler.java[start]:575) - Scheduler DefaultQuartzScheduler_$_NON_CLUSTERED started.
 INFO [2018-06-12 08:56:34,617] ({main} FolderView.java[createFolder]:107) - Create folder Zeppelin Tutorial
 INFO [2018-06-12 08:56:34,617] ({main} FolderView.java[createFolder]:107) - Create folder /
 INFO [2018-06-12 08:56:34,617] ({main} Folder.java[setParent]:168) - Set parent of / to /
 INFO [2018-06-12 08:56:34,617] ({main} Folder.java[setParent]:168) - Set parent of Zeppelin Tutorial to /
 INFO [2018-06-12 08:56:34,617] ({main} Folder.java[addNote]:184) - Add note 2C35YU814 to folder Zeppelin Tutorial
 INFO [2018-06-12 08:56:34,626] ({main} Folder.java[addNote]:184) - Add note 2BYEZ5EVK to folder Zeppelin Tutorial
 INFO [2018-06-12 08:56:34,639] ({main} Folder.java[addNote]:184) - Add note 2BWJFTXKJ to folder Zeppelin Tutorial
 INFO [2018-06-12 08:56:34,643] ({main} Folder.java[addNote]:184) - Add note 2A94M5J1Z to folder Zeppelin Tutorial
 INFO [2018-06-12 08:56:34,647] ({main} Folder.java[addNote]:184) - Add note 2C57UKYWR to folder Zeppelin Tutorial
 INFO [2018-06-12 08:56:34,711] ({main} Folder.java[addNote]:184) - Add note 2C2AUG798 to folder Zeppelin Tutorial
 INFO [2018-06-12 08:56:34,712] ({main} Notebook.java[<init>]:127) - Notebook indexing started...
 INFO [2018-06-12 08:56:35,450] ({main} LuceneSearch.java[addIndexDocs]:305) - Indexing 6 notebooks took 738ms
 INFO [2018-06-12 08:56:35,451] ({main} Notebook.java[<init>]:129) - Notebook indexing finished: 6 indexed in 0s
 INFO [2018-06-12 08:56:35,816] ({main} ServerImpl.java[initDestination]:94) - Setting the server's publish address to be /
 INFO [2018-06-12 08:56:37,632] ({main} ContextHandler.java[doStart]:744) - Started o.e.j.w.WebAppContext@27fe3806{/,file:/zeppelin/webapps/webapp/,AVAILABLE}{/zeppelin/zeppelin-web-0.7.3.war}
 INFO [2018-06-12 08:56:37,712] ({main} AbstractConnector.java[doStart]:266) - Started ServerConnector@3f702946{HTTP/1.1}{0.0.0.0:13810}
 INFO [2018-06-12 08:56:37,712] ({main} Server.java[doStart]:379) - Started @9360ms
 INFO [2018-06-12 08:56:37,713] ({main} ZeppelinServer.java[main]:197) - Done, zeppelin server started
 
 ************ Connection by DC/OS UI
 WARN [2018-06-12 09:01:36,756] ({qtp760563749-55} SecurityRestApi.java[ticket]:87) - {"status":"OK","message":"","body":{"principal":"anonymous","ticket":"anonymous","roles":"[]"}}
 
 ************ Direct connection with PORT
 WARN [2018-06-12 09:10:16,414] ({qtp760563749-65} SecurityRestApi.java[ticket]:87) - {"status":"OK","message":"","body":{"principal":"anonymous","ticket":"anonymous","roles":"[]"}}
 INFO [2018-06-12 09:10:16,533] ({qtp760563749-55} NotebookServer.java[onOpen]:157) - New connection from 10.10.50.75 : 12539
 
 ************ Connection by DC/OS UI
 WARN [2018-06-12 09:10:23,390] ({qtp760563749-65} SecurityRestApi.java[ticket]:87) - {"status":"OK","message":"","body":{"principal":"anonymous","ticket":"anonymous","roles":"[]"}}
 
 ************ Direct connection with PORT
 INFO [2018-06-12 09:12:44,083] ({qtp760563749-64} NotebookServer.java[onClose]:364) - Closed connection to 10.10.50.75 : 12539. (1001) null
 WARN [2018-06-12 09:12:44,816] ({qtp760563749-64} SecurityRestApi.java[ticket]:87) - {"status":"OK","message":"","body":{"principal":"anonymous","ticket":"anonymous","roles":"[]"}}
 INFO [2018-06-12 09:12:45,082] ({qtp760563749-68} NotebookServer.java[onOpen]:157) - New connection from 10.10.50.75 : 12713
 
 ************ Connection by DC/OS UI
 WARN [2018-06-12 09:13:48,962] ({qtp760563749-70} SecurityRestApi.java[ticket]:87) - {"status":"OK","message":"","body":{"principal":"anonymous","ticket":"anonymous","roles":"[]"}}

Best Regards,
Francisco

from dcos-zeppelin.

swoehrl-mw avatar swoehrl-mw commented on May 23, 2024

Hi Francisco,

I was able to partially reproduce this behaviour. When opening the service in Chrome sometimes the websocket connection appears to be stuck and is reported by zeppelin as disconnected.
When running the DC/OS UI behind another loadbalancer that presents a valid ssl certificate (in my case letsencrypt) instead of the self-signed ones of the adminrouter I do not encounter the problem. So I suspect that the connection problem is not a DC/OS or Zeppelin problem. Instead the browser seems to have problems when handling websocket connections over ssl connections with self-signed certificates.

Best,
Sebastian

from dcos-zeppelin.

superivaj avatar superivaj commented on May 23, 2024

Hi Sebastian,

Thank you, that makes sense. If I navigate to the service as https://[MasterNode]/service/zeppelin/ it works (after accepting insecure connection) but the default one was http://[MasterNode]/service/zeppelin/ and this one didn't.

Everything else is working as expected, I was able to use Spark on HDFS out of the box. Excellent work!!

PS: Is it possible to enable the "File" interpreter to connect with DC/OS HDFS? It is configured with http://localhost:50070/webhdfs/v1/ (Which looks like the default and it doesn't work) but I am not sure which URL to put here. I tried with the DNS http://name-0-node.hdfs.autoip.dcos.thisdcos.directory:50070/webhdfs/v1/ but still not working. I am new to all of this as you can see.

from dcos-zeppelin.

swoehrl-mw avatar swoehrl-mw commented on May 23, 2024

Hi Francisco,

cool that it is working for you.
Regarding the file interpreter: You will have to use the http port of the name node, which by default should be 9002. So the URL should be http://name-0-node.hdfs.autoip.dcos.thisdcos.directory:9002/webhdfs/v1. But please be aware that HDFS on DC/OS is running in high availability mode with one of the two namenodes in standby, and the standby node will not answer webhdfs requests. So each time the namenodes do a failover you would have to switch the URL. The only way around this I know is to use HttpFS but there is no official installation method for DC/OS so you will have to build your own.
Apart from that I would not recommend the file interpreter from zeppelin as it is very basic and at the moment only supports the "ls" command. You are better off using spark and implementing some wrapper functions around the hadoop hdfs libraries for file management.
You could also try to install the hue package and use its filebrowser but as far as I know it does not work on a current DC/OS cluster.

Best,
Sebastian

from dcos-zeppelin.

superivaj avatar superivaj commented on May 23, 2024

Thank you for your help Sebastian,

from dcos-zeppelin.

Related Issues (6)

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.