openforce / spark-mllib-scala-play Goto Github PK
View Code? Open in Web Editor NEWTwitter sentiment analysis based on Apache Spark, MLlib, Scala and Akka.
Home Page: http://sentiment.openforce.com
License: Apache License 2.0
Twitter sentiment analysis based on Apache Spark, MLlib, Scala and Akka.
Home Page: http://sentiment.openforce.com
License: Apache License 2.0
Further investigation is required since we just use the defaults until now.
So after an sbt dist and installation on a fresh server all you see is a swarm of exceptions. Need to include this in the distribution.
When starting the application I get the excpetion
Caused by: play.api.PlayException: Configuration error[Application secret not set]
at play.api.libs.CryptoConfigParser.get$lzycompute(Crypto.scala:236) ~[com.typesafe.play.play_2.11-2.4.4.jar:2.4.4]
at play.api.libs.CryptoConfigParser.get(Crypto.scala:203) ~[com.typesafe.play.play_2.11-2.4.4.jar:2.4.4]
We need to add to the documentation that the user needs to set an application secret.
With the current develop branch I cannot login via OAuth. I get the following error:
2015-12-08 19:26:02,541 - [warn] o.a.h.i.c.DefaultHttpClient - Authentication error: Unable to respond to any of these challenges: {oauth=www- authenticate: OAuth realm="https://api.twitter.com"}
2015-12-08 19:26:02,543 - [error] application -
! @6oce0039p - Internal server error, for (GET) [/authenticate] ->
play.api.http.HttpErrorHandlerExceptions$$anon$1: Execution exception[[OAuthNotAuthorizedException: Authorization failed (server replied with a 401). This can happen if the consumer key was not correct or the signatures did not match.]]
at play.api.http.HttpErrorHandlerExceptions$.throwableToUsefulException(HttpErrorHandler.scala:265) ~[play_2.11-2.4.4.jar:2.4.4]
at play.api.http.DefaultHttpErrorHandler.onServerError(HttpErrorHandler.scala:191) ~[play_2.11-2.4.4.jar:2.4.4]
at play.api.GlobalSettings$class.onError(GlobalSettings.scala:179) [play_2.11-2.4.4.jar:2.4.4]
at play.api.DefaultGlobal$.onError(GlobalSettings.scala:212) [play_2.11-2.4.4.jar:2.4.4]
at play.api.http.GlobalSettingsHttpErrorHandler.onServerError(HttpErrorHandler.scala:94) [play_2.11-2.4.4.jar:2.4.4]
Caused by: oauth.signpost.exception.OAuthNotAuthorizedException: Authorization failed (server replied with a 401). This can happen if the consumer key was not correct or the signatures did not match.
at oauth.signpost.AbstractOAuthProvider.handleUnexpectedResponse(AbstractOAuthProvider.java:243) ~[signpost-core-1.2.1.2.jar:na]
at oauth.signpost.AbstractOAuthProvider.retrieveToken(AbstractOAuthProvider.java:193) ~[signpost-core-1.2.1.2.jar:na]
at oauth.signpost.AbstractOAuthProvider.retrieveRequestToken(AbstractOAuthProvider.java:74) ~[signpost-core-1.2.1.2.jar:na]
at play.api.libs.oauth.OAuth.retrieveRequestToken(OAuth.scala:36) ~[play-ws_2.11-2.4.4.jar:2.4.4]
at controllers.Twitter$$anonfun$authenticate$1$$anonfun$apply$2.apply(Twitter.scala:38) ~[classes/:na]
Ideas:
Path for local maven repository in not compatible with Windows machines.
Querying for instance "star wars" causes no results for both online and batch model.
The provided diagram is incomplete and somewhat misleading.
@nano4711 discovered some issues on deployment and starup of this app on debian jessie. We need to investigate this further.
The OAuth mechanism has to be changed so that the user specific OAuth tokens are not send to the Director actor as it is single instance actor used by all users. Each user query has to fetch the cookie stored OAuth tokens and use them for Twitter authentication.
Make functions like tokenizeSentence
and unigramsAndBigrams
more composable.
It would assist us greatly to get feedback on the tutorials from people who were not involved with them.
The Github project description is missing, which makes it harder to find for people. Admin rights are needed to add it.
On Windows (10) and some Linux derivates the application gets stuck when dependencies to libblas and liblapack are not met. Since OSX ships those with the veclib framework, there are no issues.
The StatisticServer should remember the pushed statistics. This is needed so the batch statistics aren't lost after the batch trainer is done and also to fill the live chart of the online trainer with statistics that occurred before the user opened the frontend.
After a fresh git clone und switching to develop branch I get the following error when I try to access http://localhost:9000/
2015-11-11 10:04:08,019 - [warn] o.a.h.u.NativeCodeLoader - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
2015-11-11 10:04:08,075 - [error] o.a.s.SparkContext - Error initializing SparkContext.
org.apache.spark.SparkException: A master URL must be set in your configuration
at org.apache.spark.SparkContext.<init>(SparkContext.scala:394) [spark-core_2.11-1.5.1.jar:1.5.1]
at org.apache.spark.SparkContext.<init>(SparkContext.scala:112) [spark-core_2.11-1.5.1.jar:1.5.1]
at org.apache.spark.SparkContext$$FastClassByGuice$$119109b9.newInstance(<generated>) [guice-4.0.jar:1.5.1]
at com.google.inject.internal.cglib.reflect.$FastConstructor.newInstance(FastConstructor.java:40) [guice-4.0.jar:na]
at com.google.inject.internal.DefaultConstructionProxyFactory$1.newInstance(DefaultConstructionProxyFactory.java:61) [guice-4.0.jar:na]
2015-11-11 10:04:08,098 - [error] application -
! @6o4ml9c9d - Internal server error, for (GET) [/] ->
play.api.http.HttpErrorHandlerExceptions$$anon$1: Execution exception[[ProvisionException: Unable to provision, see the following errors:
1) Error injecting constructor, org.apache.spark.SparkException: A master URL must be set in your configuration
at org.apache.spark.SparkContext.<init>(SparkContext.scala:112)
while locating org.apache.spark.SparkContext
for parameter 1 at controllers.Application.<init>(Application.scala:23)
at controllers.Application.class(Application.scala:23)
while locating controllers.Application
1 error]]
at play.api.http.HttpErrorHandlerExceptions$.throwableToUsefulException(HttpErrorHandler.scala:265) ~[play_2.11-2.4.2.jar:2.4.2]
at play.api.http.DefaultHttpErrorHandler.onServerError(HttpErrorHandler.scala:191) ~[play_2.11-2.4.2.jar:2.4.2]
at play.core.server.Server$class.logExceptionAndGetResult$1(Server.scala:50) [play-server_2.11-2.4.2.jar:2.4.2]
at play.core.server.Server$$anonfun$getHandlerFor$4.apply(Server.scala:59) [play-server_2.11-2.4.2.jar:2.4.2]
at play.core.server.Server$$anonfun$getHandlerFor$4.apply(Server.scala:57) [play-server_2.11-2.4.2.jar:2.4.2]
Caused by: com.google.inject.ProvisionException: Unable to provision, see the following errors:
1) Error injecting constructor, org.apache.spark.SparkException: A master URL must be set in your configuration
at org.apache.spark.SparkContext.<init>(SparkContext.scala:112)
while locating org.apache.spark.SparkContext
for parameter 1 at controllers.Application.<init>(Application.scala:23)
at controllers.Application.class(Application.scala:23)
while locating controllers.Application
1 error
at com.google.inject.internal.InjectorImpl$2.get(InjectorImpl.java:1025) ~[guice-4.0.jar:na]
at com.google.inject.internal.InjectorImpl.getInstance(InjectorImpl.java:1051) ~[guice-4.0.jar:na]
at play.api.inject.guice.GuiceInjector.instanceOf(GuiceInjectorBuilder.scala:321) ~[play_2.11-2.4.2.jar:2.4.2]
at router.Routes$$anonfun$routes$1$$anonfun$applyOrElse$1$$anonfun$apply$11.apply(Routes.scala:179) ~[na:na]
at router.Routes$$anonfun$routes$1$$anonfun$applyOrElse$1$$anonfun$apply$11.apply(Routes.scala:179) ~[na:na]
Caused by: org.apache.spark.SparkException: A master URL must be set in your configuration
at org.apache.spark.SparkContext.<init>(SparkContext.scala:394) ~[spark-core_2.11-1.5.1.jar:1.5.1]
at org.apache.spark.SparkContext.<init>(SparkContext.scala:112) ~[spark-core_2.11-1.5.1.jar:1.5.1]
at org.apache.spark.SparkContext$$FastClassByGuice$$119109b9.newInstance(<generated>) ~[guice-4.0.jar:1.5.1]
at com.google.inject.internal.cglib.reflect.$FastConstructor.newInstance(FastConstructor.java:40) ~[guice-4.0.jar:na]
at com.google.inject.internal.DefaultConstructionProxyFactory$1.newInstance(DefaultConstructionProxyFactory.java:61) ~[guice-4.0.jar:na]
Generating keys/tokens for this application is tedious.
HI;
I just tried to get started with this template. A sbt run
results in the following error:
Execution exception
[ProvisionException: Unable to provision, see the following errors:
1) Error injecting constructor, java.lang.ExceptionInInitializerError
at controllers.Twitter.<init>(Twitter.scala:13)
while locating controllers.Twitter
for parameter 2 at controllers.Application.<init>(Application.scala:24)
at controllers.Application.class(Application.scala:24)
while locating controllers.Application
1 error]
Use openForce favicon instead of default.
Starting the activator ui extends the sbt build process (mainly adding jvm fork configs). This increases the startup time of the application in some cases or even halts the process.
Run app through BrowserStack. Application must work in standard browsers.
The setup instruction in README does not point out required OAuth settings on apps.twitter.com.
Remove obsolete files and directories.
Activator and Play Framework start the application by default with localhost:9000. The OAuth configuration for a Twitter App requires to set a callback URL. This cannot be localhost but has to be 127.0.0.1 instead. Therefore we have a problem when we want to access cookies which are stored with a different host.
Add emoji unicodes to emoRepl
mapping.
I would suggest to add a default config for a single user mode which uses the already required tokens in the application.conf for the Twitter querying instead of OAuth. That would simplify the scenario where someone just wants to checkout the application locally. It would also mean no callback URL has to be configured with Twitter, which would solve the issue #21 .
For our deployment to a public server we could still set this single user mode flag to false and require OAuth for each user of the application.
Launching sbt run
on Windows (10) fails with:
[info] Compiling with Traceur
Warning: node.js detection failed, sbt will use the Rhino based Trireme JavaScript engine instead to run JavaScript assets compilation, which in some cases may be orders of magnitude slower than using node.js.
[ERROR] [12/11/2015 13:04:57.805] [sbt-web-akka.actor.default-dispatcher-5] [akka://sbt-web/user/$a/process] null
akka.actor.ActorInitializationException: exception during creation
at akka.actor.ActorInitializationException$.apply(Actor.scala:166)
at akka.actor.ActorCell.create(ActorCell.scala:596)
at akka.actor.ActorCell.invokeAll$1(ActorCell.scala:456)
at akka.actor.ActorCell.systemInvoke(ActorCell.scala:478)
at akka.dispatch.Mailbox.processAllSystemMessages(Mailbox.scala:263)
at akka.dispatch.Mailbox.run(Mailbox.scala:219)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:422)
at akka.util.Reflect$.instantiate(Reflect.scala:66)
at akka.actor.ArgsReflectConstructor.produce(Props.scala:352)
at akka.actor.Props.newActor(Props.scala:252)
at akka.actor.ActorCell.newActor(ActorCell.scala:552)
at akka.actor.ActorCell.create(ActorCell.scala:578)
... 7 more
Caused by: java.io.IOException: Cannot run program "node": CreateProcess error=2, The system cannot find the file specified
at java.lang.ProcessBuilder.start(ProcessBuilder.java:1048)
at akka.contrib.process.BlockingProcess.<init>(BlockingProcess.scala:55)
... 16 more
Caused by: java.io.IOException: CreateProcess error=2, The system cannot find the file specified
at java.lang.ProcessImpl.create(Native Method)
at java.lang.ProcessImpl.<init>(ProcessImpl.java:386)
at java.lang.ProcessImpl.start(ProcessImpl.java:137)
at java.lang.ProcessBuilder.start(ProcessBuilder.java:1029)
... 17 more
Public assets folder contains unused components.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.