Logistic Regression with LBFGS in Spark 1.6 and 2.1 about spark-py-notebooks HOT 4 OPEN

Hi @colbec,

Sorry about the late reply, and thanks for having a look at the repos. I'm afraid I didn't have the time to update the repos and user a more recent version of Spark, so I can't reproduce your results. They look ver interesting indeed!

Hopefully somebody else will be able to do it and will let us know :)

But please, keep us up to date with your findings.

from spark-py-notebooks.

Hi @jadianes

No problem, Sir - unfortunately I have not been able to show any merit in the results I get with the Spark community. My post with Apache JIRA mentioned above has been dismissed as "not a problem." The result is that I for one will be using 1.6.2 for Logistic Regression since (for me, anyway) it is 3 times faster than 2.1.0 and produces a more attractive result.

from spark-py-notebooks.

For those following along, I have found some discussions related to mllib (http://apache-spark-developers-list.1001551.n3.nabble.com/Switch-RDD-based-MLlib-APIs-to-maintenance-mode-in-Spark-2-0-td17033.html) which indicate that from the Spark 2.x onwards, there is to be a preference for the newer "ml" library over the "mllib" which will be going to maintenance only mode. With much legacy tutorial support out in the wild using mllib there will inevitably be a period of transition where we just have to be a bit careful.

from spark-py-notebooks.

interactions_df = sqlContext.createDataFrame(row_data)

when i run the above in pyspark shell , I get the below error.Not able to figure out the issue here.
Please advise.

Caused by: ERROR XJ040: Failed to start database 'metastore_db' with class loade
r org.apache.spark.sql.hive.client.IsolatedClientLoader$$anon$1@2734d50, see the
next exception for details.
at org.apache.derby.iapi.error.StandardException.newException(Unknown So
urce)
at org.apache.derby.impl.jdbc.SQLExceptionFactory.wrapArgsForTransportAc
rossDRDA(Unknown Source)
... 97 more
Caused by: ERROR XSDB6: Another instance of Derby may have already booted the da
tabase C:\spark\metastore_db.
at org.apache.derby.iapi.error.StandardException.newException(Unknown So
urce)
at org.apache.derby.iapi.error.StandardException.newException(Unknown So
urce)
at org.apache.derby.impl.store.raw.data.BaseDataFileFactory.privGetJBMSL
ockOnDB(Unknown Source)
at org.apache.derby.impl.store.raw.data.BaseDataFileFactory.run(Unknown
Source)
at java.security.AccessController.doPrivileged(Native Method)
at org.apache.derby.impl.store.raw.data.BaseDataFileFactory.getJBMSLockO
nDB(Unknown Source)
at org.apache.derby.impl.store.raw.data.BaseDataFileFactory.boot(Unknown
Source)
at org.apache.derby.impl.services.monitor.BaseMonitor.boot(Unknown Sourc
e)
at org.apache.derby.impl.services.monitor.TopService.bootModule(Unknown
Source)
at org.apache.derby.impl.services.monitor.BaseMonitor.startModule(Unknow
n Source)
at org.apache.derby.impl.services.monitor.FileMonitor.startModule(Unknow
n Source)
at org.apache.derby.iapi.services.monitor.Monitor.bootServiceModule(Unkn
own Source)
at org.apache.derby.impl.store.raw.RawStore.boot(Unknown Source)
at org.apache.derby.impl.services.monitor.BaseMonitor.boot(Unknown Sourc
e)
at org.apache.derby.impl.services.monitor.TopService.bootModule(Unknown
Source)
at org.apache.derby.impl.services.monitor.BaseMonitor.startModule(Unknow
n Source)
at org.apache.derby.impl.services.monitor.FileMonitor.startModule(Unknow
n Source)
at org.apache.derby.iapi.services.monitor.Monitor.bootServiceModule(Unkn
own Source)
at org.apache.derby.impl.store.access.RAMAccessManager.boot(Unknown Sour
ce)
at org.apache.derby.impl.services.monitor.BaseMonitor.boot(Unknown Sourc
e)
at org.apache.derby.impl.services.monitor.TopService.bootModule(Unknown
Source)
at org.apache.derby.impl.services.monitor.BaseMonitor.startModule(Unknow
n Source)
at org.apache.derby.impl.services.monitor.FileMonitor.startModule(Unknow
n Source)
at org.apache.derby.iapi.services.monitor.Monitor.bootServiceModule(Unkn
own Source)
at org.apache.derby.impl.db.BasicDatabase.bootStore(Unknown Source)
at org.apache.derby.impl.db.BasicDatabase.boot(Unknown Source)
at org.apache.derby.impl.services.monitor.BaseMonitor.boot(Unknown Sourc
e)
at org.apache.derby.impl.services.monitor.TopService.bootModule(Unknown
Source)
at org.apache.derby.impl.services.monitor.BaseMonitor.bootService(Unknow
n Source)
at org.apache.derby.impl.services.monitor.BaseMonitor.startProviderServi
ce(Unknown Source)
at org.apache.derby.impl.services.monitor.BaseMonitor.findProviderAndSta
rtService(Unknown Source)
at org.apache.derby.impl.services.monitor.BaseMonitor.startPersistentSer
vice(Unknown Source)
at org.apache.derby.iapi.services.monitor.Monitor.startPersistentService
(Unknown Source)
... 94 more

from spark-py-notebooks.