neo4j-graph-analytics / ml-models Goto Github PK

View Code? Open in Web Editor NEW

64.0 14.0 23.0 3.12 MB

Machine Learning Procedures and Functions for Neo4j

Home Page: https://github.com/neo4j-graph-analytics/ml-models/releases/tag/1.0.0

License: Apache License 2.0

Java 100.00%

analytics data-science machine-learning linear-regression graph-database neo4j cypher

ml-models's People

Contributors

Stargazers

Watchers

ml-models's Issues

failed to invoke procedure embedding.deepgl

using version 1.0.2 and running this cypher code:
CALL embedding.deepgl("Address","PAYS", {
nodeFeatures: ['pagerank'],
iterations: 2
})
This error occurs >>
Neo.ClientError.Procedure.ProcedureCallFailed: Failed to invoke procedure embedding.deepgl: Caused by: java.lang.NoSuchMethodError: org.neo4j.graphalgo.core.IdMap.buildMappedIds()V

Not sure what it is!? Config file is updated for the unrestricted algo., embedding.

I tried the latest verions 1.0.3 and get a completely different error:
Neo.ClientError.Procedure.ProcedureCallFailed: Failed to invoke procedure embedding.deepgl: Caused by: java.lang.UnsatisfiedLinkError: no openblas in java.library.path

I am using neo4j version 3.5.5
happy to hear anyones ideas on what i might be doing wrong here
Many thanks!

Is this project dead?

We are considering using ml-models on Neo4j itself, instead of loading data into GraphX, for our graph node embedding implementation.

I noticed that the code base has not been update in 2 years, does that mean that the project is discontinued? If so, is it deprecated in favor of some alternative library or approach, or is it dropped altogether?

Maven fails to compile

When compiling the library from source I get the following three errors:

Failed tests:   shouldOnlyKeepFeaturesThatAddSomethingUnique(embedding.PruningTest): expected:<[9.0000, (..)
  pruneEmbeddingsAndFeatures(embedding.PruningTest): arrays first differed at element [3]; expected:<MEAN_OUT_NEIGHBOURHOOD> but was:<MEAN_BOTH_NEIGHBOURHOOD>
  makeModel(regression.LogisticTest): unable to read csv file for test data

I was able to solve the last error myself. Any idea about the first two errors? Seems that's the reason why the embedding features don't work...

Thanks, Andrej

unit test uses a local file not existing in the repo

see

ml-models/src/test/java/regression/LogisticTest.java

Line 41 in 4374056

 String csvFile = "/Users/laurenshin/documents/linreg-graph-analytics/src/test/resources/iris-full.csv"; 

Build failures with incorrect procedure names

Results :

Failed tests: shouldOnlyKeepFeaturesThatAddSomethingUnique(embedding.PruningTest): expected:<[9.0000, (..)
pruneEmbeddingsAndFeatures(embedding.PruningTest): arrays first differed at element [3]; expected:<MEAN_OUT_NEIGHBOURHOOD> but was:<MEAN_BOTH_NEIGHBOURHOOD>
makeModel(regression.LogisticTest): unable to read csv file for test data

Tests in error:
stream(embedding.DeepGLIntegrationTest): There is no procedure with the name algo.deepgl.stream registered for this database instance. Please ensure you've spelled the procedure name correctly and that the procedure is properly deployed.
write(embedding.DeepGLIntegrationTest): There is no procedure with the name algo.deepgl registered for this database instance. Please ensure you've spelled the procedure name correctly and that the procedure is properly deployed.

Tests run: 33, Failures: 3, Errors: 2, Skipped: 0

[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 16.798 s
[INFO] Finished at: 2018-10-20T21:40:28-04:00
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-surefire-plugin:2.12.4:test (default-test) on project neo4j-ml-models: There are test failures.
[ERROR]
[ERROR] Please refer to /Users/bradnussbaum/git/neo4j-graph-analytics/ml-models/target/surefire-reports for the individual test results.
[ERROR] -> [Help 1]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException

DeepGL OOM

running on machine with 16gb RAM and using dataset with 8k nodes and 26k rels : https://snap.stanford.edu/data/p2p-Gnutella09.html

import:

LOAD CSV FROM
"file:///p2p-Gnutella09.txt" as row fieldterminator ' '
WITH row SKIP 4
MERGE (h1:Host{id:row[0]})
MERGE (h2:Host{id:row[1]})
MERGE (h1)-[:CONNECTION]->(h2)

Run algo:

CALL embedding.deepgl("Host","CONNECTION", {
iterations: 4,
writePropery:'fourth'
})

Error:

Neo.ClientError.Procedure.ProcedureCallFailed: Failed to invoke procedure embedding.deepgl: Caused by: java.lang.OutOfMemoryError: Physical memory usage is too high: physicalBytes = 7G > maxPhysicalBytes = 7G

DeepGL: Could not initialize class org.nd4j.linalg.factory.Nd4j

Hello! I am trying to run the example in: https://tbgraph.wordpress.com/2018/10/17/feature-extraction-on-a-peer-to-peer-network-with-deepgl-embedding-and-neo4j/

This example uses the Gnutella dataset.

At some point, the DeepGL algorithm is called:

CALL embedding.deepgl("Host","CONNECTION", {
nodeFeatures: ['pagerank'],
iterations: 2
})

But it returns the following error:

Neo.ClientError.Procedure.ProcedureCallFailed: Failed to invoke procedure embedding.deepgl: Caused by: java.lang.NoClassDefFoundError: Could not initialize class org.nd4j.linalg.factory.Nd4j

I have tried putting the .jar inside the plugins folder, also, manually compiled the source and tried that. But something is not right.

@meltzerpete is there something that I am missing here?

Thanks for the help 👍

DeepGL Invalid shape error

Load data.

:play movies

Run algorithm:

CALL embedding.deepgl("Person" ,null)

I get an error:

Neo.ClientError.Procedure.ProcedureCallFailed: Failed to invoke procedure embedding.deepgl: Caused by: org.nd4j.linalg.exception.ND4JIllegalStateException: Invalid shape: Requested INDArray shape [0, 54] contains dimension size values < 1 (all dimensions must be 1 or more)

Failed to install with neo4j-desktop project

I downloaded neo4j-ml-models-1.0.1.jar file and copied it into the plugin directory of my neo4j-desktop project's database, when starting the database, I get the following error in my log:

2019-09-03 02:03:44.664+0000 INFO  ======== Neo4j 3.5.6 ========
2019-09-03 02:03:44.671+0000 INFO  Starting...
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/F:/neo4j/neo4jDatabases/database-d83a37c7-69f1-4177-8857-3051b633649e/installation-3.5.6/plugins/neo4j-ml-models-1.0.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/F:/neo4j/neo4jDatabases/database-d83a37c7-69f1-4177-8857-3051b633649e/installation-3.5.6/lib/slf4j-nop-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
2019-09-03 02:03:45.771+0000 INFO  Initiating metrics...
2019-09-03 02:04:02.368+0000 INFO  Sending metrics to CSV file at F:\neo4j\neo4jDatabases\database-d83a37c7-69f1-4177-8857-3051b633649e\installation-3.5.6\metrics
2019-09-03 02:04:03.481+0000 INFO  Bolt enabled on 127.0.0.1:7687.
2019-09-03 02:04:05.036+0000 INFO  Started.
2019-09-03 02:04:05.120+0000 INFO  Mounted REST API at: /db/manage
2019-09-03 02:04:05.189+0000 INFO  Server thread metrics have been registered successfully
2019-09-03 02:04:05.459+0000 INFO  Stopping...
2019-09-03 02:04:06.539+0000 INFO  Stopped.
2019-09-03 02:04:06.540+0000 ERROR Failed to start Neo4j: Starting Neo4j failed: Component 'org.neo4j.server.AbstractNeoServer$ServerComponentsLifecycleAdapter@62398a19' was successfully initialized, but failed to start. Please see the attached cause exception "com.sun.jersey.core.reflection.ReflectionHelper.classForNamePA(Ljava/lang/String;)Ljava/security/PrivilegedAction;". Starting Neo4j failed: Component 'org.neo4j.server.AbstractNeoServer$ServerComponentsLifecycleAdapter@62398a19' was successfully initialized, but failed to start. Please see the attached cause exception "com.sun.jersey.core.reflection.ReflectionHelper.classForNamePA(Ljava/lang/String;)Ljava/security/PrivilegedAction;".
org.neo4j.server.ServerStartupException: Starting Neo4j failed: Component 'org.neo4j.server.AbstractNeoServer$ServerComponentsLifecycleAdapter@62398a19' was successfully initialized, but failed to start. Please see the attached cause exception "com.sun.jersey.core.reflection.ReflectionHelper.classForNamePA(Ljava/lang/String;)Ljava/security/PrivilegedAction;".
	at org.neo4j.server.exception.ServerStartupErrors.translateToServerStartupError(ServerStartupErrors.java:45)
	at org.neo4j.server.AbstractNeoServer.start(AbstractNeoServer.java:187)
	at org.neo4j.server.ServerBootstrapper.start(ServerBootstrapper.java:124)
	at org.neo4j.server.ServerBootstrapper.start(ServerBootstrapper.java:91)
	at com.neo4j.server.enterprise.CommercialEntryPoint.main(CommercialEntryPoint.java:22)
Caused by: org.neo4j.kernel.lifecycle.LifecycleException: Component 'org.neo4j.server.AbstractNeoServer$ServerComponentsLifecycleAdapter@62398a19' was successfully initialized, but failed to start. Please see the attached cause exception "com.sun.jersey.core.reflection.ReflectionHelper.classForNamePA(Ljava/lang/String;)Ljava/security/PrivilegedAction;".
	at org.neo4j.kernel.lifecycle.LifeSupport$LifecycleInstance.start(LifeSupport.java:473)
	at org.neo4j.kernel.lifecycle.LifeSupport.start(LifeSupport.java:111)
	at org.neo4j.server.AbstractNeoServer.start(AbstractNeoServer.java:180)
	... 3 more
Caused by: java.lang.NoSuchMethodError: com.sun.jersey.core.reflection.ReflectionHelper.classForNamePA(Ljava/lang/String;)Ljava/security/PrivilegedAction;
	at com.sun.jersey.server.impl.modelapi.annotation.IntrospectionModeller.workOutPostConstructPreDestroy(IntrospectionModeller.java:225)
	at com.sun.jersey.server.impl.modelapi.annotation.IntrospectionModeller.createResource(IntrospectionModeller.java:132)
	at com.sun.jersey.server.impl.application.WebApplicationImpl.getAbstractResource(WebApplicationImpl.java:769)
	at com.sun.jersey.server.impl.application.WebApplicationImpl.createAbstractResourceModelStructures(WebApplicationImpl.java:1595)
	at com.sun.jersey.server.impl.application.WebApplicationImpl._initiate(WebApplicationImpl.java:1356)
	at com.sun.jersey.server.impl.application.WebApplicationImpl.access$700(WebApplicationImpl.java:180)
	at com.sun.jersey.server.impl.application.WebApplicationImpl$13.f(WebApplicationImpl.java:799)
	at com.sun.jersey.server.impl.application.WebApplicationImpl$13.f(WebApplicationImpl.java:795)
	at com.sun.jersey.spi.inject.Errors.processWithErrors(Errors.java:193)
	at com.sun.jersey.server.impl.application.WebApplicationImpl.initiate(WebApplicationImpl.java:795)
	at com.sun.jersey.server.impl.application.WebApplicationImpl.initiate(WebApplicationImpl.java:790)
	at com.sun.jersey.spi.container.servlet.ServletContainer.initiate(ServletContainer.java:509)
	at com.sun.jersey.spi.container.servlet.ServletContainer$InternalWebComponent.initiate(ServletContainer.java:339)
	at com.sun.jersey.spi.container.servlet.WebComponent.load(WebComponent.java:605)
	at com.sun.jersey.spi.container.servlet.WebComponent.init(WebComponent.java:207)
	at com.sun.jersey.spi.container.servlet.ServletContainer.init(ServletContainer.java:394)
	at com.sun.jersey.spi.container.servlet.ServletContainer.init(ServletContainer.java:577)
	at javax.servlet.GenericServlet.init(GenericServlet.java:241)
	at org.eclipse.jetty.servlet.ServletHolder.initServlet(ServletHolder.java:671)
	at org.eclipse.jetty.servlet.ServletHolder.initialize(ServletHolder.java:428)
	at org.eclipse.jetty.servlet.ServletHandler.lambda$initialize$0(ServletHandler.java:750)
	at java.util.stream.SortedOps$SizedRefSortingSink.end(SortedOps.java:352)
	at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482)
	at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:471)
	at java.util.stream.StreamSpliterators$WrappingSpliterator.forEachRemaining(StreamSpliterators.java:312)
	at java.util.stream.Streams$ConcatSpliterator.forEachRemaining(Streams.java:743)
	at java.util.stream.Streams$ConcatSpliterator.forEachRemaining(Streams.java:742)
	at java.util.stream.ReferencePipeline$Head.forEach(ReferencePipeline.java:580)
	at org.eclipse.jetty.servlet.ServletHandler.initialize(ServletHandler.java:744)
	at org.eclipse.jetty.servlet.ServletContextHandler.startContext(ServletContextHandler.java:369)
	at org.eclipse.jetty.server.handler.ContextHandler.doStart(ContextHandler.java:852)
	at org.eclipse.jetty.servlet.ServletContextHandler.doStart(ServletContextHandler.java:278)
	at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
	at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:167)
	at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:119)
	at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:113)
	at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
	at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:167)
	at org.eclipse.jetty.server.Server.start(Server.java:418)
	at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:110)
	at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:113)
	at org.eclipse.jetty.server.Server.doStart(Server.java:382)
	at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
	at org.neo4j.server.web.Jetty9WebServer.startJetty(Jetty9WebServer.java:329)
	at org.neo4j.server.web.Jetty9WebServer.start(Jetty9WebServer.java:159)
	at org.neo4j.server.AbstractNeoServer.startWebServer(AbstractNeoServer.java:290)
	at org.neo4j.server.AbstractNeoServer$ServerComponentsLifecycleAdapter.start(AbstractNeoServer.java:543)
	at org.neo4j.kernel.lifecycle.LifeSupport$LifecycleInstance.start(LifeSupport.java:452)
	... 5 more
2019-09-03 02:04:06.541+0000 INFO  Neo4j Server shutdown initiated by request

If I delete the neo4j-ml-models-1.0.1.jar in the database's plugin folder, the database can start successfully. How can I fix this? Thanks!

Confusion on embedding.deepgl adaptation

There exists a 1.0.2 tag, but the project seems to only build a 1.0.1 version of the jar file. Also, whether I download the existing 1.0.1 release or build the 1.0.2 tagged version of the code, then upload the jar file into the plugins folder, and update the neo4j.conf file as such:

dbms.security.procedures.whitelist=regression.*,embedding.*
dbms.security.procedures.unrestricted=regression.*,embedding.*

I'm getting a ProcedureNotFound error. It seems as if others are able to utilize the embedding.opengl procedures with some degree of success. I'm genuinely curious as to what I'm doing wrong.

Embedding size with embedding.deepgl

Hi,

I try to figure out how to properly set the number of desired features with embedding.deepgl algorithms. Currently, my setup goes as follows:

CALL
embedding.deepgl(
    null,
    null,
    {
         pruningLambda: 0.6,
         diffusions: 3,
         iterations: 3,
         embeddingSize: 128
    }
)

I explicitely set embedding size to 128, however the resulting size is 158. Could somebody please explain why?

Thanks, Andrej

DeepGL algo doesnt seem to finish

Load data:

:play movies

Run algorithm:

CALL embedding.deepgl(null ,null)

Inspecting the logs it seems that algorithm get quickly to:

2018-10-06 08:03:38.268+0000 INFO  [neo4j.BoltWorker-3 [bolt] [/127.0.0.1:54242] ] DeepGL 100% Size of combined embedding: [171, 28379]
2018-10-06 08:03:38.268+0000 INFO  [neo4j.BoltWorker-3 [bolt] [/127.0.0.1:54242] ] DeepGL 100% Number of prev features: 767
2018-10-06 08:03:38.268+0000 INFO  [neo4j.BoltWorker-3 [bolt] [/127.0.0.1:54242] ] DeepGL 100% Creating AdjacencyMatrix
2018-10-06 08:03:38.268+0000 INFO  [neo4j.BoltWorker-3 [bolt] [/127.0.0.1:54242] ] DeepGL 100% Adding columns to array
2018-10-06 08:03:38.268+0000 INFO  [neo4j.BoltWorker-3 [bolt] [/127.0.0.1:54242] ] DeepGL 100% Added columns to array
2018-10-06 08:03:38.268+0000 INFO  [neo4j.BoltWorker-3 [bolt] [/127.0.0.1:54242] ] DeepGL 100% Populating adjacency matrix 2

but the next step doesn't finish more than 10 minutes after.

Could not initialize class org.nd4j.linalg.factory.Nd4j

When copying the 3rd release .jar file to plugins folder and setting the the config:
dbms.security.procedures.whitelist=regression., embedding.
dbms.security.procedures.unrestricted=regression., embedding.
(running on neo4j 3.5)

The procedures embedding.deepWalk and embedding.deepgl
cannot be run in the neo4j browser due to the error:
neo4j Could not initialize class org.nd4j.linalg.factory.Nd4j

Seems like the compiled file lacks some dependcies.

Sorry I cannot build myself as I am not Java programmer.

Anybody has same issues?
Thx.

length of resulting embedding

Using same data as in #6
data:
https://snap.stanford.edu/data/p2p-Gnutella09.html

import:

LOAD CSV FROM
"file:///p2p-Gnutella09.txt" as row fieldterminator ' '
WITH row SKIP 4
MERGE (h1:Host{id:row[0]})
MERGE (h2:Host{id:row[1]})
MERGE (h1)-[:CONNECTION]->(h2)

Run algo with 2 iterations and pagerank node property:

CALL embedding.deepgl("Host","CONNECTION", {
nodeFeatures: ['pagerank'],
iterations: 2
})

Length of the resulting array for each node is 451 according to:

MATCH (n:Host)
WITH n limit 1
RETURN length(n.embedding)

That seems quite a lot with only 2 iterations and thus calculating cosine similarity for 8k nodes can take 5+ minutes on a 16gb and the whole process on a big graph would be quite slow for now.

Neo4j ml-models Neo.ClientError.Procedure.ProcedureRegistrationFailed

Hi I have downloaded the latest jar file and updated noe4j.config but I am still getting the following error when calling embedding.deepgl...how can I fix it?

Neo.ClientError.Procedure.ProcedureRegistrationFailed: embedding.deepgl is unavailable because it is sandboxed and has dependencies outside of the sandbox. Sandboxing is controlled by the dbms.security.procedures.unrestricted setting. Only unrestrict procedures you can trust with access to database internals.

How to develope this solution in python?

I have read this article and couldn't found any other source to solve regression or any other ML algorithm on graph data.
Could you please guide me how to use regression on graph data from scratch. I am using postgresql and I can not use Neo4j.
Thank you very much.

neo4j-graph-analytics / ml-models Goto Github PK

ml-models's People

Contributors

Stargazers

Watchers

Forkers

ml-models's Issues

Recommend Projects

Recommend Topics

Recommend Org