usc-cloud / goffish Goto Github PK
View Code? Open in Web Editor NEWUSC GoFFish Graph Analytics Framework
USC GoFFish Graph Analytics Framework
GML parser ignores newlines, and is prone to deep stack exceptions on malformed GML files. Rewrite of parser is necessary, it may be worth looking into parser generators, but this is likely overkill for now.
Currently We can't look up the subgraph id from the vertex. As a result in Gopher currently we route messages with partition id and remote vertex it to route between sub-graphs.
Ideally subgraph centric programming abstractions should provide the capability to direct communication between subgroups.
Doing this Lookup at Gopher level will be costly. So we need some mapping to do the this look up from the local node.
Different graph sizes, # of partitions, kryo/protobuf, read/write. preference to XDATA graph dataset. Run on Tsangpo.
Sample code
I m getting this at subIt.hasNext()
Iterable instances = sliceManager.readInstances(subgraph,
endTime - 5 * 60 * 1000, endTime, nodeProperties, edgeProperties);
Iterator<ISubgraphInstance> subIt = instances.iterator();
if (subIt.hasNext()) {
-----------------
}
Trace
java.lang.NullPointerException
at edu.usc.pgroup.goffish.gofs.slice.FileStorageManager.translateUUIDTo
ile(FileStorageManager.java:39)
at edu.usc.pgroup.goffish.gofs.slice.FileStorageManager.getReadStream(F
leStorageManager.java:23)
at edu.usc.pgroup.goffish.gofs.slice.SliceManager.readPropertyInstances
lice(SliceManager.java:418)
at edu.usc.pgroup.goffish.gofs.slice.InstanceIterator.advanceToNext(Ins
anceIterator.java:69)
at edu.usc.pgroup.goffish.gofs.slice.InstanceIterator.advanceToNext(Ins
anceIterator.java:13)
at edu.usc.pgroup.goffish.gofs.util.AbstractWrapperIterator.hasNext(Abs
ractWrapperIterator.java:34)
at edu.usc.pgroup.goffish.gopher.sample.CarTracer.compute(CarTracer.jav
:81)
at edu.usc.pgroup.floe.applications.gopher.BSPProcessorPellet$GraphTask
unner.run(BSPProcessorPellet.java:352)
at java.util.concurrent.ForkJoinTask$AdaptedRunnable.exec(ForkJoinTask.
ava:1265)
at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:334)
at java.util.concurrent.ForkJoinWorkerThread.execTask(ForkJoinWorkerThr
ad.java:604)
at java.util.concurrent.ForkJoinPool.scan(ForkJoinPool.java:784)
at java.util.concurrent.ForkJoinPool.work(ForkJoinPool.java:646)
at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.j
va:398)
Work with Alok.
Follow up with Sotera about the process to integrate GoFS with the XDATA cluster.
Graph namespace
logical partition name
host name for partition
daemon port number
Start with vertex centric timeseries. Later, subgraph centric timeseries. E.g. License plate detection.
To close finish.
Work with Charith, Alok.
Javadoc API, Architecture,
Decision rationale. Docs on tools, commandline, deployment, prereqs, quick start.
There a number of conceptual problems with SliceManager as it exists right now, as well as bugs in how it handles more than one partition, or two properties with the same name. This needs to be fixed ASAP.
We need an integration test for instance roundtripping to/from disk.
Hsuan-Yi to help with this
Support DIMACS graph format as well as GML.
Document our extensions to GML, and how we parse our custom GML files.
We should push the problem of the slice manager trying to group horizontally sliced instances (from GML, etc) and vertically slice them into each property to IInstanceSerializablePartition. Allow IInstanceSerializablePartition to return groups of instances at a time, rather than just one instance, so that the partition implementation is responsible for returning instances that A) can be grouped together B) will all fit in memory together.
Work with Nam and Charith
Perf Comparison of 1-3 external applications, e.g. Vector Corr, Community Detection?
Santosh to take a stab at this. Ability to compile gopher apps using just gofs api jar.
Vertex centric
subgraph centric
Time series
Define what INameNode responsibilities actually are, and implement. Convert metis format to gml after partitionimg. Send request to partitiom daemon to pull gml partitiom files from name node. Create hash for vertices with remote exges.
Review timeseires subgraph APIs. Provide wrappers (utility function) to allow access to vertices and edges for individual instances. Work with Charith.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.