usc-cloud / goffish Goto Github PK

GML parser ignores newlines, and is prone to deep stack exceptions on malformed GML files. Rewrite of parser is necessary, it may be worth looking into parser generators, but this is likely overkill for now.

Currently We can't look up the subgraph id from the vertex. As a result in Gopher currently we route messages with partition id and remote vertex it to route between sub-graphs.

Ideally subgraph centric programming abstractions should provide the capability to direct communication between subgroups.

Doing this Lookup at Gopher level will be costly. So we need some mapping to do the this look up from the local node.

https://code.google.com/p/kryo/issues/detail?id=113

Different graph sizes, # of partitions, kryo/protobuf, read/write. preference to XDATA graph dataset. Run on Tsangpo.

Sample code
I m getting this at subIt.hasNext()

Iterable instances = sliceManager.readInstances(subgraph,
endTime - 5 * 60 * 1000, endTime, nodeProperties, edgeProperties);

                Iterator<ISubgraphInstance> subIt = instances.iterator();
                if (subIt.hasNext()) {

                   -----------------
               }

Trace

java.lang.NullPointerException
at edu.usc.pgroup.goffish.gofs.slice.FileStorageManager.translateUUIDTo
ile(FileStorageManager.java:39)
at edu.usc.pgroup.goffish.gofs.slice.FileStorageManager.getReadStream(F
leStorageManager.java:23)
at edu.usc.pgroup.goffish.gofs.slice.SliceManager.readPropertyInstances
lice(SliceManager.java:418)
at edu.usc.pgroup.goffish.gofs.slice.InstanceIterator.advanceToNext(Ins
anceIterator.java:69)
at edu.usc.pgroup.goffish.gofs.slice.InstanceIterator.advanceToNext(Ins
anceIterator.java:13)
at edu.usc.pgroup.goffish.gofs.util.AbstractWrapperIterator.hasNext(Abs
ractWrapperIterator.java:34)
at edu.usc.pgroup.goffish.gopher.sample.CarTracer.compute(CarTracer.jav
:81)
at edu.usc.pgroup.floe.applications.gopher.BSPProcessorPellet$GraphTask
unner.run(BSPProcessorPellet.java:352)
at java.util.concurrent.ForkJoinTask$AdaptedRunnable.exec(ForkJoinTask.
ava:1265)
at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:334)
at java.util.concurrent.ForkJoinWorkerThread.execTask(ForkJoinWorkerThr
ad.java:604)
at java.util.concurrent.ForkJoinPool.scan(ForkJoinPool.java:784)
at java.util.concurrent.ForkJoinPool.work(ForkJoinPool.java:646)
at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.j
va:398)

Work with Alok.

Follow up with Sotera about the process to integrate GoFS with the XDATA cluster.

Graph namespace
logical partition name
host name for partition

daemon port number

Start with vertex centric timeseries. Later, subgraph centric timeseries. E.g. License plate detection.

To close finish.

1. documentation - Jython integration (dependencies, compile etc.),
2. Sample code (including Python wrappers for GMLPartition and Slice manager)

Work with Charith, Alok.
Javadoc API, Architecture,

Decision rationale. Docs on tools, commandline, deployment, prereqs, quick start.

There a number of conceptual problems with SliceManager as it exists right now, as well as bugs in how it handles more than one partition, or two properties with the same name. This needs to be fixed ASAP.

We need an integration test for instance roundtripping to/from disk.

Hsuan-Yi to help with this

Support DIMACS graph format as well as GML.

Document our extensions to GML, and how we parse our custom GML files.

We should push the problem of the slice manager trying to group horizontally sliced instances (from GML, etc) and vertically slice them into each property to IInstanceSerializablePartition. Allow IInstanceSerializablePartition to return groups of instances at a time, rather than just one instance, so that the partition implementation is responsible for returning instances that A) can be grouped together B) will all fit in memory together.

Work with Nam and Charith
Perf Comparison of 1-3 external applications, e.g. Vector Corr, Community Detection?

Santosh to take a stab at this. Ability to compile gopher apps using just gofs api jar.

Vertex centric
subgraph centric
Time series

Define what INameNode responsibilities actually are, and implement. Convert metis format to gml after partitionimg. Send request to partitiom daemon to pull gml partitiom files from name node. Create hash for vertices with remote exges.

Review timeseires subgraph APIs. Provide wrappers (utility function) to allow access to vertices and edges for individual instances. Work with Charith.