GoFFish Framework Development
dream-lab / goffish_v2 Goto Github PK
View Code? Open in Web Editor NEWPublic releases of GoFFish Distributed Graph Processing Platforms
License: Apache License 2.0
goffish_v2's Introduction
goffish_v2's People
Contributors
Stargazers
Watchers
Forkers
raksha0704goffish_v2's Issues
Algorithm: Bipartite graph
Pregel: PR, SSSP, Bipartite, Semi-clustering,
Widom VLDB 2014: SCC, WCC, MSF, Coloring, Max Weight Matching
Yan VLDB 2014: WCC, SCC, Bi-connected Components, Spanning tree, Euler tour
Quick ASNAM 2012: CC, Diameter est, Avg path length, k-cores, triangle counting, clustering coeff, k-truss finding,
Han VLDB 2014: PR, SSSP, WCC, MST
Hong TechReport 2012: PR, Conductance, SSSP, Bipartite, Appox BC
Algorithm: Approximate BC
Pregel: PR, SSSP, Bipartite, Semi-clustering,
Widom VLDB 2014: SCC, WCC, MSF, Coloring, Max Weight Matching
Yan VLDB 2014: WCC, SCC, Bi-connected Components, Spanning tree, Euler tour
Quick ASNAM 2012: CC, Diameter est, Avg path length, k-cores, triangle counting, clustering coeff, k-truss finding,
Han VLDB 2014: PR, SSSP, WCC, MST
Hong TechReport 2012: PR, Conductance, SSSP, Bipartite, Appox BC
Algorithm: Semi-clustering
Pregel: PR, SSSP, Bipartite, Semi-clustering,
Widom VLDB 2014: SCC, WCC, MSF, Coloring, Max Weight Matching
Yan VLDB 2014: WCC, SCC, Bi-connected Components, Spanning tree, Euler tour
Quick ASNAM 2012: CC, Diameter est, Avg path length, k-cores, triangle counting, clustering coeff, k-truss finding,
Han VLDB 2014: PR, SSSP, WCC, MST
Hong TechReport 2012: PR, Conductance, SSSP, Bipartite, Appox BC
GoFS: INameNode should return enhanced DataNode object
getDataNodes():
DataNode should have a logical ID, hostname (for remote data node), service endpoint (for remote data node) in addition to URI.
Algorithm: Clustering Coefficient
Pregel: PR, SSSP, Bipartite, Semi-clustering,
Widom VLDB 2014: SCC, WCC, MSF, Coloring, Max Weight Matching
Yan VLDB 2014: WCC, SCC, Bi-connected Components, Spanning tree, Euler tour
Quick ASNAM 2012: CC, Diameter est, Avg path length, k-cores, triangle counting, clustering coeff, k-truss finding,
Han VLDB 2014: PR, SSSP, WCC, MST
Hong TechReport 2012: PR, Conductance, SSSP, Bipartite, Appox BC
Algorithm: Bi-connected Components
Pregel: PR, SSSP, Bipartite, Semi-clustering,
Widom VLDB 2014: SCC, WCC, MSF, Coloring, Max Weight Matching
Yan VLDB 2014: WCC, SCC, Bi-connected Components, Spanning tree, Euler tour
Quick ASNAM 2012: CC, Diameter est, Avg path length, k-cores, triangle counting, clustering coeff, k-truss finding,
Han VLDB 2014: PR, SSSP, WCC, MST
Hong TechReport 2012: PR, Conductance, SSSP, Bipartite, Appox BC
Launch Gopher worker on alternate ports
Launch Gopher workers on alternate ports so that multiplle gopher deployments can be present
Consider HAMA as execution engine for GoFFish
Algorithm: Max Weight Matching
Pregel: PR, SSSP, Bipartite, Semi-clustering,
Widom VLDB 2014: SCC, WCC, MSF, Coloring, Max Weight Matching
Yan VLDB 2014: WCC, SCC, Bi-connected Components, Spanning tree, Euler tour
Quick ASNAM 2012: CC, Diameter est, Avg path length, k-cores, triangle counting, clustering coeff, k-truss finding,
Han VLDB 2014: PR, SSSP, WCC, MST
Hong TechReport 2012: PR, Conductance, SSSP, Bipartite, Appox BC
Add log output to GoFS load
Create log file of loading progress
Algorithm: Triangle Counting
Pregel: PR, SSSP, Bipartite, Semi-clustering,
Widom VLDB 2014: SCC, WCC, MSF, Coloring, Max Weight Matching
Yan VLDB 2014: WCC, SCC, Bi-connected Components, Spanning tree, Euler tour
Quick ASNAM 2012: CC, Diameter est, Avg path length, k-cores, triangle counting, clustering coeff, k-truss finding,
Han VLDB 2014: PR, SSSP, WCC, MST
Hong TechReport 2012: PR, Conductance, SSSP, Bipartite, Appox BC
Algorithm: k-cores
Pregel: PR, SSSP, Bipartite, Semi-clustering,
Widom VLDB 2014: SCC, WCC, MSF, Coloring, Max Weight Matching
Yan VLDB 2014: WCC, SCC, Bi-connected Components, Spanning tree, Euler tour
Quick ASNAM 2012: CC, Diameter est, Avg path length, k-cores, triangle counting, clustering coeff, k-truss finding,
Han VLDB 2014: PR, SSSP, WCC, MST
Hong TechReport 2012: PR, Conductance, SSSP, Bipartite, Appox BC
Add init method on BSPProcessor
init() should be called once for partition after loading the partition
Algorithm: KMeans Clustering
Pregel: PR, SSSP, Bipartite, Semi-clustering,
Widom VLDB 2014: SCC, WCC, MSF, Coloring, Max Weight Matching
Yan VLDB 2014: WCC, SCC, Bi-connected Components, Spanning tree, Euler tour
Quick ASNAM 2012: CC, Diameter est, Avg path length, k-cores, triangle counting, clustering coeff, k-truss finding,
Han VLDB 2014: PR, SSSP, WCC, MST
Hong TechReport 2012: PR, Conductance, SSSP, Bipartite, Appox BC
Algorithm: Graph Coloring
Pregel: PR, SSSP, Bipartite, Semi-clustering,
Widom VLDB 2014: SCC, WCC, MSF, Coloring, Max Weight Matching
Yan VLDB 2014: WCC, SCC, Bi-connected Components, Spanning tree, Euler tour
Quick ASNAM 2012: CC, Diameter est, Avg path length, k-cores, triangle counting, clustering coeff, k-truss finding,
Han VLDB 2014: PR, SSSP, WCC, MST
Hong TechReport 2012: PR, Conductance, SSSP, Bipartite, Appox BC
Algorithm: Spanning Tree
Pregel: PR, SSSP, Bipartite, Semi-clustering,
Widom VLDB 2014: SCC, WCC, MSF, Coloring, Max Weight Matching
Yan VLDB 2014: WCC, SCC, Bi-connected Components, Spanning tree, Euler tour
Quick ASNAM 2012: CC, Diameter est, Avg path length, k-cores, triangle counting, clustering coeff, k-truss finding,
Han VLDB 2014: PR, SSSP, WCC, MST
Hong TechReport 2012: PR, Conductance, SSSP, Bipartite, Appox BC
Algorithm: Conductance
Pregel: PR, SSSP, Bipartite, Semi-clustering,
Widom VLDB 2014: SCC, WCC, MSF, Coloring, Max Weight Matching
Yan VLDB 2014: WCC, SCC, Bi-connected Components, Spanning tree, Euler tour
Quick ASNAM 2012: CC, Diameter est, Avg path length, k-cores, triangle counting, clustering coeff, k-truss finding,
Han VLDB 2014: PR, SSSP, WCC, MST
Hong TechReport 2012: PR, Conductance, SSSP, Bipartite, Appox BC
Algorithm: Euler Tour
Pregel: PR, SSSP, Bipartite, Semi-clustering,
Widom VLDB 2014: SCC, WCC, MSF, Coloring, Max Weight Matching
Yan VLDB 2014: WCC, SCC, Bi-connected Components, Spanning tree, Euler tour
Quick ASNAM 2012: CC, Diameter est, Avg path length, k-cores, triangle counting, clustering coeff, k-truss finding,
Han VLDB 2014: PR, SSSP, WCC, MST
Hong TechReport 2012: PR, Conductance, SSSP, Bipartite, Appox BC
Algorithm: Diameter, Average Path Length, Other Graph Statistics
Pregel: PR, SSSP, Bipartite, Semi-clustering,
Widom VLDB 2014: SCC, WCC, MSF, Coloring, Max Weight Matching
Yan VLDB 2014: WCC, SCC, Bi-connected Components, Spanning tree, Euler tour
Quick ASNAM 2012: CC, Diameter est, Avg path length, k-cores, triangle counting, clustering coeff, k-truss finding,
Han VLDB 2014: PR, SSSP, WCC, MST
Hong TechReport 2012: PR, Conductance, SSSP, Bipartite, Appox BC
Message passing fails when sending messages to all partitions
When sending messages to every partition, each subgraph should receive exactly the same number of messages as the subgraph count in the graph. The following code performs this test for large number of supersteps:
public class TestGoffish extends GopherSubGraph {
int subgraphCount = 5, iterations = 1, maxIterations = 500;
boolean testFail = false;
String reason = new String();
@Override
public void compute(List<SubGraphMessage> messageList) {
System.out.println("Superstep " + getSuperStep());
if (iterations == maxIterations) {
System.out.println("Test passed!");
voteToHalt();
return;
}
String s = new String();
for (int i = 0; i < 100; i++) s += "1";
if (getSuperStep() % 2 == 0) {
SubGraphMessage m = new SubGraphMessage(s.getBytes());
for (long part : partitions)
sendMessage(part, m);
}
else {
for (SubGraphMessage messageItem : messageList) {
String ms = new String(messageItem.getData());
if (!s.equals(ms)) {
testFail = true;
reason = "Received wrong message";
}
}
if (messageList.size() != subgraphCount) {
testFail = true;
reason = "Received unequal message " + messageList.size();
}
}
if (testFail) {
System.out.println("Test failed!");
System.out.println("Reason: " + reason);
System.exit(-1);
}
iterations++;
}
}
I have tested for few graphs including fbgraph and google undirected graph. For small graphs it takes a large number of supersteps (48 for fbgraph) to fail but for large graphs the number of superstep can be low (3 for google-undirected). In all the runs, it received less number of messages than expected and the content of the message was correct.
It can be tested for any graph by changing the subgraphCount accordingly.
Algorithm: Min Spanning Forest
Pregel: PR, SSSP, Bipartite, Semi-clustering,
Widom VLDB 2014: SCC, WCC, MSF, Coloring, Max Weight Matching
Yan VLDB 2014: WCC, SCC, Bi-connected Components, Spanning tree, Euler tour
Quick ASNAM 2012: CC, Diameter est, Avg path length, k-cores, triangle counting, clustering coeff, k-truss finding,
Han VLDB 2014: PR, SSSP, WCC, MST
Hong TechReport 2012: PR, Conductance, SSSP, Bipartite, Appox BC
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.