okkevaneck / distributed_systems_lab Goto Github PK

Use some library or something to perform graph analysis

Vertex IDs in Graphalytics dataset are actually random number, not just start from some number n and end with n + n_vertices, which leads to the bug in the converter.

Multiple functions need the decorator.

graph.spread should probably be measured

• list how many nodes has been burned per X time units

• Compute node should measure how many messages are sent, how many messages are received.

• Head node should keep track of when heartbeats are received, how many nodes are burned per heartbeat.

Head Node is responsible for

• Stitching together all graph sampled from compute nodes
• listening for heartbeats
• • send kill signal to compute nodes to stop burning
• writing resulting graph to a file

Issue to implement MPI communication in compute node classes

• Implement a thread to listen for communication from other compute nodes
• Implement a thread to listen for communication from the HEAD node telling the compute node to stop
• Implement a function to send the head node heartbeats that communicate how many nodes have been sampled by the compute node
• Implement function to listen for restart call

Because we will have precomputed partitions with clustering, we will need each compute node to handle multiple partitions.

(I.e 8 partitions for a graph & 2 compute nodes means each compute node gets 4 partitions)

This task is to implement that functionality

for faster set_vertex_status calls
and faster get_vertex_status calls

shouldn't these functions be in the vertex class anyway?
What am I doing?

When ComputeNode is blocking on send (heartbeat), after the head send a kill request. The head will never accept that send and thus the ComputeNode wont stop by it self.

We should read 2 graph files.
a .v file, and a .e file.

Changes to graph interpreter and Graph.add_vertex_and_neighbor

It's pretty underperformant right now

To upsacle X2

run downscaling to (0.5) algorithm 4 times
Then stitch together all (0.5) samples.

head node sends a restart command when downscaling iteration is done (i.e 0.5 nodes collected)

So when we run halted vs wild, we can just pass a different machine_with_vertex function to the compute node.

For halted fires we want clustering in the partitions

• manage difference in environment variables. (i.e "LOCAL" variable)
• Create all the compute node classes for each simulated node.
• run the node instance in it's own thread
• handle communication between compute nodes with MPI??

Current data storage structure:

data/
|
| ---- kgs/
|        |
|        | ---- kgs-2-partitions/
|        |        |
|        |        | ---- node1.e
|        |        |
|        |        | ---- node1.p
|        |        |
|        |        | ---- node2.p 
|        |        |
|        |        | ---- node2.e
...
...
|        | ---- kgs-16-partitions/
|        |        |
|        |        | ---- node1.e
|        |        |
|        |        | ---- node1.p
...
...

|        |        | ---- node16.p 
|        |        |        
|        |        | ---- node16.e 

Update:
Memory is still a problem especially when doing partitioning on a large graph (like S scale in LDBC datasets)

Compute nodes are responsible for sending messages to other compute nodes

In a halted forest fire there are two ideas that need be implemented.

1. Don't pass messaged
2. Read from assigned partition files. <- might be it's own task
3. Run algorithm on assigned graph
4. Parse scale up call vs scale down call, what the percentage of the scale up/down call is.