okkevaneck / distributed_systems_lab Goto Github PK

Compute nodes are responsible for sending messages to other compute nodes

In a halted forest fire there are two ideas that need be implemented.

1. Don't pass messaged
2. Read from assigned partition files. <- might be it's own task
3. Run algorithm on assigned graph
4. Parse scale up call vs scale down call, what the percentage of the scale up/down call is.

for faster set_vertex_status calls
and faster get_vertex_status calls

shouldn't these functions be in the vertex class anyway?
What am I doing?

Current data storage structure:

data/
|
| ---- kgs/
|        |
|        | ---- kgs-2-partitions/
|        |        |
|        |        | ---- node1.e
|        |        |
|        |        | ---- node1.p
|        |        |
|        |        | ---- node2.p 
|        |        |
|        |        | ---- node2.e
...
...
|        | ---- kgs-16-partitions/
|        |        |
|        |        | ---- node1.e
|        |        |
|        |        | ---- node1.p
...
...

|        |        | ---- node16.p 
|        |        |        
|        |        | ---- node16.e 

Update:
Memory is still a problem especially when doing partitioning on a large graph (like S scale in LDBC datasets)

• manage difference in environment variables. (i.e "LOCAL" variable)
• Create all the compute node classes for each simulated node.
• run the node instance in it's own thread
• handle communication between compute nodes with MPI??

It's pretty underperformant right now

Vertex IDs in Graphalytics dataset are actually random number, not just start from some number n and end with n + n_vertices, which leads to the bug in the converter.

For halted fires we want clustering in the partitions

When ComputeNode is blocking on send (heartbeat), after the head send a kill request. The head will never accept that send and thus the ComputeNode wont stop by it self.

Use some library or something to perform graph analysis

Issue to implement MPI communication in compute node classes

• Implement a thread to listen for communication from other compute nodes
• Implement a thread to listen for communication from the HEAD node telling the compute node to stop
• Implement a function to send the head node heartbeats that communicate how many nodes have been sampled by the compute node
• Implement function to listen for restart call

So when we run halted vs wild, we can just pass a different machine_with_vertex function to the compute node.

Multiple functions need the decorator.

graph.spread should probably be measured

• list how many nodes has been burned per X time units

• Compute node should measure how many messages are sent, how many messages are received.

• Head node should keep track of when heartbeats are received, how many nodes are burned per heartbeat.

We should read 2 graph files.
a .v file, and a .e file.

Changes to graph interpreter and Graph.add_vertex_and_neighbor

Head Node is responsible for

• Stitching together all graph sampled from compute nodes
• listening for heartbeats
• • send kill signal to compute nodes to stop burning
• writing resulting graph to a file

Because we will have precomputed partitions with clustering, we will need each compute node to handle multiple partitions.

(I.e 8 partitions for a graph & 2 compute nodes means each compute node gets 4 partitions)

This task is to implement that functionality

To upsacle X2

run downscaling to (0.5) algorithm 4 times
Then stitch together all (0.5) samples.

head node sends a restart command when downscaling iteration is done (i.e 0.5 nodes collected)