usc-cloud / floe2 Goto Github PK

*Message dispatcher should not have to de-serialize the messages. Let the transport do serialization ( Do serialization at the end just before sending out message)

Multiple containers on a single machine use same port numbers to assign to the flakes. Allow an option to set it at container level rather than at coordinator level.

Currently messages are duplicated to different pellets (and-split). e.g. (task parallelism)

This feature will add an ability to selectively send messages to desired downstream pellets. (multi-select split)

We will use named streams to implement this feature.

i.e. a pellet can generate more than one named stream. The emitter interface will have an additional parameter which refers to the stream name. The stream names should static and be predefined by the pellet and cannot be changed at runtime.

A downstream neighbour can then subscribe not only to the pellet, but also to the particular named stream of the pellet.

By default all pellets generate a stream named "DEFAULT_STREAM" and when subscribed without any parameters subscribe to the stream of the same name.

depends on #11, #19, #20, #21

Currently, once the resource mapping is obtained from the resource manager, it is stored in ZK where individual containers monitor the ZK location and deploy any flakes assigned to them as required.

However, there is no way for the coordinator (and in turn, the user) to know whether all flakes have been deployed successfully.

*Simple interface
*Plugable model

Scale down does not work.
The resource mapping is correctly updated. The signal is received by the container but scale down action does not decrement the pellet.

Currently..
The predecessor pellet instance creates and send a message meant directly for the succeeding pellet's instance. This is nice since the notion of "flake" is completely hidden from the preceding flake's dispersion strategy. It works directly with the instances.

But, on the flip side, even though flake is the unit of communication, multiple subscriptions and backchannel messages are sent per flake since each flake creates multiple pellet instances.

This also has implications on fault tolerance (e.g. in case of reducer pellets). (Will add more details later).

Add the feature to perform scale down at runtime (i.e. gracefully terminate PE instances)
-Add command line api to do so.
-Use the updated data structure from scale up to hold resource mapping 'delta'
-Update Container/AppsAssignmentMonitor to look for changes in the app assignment and respond to the scale down request.

Note: It is important to cleanly close data connections to ensure no messages are lost during scale down, and any messages received but pending in the queue should be processed before terminating the instance.

Launching Flake Phase - launch all flakes and start listeners
Connect Flake Phase - Issue connection requests
Launch Pellet Phase - Launch required number of pellet instances per flake
Start Execution Phase - Signal "source" pellets to start execution.

Cntrl-C should be handled gracefully without effecting any other components and should cleanly shutdown all sockets (tcp, inproc and most importantly ipc).

Multiple features need to be added as part of this.

NOTE: This issue is only for scalable MR (and not elastic MR). i.e. the number of reducers are decided at deployment.

1. channel grouping
2. channel dispersion interface and strategies
3. special pellet types for mapper and reducers
4. appbuilder apis

This is a stupid regression caused due to an effort to optimize by eliminating the middle layer of communication between the message emitter and the multiple backends (one per edge in the application).

Fix: bring the middle layer back!! (needs more re-factoring to keep the back channel and other things intact)

This is a timing issue. We terminate the flake before the message is actually sent. Need to fix #27 to fix this issue (and other possible message loss issues).

Currently Low priority since we are focusing on RR and MapReduce and both these strategies do not require an special message. Later to implement active load balancing, we will need this feature and will add it then.

UPDATE: The elastic MR strategy requires this feature since we will send the position on the ring on the backchannel.

Use Mesos/JClouds to acquire resources and resource management/mapping.

depends on #24

Ability to send signals to running pellets.

1. Note this should be on the same thread as data messages. This will let the user to respond to the signal without worrying about threading issues.
2. side effect is that during the time the signal is being processed, the data messages will start queuing in the zmq queues.

Need a clean API for flake token/data handler while scaling in/out

The flake first connects to the preceding flakes and then creates the pellet instance. But given the ZMQ's PUSH policy that the send method will block untill atleast one client connects, it leads to an issue since both the backend send and the control signal receive are on the same thread. Since backend send blocks, the control signal to increment pellet is never received and a deadlock occurs.

ApplicationBuilder API as well as updates to the thrift definitions.
Update ClusterResourceManager to account for the requested parallelism.

Add support for dynamic pellets with AppBuilderAPI to easily create such dynamic tasks.

Add runtime support for dynamically switching the active alternate for a pellet.

NOTE: This issue is only related to dynamic alternate switching without synchronization/consistency issues. A different issue will be opened for that.

Add the feature to perform scale up/out at runtime depending on available resources.
-Add command line api to do so.
-Update data structure to hold resource mapping 'delta'
-Update Container/AppsAssignmentMonitor to look for changes in the app. assignment and respond to the scale up request.

Note: Active load balancing is not part of this and will be handled separately.

Decoupling from scale down feature since scaling down needs additional work to cleanly close data connections to ensure no messages are lost during scale down.

There may be message loss during scale down if there are messages in the flakes zmq input or output queue and we call a disconnect. This is a know issue and will be fixed as part of the fault tolerance framework.

Also need to add a special back channel message to do so.

This is required to make the "createFlake" and "terminateFlake" functions synchronous so that we do not start pellets before the flakes are ready.

When multiple subscribers subscribe to the same stream, the stream should be duplicated across the multiple subscribers.
Currently, the stream is split (in a round robin way) across them.

Expected:

1. duplicate stream across multiple subscribers. (task parallelism)
2. split (in round robin, or load balanced way) across multiple instances of the same subscriber pellet (data parallelism).

Issue is with heartbeats.
(done) Fix for previously terminated flakes: The last heartbeat sent by the flake will mark the flake as terminated which will make the container cleanup the flake and hence on scale up initialize a new flake.

Fix for failed flakes (where the heartbeat may stop without the terminate flag): Reduce flake-to-container heartbeat duration. Periodic cleanup if heartbeat is not received in time. (will file and fix as a separate issue as part of the fault tolerance feature).