extension towards paralell map / reduce that maps eventually about swarm HOT 5 OPEN

Well, seems too theoretical for me.
Can you illustrate that with some real-life workload scenario?

from swarm.

real life workload. when you need to increase the throughput of a sysytem you run many of them in parallel. this is what Spark & Storm do in slight different fundamental ways.
so the real world workload is a systems requirement for faster results.

the way reactive framework operate is a DAG. a single node acts as a computational node.
but to achieve the real world requirement we need to have 100 of them (for example).
in order for each of the 100 to do the computation, the data stream is sliced into 100 separate chunks.
This what cassandra and spark do using Dstreams.
Do this help ?

from swarm.

Regarding multi-server/load-distribution scenarios, the only thing currently planned is consistent hashing.
Using CRDTs for large scale computing is definitely out of the project's scope at this stage.
Eventually, Swarm may expand in that direction, but that is too different from what we are doing now.
There are some immediate practical challenges, like distributed counters. Those seem to be resolvable by the current means.

from swarm.

P.S. The next step in that direction is adding server-side event filters/listeners.

from swarm.

looking forward to playing with it and seeing where i can push it.
this wll integrate with love field nicely btw

from swarm.