How to let disco generate input data in real-time and then do map-reduce? about disco HOT 3 CLOSED

I knew Yarn but it still separates real time raw input data generation from map-reduce? In other words, it does my job in two individual steps: 1. run my python module and generate raw inputs and store then in text files 2. use these text files in map-reduce-cluster.

from disco.

In theory you could run the map on empty blobs (as many as your module needs) and just yield the results of your module to the reduce function instead of anything from the file.

from disco.

thanks. it seems work on me.

from disco.