filipegmiranda / typed-mapreduce Goto Github PK

As stated above. The Project needs a README.md that describes clearly the project goals, what it is about, how to download it, get started with it and run it.

It is also important to describe how to contribute to ir, including the code of conduct that should be followed

As in MapReduce, it should be possible to give more than one inputPath to a Job, and associate this
Input to a Mapper.

Initially, what I think is that the inputs and mappers may be represented in a Seq[(Input, Mapper)]

Each Input and a mapper is used by a Mapper Coordinator(new Actor at this point), and this actor reads the input data with the respect chinks and passes to the Mappers under it.

This allows also the Input and Mappers to be recovered at this level of granularity, so that, the MapReduceCoordinator Actor, upon Failure of one of its Mappers Coordinators, restarts that Mappers Coordinator. (Need to think carefully here, maybe this should be configured)

Today the Job starts and loads the Actors Mappers with "income" data, which may lead OutOfMemory errors. In other words, it just throws data to the mappers as it is capable of reading it.

We should check whether a back pressure algorithm may help better control the data flow, and actually make the Mappers and Reducers ask for work instead.

To test, increase the amount of lines in InMemoryDataSetInput(), the property: multiplyEachBy = 1000000
or more, the default can still be 1000 (one thousand)