mrExec on a single part file about datadr HOT 2 CLOSED

The approach that I take is to take advantage of datadr's backend-agnostic design and do something like this:

dat <- ddf(hdfsConn("_path_"))
localdat <- ddf(dat[1:10])

Now you have a local in-memory subset and you can run your mrExec or any other datadr code on localdat and test it. And you have the added benefit to stick browser() statements anywhere in your code and step in - something you can't do on a cluster with rhipe.

from datadr.

That worked pretty well for code debugging. (Tried it). Useful.

The other issue is that new Yarn containers are super strict about going over mapreduce.reduce.memory.mb . Any reducer that tries to go over for a second is turned into a failed reducer container and restarted. I am finding that to be a blessing and a curse. It is a blessing because you never induce swapping, but you have to nail it, but I can't figure out how to guess how high it needs to be other than to start jobs and see what happens---which was the other part of why part files were in my head.

from datadr.