Current: N+X map-reduce jobs. Each job write result on HDFS. Only the last result is needed.

Possible steps:

• read the book of yarn
• write example application master
• write example tez application
• dive on tez

dive doesn't have execute plan, only generate by hand-written plan.
If operator tree is constructed, some works can be started:

• sub-query(SelectJob do not support sub-select job)
• optimization such as push-down projection and selection

Current: the serialization format of Row is BytesWritable. Each time dive need to read or write, dive create a new byte array. A possible solution is to create instance from DataInput. (Write to output is easy, because dive should not write schema in it)

Possible Solution
Rewrite the Mapper/Reducer or upper task generator
As long as we set-up the schema before de-serializing the Row, we can use Row#read(in)
Maybe by rewrite the task generator
See http://avro.apache.org/docs/1.7.6/mr.html
'''
AvroJob.setInputSchema(conf, User.getClassSchema());
'''