Application reads multiple CSV or TSV files in the same directory handling their extensions particularities. Group By the key and value in all file sources and counting them, keeping only those with Odd count. After that Writes all content in a new TSV file.

Test Application:

sbt test

Run with SBT:

sbt run 
    --input | -i {S3_INPUT_PATH:'s3a://bucket/input'}
    --output | -o {S3_OUTPUT_PATH:'s3a://bucket/output'}
    --aws-profile | -a {AWS_PROFILE:default}

Submit application:

spark-submit --master {yarn/local/spark:HOST:PORT} --class com.vigil.VigilGeneralDataEngineerApp {JAR_PATH} \ 
    --input | -i {S3_INPUT_PATH:'s3a://bucket/input'}
    --output | -o {S3_OUTPUT_PATH:'s3a://bucket/output'}
    --aws-profile | -a {AWS_PROFILE:default}

It was developed two approaches for the exercise. Firstly I started with DataFrame API, and after I reimplemented with RDD API:

• This approach executes in one Stage
• And executes with all optimization of Catalyst
• Processed in 0.9 seconds on my local machine*

• This other approach execute with two Stages, because off reduceByKey (which is better than groupByKey)
• And executed in 0.6 seconds on my local machine*

*These times are not that relevant because the files sizes are too small

• Create SparkSession by Environment