This is solution of https://github.com/Kyotu-Technology/kyotu/tree/main/recruitment-challenges/large-file-reading-challenge.
I know that I could use DB. Yes then you just need to write something like this:
SELECT date_part('year', mesure_time) as year, round(avg(temperature)::numeric, 2) as temperature
FROM data_points
WHERE city = '$city'
GROUP BY city, year
ORDER BY year;
But, where is fun? More information about that approach you can find in database folder.
- Java 21
- Maven 3.9.6
You can use SDKMAN.io to configure your environment.
$ sdk env install
To build this project you need get it from repository and run maven.
$ git clone [email protected]:Koziolek/big-files.git
$ cd big-files
$ mvn clean verify
That will run build process and test.
In target
directory there will be big-files-1.0-SNAPSHOT-jar-with-dependencies.jar
file. Run it:
$ java -jar target/big-files-1.0-SNAPSHOT-jar-with-dependencies.jar -c Warszawa -f ./src/test/resources/example_file.csv
It will calculate avg for Warszawa from example file from original task repo. For help use -h
option.
You could generate big file (almost 4GB) for testing:
for i in {1..10000}; do cat ./src/test/resources/example_file.csv >> ./src/test/resources/long.csv ; done;
All options are available by using -h
or --help
switch.
Name of city that you interested in.
Path to data file
Type of calculator NAIVE or INPLACE. Default is INPLACE.
Prints help
Work in parallel mode (faster but need more memory). Default false.
If set prints data in given format: JSON or CSV. Default format is plain text
If set ignores -C and -p. Calculator is picked in smart way based on number of CPU and memory.
What to do when file change during processing. Default is just log. Use ERROR to stop processing and throw.
Very small memory and very big file example. This will take time, but is possible to finish.
$ time java -Xmx8m -jar target/big-files-1.0-SNAPSHOT-jar-with-dependencies.jar -c Warszawa -f ./src/test/resources/long.csv -C INPLACE
2018 » 13,56°C
2019 » 13,81°C
2020 » 16,12°C
2021 » 15,61°C
2022 » 14,68°C
2023 » 15,46°C
0:13.61 real, 13.53 user, 1.34 sys, 0 avg_mem, 87256 max_mem
$ time java -Xmx4m -jar target/big-files-1.0-SNAPSHOT-jar-with-dependencies.jar -c Warszawa -f ./src/test/resources/long.csv -C INPLACE
2018 » 13,56°C
2019 » 13,81°C
2020 » 16,12°C
2021 » 15,61°C
2022 » 14,68°C
2023 » 15,46°C
1:53.72 real, 114.02 user, 5.24 sys, 0 avg_mem, 74196 max_mem,
JSON printer and CSV printer
$ avg_mem, 74196 max_mem, %I IO_ins 288 IO_outs
koziolek@koziolek-desktop3 ~/workspace/codest/big-files $ time java -Xmx4m -jar target/big-files-1.0-SNAPSHOT-jar-with-dependencies.jar -c Warszawa -f ./src/test/resources/example_file.csv -C INPLACE -P JSON
[
{"year": 2018, "temp": "13,52°C"}
{"year": 2019, "temp": "13,81°C"}
{"year": 2020, "temp": "16,12°C"}
{"year": 2021, "temp": "15,61°C"}
{"year": 2022, "temp": "14,68°C"}
{"year": 2023, "temp": "15,46°C"}
]
0:00.46 real, 0.37 user, 0.03 sys, 0 avg_mem, 63184 max_mem,
$ time java -Xmx4m -jar target/big-files-1.0-SNAPSHOT-jar-with-dependencies.jar -c Warszawa -f ./src/test/resources/example_file.csv -C INPLACE -P CSV
2018;13,52°C
2019;13,81°C
2020;16,12°C
2021;15,61°C
2022;14,68°C
2023;15,46°C
0:00.46 real, 0.35 user, 0.03 sys, 0 avg_mem, 59004 max_mem
Using file watcher (ERROR
mode)
$ time java -jar target/big-files-1.0-SNAPSHOT-jar-with-dependencies.jar -c Warszawa -f ./src/test/resources/long.csv -C INPLACE -w ERROR -p
Exception in thread "main" java.lang.IllegalStateException: File changed during processing
at com.thecodest.bigfiles.calculators.watch.ThrowingStrategy.finisher(ThrowingStrategy.java:22)
at com.thecodest.bigfiles.calculators.AbstractFileStreamCalculator.getData(AbstractFileStreamCalculator.java:36)
at com.thecodest.bigfiles.calculators.AvgCalculator.getData(AvgCalculator.java:14)
at java.base/java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:197)
at java.base/java.util.stream.SliceOps$1$1.accept(SliceOps.java:200)
at java.base/java.util.stream.Streams$StreamBuilderImpl.tryAdvance(Streams.java:397)
at java.base/java.util.stream.ReferencePipeline.forEachWithCancel(ReferencePipeline.java:129)
at java.base/java.util.stream.AbstractPipeline.copyIntoWithCancel(AbstractPipeline.java:527)
at java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:513)
at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:499)
at java.base/java.util.stream.ForEachOps$ForEachOp.evaluateSequential(ForEachOps.java:151)
at java.base/java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(ForEachOps.java:174)
at java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
at java.base/java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:596)
at com.thecodest.bigfiles.Application.main(Application.java:46)
Command exited with non-zero status 1
0:03.50 real, 15.39 user, 3.48 sys, 0 avg_mem, 2911992 max_mem
But in another terminal you run
for i in {1..10000}; do cat ./src/test/resources/example_file.csv >> ./src/test/resources/long.csv ; done;
Just after program starts. In this mode watcher throws exception immediately when detect file change.
Using file watcher (LOG
mode)
$ time java -jar target/big-files-1.0-SNAPSHOT-jar-with-dependencies.jar -c Warszawa -f ./src/test/resources/long.csv -C INPLACE -p
File was changed during processing. You need to manually restart program.
Returned result could be inconsistent with current state of file.
2018 » 13.56°C
2019 » 13.81°C
2020 » 16.12°C
2021 » 15.61°C
2022 » 14.68°C
2023 » 15.46°C
0:04.70 real, 34.77 user, 14.43 sys, 0 avg_mem, 8444888 max_mem
But in another terminal you run
for i in {1..10000}; do cat ./src/test/resources/example_file.csv >> ./src/test/resources/long.csv ; done;
Just after program starts. In this mode watcher will log information about change at the end of processing. Results depends on change type.
- If record that changed was already processed then that change will be ignored.
- If record that changed is not yet processed then new value will be taken.
- If new record append at the end of file then will be processed as normal.
- If new record append before currently processed record then that change will be ignored and currently processed record will be processed one more time in next processing step.