tarensanders / gt3x2csv Goto Github PK

I think this is what is causing the crashes when converting large numbers of files on DMC. vroom by default uses all available threads to perform the writes, but this seems to be clashing with the number of files being processed.

Accessible from

X <- read.gt3x("C:/Users/taren/Downloads/Test gt3x Files/SS_left_19.5.2017-05-22.gt3x")
attributes(X)$sample_rate

{read.gt3x} supports the old file format. At the moment validate_gt3x_files() fails for the old files because it only tests for the newer file format.

It might be safe enough to just remove the check and let {read.gt3x} fail if the files are invalid.

I can't remember which package I saw that did this, but the large test files were hosted on a public URL, and a function checked if they existed and downloaded them before testing. That would make it easier to develop on different machines, and for contributors to run the longer tests locally.

Should also switch the Rbuildignore to be for 'large_tests' folder.

Most of the issues I run into are due to running out of system memory. The current methods are a bit crude - full data is read in at once, there's no manual garbage collection, etc. Most annoyingly this leaves a lot of performance on the table - you need to decrease the number of cores used in order to reserve enough memory.

There's definitely a need to do memory profiling and see where improvements could be made.

I think maybe it should just be ditched?

I think this is because read.gt3x changed their method, but the columns are output in the wrong order compared to Actilife