Dealing with very large files about mdfreader HOT 4 CLOSED

Dear,
I would say that without increasing your machine memory, you will have always difficulties to read big files.
But there are ways to optimise memory consumption:

1. you can use parameter convertAfterRead=False to have same memory use of your file (conversion to floating inflates memory use)
2. you can choose to only read a list of channels. To know the file list of channels, you can refer to issue #27 and then use the listchannel parameter
3. exporting to CSV is generally a bad idea for big files. You could export to HDF5 which is a good and widely used format (you can read it in Matlab for instance). But generally for whatever export, it is needed to copy some data so you also need memory margin.
   Regards

from mdfreader.

Hello,
Thank you very much for getting back to me. I am fairly new to all of this and appreciate your help.
On the third point, the files are so big that it is not even getting as far as reading them in, even without trying to convert them and that is with convertAfterRead=False.
So am I right in thinking that, if a CAN log was taken over 8 hours, there is no way of reading in only (say) the first hour or the second hour of data?
If not, I will definitely look at limiting the channels that I read in or splitting the file using different software.
Thanks again,
Peter

from mdfreader.

Hi,
There is no functionality to have partial read of the data in mdfreader so I as you proposed, you should read a list of channels
Regards

from mdfreader.

Thank you very much for your help. I'll close the issue now but I'm very grateful.
Yours,
Peter

from mdfreader.