Read error, probably caused by write error? about mdfreader HOT 7 CLOSED

Hi,
Your code seems correct, there might be a bug in the write method, it is recent feature for mdf4 version. (Which of mdf are you dealing wth ?)
It seems pointer in one of the headers goes out of file, this is meaning of the error.
Same as in issue #15, could you check structure of the file with mdfvalidator program, downloadable from vector site ? -> then we could understand which header is wrong and try to narrow down the bug.
Regards
Aymeric

from mdfreader.

Hi,
thanks for the quick response!
The mdf version seems to be 3.3 (according to mdf_file.MDFVerionNumber = 330). Unfortunately I don't have a windows machine available at the moment, so I can't install the MDF Validator :/
But maybe the complete error output can help a bit:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/lib/python3.5/dist-packages/mdfreader/mdf.py", line 95, in __init__
    self.read(fileName, channelList=channelList, convertAfterRead=convertAfterRead, filterChannelNames=filterChannelNames)
  File "/usr/local/lib/python3.5/dist-packages/mdfreader/mdfreader.py", line 325, in read
    self.read3(self.fileName, info, multiProc, channelList, convertAfterRead, filterChannelNames=False)
  File "/usr/local/lib/python3.5/dist-packages/mdfreader/mdf3reader.py", line 700, in read3
    buf.read(channelList) # reads datablock potentially containing several channel groups
  File "/usr/local/lib/python3.5/dist-packages/mdfreader/mdf3reader.py", line 559, in read
    self[recordID]['data'] = self.loadSorted(self[recordID]['record'], zip=None, nameList=channelList)
  File "/usr/local/lib/python3.5/dist-packages/mdfreader/mdf3reader.py", line 579, in loadSorted
    return record.readSortedRecord(self.fid, self.pointerToData, nameList)
  File "/usr/local/lib/python3.5/dist-packages/mdfreader/mdf3reader.py", line 476, in readSortedRecord
    return fromfile(fid, dtype=self.numpyDataRecordFormat, shape=self.numberOfRecords, names=self.dataRecordName)
  File "/usr/local/lib/python3.5/dist-packages/numpy/core/records.py", line 777, in fromfile
    "Not enough bytes left in file for specified shape and type")
ValueError: Not enough bytes left in file for specified shape and type

Btw: The output is the same if I use python 2.7 instead.
Thanks for your help! :)
Bibo

from mdfreader.

Hi,
It seems number of records defined is wrong or pointer of datablock is wrong but I can not understand why. With the files I have, I could not see any error -> your file has probably some specificity that I did not yet test, so hard to troubleshoot.
Regarding mdfvalidator and if you use linux, it can be installed and runs using wine.
Other solution is that you send me a pickle of your data before writing it but as usual I guess it can be confidential.
Aymeric

from mdfreader.

Good news :)
I managed to view the files with mdfvalidator using wine and found some errors.
The one causing the trouble was: The type of the added channel was int64, which apparently can't be handled. After changing the code to

data = np.asarray(assignments[last:last+dim_channel]).astpye(np.int32)

I can now read the files without problems.
There still are some errors mdfvalidator points out ("raster will only be used for virtual time channel!" for all channels and "Textual time differs exactly by one hour: probably daylight saving time" for the header block) but those don't seem to have an notable impact on working with the files.
Thanks again for your help!
Bibo

from mdfreader.

Hi,
Happy you could solve your issue.
Indeed for version 3.x fo mdf, it is limited up to int16 or uint64.
However mdf4.x can handle int64 and unint64.
Aymeric

from mdfreader.

Hi,
I actually checked the specification more in detail and int64 should be possible. When you mention int64 is not apparently handled, it is not mdfvalidator mentionning it but rather a capability issue of mdfreader to read it ?

from mdfreader.

Well, mdfreader can't read the files with int64 but mdfvalidator does raise an error message, too. The message seems to refer to the software, which recorded the data, so maybe that's where something doesn't work...

from mdfreader.