ratal / mdfreader Goto Github PK

Hi,
I was trying to install the new version (0.1.9.1) of mdfreader. Therefore I installed cython and its dependencies for Windows (https://github.com/cython/cython/wiki/CythonExtensionsOnWindows). Unfortunately, the package installation exited with the following error:

building 'mdfreader.dataRead' extension
C:\Users\zzz\AppData\Local\Programs\Common\Microsoft\Visual C++ for Python\9.0\VC\Bin\cl.exe /c /nologo /Ox /MD /W3 /GS- /DNDEBUG "-IC:\Program Files\Anaconda\include" "-IC:\Program Files\Anaconda\PC" /Tcmdfreader/dataRead.c /Fobuild\temp.win-amd64-2.7\Release\mdfreader/dataRead.obj
dataRead.c
mdfreader/dataRead.c(250) : fatal error C1083: Cannot open include file: 'numpy/arrayobject.h': No such file or directory
error: command 'C:\Users\zzz\AppData\Local\Programs\Common\Microsoft
Visual C++ for Python\9.0\VC\Bin\cl.exe' failed with exit status 2

Is there a special numpy-package, that I need to install first?

EDIT:
I solved the issue by importing numpy in setup.py and adding "include_dirs=[numpy.get_include()]" in line 104 after "ext_modules=cythonize('mdfreader/dataRead.pyx'),"

Hello, I'm trying to read an MDF4 file and have 2 issues.

1. In mdfinfo4 at line 536
   self['cc_ref'][i] = CCBlock(fid, self['cc_ref'][i])['unit']['Comment']
   I receive a KeyError for 'name'. As the only entry of CCBlock with a Comment key was an entry called unit I replaced those. Several CCBlocks didn't have unit either which I countered by introducing

try:
     self['cc_ref'][i] = CCBlock(fid, self['cc_ref'][i])['unit']['Comment']
except Exception, e:
     print repr(e)

I'm not sure how much this impacts the further processing. mdfreader.mdfinfo() is working now though as it seems

2. Continuing mdfreader.mdf() I receive the following errors:

('Unexpected error:', (<type 'exceptions.ImportError'>, ImportError('No module named dataRead',), <traceback object at 0x7fbe92be2f38>))
dataRead crashed, back to python data reading
('Unexpected error:', (<type 'exceptions.ImportError'>, ImportError('No module named dataRead',), <traceback object at 0x7fbe8ac113b0>))
dataRead crashed, back to python data reading
Traceback (most recent call last):
File "/workspace/Eclipse/Python/ReadMDF4/src/ReadMDF4.py", line 12, in
yop2 = mdfreader.mdf("./file.MF4")
File "/Programme/anaconda/lib/python2.7/site-packages/mdfreader/mdf.py", line 106, in init
self.read(fileName, channelList=channelList, convertAfterRead=convertAfterRead, filterChannelNames=filterChannelNames)
File "/Programme/anaconda/lib/python2.7/site-packages/mdfreader/mdfreader.py", line 347, in read
self.read4(self.fileName, info, multiProc, channelList, convertAfterRead, filterChannelNames=False)
File "/Programme/anaconda/lib/python2.7/site-packages/mdfreader/mdf4reader.py", line 1282, in read4
buf.read(channelList) # reads raw data from data block with DATA and DATABlock classes
File "/Programme/anaconda/lib/python2.7/site-packages/mdfreader/mdf4reader.py", line 371, in read
self[recordID]['data'] = self.load(record, zip=None, nameList=channelList, sortedFlag=True)
File "/Programme/anaconda/lib/python2.7/site-packages/mdfreader/mdf4reader.py", line 453, in load
temps['data'] = DATABlock(record, parent_block=data_block, channelList=nameList, sortedFlag=sortedFlag)
File "/Programme/anaconda/lib/python2.7/site-packages/mdfreader/mdf4reader.py", line 92, in DATABlock
return fromstring(parent_block['data'], dtype=record.numpyDataRecordFormat, shape=record.numberOfRecords, names=record.dataRecordName)
File "/Programme/anaconda/lib/python2.7/site-packages/numpy/core/records.py", line 709, in fromstring
_array = recarray(shape, descr, buf=datastring, offset=offset)
File "/Programme/anaconda/lib/python2.7/site-packages/numpy/core/records.py", line 423, in new
strides=strides, order=order)
TypeError: buffer is too small for requested array

Error in polynomial conversion 

Original issue reported on code.google.com by [email protected] on 19 Sep 2013 at 9:28

Attachments:

• mdfreader.py.patch

Try to read a mf4 file (version 4.10) But get the following exception.
Unexpected error: (<class 'ValueError'>, ValueError('Big-endian buffer not supported on little-endian compiler',),<traceback object at 0x055D8328>)

In order to further improve performance on mdf-files with a large header (large number of channels), you could consider using lazy initialization for some mdfblocks (which are not always read anyway).

In my case, runtime was about halved, when using lazy init for:

• CommentBlock
• CCBlock
• CABlock
• ATBlock

I didn't file a pull request for this issue, as I don't know if this has negative effects in some cases.
I setup a branch to show my implementation: https://github.com/mattispasch/mdfreader/tree/lazy_init_blocks

EDIT: My Implementation is not thread-safe!

When I import my mdf4 data using
yop = mdfreader.mdf(filename)
everything works great but it takes a long time to process all the data
when I use:

channelList = ['example','#latitude','of','#data']
yop = mdfreader.mdf()
yop.read(fileName=filename, channelList=channelList)

The program fails on the channels that start with '#' symbols
here is the traceback:

Traceback (most recent call last):
  File "C:\Users\420278\AppData\Local\Programs\Python\Python36\lib\site-packages\numpy\core\records.py", line 444, in __getattribute__
    res = fielddict[attr][:2]
KeyError: 'latitude'

here is an additional traceback of an exception that occurs during the handeling of the one above,

  File "C:\Users\420278\AppData\Local\Programs\Python\Python36\lib\site-packages\mdfreader\mdfreader.py", line 362, in read
    self.read4(self.fileName, info, multiProc, channelList, convertAfterRead, filterChannelNames=False)
  File "C:\Users\420278\AppData\Local\Programs\Python\Python36\lib\site-packages\mdfreader\mdf4reader.py", line 1393, in read4
    temp = buf[recordID]['data'].__getattribute__(recordName)  # extract channel vector
  File "C:\Users\420278\AppData\Local\Programs\Python\Python36\lib\site-packages\numpy\core\records.py", line 446, in __getattribute__
    raise AttributeError("recarray has no attribute %s" % attr)
AttributeError: recarray has no attribute latitude

Once again, this problem does not happen when importing without a channelList and it prints out the proper names when calling yop.keys().
calling:

    info=mdfreader.mdfinfo()
    print(info.listChannels(filename))

produces the expected output with properly formatted channel names

The problem is somehow related to your function:
_gen_valid_identifier
in mdf.py
I was able to implement a workaround by adding

or '#'

to the sanitizer, resulting in the code below (around line 638 of mdf.py):

def _gen_valid_identifier(seq):
    # get an iterator
    itr = iter(seq)
    # pull characters until we get a legal one for first in identifer
    for ch in itr:
        if ch == '_' or '#' or ch.isalpha():
            yield ch
            break
        elif ch.isdigit():
            itr = chain(itr,ch)

    # pull remaining characters and yield legal ones for identifier
    for ch in itr:
        if ch == '_' or '#' or ch.isalpha() or ch.isdigit():
            yield ch

What steps will reproduce the problem?
1. Program does not show *.mdf files within the "Select Measurement Files" 
window

What is the expected output? What do you see instead?


What version of the product are you using? On what operating system?


Please provide any additional information below.

Original issue reported on code.google.com by [email protected] on 11 Jan 2013 at 9:17

Hi,

a co-worker of mine is using this library to open a MDF3 file, change some values and write it back to a file.
He noticed, that textual conversions get lost after writing the file.
Steps to reproduce:

• Open a MDF3 file
• Print a channel with textual conversions
• call 'write' method to save to new file
• Open new file
• Print same channel

The output looks like this:

OpenFile, Print Channel:
 {   'conversion': {   'parameters': {   0: {   'Textrange': '',
                                                'lowerRange': 0.0,
                                                'pointerToTXBlock': 0,
                                                'upperRange': 0.0},
                                         1: {   'Textrange': 'Vorwaerts',
                                                'lowerRange': 0.0,
                                                'pointerToTXBlock': 115816,
                                                'upperRange': 0.0},
                                         2: {   'Textrange': 'Rueckwaerts',
                                                'lowerRange': 1.0,
                                                'pointerToTXBlock': 115830,
                                                'upperRange': 1.0},
                                         3: {   'Textrange': 'Init',
                                                'lowerRange': 2.0,
                                                'pointerToTXBlock': 115846,
                                                'upperRange': 2.0},
                                         4: {   'Textrange': 'ungueltig_oder_nicht_verbaut',
                                                'lowerRange': 3.0,
                                                'pointerToTXBlock': 115855,
                                                'upperRange': 3.0}},
                       'type': 12},
     'data': array([0, ..., 3], dtype=uint8),
     'description': 'Drehrichtung Rad HL',
     'master': 't_27',
     'masterType': 1,
     'unit': ''}
 
 Save File
 
 Open New File, Print Channel:
 {   'conversion': {   'parameters': {   }, 'type': 65535},
     'data': array(['V', ..., 'u'], dtype='|S1'),
     'description': 'Drehrichtung Rad HL',
     'master': 't_27',
     'masterType': 1,
     'unit': ''}

Before writing, the channel contains a textual conversion and the data array contains numerical values.

After writing and reopening the file the conversion is lost and the data array contains only the first letter of the String representation.

Debugging showed that data is correct until this line is executed (in mdf3reader.py):
fid.write(pack('<' + dataTypeList * nRecords, *records)) # dumps data vector from numpy

Regards,
Johannes

Hello,
I am loving your tool for the awesome speed it provides, however I noticed one or two bugs or something I don't know why it happens. If it's just me please correct me.

Some of the mdf files I'd like to read don't work at all and I get the following error:
"ValueError: Not enough bytes left in file for specified shape and type".
My guess is, the mdf is not totally correct, however other tools like CANape read it without any problems.

The seccond problem happens with the files that work to read (other file origin).
Here some signals are read in prefectly and some aren't. The wrong signals do have weird timestamps in the master channel and also wrong values in the signal channel. However it seems like every 8th (or for some every 16th) value is correct. If a signal is wrong every other signal that uses the same master channel is also not correct.

Here is an example of what I mean (master left, signal right):
0.7816270000000001-----------8709
8.511752513479514E-154-----524288
1.8481E-319-----------------------65288
-8.229186798482021E294-----39680
-1.3107045803846904E-187--0
8.4203E-319-----------------------962816
1.77266332229639E268--------851560
4.85948271492081E121-------344576
9.780756----------------------------8709
8.512006343632276E-154-----524288
1.8481E-319-----------------------65288
-8.229186798482054E294-----39936
-8.589833538009107E-183---0
8.4746E-319-----------------------606784
3.428829535692728E293------999731
-1.9219814427611343E-245--344592
18.779568--------------------------8709
8.512093962532729E-154-----524288
1.8481E-319-----------------------65288
-8.22918679848207E294------40192
-5.629433307469648E-178----0
8.52896E-319---------------------153600
8.121418629503833E-124-----524860
-4.1477461483712573E225---344600
27.778571000000003-----------8709
8.512145166432896E-154-----524288
1.8481E-319-----------------------65288

The Value 8709 is correct, all the others aren't. Also the according times seem to be the only ones to make sense.

Is it possible that at some point within the raw data reading something goes wrong depending on the exact mdf file format? To me it looks like a shift of one bit or so.
Another guess is that something about 64bit isn't supported somehow. I am using python 64bit and also compiled Cython for 64bit use.

Thanks in advance!
Jonathan

I am trying to do the simplest thing: loading a mdf file as follows

yop=mdfreader.mdf('myfile.mdf')

but I get the same error even though I tried with 3 different files

myfile.mdf
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "mdfreader/mdf.py", line 106, in __init__
    self.read(fileName, channelList=channelList, convertAfterRead=convertAfterRead, filterChannelNames=filterChannelNames)
  File "mdfreader/mdfreader.py", line 341, in read
    info = mdfinfo(self.fileName, filterChannelNames)
  File "mdfreader/mdfreader.py", line 147, in __init__
    self.readinfo(fileName, filterChannelNames)
  File "mdfreader/mdfreader.py", line 177, in readinfo
    self.update(info4(None, fid))
  File "mdfreader/mdfinfo4.py", line 830, in __init__
    self.readinfo(fid)
  File "mdfreader/mdfinfo4.py", line 844, in readinfo
    self['HDBlock'].update(HDBlock(fid))
  File "mdfreader/mdfinfo4.py", line 218, in __init__
    self['Comment'] = CommentBlock(fid, self['hd_md_comment'], 'HD')
  File "mdfreader/mdfinfo4.py", line 278, in __init__
    self.loadHeader(fid, pointer)
  File "mdfreader/mdfinfo4.py", line 84, in loadHeader
    fid.seek(pointer)
IOError: [Errno 22] Invalid argument

I just want to convert the file to another format but I can't even open it.
How is it that I cannot do the simplest thing ? Please any help would be appreciated

What steps will reproduce the problem?
1. start mdfconverter.py
2. browser to a file
3.

What is the expected output? What do you see instead?
channel list should contain names of channels
instead only the first letter is shown

What version of the product are you using? On what operating system?
latest version. same result on python 3.4 and 2.7 on ubuntu 14.04

Please provide any additional information below.
problem can be fixed by changing line 317 from mdfinfo3.py to
        channelNameList.append( signalname ) # BUG! signalname[0] )  

Original issue reported on code.google.com by [email protected] on 18 Nov 2014 at 11:30

Hello Ratal,
I have installed this python package to read an mdf file and export to a CSV file.
When trying to read it "yop=mdfreader.mdf('NameOfFile')"
I am getting the following error. Kindly look into it and let me know the problem, that could be.

log:

Min/max values changed
<tool_id>CANape</tool_id>
<tool_vendor>Vector Informatik GmbH</tool_vendor>
<tool_version>11.0.70.53107</tool_version>
<user_name>Hemanth</user_name>
>
Traceback (most recent call last):
File "C3PO_Trials.py", line 20, in
yop=mdfreader.mdf('XXXMeasurementXXX.MDF')
File "/usr/local/lib/python2.7/dist-packages/mdfreader/mdf.py", line 106, in init
self.read(fileName, channelList=channelList, convertAfterRead=convertAfterRead, filterChannelNames=filterChannelNames)
File "/usr/local/lib/python2.7/dist-packages/mdfreader/mdfreader.py", line 353, in read
self.read4(self.fileName, info, multiProc, channelList, convertAfterRead, filterChannelNames=False)
File "/usr/local/lib/python2.7/dist-packages/mdfreader/mdf4reader.py", line 1352, in read4
temp.loadInfo(info) # load all info related to record
File "/usr/local/lib/python2.7/dist-packages/mdfreader/mdf4reader.py", line 1034, in loadInfo
self.recordToChannelMatching[Channel.name] = self.recordToChannelMatching[self[-2].name]
KeyError: u'_build_id'

What steps will reproduce the problem?
1. I wrote python mdfconverter.py in console. 
2. In UI i selected an mdf file and try to plot the signal.

What is the expected output? What do you see instead?
I got an error 
Traceback (most recent call last):
  File "C:\MDF-READER\mdfreaderui.py", line 255, in plotSelected
    self.mdfClass.__init__(str(self.FileList[0].text())) # read file
TypeError: 'QListWidget' object does not support indexing

What version of the product are you using? On what operating system?
pyhton 2.7

Please provide any additional information below.
I have tried with python 3.4 also. It is giving the same error.

Original issue reported on code.google.com by [email protected] on 1 Apr 2014 at 12:07

What is causing this problem?
As defined the function mdfblockread expects blockFormat to be a tuple of 
tuples. In case the blockFormat contains just a single tuple, it then starts 
considering the elements of the tuple as tuples themselves causing unexpected 
results. 

What can solve this problem?
I had added the following lines of code to overcome this problem:
----------------------------------------
if type(blockFormat[0])!=tuple:
    blockFormat=tuple([blockFormat])
----------------------------------------


What is the expected output? What do you see instead?
Expecting to read an MDF file. Getting an "out of range" error instead


What version of the product are you using? On what operating system?
Using Python 2.6 on Windows 7

Please provide any additional information below.

Original issue reported on code.google.com by [email protected] on 28 Mar 2013 at 1:23

suggested change (inline with how you did some other imorts):
try:
from .mdfinfo3 import info3
except:
from mdfinfo3 import info3

Hi ratal,

First of all thank you for the great work!

I'm struggling to concatenate multiple MDF files (all of them are measured with same configuration hence sample rates, etc.). Do you know a way to do so?

Thanks!

Hello Ratal,

there seems to be an issue reading MDF4 files that are using cn_composition.
If I try to read a file I get the following Stacktrace:

`...
C:\Program Files\Anaconda3\lib\site-packages\mdfreader\mdfinfo4.py in readCNBlock(self, fid, dg, cg, channelNameList)
1351 if fid.read(4) in ('##CA', b'##CA'):
1352 self['CNBlock'][dg][cg][cn]['CABlock'] = CABlock(fid, self['CNBlock'][dg][cg][cn]['cn_composition'])
-> 1353 elif self.mdfblockreadCHAR(fid, 4) in ('##CN', b'##CN'):
1354 self['CNBlock'][dg][cg][cn]['CNBlock'] = CNBlock(fid, self['CNBlock'][dg][cg][cn]['cn_composition'])
1355 else:

AttributeError: 'info4' object has no attribute 'mdfblockreadCHAR'`

This method is not available in info4 class as it does not inherit from MDFBlock.

Regards,
Johannes

When I install using setup.py, the file does not show up as accessible unless running code from the cloned directory. Install via pip or setup.py should install the necessary files into the site-packages folder without the need to run the code within the git-cloned directory.

Problem:
When using mdfreader with mdf4-files which contain a huge number of channels (> 100.000) processing takes a long time.
This is about an hour for mdf4info and mdf4reader each.
I my case this is particulary annoying since the files are very small (short time frame)

Cause:
The main cause is that the software is checking for duplicate channel names based on lists.
Using sets for those lookups reduces runtime complexity from O(n^2) to O( n*log(n) ).
=> Runtime is reduced to about 1/100th in my case.

Fix:
I'll setup a pull request referencing this issue. I wasn't able to test this exact software version due to a bunch of bugs concerning my test mdf files. So please accept with caution. I hope it's still useful to point to the required changes

Hello,
calling the mdfreader.mdf on a sorted mdf4 file, I face the error:

Unexpected error: (<class 'SystemError'>, SystemError("Parent module '' not loaded, cannot perform relative import",), <traceback object at 0x000000000E4F4108>)
dataRead crashed, back to python data reading

I am using python 3.52 with Anconda3 distribution on a 64bit Windows machine.
I am running the mdfreader downloaded from github yesterday, 05.04.2017.

I tracked it down to come possibly from line 1180 in mdf4reader.py:
from .dataRead import dataRead

Looking for this module on my system I found
dataRead.cp35-win_amd64.pyd
in Lib\site-packages.

Just to make sure, I updated cython to 0.25.2, checked that system path includes this folder, however the import of dataRead module keeps failing.

Any suggestions? I would appreciate that!

Hi ratal,

i am trying to reduce the memory consumption of my script by reading just specific channels from my mdf-file. The code I used is:

inca_mdf = mdfreader.mdf()
inca_mdf.read(inca_filepath, channelList=cList)

where cList is an array of strings. Unfortunately this does not work, and I get the following error-message:

File "aaa", line 43, in Main
inca_mdf.read(inca_filepath, channelList=cList)

File "build\bdist.win-amd64\egg\mdfreader\mdfreader.py", line 371, in read
self.read4(self.fileName, info, multiProc, channelList, convertAfterRead)

File "build\bdist.win-amd64\egg\mdfreader\mdf4reader.py", line 1162, in read4
buf.read(channelList) # reads raw data with DATA and DATABlock classes

File "build\bdist.win-amd64\egg\mdfreader\mdf4reader.py", line 348, in read
self[recordID]['data'] = self.load(record, zip=None, nameList=channelList, sortedFlag=True)

File "build\bdist.win-amd64\egg\mdfreader\mdf4reader.py", line 422, in load
temps['data'] = DATABlock(record, parent_block=data_block, channelList=nameList, sortedFlag=sortedFlag)

File "build\bdist.win-amd64\egg\mdfreader\mdf4reader.py", line 88, in DATABlock
return record.readBitarray(parent_block['data'], channelList)

File "build\bdist.win-amd64\egg\mdfreader\mdf4reader.py", line 992, in readBitarray
buf = recarray(self.numberOfRecords, format)

File "C:\Program Files\Anaconda\lib\site-packages\numpy\core\records.py", line 401, in new
self = ndarray.new(subtype, shape, (record, descr), order=order)

ValueError: data-type with unspecified variable length

I am using version 0.1.8 of mdfreader. Is there any way I can solve this?

Best regards

Hi ratal,

I recently found this module and played with it for some time - I like it so far!
Please note that I'm not really a python-guy, I'm trying to use this from Julia via PyCall (I'm also not really a Julia-guy either, but whatever...)

Here is my system:

• Win7 x64
• Anaconda python 2.7
• built your lib using VS 2013
• Julia 0.3.11
• Jupyter via Anaconda

Some things I noticed while testing:

• Channel names are not unique (in many measurement files I got around here). Maybe a list would be a better data structure to store the channels. This issue also can raise exceptions when, for example, the resampling method is invoked. Having two signal-channels with the same name, one signal-channel will get overriden by another one when the file is read, leaving a random master-channel behind which might be the issue here, but I could not verify that yet.
• Some encoding-exception is thrown when I try to resample the channels because of german umlaut characters (äöü) in the channel description. This only occurs when I use the lib via Julia. I could fix that by changing line (this might be not a good solution but it works for me):

  mdfreader/mdfreader/mdfinfo3.py

  Line 637 in b44231d

  Block[fieldName] = value.replace('\x00', '')

if PythonVersion < 3:
    if removeTrailing0:
       # original
       Block[fieldName] = value.replace('\x00', '')
       # changed
       Block[fieldName] = value.decode('latin1').encode('utf-8').rstrip('\x00')

• Again also only relevant when using Julia: It is really difficult to call the methods of the mdf object as it is automatically converted to a dictionary. I could fix that by not inheriting from dict but to aggregate a dict inside the class, so every self[channelName] becomes self.channels[channelName] instead.

A lot of trial and error was involved as I consider myself a beginner in both languages, so maybe there are no issues and I'm just stupid :)
I'm not really familiar with the git process, so if you need the code I'll send it to you. During the next few days I'll try to convert the lib (mdf3 only for now) to a list-based structure - maybe this will work.

Regards
Sebastian

Hi, I am trying to read a sorted mdf4.1 with mdfreader version master0.2.4 from GITHub.
The following error is thrown:

File "C:\Anaconda2\lib\site-packages\mdfreader\mdf4reader.py", line 1128, in readSortedRecord
rec[channel.name][r] = channel.CFormat.unpack(buf[channel.posByteBeg:channel.posByteEnd])[0]
AttributeError: channel instance has no attribute 'CFormat'

I tracked that the error is caused, when it comes to read the channel 'ms', which is added to the channelList from the record.loadInfo() function.
Looking in the debugging mode into the channel objects, CFormat really does not exist for channel 'ms'. It exists only Format, which is however an empty string. This struct, CFormat, should be created in channel.set() from channel.Format which has been set by datatypeformat4(). However, channel.set is never called for the added channel 'ms', so that Format remains empty and CFormat is never created.
Channel.set() is called in line 990 in fct loadInfo() of class record and before channel ms is added.
For channel 'ms' (and 'hour, 'day', etc) the function channel.setCANOpen() is called. In this function Format is not filled by datatypeformat4() and CFormat is not created.
I added the following lines of code, which I copied from channel.set(), to the end of function channel.setCanopen():
if self.signalDataType not in (13, 14): # processed by several channels if not self.channelType == 1: # if not VSLD self.Format = datatypeformat4(self.signalDataType, self.bitCount) else: # VLSD self.Format = datatypeformat4(0, self.bitCount) self.CFormat = Struct(self.Format)
The script then runs without throwing this error, but I still need to check if data Import works correctly ...

We're parsing thousands of files.

Please suppress the printing of the file path.

When I read in my mf4 data I get the following error:

lib/python2.7/site-packages/mdfreader/mdf4reader.py in read4(self, fileName, info, multiProc, channelList, convertAfterRead, filterChannelNames)
   1303                                             buf[recordID]['data'] is not None: # no data in channel group
   1304                                         if isinstance(buf[recordID]['data'],recarray):
-> 1305                                             temp = buf[recordID]['data'].__getattribute__(convertName(recordName))  # extract channel vector
   1306                                         else:
   1307                                             temp = buf[recordID]['data'][convertName(recordName)]
lib/python2.7/site-packages/numpy/core/records.pyc in __getattribute__(self, attr)
    444             res = fielddict[attr][:2]
    445         except (TypeError, KeyError):
--> 446             raise AttributeError("recarray has no attribute %s" % attr)
    447         obj = self.getfield(*res)
    448 

AttributeError: recarray has no attribute Analog: Time_abs_title

It is caused by having besides a normal entry Time with signalDataType 0 an additional entry Time_abs with signalDataType 13. The class recordChannel in mdf4reader.py however doesn't handle this signalDataType. When I manually set the signalDataType of Time_abs to 0 (like Time), the data can be read in, however, I get the following error:

lib/python2.7/site-packages/mdfreader/mdf4reader.py in load(self, record, zip, nameList, sortedFlag)
    465             temps['data'] = self.load(record, zip=temps['hl_zip_type'], nameList=nameList, sortedFlag=sortedFlag)
    466         elif temps['id'] in ('##DT', '##RD', b'##DT', b'##RD'):  # normal sorted data block, direct read
--> 467             temps['data'] = record.readSortedRecord(self.fid, self.pointerTodata, channelList=nameList)
    468         elif temps['id'] in ('##SD', b'##SD'):  # VLSD
    469             temps['data'] = self.fid.read(temps['length'] - 24)

lib/python2.7/site-packages/numpy/core/records.pyc in fromfile(fd, dtype, shape, offset, formats, names, titles, aligned, byteorder)
    781     nbytesread = fd.readinto(_array.data)
    782     if nbytesread != nbytes:
--> 783         raise IOError("Didn't read as many bytes as expected")
    784     if name:
    785         fd.close()

IOError: Didn't read as many bytes as expected

Do you have any suggestion on how to handle the Time_abs entry with signalDataType 13 properly?

Thank you.

Encountered error when importing an mdf v3.20 file.
Line 720 of mdf3reader.py.

Python complains that the inputs to right_shift are not supported.
The code fails when temp is a 2D array with type |V8.

Hi,
iam trying your cod example on amdf file i have and i cant figure out whats actually going wrong, as it literallay has no code:

import mdfreader
yop = mdfreader.mdf('2015-09-29_10-53_JLR_Busbartest_50degC_000.MDF') # loads whole mdf file content in yop mdf object
yop.keys() # list channels names
yop.masterChannelList() # list channels grouped by raster or master channel
yop = mdfreader.mdfinfo() # create a mdfinfo instance
yop.listChannels('2015-09-29_10-53_JLR_Busbartest_50degC_000.MDF') # returns the list of channels

But when i execute the code i get an error when trying to list the channels:

runfile('D:/usr/bla/mdfopener.py', wdir='D:/usr/bla/')
2015-09-29_10-53_JLR_Busbartest_50degC_000.MDF
Traceback (most recent call last):

  File "<ipython-input-5-0536134f7c94>", line 1, in <module>
    runfile('D:/usr/bla/mdfopener.py', wdir='D:/usr/bla/Arbeit/CAE/E33/E33_Busbartest')

  File "D:\usr\bla\Apps\PortableApps\WinPython-64bit\python-3.4.4.amd64\lib\site-packages\spyderlib\widgets\externalshell\sitecustomize.py", line 845, in runfile
    execfile(filename, namespace)

  File "D:\usr\bla\Apps\PortableApps\WinPython-64bit\python-3.4.4.amd64\lib\site-packages\spyderlib\widgets\externalshell\sitecustomize.py", line 103, in execfile
    exec(compile(f.read(), filename, 'exec'), namespace)

  File "D:/usr/bla/mdfopener.py", line 18, in <module>
    yop.masterChannelList() # list channels grouped by raster or master channel

TypeError: 'dict' object is not callable

What is it what iam doing wrong? i really cant figure out whats wrong :(

Sincerely,
Blubbsy

What steps will reproduce the problem?
1. INCA mda file size of 11Mb 
2. Try to convert file to CSV format with Unified Scaling
* Smaller size file works good without issue

What is the expected output? What do you see instead?


What version of the product are you using? On what operating system?

WinXP

Please provide any additional information below.

----Error Message capture------------

c:\Util\MDF_convert\mdfreader.py:121: RuntimeWarning: invalid value encountered
in multiply
  L[channelName] = L[channelName] * P2 + P1
Error, following file might be corrupted : E:/Doc/mmm.dat
Please re-try by removing this file from the list and restart mdfconverter to 
kill processes and clean memory
Traceback (most recent call last):
  File "c:\Util\MDF_convert\mdfreaderui.py", line 106, in on_Convert_clicked
    result.get() # waits until finished
  File "C:\Python27\lib\multiprocessing\pool.py", line 528, in get
    raise self._value
MemoryError

Original issue reported on code.google.com by [email protected] on 4 Mar 2013 at 5:42

Hello,
I want to put this issue again. I used last committed version of 'mdf reader', you made it.

#################################

Current working script

#################################
import sys
import os.path
from mdfreader import mdfreader
qt = sys.argv[1]
filepath = '%s' %qt
print(filepath)
filepathToCsv = filepath+'.csv'

yop = mdfreader.mdf(filepath)
yop.keys()
yop.exportToCSV(filepathToCsv)
##################################
--> this works fine but number type data format have bee cut.

There is the key which name is 'VS' in my mdf(version3).
When I check my 'VS' data using the below script.
'vs = yop.getChannelData('VS') -->
4.500000000000000000e+01
4.500000000000000000e+01
4.500000000000000000e+01
4.400000000000000000e+01
...
'vs = yop.getChannelData('VS').astype(int) -->
45
45
45
44
...
After finished convert the mdf, I can only see the result '0.0' values for VS column in the CSV result.
I think getChannelData works properly, but ExportToCSV Function has little bug.
Could you please check this?

Best Regards,

MDF 3.20 VDMS2^@^@^@^@^@^@^@@^A^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@hdÐ^@^P^A^@^@þ7^K^@8^K^@¤^@20:01:201714:36:48PentaSystems^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@hkmc Namyang Research Center^@^@^@^@hkmc VDMS 2nd Construction^@^@^@^@^@^@hkmc VDMS Test Vehicle Data^@^@^@^@^@^@^@¥º<9b><81><9b>^T ^@
^@hkmc VDMS Device Reference Time^@dg^^@/>^@^@,^A^@^@^@^@^@^@^@^@^@^@^A^@^@^@^@^@^@^@cg^Z^@^@^@^@^@£^A^@^@f^A^@^@^@^@^R^@^P^@^@^@^@^@tx]^@{"channel":2,"message_id":550,"message_name":"NAVI_STD_STUB_E","comment":"[E] On Event"}^@cnä^@Å^B^@^@<87>^B^@^@^@^@^@^@^@^@^@^@^@^@^@^@^A^@Timechannel^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@@^@^P^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@cc>^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@seconds^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^B^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@ð?CNä^@r^E^@^@ä^C^@^@^@^@^@^@^@^@^@^@©^C^@^@^@^@STUB_MessageType^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@this is Stub Message for Map Based ADAS.^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@@^@^C^@^M^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@tx;^@{"comment":"This is Stub Message for Map Based ADAS."}^@ccn^A^@^A^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^K^@^H^@^@^@^@^@^@^@^@^@reserved^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@ð?Position^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@@segment^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^H@Stub^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^P@Profile Short^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^T@Profile Long^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^X@Meta Data^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@invalid^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@cnä^@Æ^F^@^@p^F^@^@^@^@^@^@^@^@^@^@6^F^@^@^@^@STUB_Offset^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@this is Stub Offset for Map Based ADAS.^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@C^@^M^@^M^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@tx:^@{"comment":"This is Stub Offset for Map Based ADAS."}^@CCV^@^@^A^@^@^@^@^@^@^@^@^@^@^@^@^@ÿ¿@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^K^@^A^@^@^@^@^@^@ÿ¿@invalid^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@cnä^@º^H^@^@ì^G^@^@^@^@^@^@^@^@^@^@ª^G^@^@^@^@STUB_CyclicCounter^@^@^@^@^@^@^@^@^@^@^@^@^@^@this is Stub Cyclic Counter for Map Based ADAS.^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@p^@^B^@^M^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@txb^@{"comment":"This is Stub Cyclic Counter for Map Based ADAS."}^@ccÎ^@^@^A^@^@^@^@^@^@^@^@^@^@^@^@^@^@^H@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^K^@^D^@^@^@^@^@^@^@^@^@Cyclic Counter^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@ð?Cyclic Counter^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@@Cyclic Counter^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^H@Cyclic Counter^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@cnä^@^G^T^@^@Ù ^@^@^@^@^@^@^@^@^@^@<9e> ^@^@^@^@STUB_PathIndex^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@this is Stub Message for Map Based ADAS.^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@r^@^F^@^M^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@tx;^@{"comment":"This is Stub Message for Map Based ADAS."}^@cc.
^@^A^@^@^@^@^@^@^@^@^@^@^@^@^@<80>O@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^K^@@^@^@^@^@^@^@^@^@^@FIRST path in the Horizon^@^@^@^@^@^@^@^@^@^@^@^@^@ð?Reserved^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@@reserved^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^H@Reserved^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^P@Reserved^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^T@Reserved^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^X@Reserved^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@reserved^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@ @general Path Index^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@"@general Path Index^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@$@general Path Index^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@&@general Path Index^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@(@general Path Index^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@@general Path Index^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@,@general Path Index^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@.@general Path Index^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@0@General Path Index^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@1@General Path Index^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@2@General Path Index^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@3@General Path Index^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@4@General Path Index^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@5@General Path Index^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@6@General Path Index^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@7@General Path Index^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@8@General Path Index^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@9@General Path Index^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@:@general Path Index^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@;@general Path Index^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@<@general Path Index^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@=@general Path Index^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@>@general Path Index^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@?@general Path Index^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@@@general Path Index^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@<80>@@general Path Index^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@A@General Path Index^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@<80>A@General Path Index^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@b@General Path Index^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@<80>B@General Path Index^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@C@General Path Index^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@<80>C@General Path Index^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@d@General Path Index^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@<80>D@General Path Index^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@e@General Path Index^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@<80>E@General Path Index^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@f@General Path Index^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@<80>F@General Path Index^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@g@General Path Index^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@<80>G@General Path Index^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@h@General Path Index^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@<80>H@General Path Index^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@i@General Path Index^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@<80>I@General Path Index^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@j@General Path Index^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@<80>J@General Path Index^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@k@General Path Index^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@<80>K@General Path Index^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@l@General Path Index^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@<80>L@General Path Index^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@m@General Path Index^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@<80>M@General Path Index^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@n@General Path Index^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@<80>N@General Path Index^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@O@General Path Index^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@<80>O@General Path Index^@^@^@^@^@^@^@^@^@^@^@^@^@^@cnä^@[^_^@^@-^U^@^@^@^@^@^@^@^@^@^@ë^T^@^@^@^@STUB_SubPathIndex^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@this is Stub Sub Path Index for Map Based ADAS.^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@X^@^F^@^M^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@txb^@{"comment":"This is Stub Sub Path Index for Map Based ADAS."}^@cc.

While making use of exportToHDF(), I have seen that the signals get allocated to wrong master channels. I have looked into the function exportToHDF() and managed to add a couple of lines of code with an 'elif' condition, thereby fixing the problem. However, I have tested the fix on only one mdf file, and am not sure if it solves the bug for all cases. I have put a screenshot below from the method 'exportToHDF' in mdfreader.py.

I was trying to go through the code and understand how exactly are you handling the channel data which are sampled at different frequencies. Are you up-sampling it or down-sampling it?

Thanks in advance.

KeyError Traceback (most recent call last)
in ()
----> 1 data = mdfreader.mdf('C:/Users/q400300/Desktop/horor/measures_10.11.2016_14_20_32.mf4')

C:\Users\q400300\Desktop\work\mdf.pyc in init(self, fileName, channelList, convertAfterRead, filterChannelNames)
104 self.fileName = fileName
105 if fileName is not None:
--> 106 self.read(fileName, channelList=channelList, convertAfterRead=convertAfterRead, filterChannelNames=filterChannelNames)
107
108

C:\Users\q400300\Desktop\work\mdfreader.pyc in read(self, fileName, multiProc, channelList, convertAfterRead, filterChannelNames)
345
346 # read file blocks
--> 347 info = mdfinfo(self.fileName, filterChannelNames)
348
349 self.MDFVersionNumber = info.mdfversion

C:\Users\q400300\Desktop\work\mdfreader.pyc in init(self, fileName, filterChannelNames)
147 self.mdfversion = 0
148 if fileName is not None:
--> 149 self.readinfo(fileName, filterChannelNames)
150
151 def readinfo(self, fileName=None, filterChannelNames=False):

C:\Users\q400300\Desktop\work\mdfreader.pyc in readinfo(self, fileName, filterChannelNames)
180 else: # MDF version 4.x
181 from mdfinfo4 import info4
--> 182 self.update(info4(None, fid))
183
184 def listChannels(self, fileName=None):

C:\Users\q400300\Desktop\work\mdfinfo4.py in init(self, fileName, fid)
1148 fid.close()
1149 elif fileName is None and fid is not None:
-> 1150 self.readinfo(fid)
1151
1152 def readinfo(self, fid):

C:\Users\q400300\Desktop\work\mdfinfo4.py in readinfo(self, fid)
1195
1196 # reads Data Group Blocks and recursively the other related blocks
-> 1197 self.readDGBlock(fid)
1198
1199 def readDGBlock(self, fid, channelNameList=False):

C:\Users\q400300\Desktop\work\mdfinfo4.py in readDGBlock(self, fid, channelNameList)
1218 self['DGBlock'][dg].update(DGBlock(fid, self['DGBlock'][dg - 1]['dg_dg_next']))
1219 # reads Channel Group blocks
-> 1220 self.readCGBlock(fid, dg, channelNameList)
1221
1222 def readCGBlock(self, fid, dg, channelNameList=False):

C:\Users\q400300\Desktop\work\mdfinfo4.py in readCGBlock(self, fid, dg, channelNameList)
1252 if not self['CGBlock'][dg][cg]['cg_flags'] & 0b1: # if not a VLSD channel group
1253 # reads Channel Block
-> 1254 self.readCNBlock(fid, dg, cg, channelNameList)
1255 else:
1256 VLSDCGBlock.append(cg)

C:\Users\q400300\Desktop\work\mdfinfo4.py in readCNBlock(self, fid, dg, cg, channelNameList)
1402
1403 # reads Channel Conversion Block
-> 1404 self['CCBlock'][dg][cg][cn] = CCBlock(fid, self['CNBlock'][dg][cg][cn]['cn_cc_conversion'])
1405
1406 MLSDChannels = self.readComposition(fid, dg, cg, MLSDChannels, channelNameList=False)

C:\Users\q400300\Desktop\work\mdfinfo4.py in init(self, fid, pointer)
818 def init(self, fid=None, pointer=None):
819 if fid is not None:
--> 820 self.read(fid, pointer)
821
822 def read(self, fid, pointer):

C:\Users\q400300\Desktop\work\mdfinfo4.py in read(self, fid, pointer)
855 elif ID in ('##CC', b'##CC'): # for table conversion
856 # much more complicated nesting conversions !!!
--> 857 self['cc_ref'][i] = CCBlock(fid, self['cc_ref'][i])['name']['Comment']
858 if self['cc_md_comment']: # comments exist
859 self['Comment'] = CommentBlock(fid, self['cc_md_comment'], MDType='CC')

KeyError: 'name'

value.decode does not work if value is a str, which it is if it is set to = '' as on row 653
tested on Python3.

suggested fix, only run block 654-663 if value != ''

The console always shows an error, when trying to convert the mdf file to .m - converting to csv works fine

Operating system: Win 7, 64 Bit.
Converter version: newest, freshly pulled

Error code:
Error, following file might be corrupted : /cygdrive/d/testHILAPI/stress_test-08_06_2015-11_32_21.MDF
Traceback (most recent call last):
File "/cygdrive/C/mdfreader-master/mdfconverter/mdfreaderui.py", line 111, in on_Convert_clicked
result = list(map(processMDFstar, args))
File "/cygdrive/C/mdfreader-master/mdfconverter/mdfreaderui.py", line 333, in processMDFstar
raise Exception('Please re-try by removing this file from the list and restart mdfconverter to kill processes and clean memory')
Exception: Please re-try by removing this file from the list and restart mdfconverter to kill processes and clean memory

As stated in Response - I set MultiProc to False

The specific file mentioned in the Error can be sent via Mail/Link if necessary for testing purposes.

Regards David (Argyll)

Hello,

I want to read some MDF files with the mdfreader tool. My MDF files have some fields which are called "Dauermesung_1: Time", "Dauermessung_2: Time", etc. If I read these files with mdfreader.mdf() I get the error

ValueError: ':' is not an allowed character in buffer field names

Is there a chance to fix this?

Best regards, donvicario

Installing under cygwin (Windows7, 64bit OS) or Linux (Kubuntu 12.04) doesn't work and comes up with the following error:

$ python setup.py install
Traceback (most recent call last):
File "setup.py", line 6, in
from Cython.Build import cythonize
ImportError: No module named Cython.Build

Hello,

got a problem when i start setup.py, seems to be with dataRead.pyx. I am using the Winpython Package.

Hello,

I wanted to read a mdf file from CANoe (mdf version 3.x).

Code:
import mdfreader
file = mdfreader.mdf("CANOE.MDF")

But it failed with the following python traceback:
Traceback (most recent call last):
File "C:\Users\XXX\Desktop\test.py", line 16, in
file = mdfreader.mdf("CANOE.MDF")
File "C:\Python27\lib\site-packages\mdfreader-0.2.4-py2.7-win32.egg\mdfreader\mdf.py", line 114, in init
self.read(fileName, channelList=channelList, convertAfterRead=convertAfterRead, filterChannelNames=filterChannelNames)
File "C:\Python27\lib\site-packages\mdfreader-0.2.4-py2.7-win32.egg\mdfreader\mdfreader.py", line 360, in read
self.read3(self.fileName, info, multiProc, channelList, convertAfterRead, filterChannelNames=False)
File "C:\Python27\lib\site-packages\mdfreader-0.2.4-py2.7-win32.egg\mdfreader\mdf3reader.py", line 825, in read3
buf.read(channelSet) # reads datablock potentially containing several channel groups
File "C:\Python27\lib\site-packages\mdfreader-0.2.4-py2.7-win32.egg\mdfreader\mdf3reader.py", line 651, in read
data = self.loadUnSorted(nameList=channelSet)
File "C:\Python27\lib\site-packages\mdfreader-0.2.4-py2.7-win32.egg\mdfreader\mdf3reader.py", line 696, in loadUnSorted
for recordID in record:
TypeError: 'type' object is not iterable

Could you please help me with this problem.

Rolf

Sorry I am new to python but I receive folloing error When I want to run following lines:

import mdfreader
FILENAME = 'm35_1.dat'
yop=mdfreader.mdf(FILENAME)

runfile('C:/Users/yyazicio/Desktop/Works/07.Python/mdfreader.py', wdir='C:/Users/yyazicio/Desktop/Works/07.Python')
Traceback (most recent call last):

File "", line 1, in
runfile('C:/Users/yyazicio/Desktop/Works/07.Python/mdfreader.py', wdir='C:/Users/yyazicio/Desktop/Works/07.Python')

File "C:\Users\yyazicio\AppData\Local\Continuum\Anaconda3\lib\site-packages\spyder\utils\site\sitecustomize.py", line 866, in runfile
execfile(filename, namespace)

File "C:\Users\yyazicio\AppData\Local\Continuum\Anaconda3\lib\site-packages\spyder\utils\site\sitecustomize.py", line 102, in execfile
exec(compile(f.read(), filename, 'exec'), namespace)

File "C:/Users/yyazicio/Desktop/Works/07.Python/mdfreader.py", line 8, in
import mdfreader

File "C:\Users\yyazicio\Desktop\Works\07.Python\mdfreader.py", line 10, in
yop=mdfreader.mdf(FILENAME)

AttributeError: module 'mdfreader' has no attribute 'mdf'

and install version is
python 3.6
scipy: 0.18.1
numpy: 1.11.3
matplotlib: 2.0.0
pandas: 0.19.2
statsmodels: 0.6.1
sklearn: 0.18.1

Hello,

I had a problem importing some signals based on 1 tab interpolation (extra temperature/pressure sensors). The code does not work, here my fix :

def tabInterpConv(data, conv):  # 1 Tabular with interpolation
    """ apply Tabular interpolation conversion to data

    Parameters
    ----------------
    data : numpy 1D array
        raw data to be converted to physical value
    conv : mdfinfo3.info3 conversion block ('CCBlock') dict

    Returns
    -----------
    converted data to physical value
    """
    _K = sorted(conv.keys())
    _x = [conv[_k]['int'] for _k in _K]
    if (all(diff(_x)) > 0):
        _y = [conv[_k]['phys'] for _k in _K]
        return interp(data, _x, _y)
    else:
        print(('X values for interpolation of channel are not increasing'))
        return data

Kind regards and thanks for the great work!

Emmanuel

I've been working on a python library that generates an MDF file from scratch, given the inputs for channel groups. I'm noticing that when I use Vector's MDF Validator tool I'm not getting any errors and all of my channel groups are recognized just fine. I am, however, producing unsorted MDFs and am wondering if mdfreader only handles sorted files?

When I try to upload an MDF using your package I am seeing a KeyError: 'data' error. https://imgur.com/a/Bjapk <-- Screenshot of console

Any ideas what this might be?

Hey i am getting following error when i try to import mdfreader

import mdfreader
Traceback (most recent call last):
File "", line 1, in
File "mdfreader/init.py", line 25, in
from .mdf4reader import mdf4
File "mdfreader/mdf4reader.py", line 38, in
from mdfreader.mdfinfo4 import info4, MDFBlock, ATBlock, IDBlock, HDBlock, DGBlock, CGBlock, CNBlock, MDFBlock, FHBlock, CommentBlock
File "mdfreader/mdfreader.py", line 42, in
from mdf4reader import mdf4
ImportError: cannot import name mdf4

Hi,

I installed the latest version using pip:

$ pip install git+https://github.com/ratal/mdfreader
Collecting git+https://github.com/ratal/mdfreader
Cloning https://github.com/ratal/mdfreader to c:\users\stoev\appdata\local\temp\pip-jv3atn6p-build
Requirement already satisfied: numpy>=1.7 in c:\users\stoev\appdata\local\continuum\anaconda3\lib\site-packages (from mdfreader==0.2.4)
Requirement already satisfied: sympy in c:\users\stoev\appdata\local\continuum\anaconda3\lib\site-packages (from mdfreader==0.2.4)
Requirement already satisfied: cython>=0.21 in c:\users\stoev\appdata\local\continuum\anaconda3\lib\site-packages (from mdfreader==0.2.4)
Installing collected packages: mdfreader
Running setup.py install for mdfreader: started
Running setup.py install for mdfreader: finished with status 'done'
Successfully installed mdfreader-0.2.4

Now an attempt to import the mdfreader results in an ImportError:

import mdfreader
Traceback (most recent call last):
File "..\Continuum\Anaconda3\lib\site-packages\IPython\core\interactiveshell.py", line 2881, in run_code
exec(code_obj, self.user_global_ns, self.user_ns)
File "", line 1, in
import mdfreader
File "C:\Program Files (x86)\JetBrains\PyCharm Community Edition 2016.3.2\helpers\pydev_pydev_bundle\pydev_import_hook.py", line 21, in do_import
module = self.system_import(name, *args, **kwargs)
File "..\Continuum\Anaconda3\lib\site-packages\mdfreader_init.py", line 30, in
from mdfreader import mdf,mdfinfo
ImportError: cannot import name 'mdfinfo'

The version from https://pypi.python.org/pypi/mdfreader/0.2.3 runs without any problems

Dear Aymeric,

Unfortunately, I found a corner case in some of my files, that causes an numpy core records error:

lib/python2.7/site-packages/numpy/core/records.pyc in fromfile(fd, dtype, shape, offset, formats, names, titles, aligned, byteorder)
    781     nbytesread = fd.readinto(_array.data)
    782     if nbytesread != nbytes:
--> 783         raise IOError("Didn't read as many bytes as expected")
    784     if name:
    785         fd.close()

It is triggered, when numberOfRecords == 0.
In the readSortedRecord function, I'm in the case channelList is None and self.byte_aligned. Then the call
return fromfile(fid, dtype=self.numpyDataRecordFormat, shape=self.numberOfRecords, names=self.dataRecordName) throws the numpy error.

Do you have a suggestion how to handle this corner case? It is a measurement, which is written to file, but no data is actually recorded.

Thanks,
Katharina

problem parsing metadata for CC block
Def_float<common_properties>%6.3f</common_properties>

I come up with this type of issue all the time while trying to read mdf data

Hi,

being new to mdf files I'm doing my first steps with your tool. After having successfully altered some channel data and written new channels to a file, I get some issues in another program, similar to a problem reported in issue #15. First the code:
So what I wanna do is read some mdf files and for each file add a new channel and write to a new mdf file.

#some stuff, mainly creating assignements
last = 0
#run through all mdf files I want to consider
#all files have only channels of the same length
for name in file_names:
    mdf_file = mdfr.mdf(name)

    #get size of channels and read corresponding part of assignements
    #since assignements has size mdf_file1_channelsize + mdf_file2_channelsize + ...
    #channel_names is a list of channels, contained in all files
    dim_channel = mdf_file.getChannelData(channel_names[0]).shape[0]
    data = np.asarray(assignments[last:last+dim_channel])
    last = last + dim_channel

    #get master channel and set other metadata
    master_channel = mdf_file.getChannelMaster(channel_names[0])
    channel_name = 'Assignements'
    description = 'Assignes data to different categories'

    #add channel
    mdf_file.add_channel(dataGroup=1, channel_name=channel_name, data=data, master_channel=master_channel, master_type=1, unit='',description=description)

    #if I do print(mdf_file) here, everything looks fine
    #print(mdf_file)

    #write
    mdf_file.write('./something.dat')

The program runs fine, no errors are raised. Still, if I want to open the created mdf files, I get "ValueError: Not enough bytes left in file for specified shape and type".
Also I realized that the data of the channels is heavily altered (somehow managed to read the previously existing files with another program).
Any ideas on what I'm doing wrong?
Greetings,
Bibo

What is the expected output? What do you see instead?
How do you convert an MDF file into Pandas from the command line?

What version of the product are you using? On what operating system?
Ubuntu, python 2.7

Please provide any additional information below.

Original issue reported on code.google.com by [email protected] on 25 Nov 2014 at 6:20

Firstly, my sincerest apologies if this is not the place to post this - I am new to Github and wanted to raise a question but could not see where else to do it.
I am working with some very large MDF files, >1 or 2GB. As such, my machine crashes when trying to convert them to CSV. If I pull in a few channels, it works fine, so the software is installed correctly and working. Is there any way to convert a portion of a file and so not to have to read in the whole thing? I can use other CAN tools to do this but, if I could do all of my processing in Python, that would be amazing.
Many thanks for this software - it has been good to use so far.
Yours,
Peter

Hello,

If I use below script the VS data number of data have been cut.
yop.exportToCSV(filepathToCsv)
value : 2.500000000000000000e+01(exponential expression)
After using exportToCSV
this the record expressed 0.

Information have lost. Do you know how I can fix it?

Best Regards,

Hi,

sorry if I am not using the typical way of bug reporting but I am a very newbie on GitHub. I just tried the mdfreader on my MDF 4.1 files (measurement files from endurance tests) just to see if it was working and:

• I was capable to get an info object and use it, get a list of channels and so on
• by trying to read the data(mdfreader.mdf('file name'), I am getting an error message: IndexError: bitarray index out of range. The file is 1.4GB big.

My working environment is Anaconda 3.4.2.0 64bits on Win7.

Here the whole error message:
f = mdfreader.mdf(r'C:\devl\data\Loadbox\mf4_data\2016-09-23 00-00-00 (4).mf4')

C:\devl\data\Loadbox\mf4_data\2016-09-23 00-00-00 (4).mf4
Unexpected error: (<class 'ImportError'>, ImportError("No module named 'mdfreader.dataRead'",), <traceback object at 0x0000000009696208>)
dataRead crashed, back to python data reading

ImportError Traceback (most recent call last)
C:\devl\devltools\Anaconda3\lib\site-packages\mdfreader\mdf4reader.py in readBitarray(self, bita, channelList)
1178 try: # use rather cython compiled code for performance
-> 1179 from .dataRead import dataRead
1180 for chan in range(len(self)):

ImportError: No module named 'mdfreader.dataRead'

During handling of the above exception, another exception occurred:

IndexError Traceback (most recent call last)
in ()
----> 1 f = mdfreader.mdf(r'C:\devl\data\Loadbox\mf4_data\2016-09-23 00-00-00 (4).mf4')

C:\devl\devltools\Anaconda3\lib\site-packages\mdfreader\mdf.py in init(self, fileName, channelList, convertAfterRead, filterChannelNames)
104 self.fileName = fileName
105 if fileName is not None:
--> 106 self.read(fileName, channelList=channelList, convertAfterRead=convertAfterRead, filterChannelNames=filterChannelNames)
107
108

C:\devl\devltools\Anaconda3\lib\site-packages\mdfreader\mdfreader.py in read(self, fileName, multiProc, channelList, convertAfterRead, filterChannelNames)
351 self.read3(self.fileName, info, multiProc, channelList, convertAfterRead, filterChannelNames=False)
352 else: # MDF version 4.x. Channel by channel reading implemented
--> 353 self.read4(self.fileName, info, multiProc, channelList, convertAfterRead, filterChannelNames=False)
354
355 def write(self, fileName=None):

C:\devl\devltools\Anaconda3\lib\site-packages\mdfreader\mdf4reader.py in read4(self, fileName, info, multiProc, channelList, convertAfterRead, filterChannelNames)
1364 channelList.extend(['ms', 'min', 'hour', 'day', 'month', 'year'])
1365
-> 1366 buf.read(channelList) # reads raw data from data block with DATA and DATABlock classes
1367
1368 # processing data from buf then transfer to self

C:\devl\devltools\Anaconda3\lib\site-packages\mdfreader\mdf4reader.py in read(self, channelList, zip)
367 recordID = list(self.keys())[0]
368 record = self[recordID]['record']
--> 369 self[recordID]['data'] = self.load(record, zip=None, nameList=channelList, sortedFlag=True)
370 for cn in record.VLSD: # VLSD channels
371 if channelList is None or record[cn].name in channelList:

C:\devl\devltools\Anaconda3\lib\site-packages\mdfreader\mdf4reader.py in load(self, record, zip, nameList, sortedFlag)
449 data_block['id'] = '##DT' # do not uncompress in DATABlock function
450 data_block['data'] = buf
--> 451 temps['data'] = DATABlock(record, parent_block=data_block, channelList=nameList, sortedFlag=sortedFlag)
452 else: # empty datalist
453 temps['data'] = None

C:\devl\devltools\Anaconda3\lib\site-packages\mdfreader\mdf4reader.py in DATABlock(record, parent_block, channelList, sortedFlag)
94 return fromstring(parent_block['data'], dtype=record.numpyDataRecordFormat, shape=record.numberOfRecords, names=record.dataRecordName)
95 else: # record is not byte aligned or channelList not None
---> 96 return record.readBitarray(parent_block['data'], channelList)
97 else: # unsorted reading
98 print('not implemented yet unsorted data block reading', file=stderr) # to be implemented if needed, missing example file

C:\devl\devltools\Anaconda3\lib\site-packages\mdfreader\mdf4reader.py in readBitarray(self, bita, channelList)
1227 temp[i].extend(byte)
1228 else: # signed integer (two's complement), keep sign bit and extend with bytes
-> 1229 temp = signedInt(temp, byte)
1230 nTrailBits = self[chan].nBytes*8 - self[chan].bitCount
1231 if self[chan].signalDataType in (2, 3) and \

C:\devl\devltools\Anaconda3\lib\site-packages\mdfreader\mdf4reader.py in signedInt(temp, extension)
1203 extensionInv.setall(True)
1204 for i in range(self.numberOfRecords): # extend data of bytes to match numpy requirement
-> 1205 signBit = temp[i][-1]
1206 if not signBit: # positive value, extend with 0
1207 temp[i].extend(extension)

IndexError: bitarray index out of range

If you are interested, I may try to give you the file leading to this failure - if it is possible to upload such big file on GitHub...

Thanks!