1 Tabular with interpolation about mdfreader HOT 9 CLOSED

Thanks, I tested it and it is ok to me. It has little impact on mdf3reader, except maybe performance but I think marginal and not so common case.

However, according to specification, conv[]['int'] should be in increasing order. Below is an extract of it:
Base is a table with n break points. The int values are given in an increasing order (strictly
monotonous increasing). If a value x is located between two table values, (tablevalue[i] ≤ x <
tablevalue[i+1]), linear interpolation between physvalue[i] and physvalue[i+1] is used. Is x < than Int
value 1, Physvalue 1 will be returned. If x is larger than or equal to the greatest Int value, Phys value n
will be returned.

Then I am even wondering in your case if the int values are correctly paired with the physvalue.
I could commit your proposal but I would to understand first what software generated this mdf3 file ?

from mdfreader.

Actually, test for monotonously increasing int value (all(diff(_x)) > 0) is not needed anymore, the else part also if we apply sorting.
Once again, I am not sure if we should sort pair of int&physvalue or only int... do I misunderstand your modification ?

from mdfreader.

The file is written by etas inca.
The all and sort should not be necessary, indeed. My modification was only meant to take in account the data structure of conv

Tested successfully with :

def tabInterpConv(data, conv):  # 1 Tabular with interpolation
    """ apply Tabular interpolation conversion to data

    Parameters
    ----------------
    data : numpy 1D array
        raw data to be converted to physical value
    conv : mdfinfo3.info3 conversion block ('CCBlock') dict

    Returns
    -----------
    converted data to physical value
    """
    _K = conv.keys()
    _x = [conv[_k]['int'] for _k in _K]
    _y = [conv[_k]['phys'] for _k in _K]
    return interp(data, _x, _y)

from mdfreader.

Actually, I realized I could never try this conversion as I do not have any file using it. Would you have a dummy file to share that I can confirm and optimize the code ?
For optimization, interp is from numpy, _x and _y should rather be numpy arrays and not lists. To convert the nested dict conv, I think it would be more efficient to use something like:

tmp = array([(key, val['int'], val['phy']) for (key, val) in conv.items()])
return interp(data, tmp[:,1], tmp[:,2])

Can you give a try ?
tabConv function (conversion type 2, no interpolation but nearest low int value) should also be modified, tentatively (if you also have occasion to try?):

tmp = array([(key, val['int'], val['phy']) for (key, val) in conv.items()])
indexes = searchsorted(tmp[:, 1], data)
return tmp[indexes, 2]

from mdfreader.

Hello,

I will try to give you a (small) dummy file in the next days. In fact, the very big problem is yes the optimization… because such signals can be recorded at 2kHz (sometimes 10kHz or more) and the module can not proceed such data (python complains about memory issues!).

I'll try the interpolation and nearest low int value; however in MDA, the signal are smoothed.

Regards,
Emmanuel

from mdfreader.

No feed back since a month, I close the issue.

from mdfreader.

Hello,

sorry to relaunch the topic (I had one month vacation inbetween). I just tested the last release and there is still an error ('phy' instead of 'phys' in the dictionary key).

Apart from that, the extraction of 'int' and 'phys' pairs is a little bit quicker with two list comprehension, even if I create two numpy arrays in the returned value of the "modif" function. Tested on an old laptop of 2011...

tableOfConversion = {
   1 : {'int' : 1.2,
        'phys' : 25
   },
   2 : {'int' : 1.9,
        'phys' : 158.6
   },
   3 : {'int' : 2.8,
        'phys' : 333
   },
   4 : {'int' : 3.8,
        'phys' : 400
   },
   5 : {'int' : 3.99,
        'phys' : 458
   },
   6 : {'int' : 3.8,
        'phys' : 785.3
   },
   7 : {'int' : 4.9,
        'phys' : 997.5
   },
}
def orig(conv):  # Original code
   tmp = np.array([(key, val['int'], val['phys']) for (key, val) in conv.items()])
   return (tmp[:,1], tmp[:,2])
def modif(conv):  # Modification
   _x = [conv[_k]['int'] for _k in conv.keys()]
   _y = [conv[_k]['phys'] for _k in conv.keys()]
   return (_x, _y)

%timeit orig(tableOfConversion)
# returns : 100000 loops, best of 3: 7.87 µs per loop

%timeit modif(tableOfConversion)
# returns: 100000 loops, best of 3: 2.25 µs per loop

I guess the function tabConv is impacted too.

Concerning the sample dat file, I can not anonymize the file, I can not provide anything unfortunately.

Regards
Emmanuel

from mdfreader.

Dear fju5jdjhue,
Thanks for your contribution. If you tell me it is quicker your way, I believe it, anyway I can not test it ;)
Could you please issue then a pull request, I will accept it ?
About memory issues, do not forget to try using convertAfterRead=False argument in read method, it should save you generally 1/3 to 1/2 of memory (theoritically uses same size of mdf file in memory, only converts data on demand using getChannelData method).
Regards
Aymeric

from mdfreader.

Good evening,
thank you for the correction in the version 0.2.0. No need to patch any longer
Regards
Emmanuel

from mdfreader.