costerwi / dwdatareader Goto Github PK

The DWChannel.scaled() function eagerly reads all channel data into a pandas.Series. For big .d7d files, this leads to MemoryError. I am not familiar with the DWDataReaderLib64.dll, but as a pure Python solution, one could present channel data as a generator, i.e. using yield for every measurement or chunk of measurements.

This would enable using the .d7d as a streaming source of data (e.g. to put into a data base or analyze on the fly). What do you think?

.dxd files generated with new Dewesoft release (2023.4 RELEASE-230724) could not be opened with dwdatareader: "DWError: d7d file corrupt".

Since 8/31/2023 there is a new v4_2_0_25 Dewesoft Data Reader Library: https://downloads.dewesoft.com/developers/DWDataReader.zip
The .so and DLL files should be updated to the newest version. Hopefully, this will fix the issue.

Thank you in advance.

We are using your library to batch process recordings made by others and supplied to us as d7d files. For quality assurance purposes we are collecting and archiving the metadata of the recordings (example code included below). We can pull out most of what we need, but our client recently asked us to also capture the LP Filter and the LP Filter data fields. We can see these in DevesoftX but can’t find them in the data structures available under fh below.

import dwdatareader as dw # from https://github.com/costerwi/dwdatareader
 
fh = dw.open(fname)
my_channels_info = [] # empty list to hold the signals info
for ch in fh.values():
      hkeys = fh.header.keys()
      customer = fh.header['CUSTOMER'] if 'CUSTOMER' in hkeys else 'unknown customer'
      testnum = fh.header['TESTNUMBER'] if 'TESTNUMBER’ in hkeys else 'undefined'
      row = {'name': ch.name, # these are added into a summary dataframe later.
            'unit': ch.unit,
            'desc': ch.description,
            'nsam': ch.number_of_samples,
            'customer': customer,
            'testnum': testnum,
            't_start': fh.info.start_store_time,
            'rate': fh.info.sample_rate,
            'duration': fh.info.duration,
            'fname': raw_fname
      }
      my_channels_info.append(row)
 
# transfer channels info into a dataframe
df1ch = pd.DataFrame(my_channels_info)

Here is the pertinent screen shot from DewesoftX

What we need is the stuff that opens to the right when hovering over these cells. Any guidance you can offer is welcome.
Thanks in advance!

Hi,
we're having trouble with older .dxd files, that can't be read with the current library.

Could you please upgrade to the new library provided by dewesoft?

https://developer.dewesoft.com/questions/issue-in-dwdatareader-version-4020021

Thanks in advance,
Alex

In my usecase reading in a channel with +50million signals I sometimes face memory issues on the system the code is running.

An idea would be to offer this pagination with an offset and count in the scaled method like

def scaled(self,offset:int, count:int):

so one channel could be read in a parallel distributed way.

The C API has a DWGetChannelProps to access meta information of a channel, e.g. the module it was associated with in the .d7s Setup file. It would be good to have a binding for these properties. If you agree, I will follow up a PR, I've already implemented this.

I have been able to get the field names in the dxd file, how can I get the data in each field name?

I am able to import analog channel data and counter channel data but not CAN data from the same Dewesoft project files. They are all assigned NaN values. I am using version 0.15.0 4020021. Thank you for the assistance.

Hello, i wanted to check if there is a solution to get dwdatareader to work under macOS operating environment?

Error:
File "..\Miniconda3\lib\site-packages\dwdatareader_init_.py", line 532, in open
return DWFile(source)
File "..\Miniconda3\lib\site-packages\dwdatareader_init_.py", line 360, in init
self.open(source) # If this fails then the instance is not constructed
File "..\Miniconda3\lib\site-packages\dwdatareader_init_.py", line 390, in open
raise DWError(stat)
dwdatareader.DWError: error in DLL

I am getting this error when trying read the newest files created from dewesoft. The ones we collected later last year seem to read in fine :(.

In the script it is failing to just open the files
with dw.open(filepath) as f:
df = f.dataframe()

Hi,

I'd like to understand how it is possible, that the current codebase in master isn't able to run all unittests successfully on Linux?
Is this system-related or are the tests deprecated?

Thank you.
Martin

There is an implied request on the Dewesoft forum to upgrade the libraries so that .dxd files will be supported.

Hi, this is my code:
.
.
import dwdatareader as dw
print(dw.version, dw.getVersion())
for dxd_key in dxd_list:
with dw.open(Path(dxd_key)) as f:

and this is the error I get. any idea?
AttributeError: 'WindowsPath' object has no attribute 'encode'

Looking through a bunch of Dewesoft data that was stored based on a trigger events.

Is there a way that I could read in the information that triggered the event for each file imported into Python? This would not include the "Storing started" or "Storing stopped", just the "Data trigger events" line

Was reading through the init.py to determine if I could get this information myself, however, it is above my current level of skill.

First off, thanks for the great Dewesoft reader library.
I was recently using it for my datafiles which are DXD and are created on a Windows x64, en-US machine.

The units had some unicode characters for degree symbol and ohms. When I imported it with this library it had the classic Å symbol which is the give away of reading UTF-8 binary data but assuming it should be decoded according to Windows codepage (looks like you have ISO-8859-1 chosen).

A quick peek into the python code and I saw this is extremely easy to fix in this library - just call dwdatareader.encoding = 'utf-8' and it gives the correctly decoded strings.

I just wanted to file an issue to bring up the fact that it appears that DewesoftX is encoding strings in UTF-8 and perhaps this library should change the default encoding to match?

Unfortunately I am only sample size of one and have not tested other locales or versions of Dewesoft, so I am not sure if this default encoding applies everywhere. Thanks for your time!

With the latest versions of the DWDataReaderLib and a file generated with DEWESoft X3 SP2, the channels informations can be retrieved.

The new library adds one variable to the DWChannel structure which is "data_type".

The issue is corrected by changing the DWChannel fields variable to :

_fields_ = [("index", ctypes.c_int), ("_name", ctypes.c_char * 100), ("_unit", ctypes.c_char * 20), ("_description", ctypes.c_char * 200), ("color" , ctypes.c_uint), ("array_size", ctypes.c_int), ("data_type", ctypes.c_int)]

Hi,
i have ran this code: import dwdatareader as dw with dw.open('Essais_vol_0011.d7d') as f: print(f.info) ch1 = f['chname1'].series() ch1.plot() for ch in f.values(): print(ch.name, ch.series().mean())
And the code give me this error "DWError: error in DLL"

Traceback (most recent call last): File "C:\Users\2101550\Documents\Projects\Python\Test.py", line 11, in import dwdatareader as dw File "C:\Users\2101550\Documents\Projects\Python\dwdatareader__init__.py", line 524, in loadDLL() File "C:\Users\2101550\Documents\Projects\Python\dwdatareader__init__.py", line 505, in loadDLL raise DWError(stat) DWError: error in DLL.

This is the error that is written in my console.

I've tried the example on the website but it doesn't work either, can you help me?

Hello,

The DLL seems to be failing when updating to the latest release of DewesoftX, 2023.1.

Traceback (most recent call last):

File "C:\ProgramData\Anaconda\lib\site-packages\spyder_kernels\py3compat.py", line 356, in compat_exec
exec(code, globals, locals)

File "c:\users\jesper\documents\python\projects\dewe_dyno\process_dxd.py", line 18, in
fdw = dw.open("C:/Users/Public/Documents/Dewesoft/Data/12-01-2023/Run_08.dxd")

File "C:\ProgramData\Anaconda\lib\site-packages\dwdatareader_init_.py", line 538, in open
return DWFile(source)

File "C:\ProgramData\Anaconda\lib\site-packages\dwdatareader_init_.py", line 360, in init
self.open(source) # If this fails then the instance is not constructed

File "C:\ProgramData\Anaconda\lib\site-packages\dwdatareader_init_.py", line 390, in open
raise DWError(stat)

DWError: error in DLL

Hi,

using the dwdatareader lib we ascertained two bugs in the DWDataReaderLib

• DWGetScaledSamplesCount() is zero when using big data file with asynchronous channels
• Wrong time values in files with lost data ('.lost.dxd')

We issued them and Dewesoft released a new version that solved those issues. Please update the repo with the newest files...

Thanks and best wishes

/chr

Looks like dewesoft has released DLL v4.0.2.23, causing some version mismatch issues when trying to use this library with newly collected data.

Hi,

we are using the power module function a lot, with several AC and DC power modules in one measurement. The problem is, that the dwdatareaderlib-API gives only the channel name (like U_L1, U_L2, P_H1, ...), but not the related power module name itself. Hence there are several channels with the same name

This has several implications:

• __getitem__ gives only the first occurrence of a channel with the given name. We cannot, for example, get the second occurrence of the name
• __iter__ also returns the name, which results in the same problem. One cannot iterate over DWFile and then index with the channel name you get in the iteration
• Indexing by number results in two nested for loops (we have data files with >400 channels and > a few GB, this is just super long)
• The only way you can find out more about the channel is by requesting the channel-XML. But this only gives you the information that the channel is part of a power module with its according index, not the name of the module itself

Our solution/dirty hack is:

• Extract header file from data file
• Analyze this XML file, go through all math channels and then through all power modules
• Make a list of all power modules with name and index
• Iterate through all channels by index, check if name (like U_L1) and power module index matches with our name/index-list
• If yes, read data
• When done, delete XML header file

The Dewesoft support knows the problem and affirmed that our solution is at the moment the only way to get the power module name. And it sounded that there won't be an update in the dwdatareaderlib in the near future that solves the problem in an elegant way. So I wanna start a discussion if and how we can integrate it in the dwdatareader

One point is, how to get channel data when the name is not unique. Then, handling huge files with a lot of channels. Finally and most important, how we can implement something like DWChannel.get_math_module_number(), DWChannel.get_math_module_type() and DWChannel.get_math_module_name() ?

Best wishes
/chr

I think you are missing pytz in the requirements.

Traceback (most recent call last):
    print(f.info)
  File "[...]/python3.8/site-packages/dwdatareader/__init__.py", line 44, in __str__
    return "{0.start_store_time} {0.sample_rate} Hz {0.duration} s".format(self)
  File "[...]/python3.8/site-packages/dwdatareader/__init__.py", line 50, in start_store_time
    import pytz
ModuleNotFoundError: No module named 'pytz'

I tested it with pytz==2021.1
To auto install add
install_requires=["pytz==2021.1"]
into the setup.py setup kwargs.

First of all, thanks for this great library, it improved so many things for me.

I noticed I cannot access certain channels, e.g. when we do Flicker-measurements. The channels are called e.g. "I_Pst_L1", and in Dewesoft they have sub-variables with certain angles, e.g. 30°, 50°, ...
When I manually export to csv in Dewesoft, those angles can be exported, while it seems to not be possible for the library / the dll.

Any experience with this, or even a way to do it?

Best, Stef