Broken export_hdf5 about vaex HOT 12 CLOSED

That is odd. Can you have me the output h5dump -H <export_filename>?

from vaex.

It crashes without making any content to that file. How's external command would help?

from vaex.

It should create an empty file (as in, the data is all zeros), and then it will open it with write access. That last part fails, but the first part should already create the whole structure of the file, and I want to see what goes wrong there.

from vaex.

I am having the same issue. The file is generated but it is not readble by vaex.

Here is my h5dump of the file created

https://gist.github.com/balbinot/5dd60eeebcc3cf8f50143504a3c6b954

from vaex.

I will close this as stale. Since than vaex changed/improved a number of times.
Please re-open this issue anew if the problem persists.

from vaex.

Hi all,

I am experiencing this issue. Here is a snippet of the code

vaex_df = vaex.from_pandas(df=df, copy_index=False)
vaex_df.export_hdf5("data/data.hdf5")

which causes KeyError: "Unable to open object (object 'data' doesn't exist)"

from vaex.

On the export line?

from vaex.

A full stacktrace would be helpful.

from vaex.

I have no idea what happened but it seems the export function is looking for groups that do not exist.

~/anaconda/lib/python3.6/site-packages/vaex/dataset.py in export_hdf5(self, path, column_names, byteorder, shuffle, selection, progress, virtual, sort, ascending)
   4993         """
   4994         import vaex.export
-> 4995         vaex.export.export_hdf5(self, path, column_names, byteorder, shuffle, selection, progress=progress, virtual=virtual, sort=sort, ascending=ascending)
   4996 
   4997     def export_fits(self, path, column_names=None, shuffle=False, selection=False, progress=None, virtual=False, sort=None, ascending=True):

~/anaconda/lib/python3.6/site-packages/vaex/export.py in export_hdf5(dataset, path, column_names, byteorder, shuffle, selection, progress, virtual, sort, ascending)
    235     kwargs = locals()
    236     import vaex.hdf5.export
--> 237     vaex.hdf5.export.export_hdf5(**kwargs)
    238 
    239 

~/anaconda/lib/python3.6/site-packages/vaex/hdf5/export.py in export_hdf5(dataset, path, column_names, byteorder, shuffle, selection, progress, virtual, sort, ascending)
    174 
    175         # after this the file is closed,, and reopen it using out class
--> 176         dataset_output = vaex.hdf5.dataset.Hdf5MemoryMapped(path, write=True)
    177 
    178 	column_names = vaex.export._export(dataset_input=dataset, dataset_output=dataset_output, path=path, random_index_column=random_index_name,

~/anaconda/lib/python3.6/site-packages/vaex/hdf5/dataset.py in __init__(self, filename, write)
     61                 self._version = 1
     62                 try:
---> 63                         self._load()
     64                 finally:
     65                         self.h5file.close()

~/anaconda/lib/python3.6/site-packages/vaex/hdf5/dataset.py in _load(self)
    145                 if "table" in self.h5file:
    146                         self._version = 2
--> 147                         self._load_columns(self.h5file["/table"])
    148                         self.h5table_root_name = "/table"
    149                 # TODO: shall we rename it vaex... ?

~/anaconda/lib/python3.6/site-packages/vaex/hdf5/dataset.py in _load_columns(self, h5data, first)
    228                                         if unitname == "system.get('S.I.').base('mass')":
    229                                                 self.units[column_name] = astropy.units.Unit("kg")
--> 230                                 data = column if self._version == 1 else column['data']
    231                                 if hasattr(data, "dtype"):
    232                                         #print column, column.shape

h5py/_objects.pyx in h5py._objects.with_phil.wrapper()

h5py/_objects.pyx in h5py._objects.with_phil.wrapper()

~/anaconda/lib/python3.6/site-packages/h5py/_hl/group.py in __getitem__(self, name)
    165                 raise ValueError("Invalid HDF5 object reference")
    166         else:
--> 167             oid = h5o.open(self.id, self._e(name), lapl=self._lapl)
    168 
    169         otype = h5i.get_type(oid)

h5py/_objects.pyx in h5py._objects.with_phil.wrapper()

h5py/_objects.pyx in h5py._objects.with_phil.wrapper()

h5py/h5o.pyx in h5py.h5o.open()

KeyError: "Unable to open object (object 'data' doesn't exist)"

@mfouesneau
I had this issue when using your ezpadova package. To solve it I had to rename columns like

log(age/yr) to something without () or /

from vaex.

Noted. I cannot sadly parse all the various names PARSEC decide to use/change regularly. But should this be somewhat included in vaex? Something to check confirm names?

from vaex.

i have a same error

it can not work

case 1

df = vaex.open("./test.csv",convert=True)

case 2

df = vaex.from_csv("./test.csv",convert= False)
df.export_hdf5("./test.csv.hdf5")
del df
vaex.open("./test.csv.hdf5")

but it can work

df =pd.read_csv("./test.csv")
df2 = vaex.from_pandas(df)
df2.export_hdf5("./test.csv.hdf5")
vaex.open("./test.csv.hdf5")

version

python 3.7
vaex-core                     4.1.0
vaex-hdf5                     0.7.0

from vaex.

This message is not very helpful as such.

Can you please describe what fails, i.e. what is the error message or the stacktrace would be better. Also example of what's inside the csv would be helpful.

Vaex is using pandas for reading csv files so it should not make any difference whether you are using pd.read_csv or vaex.read_csv..

Thanks!

from vaex.