Setting and reading dimscale in hdf5 files correctly in python

Question

I am trying to attach dimension scales to datasets I want to store in hdf5 files with python, but get an error when I try to print the attributes after setting them. The relevant code snippet reads as follows:

import h5py
import numpy as np

# create data and x-axis
my_data = np.random.randint(10, size=(100, 200))
x_axis  = np.linspace(0, 1, 100)

h5f = h5.File('my_file.h5','w')
h5f.create_dataset( 'data_1', data=my_data )
h5f['data_1'].dims[0].label = 'm'
h5f['data_1'].dims.create_scale( h5f['x_axis'], 'x' )

# the following line is creating the problems
h5f['data_1'].dims[0].attach_scale( h5f['x_axis'] )

# this is where the crash happens but only if the above line is included
for ii in h5f['data_1'].attrs.items():
    print ii

h5f.close()

The command print(h5.version.info) prints the following output:

Summary of the h5py configuration
---------------------------------

h5py    2.2.1
HDF5    1.8.11
Python  2.7.6 (default, Jun 22 2015, 17:58:13) 
[GCC 4.8.2]
sys.platform    linux2
sys.maxsize     9223372036854775807
numpy   1.8.2

The error message is the following:

Traceback (most recent call last):
  File "HDF_write_dimScales.py", line 16
    for ii in h5f['data_1'].attrs.items():
  File "/usr/lib/python2.7/dist-packages/h5py/_hl/base.py", line 347, in items
    return [(x, self.get(x)) for x in self]
  File "/usr/lib/python2.7/dist-packages/h5py/_hl/base.py", line 310, in get
    return self[name]
  File "/usr/lib/python2.7/dist-packages/h5py/_hl/attrs.py", line 55, in __getitem__
    rtdt = readtime_dtype(attr.dtype, [])
  File "h5a.pyx", line 318, in h5py.h5a.AttrID.dtype.__get__ (h5py/h5a.c:4285)
  File "h5t.pyx", line 337, in h5py.h5t.TypeID.py_dtype (h5py/h5t.c:3892)
TypeError: No NumPy equivalent for TypeVlenID exists

Any ideas or hints are appreciated.

Yossarian · Accepted Answer · 2015-11-20 18:02:40Z

1

It works with some slight adjusts for me on h5py 2.5.0. The problem may be to do with when you call create_scale. With h5py 2.5.0, I get a KeyError for h5f['x_axis'] in your create_scale() call. To get your example to work, I had to explicitly create the x_axis dataset first.

import h5py
import numpy as np

# create data and x-axis
my_data = np.random.randint(10, size=(100, 200))

# Use a context manager to ensure h5f is closed
with h5py.File('my_file.h5','w') as h5f:
    h5f.create_dataset( 'data_1', data=my_data )

    # Create the x_axis dataset directly in the HDF5 file
    h5f['x_axis']  = np.linspace(0, 1, 100)

    h5f['data_1'].dims[0].label = 'm'

    # Now we can create and attach the scale without problems
    h5f['data_1'].dims.create_scale( h5f['x_axis'], 'x' )
    h5f['data_1'].dims[0].attach_scale( h5f['x_axis'] )

    for ii in h5f['data_1'].attrs.items():
        print(ii)

# Output
#(u'DIMENSION_LABELS', array(['m', ''], dtype=object))
#(u'DIMENSION_LIST', array([array([<HDF5 object reference>], dtype=object),
#       array([], dtype=object)], dtype=object))

If you're still having problems, you may have to upgrade to h5py 2.5.0, which has better handling (although still not perfect) of VLEN types.

answered Nov 20, 2015 at 18:02

Yossarian

5,5201 gold badge39 silver badges60 bronze badges

Sign up to request clarification or add additional context in comments.

4 Comments

Alf Over a year ago

thanks for the answer, but it does not solve my problem. I guess I will check the update possibilities of the server I am working on (where I have no superuser/root access).

Yossarian Over a year ago

@Alf If you have access to pip, you can install things in your home directory with pip install --user h5py.

Alf Over a year ago

I just did this a few minutes before you posted it ;) It is working now (with my code above) and the output reads:

Alf Over a year ago

the output reads:

(u'DIMENSION_LABELS', array(['m', ''], dtype=object))         (u'DIMENSION_LIST', array([array([<HDF5 object reference>], dtype=object),        array([], dtype=object)], dtype=object))

Community · Accepted Answer · 2017-05-23 10:27:17Z

0

This is just a guess, but since the error references TypeVlenID, it may have something to do with an incomplete implementation of vlen in h5py (especially in our version of the module).

Inexplicable behavior when using vlen with h5py

Writing to compound dataset with variable length string via h5py (HDF5)

edited May 23, 2017 at 10:27

CommunityBot

11 silver badge

answered Nov 19, 2015 at 20:29

hpaulj

233k14 gold badges260 silver badges392 bronze badges

1 Comment

Alf Over a year ago

hmm, thanks for pointing me to that problem, I will read up on that.

Collectives™ on Stack Overflow

Setting and reading dimscale in hdf5 files correctly in python

2 Answers 2

4 Comments

1 Comment

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

4 Comments

1 Comment

Your Answer

Sign up or log in

Post as a guest

Linked

Related