Can't load HDF5 files in Python

Question

I'm having a weird problem with reading hdf5 files. This is with the latest version of Python and Numpy, as well as other packages. I'm on Ubuntu 18.0.4. I can write hdf5 files just fine, but when I try to read hdf5 files, it throws this error:

ValueError                                Traceback (most recent call last)
<ipython-input-1-f757457e8111> in <module>
      2 df = pd.DataFrame(data={'d': ['a', 'b', 'c']})
      3 df.to_hdf('test.hdf', key='data')
----> 4 df = pd.read_hdf('test.hdf')

~/numpy_bug/lib/python3.7/site-packages/pandas/io/pytables.py in read_hdf(path_or_buf, key, mode, **kwargs)
    392                                      'contains multiple datasets.')
    393             key = candidate_only_group._v_pathname
--> 394         return store.select(key, auto_close=auto_close, **kwargs)
    395     except:
    396         # if there is an error, close the store

~/numpy_bug/lib/python3.7/site-packages/pandas/io/pytables.py in select(self, key, where, start, stop, columns, iterator, chunksize, auto_close, **kwargs)
    739                            chunksize=chunksize, auto_close=auto_close)
    740 
--> 741         return it.get_result()
    742 
    743     def select_as_coordinates(

~/numpy_bug/lib/python3.7/site-packages/pandas/io/pytables.py in get_result(self, coordinates)
   1481 
   1482         # directly return the result
-> 1483         results = self.func(self.start, self.stop, where)
   1484         self.close()
   1485         return results

~/numpy_bug/lib/python3.7/site-packages/pandas/io/pytables.py in func(_start, _stop, _where)
    732             return s.read(start=_start, stop=_stop,
    733                           where=_where,
--> 734                           columns=columns)
    735 
    736         # create the iterator

~/numpy_bug/lib/python3.7/site-packages/pandas/io/pytables.py in read(self, start, stop, **kwargs)
   2935             blk_items = self.read_index('block%d_items' % i)
   2936             values = self.read_array('block%d_values' % i,
-> 2937                                      start=_start, stop=_stop)
   2938             blk = make_block(values,
   2939                              placement=items.get_indexer(blk_items))

~/numpy_bug/lib/python3.7/site-packages/pandas/io/pytables.py in read_array(self, key, start, stop)
   2487 
   2488         if isinstance(node, tables.VLArray):
-> 2489             ret = node[0][start:stop]
   2490         else:
   2491             dtype = getattr(attrs, 'value_type', None)

~/numpy_bug/lib/python3.7/site-packages/tables/vlarray.py in __getitem__(self, key)
    679                 key += self.nrows
    680             (start, stop, step) = self._process_range(key, key + 1, 1)
--> 681             return self.read(start, stop, step)[0]
    682         elif isinstance(key, slice):
    683             start, stop, step = self._process_range(

~/numpy_bug/lib/python3.7/site-packages/tables/vlarray.py in read(self, start, stop, step)
    819             listarr = []
    820         else:
--> 821             listarr = self._read_array(start, stop, step)
    822 
    823         atom = self.atom

tables/hdf5extension.pyx in tables.hdf5extension.VLArray._read_array()

ValueError: cannot set WRITEABLE flag to True of this array

This appears to be line 2021 in the _read_array() function here.

Minimal reproducing code:

import pandas as pd
df = pd.DataFrame(data={'d': ['a', 'b', 'c']})
df.to_hdf('test.hdf', key='data')
df = pd.read_hdf('test.hdf')

It looks like this is related: github.com/nipy/nibabel/issues/697 — wordsforthewise
– wordsforthewise, Commented Jan 18, 2019 at 0:34

wordsforthewise · Accepted Answer · 2019-03-20 17:29:09Z

1

This turns out to be a bug in the tables library, which has since been fixed: https://github.com/PyTables/PyTables/issues/719

Upgrade your pytables library with conda update pytables or pip --upgrade pytables and it should be good to go.

Old way of fixing it before the new release for reference:

Once there is a new release, it should be fixed. Until then, you have to stick with numpy version 1.15.4 by doing:

pip install numpy==1.15.4

edited Mar 20, 2019 at 17:29

answered Feb 12, 2019 at 20:38

wordsforthewise

16.1k6 gold badges96 silver badges126 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

Can't load HDF5 files in Python

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related