Python Pandas storing error: (HDF5ExtError: Problems creating the Array)

Question

I'm having problems storing a large amount of data using HDF5 functionality in Pandas.

What I'm trying to do is download a huge amount of data (Millions of rows) from a SQL sever and have it stored locally for easy and quick access.

I defined some functions to create a connection to the SQL server and run the queries. The code works fine until it tries to store the data. I section the data in different datasets and .h5 files. Some of the datasets are stored correctly, but I think the bigger ones are causing problems and I receive the following error.

HDF5ExtError: Problems creating the Array pandas

Is there a way to fix this?

This is the code I'm using:

import pymssql
import pandas as pd
import time

user = 'xxx'
password = '123'
server='SQL_Server'

def connect():
"""
Connects to SQL database and return a connection object.
"""
    connection = pymssql.connect(host=server, user=user, password=password, database=user+"_db")
    return connection

def query(query, connection, index = None):
"""
Execute an SQL query and return the result as a data frame.
"""
df =  pd.read_sql_query(query, connection, index_col=index)
return df

def store_data(connection):
for i in reversed(range(0,8)):
    st = 'store' + str(i) + '.h5'
    store = pd.HDFStore(st)
    for j in reversed(range(i*8,i*8 + 8)):
        q = "Some query here"
        df = query(q, connection)
        name = 'snap' + str(j)
        print 'Storing data for ' + name + " in " + st
        store[name] = df
        time.sleep(30)
    print '[DONE] - Closing ' + st
    store.close()

Unfortunately, since I was running the code in a remote computer in screen mode (Linux) the trace was not fully displayed and I close the program, so I don't have it available. Is there a way to retrieve it? — Jemme
– Jemme, Commented Feb 4, 2015 at 12:21

tim654321 · Accepted Answer · 2017-02-23 04:13:10Z

5

This error could be caused by running out of space on the drive you are trying to save to - check the free space on your target drive.

answered Feb 23, 2017 at 4:13

tim654321

2,2682 gold badges19 silver badges20 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

Python Pandas storing error: (HDF5ExtError: Problems creating the Array)

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related