how to manipulate numpy arrays for use with ESRI's arcpy.da.NumPyArrayToTable

Question

ESRI gives access to moving data from tables to arrays and back. I have a script that takes census data from an api call and converts it into arrays, does some simple math, and then, ideally, puts it out into a table. To do the math, the array cannot be a rec array. No combination of vstack, hstack, or concatenate seemed to give a good result. I resorted to creating individual 1-d arrays as recarrays, and then using the merge function in np.lib.recfunctions.merge_arrays. surely there is a better way.

ESRI's return from TableToNumPyArray:

>>> testArray
array([ (41039000100.0, 2628.0, 100.0, 2339.0, 135.0, 18.0, 22.0, 16.0, 25.0, 0.0, 92.0, 0.0, 92.0, 0.0, 92.0, 0.0, 92.0, 6.0, 9.0, 249.0, 90.0, 0.0, 92.0, 1, u'41039000100'),
...
dtype=[('Geo_id', '<f8'), ('TotalUnits', '<f8'), ('MOE_Total', '<f8'), >('Total_1_detached', '<f8'), ('MOE_Total_1_detached', '<f8'), ('Total_1_attached', >'<f8'), ('MOE_Total_1_attached', '<f8'), ('Total_2', '<f8'), ('MOE_Total_2', '<f8'), >('Total_3_or_4', '<f8'), ('MOE_Total_3_or_4', '<f8'), ('Total_5_to_9', '<f8'), >('MOE_Total_5_to_9', '<f8'), ('Total_10_to_19', '<f8'), ('MOE_Total_10_to_19', '<f8'), >('Total_20_to_49', '<f8'), ('MOE_Total_20_to_49', '<f8'), ('Total_50_or_more', '<f8'), >('MOE_Total_50_or_more', '<f8'), ('Total_Mobile_home', '<f8'), ('MOE_Total_Mobile_home', '<f8'), ('Total_Boat_RV_van_etc', '<f8'), ('MOE_Total_Boat_RV_van_etc', '<f8'), >('ObjectID', '<i4'), ('geo_id_t', '<U50')])

My snippet of code looks like

try:

    # Assign Geo_id array
    Geo_id_array = B25008_001E_array[...,0]
    Tpop_array = B25008_001E_array[...,1]
    Tunits_array = B25024_001E_array[...,1]
    # divide by sero is possible for real rowns and definite for the end-of-file
    # tract, so convert nan's in the HHsize_array to zero's with nan_to_num
    # HHsize_array = Tpop_array.view(np.float32)/Tunits_array.view(np.float32)
    HHsize_array = Tpop_array/Tunits_array
    HHsize_array = nan_to_num(HHsize_array)
    # Table_array = array(vstack((Geo_id_array, Tpop_array, Tunits_array, HHsize_array)), dtype = ([('Geo_id', '|S13'), ('Tpop', np.int32), ('Tunits_array', np.int32), ('HHsize', np.float32)]))
    # Table_array = np.hstack((Geo_id_array, Tpop_array, Tunits_array, HHsize_array))
    Geo_id_recarray = np.array(Geo_id_array, dtype = ([('Geo_id', '|S13')]))
    Tpop_recarray = np.array(Tpop_array, dtype = ([('Tpop', np.int32)]))
    Tunits_recarray = np.array(Tunits_array, dtype = ([('Tunits_array', np.int32)]))
    HHsize_recarray = np.array(HHsize_array, dtype = ([('HHsize', np.float32)]))
    arrays = [Geo_id_recarray, Tpop_recarray, Tunits_recarray, HHsize_recarray]
    MergedArray = np.lib.recfunctions.merge_arrays(arrays, usemask=False)
    print
    print



except Exception as e:
    # If an error occurred, print line number and error message
    import traceback, sys
    tb = sys.exc_info()[2]
    print "An error occured on line %i" % tb.tb_lineno
    print str(e)

I'd prefer to merge/join/stack the arrays before structuring them, I think. Thoughts?

It looks like all the information you have is in your testArray. Why don't you just use the fields from testArray that you need instead of creating several more one-field arrays? — askewchan
– askewchan, Commented Feb 16, 2013 at 0:00

askewchan · Accepted Answer · 2013-02-18 03:54:41Z

1

You should be able to use the structured arrays (technically you're not using recarrays) to do the "simple math." I'm not sure if you're showing the math you'd like to do, but for example if you want to do:

HHsize_array = Tpop_array/Tunits_array

But don't want to have all those separate arrays, you could simply do the math on views of the main (merged array), let's call it data:

data['HHsize'] = data['Tpop']/data['Tunits']

where HHsize, Tpop, and Tunits are all field names in one structured array called data, such that you'd have

>>> data.dtype
dtype([('Geo_id', '|S13'), ('Tpop', np.int32), ('Tunits_array', np.int32), ('HHsize', np.float32)])

answered Feb 18, 2013 at 3:54

askewchan

46.7k18 gold badges125 silver badges135 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

how to manipulate numpy arrays for use with ESRI's arcpy.da.NumPyArrayToTable

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related