append rows from a 'for loop' to a dataframe using pd.concat with python

Question

After an update, I am getting the following message:

'The frame.append method is deprecated and will be removed from pandas in a future version. Use pandas.concat instead.'

I am having trouble re-writing my code with pd.concat() instead of df.append(). Please help!

I am conducting an artificial star experiment, where I have three files with columns:

b1: ['Id', 'x', 'y', 'bmag'] - artificial stars with b-filter

i1: ['Id', 'x', 'y', 'imag'] - artificial stars with I-filter

biart: ['Id', 'x', 'y', 'bmag', 'imag'] - measured stars

I have calculated the minimum radial distance between stars in b1 (artificial) and biart (measured) by keeping the 'Id', 'bmag' and 'imag' for both files if the minimum distance is satisfied.

extract = pd.DataFrame(columns=['Id_art', 'x_art', 'y_art', 
                                  'bmag_art', 'imag_art', 
                                  'dist_d',
                                  'Id_meas', 'x_meas', 'y_meas',                                 
                                  'bmag_meas', 'imag_meas'])

for i in range(len(b1.index)):
    
    x = b1['x'].iloc[i]
    y = b1['y'].iloc[i]
    
    dist = np.sqrt((x - biart['x'])**2 + (y - biart['y'])**2)
    
    if (min(dist))<=1/2:
        
        print(b1['Id'].iloc[i], 
              b1['x'].iloc[i], 
              b1['y'].iloc[i],        
              b1['bmag'].iloc[i], 
              i1['imag'].iloc[i], 
              min(dist),
              biart['Id'].iloc[dist.idxmin()],
              biart['x'].iloc[dist.idxmin()],
              biart['y'].iloc[dist.idxmin()], 
              biart['bmag'].iloc[dist.idxmin()],
              biart['imag'].iloc[dist.idxmin()])
            
        extract = extract.append({'Id_art': b1['Id'].iloc[i],
                                    'x_art':b1['x'].iloc[i], 
                                   'y_art': b1['y'].iloc[i], 
                                 'bmag_art':b1['bmag'].iloc[i],
                                 'imag_art':i1['imag'].iloc[i],
                                   'dist_d':min(dist), 
                         'Id_meas':biart['Id'].iloc[dist.idxmin()], 
                          'x_meas':biart['x'].iloc[dist.idxmin()],
                          'y_meas':biart['y'].iloc[dist.idxmin()], 
                     'bmag_meas':biart['bmag'].iloc[dist.idxmin()],
                    'imag_meas':biart['imag'].iloc[dist.idxmin()]},
                                     ignore_index=True)

how do I rewrite the code below to pd.concat()?

extract.append({'Id_art': b1['Id'].iloc[i],
                  'x_art':b1['x'].iloc[i], 
                 'y_art': b1['y'].iloc[i],
               'bmag_art':b1['bmag'].iloc[i],
               'imag_art':i1['imag'].iloc[i],
               'dist_d':min(dist),
               'Id_meas':biart['Id'].iloc[dist.idxmin()], 
                'x_meas':biart['x'].iloc[dist.idxmin()],
                'y_meas':biart['y'].iloc[dist.idxmin()], 
             'bmag_meas':biart['bmag'].iloc[dist.idxmin()],
             'imag_meas':biart['imag'].iloc[dist.idxmin()]},
                                     ignore_index=True)

``

C.Nivs · Accepted Answer · 2022-06-24 17:25:20Z

Change the dataframe to a list, and have the pd.DataFrame constructor consume the list at the end:

# Make a list first
extract = []

for i in range(len(b1.index)):
    
    x = b1['x'].iloc[i]
    y = b1['y'].iloc[i]
    
    dist = np.sqrt((x - biart['x'])**2 + (y - biart['y'])**2)
    
    if (min(dist))<=1/2:
        
        print(b1['Id'].iloc[i], b1['x'].iloc[i], b1['y'].iloc[i],        
              b1['bmag'].iloc[i], i1['imag'].iloc[i], 
              min(dist),
              biart['Id'].iloc[dist.idxmin()],
              biart['x'].iloc[dist.idxmin()],
              biart['y'].iloc[dist.idxmin()], 
              biart['bmag'].iloc[dist.idxmin()],
              biart['imag'].iloc[dist.idxmin()])
            
        # Append to the list here
        extract.append({'Id_art': b1['Id'].iloc[i],
                                      'x_art':b1['x'].iloc[i], 
                                      'y_art': b1['y'].iloc[i], 
                                      'bmag_art':b1['bmag'].iloc[i],
                                      'imag_art':i1['imag'].iloc[i],
                                      'dist_d':min(dist), 
                            'Id_meas':biart['Id'].iloc[dist.idxmin()], 
                              'x_meas':biart['x'].iloc[dist.idxmin()],
                              'y_meas':biart['y'].iloc[dist.idxmin()], 
                        'bmag_meas':biart['bmag'].iloc[dist.idxmin()],
                       'imag_meas':biart['imag'].iloc[dist.idxmin()]})

Then, after the loop is finished filling the list:

# Consume the data here
extract = pd.DataFrame(extract)

Can you please clarify how I would append with pd.concat() ?

Collectives™ on Stack Overflow

append rows from a 'for loop' to a dataframe using pd.concat with python

1 Answer 1

1 Comment

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

1 Comment

Your Answer

Sign up or log in

Post as a guest

Related