0

I have multiple dbf files in a directory that I am trying to loop through and convert into excel files. However, because the loop times out after running for a while I divided the array list into ten subarrays of the same size.

import arcpy
import os
import glob
from arcpy.sa import *
from arcpy import env
import numpy as np
import pandas as pd

dbf_filenames = sorted(glob.glob("*.dbf"))

# Split the array into ten equal parts
pieces = 10
new_arrays = np.array_split(dbf_filenames, pieces)

# Set
array_list = new_arrays[0]

for file in array_list:
   basefilename = os.path.basename(file)

    # Name of File
    CleanBaseStr = os.path.splitext(basefilename)[0]

    # Create Variable for Year
    yearmonth = os.path.splitext(basefilename)[0][16:22]

    # Create Variable for Gage Number
    Gage = '_'.join(CleanBaseStr.split('_')[1:-1])
    # print(Gage)

    # Create New Field
    field_name = "Year_Month"
    field_type = "DOUBLE"

    field = ["Year_Month"]
    # arcpy.management.DeleteField(file, field)
    arcpy.management.AddField(file, field_name, field_type, field_length = 6)

    with arcpy.da.UpdateCursor(file, field) as cursor:

        try:
            for row in cursor:
                # print (row)
                row[0] = yearmonth
                cursor.updateRow(row)

                ### Execute table to excel
                # Name Excel file
                out_xls = CleanBaseStr + ".xlsx"
                output_file = os.path.join(path,out_xls)

                # Open dbf files
                new_df = Dbf5(file)

                # Conver dbf to databframe
                df = new_df.to_dataframe()
                df['Year_Month'] = yearmonth
                # print(df)

                # Save dataframe to excel
                df.to_excel(output_file)

        except:
            continue

The loop works as I want it to, but I would like to know if there is a way to automate the for loop to move between the subarrays rather than doing the change manually. Currently, I am waiting for the loop to finish and then manually changing the subarrays fromnew_arrays[0] to new_arrays[1] and so forth until I reach the last subarray new_arrays[9].

2
  • I'm not sure what 'there are more than the loop can handle in a single go' means, but why not just use an outer loop: for array_list in new_arrays: ? Commented Sep 20, 2022 at 3:15
  • Sorry for the confusion. I have 1 million observations that I need to loop over. The loop times out after about 100 thousand iterations. That is why I had to break it up into 10 different segments. But thank you for the idea, I will try it out. Commented Sep 20, 2022 at 15:24

0

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.