dynamically append N-dimensional array

Question

If each array has the shape (1000, 2, 100), it is easy to use

con = np.concatenate((array_A, array_B))

to concatenate them, thus con has the shape (2000, 2, 100).

I want to dynamically append or concatenate "con" in a function. The step is described as following:

First, read data from the first file and process data to generate an array.

Secondly, read date from the second file and append generated array into the first array

....

def arrayappend():
     for i in range(n):
         #read data from file_0 to file_n-1
         data = read(file_i)
         #data processing to generate an array with shape (1000, 2, 100)
         con = function(data)
         # append con

Why can't you do the processing, append the processed data to a normal python list, then finally once the loop is done call np.concatenate() on that list? — Silver
– Silver, Commented Nov 16, 2016 at 14:44
@SilverSlash, Yes, I was think of that solution. But is there a good way to append the array each time after I process one file? — Kun
– Kun, Commented Nov 16, 2016 at 14:47
Not sure what is the problem. Why not con = np.concatenate((con, data)) after # append con — Balzola
– Balzola, Commented Nov 16, 2016 at 15:33
@Balzola, data is the raw data, like strings in each file, con is generated or converted from data — Kun
– Kun, Commented Nov 16, 2016 at 15:36

hpaulj · Accepted Answer · 2016-11-16 19:49:07Z

1

Assuming all your files produce the same shape objects and you want to join them on the 1st dimension, there are several options:

 alist = []
 for f in files:
    data = foo(f)
    alist.append(f)
 arr = np.concatenate(alist, axis=0)

concatenate takes a list. There are variations if you want to add a new axis (np.array(alist), np.stack etc).

Append to a list is fast, since it just means adding a pointer to the data object. concatenate creates a new array from the components; it's compiled but still relatively slower.

If you must/want to make a new array at each stage you could write:

 arr = function(files[0])
 for f in files[1:]:
     data = foo(f)
     arr = np.concatenate((arr, data), axis=0)

This probably is slower, though, if the file loading step is slow enough you might not notice a difference.

With care you might be able start with arr = np.zeros((0,2,100)) and read all files in the loop. You have to make sure the initial 'empty' array has a compatible shape. New users often have problems with this.

answered Nov 16, 2016 at 19:49

hpaulj

233k14 gold badges260 silver badges392 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

Kun Over a year ago

Thanks, I tried the first method and it worked. So the basic idea is to append them into a list and then concatenate the list.

Silver · Accepted Answer · 2016-11-16 14:55:06Z

0

If you absolutely want to do it during iteration then:

def arrayappend():
  con = None
  for i, d in enumerate(files_list):
    data = function(d)
    con = data if i is 0 else np.vstack([con, data])

This should stack it vertically.

answered Nov 16, 2016 at 14:55

Silver

1,4991 gold badge13 silver badges27 bronze badges

Comments

Balzola · Accepted Answer · 2016-11-16 16:19:10Z

0

Very non pretty, but does it achieve what you want? It is way unoptimized.

def arrayappend():
    for i in range(n):
        data = read(file_i)
        try:
            con
            con = np.concatenate((con, function(data)))
        except NameError:
            con = function(data)
    return con

First loop will take the except branch, subsequent wont.

answered Nov 16, 2016 at 16:19

Balzola

4763 silver badges11 bronze badges

Collectives™ on Stack Overflow

dynamically append N-dimensional array

3 Answers 3

1 Comment

Comments

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

3 Answers 3

1 Comment

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related