Efficiently assign arrays along an axis (common)number in Python

Question

I have an image which I subsample

Count=0
classim = np.zeros([temp1.shape[0],temp1.shape[1]])
for rows in range(int(np.floor(temp1.shape[0]/SAMPLE_SIZE))):
    for cols in range(int(np.floor(temp1.shape[1]/SAMPLE_SIZE))):

        classim[np.multiply(rows,SAMPLE_SIZE):np.multiply(rows+1,SAMPLE_SIZE),
                np.multiply(cols,SAMPLE_SIZE):np.multiply(cols+1,SAMPLE_SIZE)] = predict.argmax(axis=-1)[Count]
        Count = np.add(Count,1)

This is terribly slow. I get the labels from "predict.argmax(axis=-1)[Count]", but can of course have it in vector form. In other words, how can I vectorise the above loop?

Colin Dickie · Accepted Answer · 2018-07-19 23:30:15Z

1

Taking your row calculations outside the inner loop would help a little. Therefore these calculations will only be made once for each row.

A few other tidy-ups gives:

classim = np.zeros_like(temp1)
predict_args = predict.argmax(axis=-1)
for rows in range(temp1.shape[0]//SAMPLE_SIZE):
    row_0 = rows * SAMPLE_SIZE
    row_1 = (rows+1) * SAMPLE_SIZE
    for cols in range(temp1.shape[1]//SAMPLE_SIZE):
        col_0 = cols * SAMPLE_SIZE
        col_1 = (cols+1) * SAMPLE_SIZE
        classim[row_0:row_1,col_0:col_1] = predict_args[Count]
        Count+=1

You would need to tell us more about the predict object before I could do much more. But these changes will help a little.

--EDIT--

You could take advantage of the numpy.repeat function. Then there is no need to iterate through the whole classim:

SAMPLE_SIZE = 2
temp1 = np.arange(20*20).reshape((20,20))

sample_shape = (temp1.shape[0]//SAMPLE_SIZE, temp1.shape[0]//SAMPLE_SIZE)

#This line should work as per your question, but returns a single value
#predict_args = predict.argmax(axis=-1)
#Use this for illustration purposes
predict_args = np.arange(sample_shape[0] * sample_shape[1])


subsampled = predict_args.reshape(sample_shape)
classim = np.repeat(np.repeat(subsampled,SAMPLE_SIZE,axis =1),SAMPLE_SIZE, axis=0)


print(subsampled)
print(classim)

edited Jul 19, 2018 at 23:30

answered Jul 19, 2018 at 21:30

Colin Dickie

9104 silver badges9 bronze badges

Sign up to request clarification or add additional context in comments.

4 Comments

CupinaCoffee Over a year ago

The predict object is a 1D array containing the value to be assigned, e.g. [1, 4, 5, 1, 9, 11, 1, 4, ...]. The length of predict is rows*cols (end values..) Example: for a SAMPLE_SIZE = 30, classified on an image of 10000x10000 pixels, about (10000/30) 333*333 classification instances occurs. The length of predict is then 333*333 = 110k, and contains the class labels stored contiguously. Thank you for the tidy-up, any bit helps. This process is very slow at the moment due to shear numbers.

Colin Dickie Over a year ago

I've updated my answer, but you may need to look at your predict array. If it's 1D as you say, predict.argmax(axis=-1) will return a single value. Unless you mean argsort?

CupinaCoffee Over a year ago

You're right. My bad. predict is nD, predict.argmax(axis=-1) is 1D. I will try your updated answer. I'll accept your answer and get back with a performance update no matter the result. I appreciate your time, thanks.

CupinaCoffee Over a year ago

Hello again. I finally got to implementing the np.repeat()-method, and for a few other loops that were smaller bottlenecks. For the loop in the post, it showed a 3 times improvement for medium sized arrays (50,000 x 50,000). After refactoring the code for a day, and feeding nominal data (about (!Mx1M)) it showed an improvement of about 20x. That is crazy. Thanks a lot Colin.

Collectives™ on Stack Overflow

Efficiently assign arrays along an axis (common)number in Python

1 Answer 1

4 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

4 Comments

Your Answer

Sign up or log in

Post as a guest

Related