Loop through mini batches of numpy array?

Question

I have a numpy array that contains 813698 rows:

len(df_numpy)
Out[55]: 813698

I want to loop through this array using mini batches of 5000.

mini_batch = 5000
i = 0
for each batch in df_numpy:
   mysubset = df_numpy[i:mini_batch+i]
   # …
   i = i + mini_batch

The problem is that (len(df_numpy)-1)/mini_batch is not an integer. So, the last mini batch is not equal to 5000.

How can I loop though df_numpy so that all records of df_numpy are included?

Well what do you want to do with the missing values? Fill with zeros, fill with random values from the original array, or just drop the ones that don't fit in a last batch? — Nils Werner
– Nils Werner, Commented Feb 27, 2020 at 11:46
@NilsWerner: I want to get all mini batches + smaller last mini batch into mysubset in the for loop. Then I perform some operations on each mysubset. — Fluxy
– Fluxy, Commented Feb 27, 2020 at 11:47
Slicing beyond the end of the array is legal in Python and will create the last "smaller mini batch" — Jussi Nurminen
– Jussi Nurminen, Commented Feb 27, 2020 at 11:54

Community · Accepted Answer · 2020-06-20 09:12:55Z

2

This code should get the job done:

mini_batch = 5000
for first in range(0, len(df_numpy), mini_batch):
    mysubset = df_numpy[first:first+mini_batch]
    # ...

Demo

In [2]: import numpy as np

In [3]: df_numpy = np.arange(13)

In [4]: mini_batch = 5

In [5]: for first in range(0, len(df_numpy), mini_batch):
   ...:    mysubset = df_numpy[first:first+mini_batch]
   ...:    print(mysubset)
[0 1 2 3 4]
[5 6 7 8 9]
[10 11 12]

edited Jun 20, 2020 at 9:12

CommunityBot

11 silver badge

answered Feb 27, 2020 at 11:52

Tonechas

13.8k16 gold badges52 silver badges85 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

Fluxy Over a year ago

Thanks. In this case what happens with the last batch that is smaller than mini_batch?

Tonechas Over a year ago

Added a small demo to make it clear what happens with the last mini batch

Collectives™ on Stack Overflow

Loop through mini batches of numpy array?

1 Answer 1

Demo

2 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Demo

2 Comments

Your Answer

Sign up or log in

Post as a guest

Related