Appending a series of 2d numpy array to create a 3d numpy array

Question

I thought this was an easy problem, but I have been struggling with it.

I have a dataframe with 4 columns (Open, High, Low, Close).

I need to iteratively select

for 100 times
a batch of 75 rows
each having 4 columns. Such that the final shape is (100,75,4)

I have tried np.append, np.stack, np.dstack, np.concatenate. None of it works.

In np.append i get a shape (7500,4)

In np.stack in the second iteration there is error that all input arrays must have the same shape (since after first stack the original arrays shape is different).

My last code with np.stack (not put other attempts):

for i in range (100):
  print(i)
  if (i==0):
    temp_array=timeseries[['Open','High','Low','Close']].iloc[i:i+75].to_numpy()
  else:
    temp_temp_array=np.stack([temp_array,timeseries[['Open','High','Low','Close']].iloc[i:i+75].to_numpy()])
    temp_array=temp_temp_array
  print(temp_array.shape)

It seems stackoverflow/internet does not have an answer (or may be I am not asking the right questions).

np.array([df.iloc[i:i+75].to_numpy() for i in range(100)]), could be improved with as_stride though. — Quang Hoang
– Quang Hoang, Commented Sep 18, 2023 at 15:29

Learning is a mess · Accepted Answer · 2023-09-18 20:37:29Z

1

As suggested by Quang, you can use strides here for speed and memory (!) efficiency:

X = df.values
rolling_X = np.lib.stride_tricks.as_strided(X, shape=(X.shape[0],75,X.shape[1]), strides=(X.strides[0], X.strides[0], X.strides[1]))

edit:

To mask out some columns, you can do it on the dataframe first:

X = df[['High']].values

or, for speed, you can use numpy index lookups:

X = df.values[:,[1]]

edited Sep 18, 2023 at 20:37

answered Sep 18, 2023 at 15:42

Learning is a mess

8,3178 gold badges42 silver badges82 bronze badges

Sign up to request clarification or add additional context in comments.

3 Comments

user1517108 Over a year ago

Wow thanks!!! Magic! Cant understand much of it, but wow. A followup query - If I wanted just one column from X.. lets say X['High'] what changed shud i do. Code isnt working for that.

Learning is a mess Over a year ago

Have edited my post to answer your query, let me know if you want to know more

Learning is a mess Over a year ago

Btw, here is a insightfult comment to understand the underlying "magic" (it's not) stackoverflow.com/a/53099870/3275464

Collectives™ on Stack Overflow

Appending a series of 2d numpy array to create a 3d numpy array

1 Answer 1

3 Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

3 Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related