18

Let's say we have a 1d numpy array filled with some int values. And let's say that some of them are 0.

Is there any way, using numpy array's power, to fill all the 0 values with the last non-zero values found?

for example:

arr = np.array([1, 0, 0, 2, 0, 4, 6, 8, 0, 0, 0, 0, 2])
fill_zeros_with_last(arr)
print arr

[1 1 1 2 2 4 6 8 8 8 8 8 2]

A way to do it would be with this function:

def fill_zeros_with_last(arr):
    last_val = None # I don't really care about the initial value
    for i in range(arr.size):
        if arr[i]:
            last_val = arr[i]
        elif last_val is not None:
            arr[i] = last_val

However, this is using a raw python for loop instead of taking advantage of the numpy and scipy power.

If we knew that a reasonably small number of consecutive zeros are possible, we could use something based on numpy.roll. The problem is that the number of consecutive zeros is potentially large...

Any ideas? or should we go straight to Cython?

Disclaimer:

I would say long ago I found a question in stackoverflow asking something like this or very similar. I wasn't able to find it. :-(

Maybe I missed the right search terms, sorry for the duplicate then. Maybe it was just my imagination...

Improve this question
4
  • 4
    If you don't mind using pandas, have a look at the ffill method (or see fillna for the complete story). However, there's no "forward fill" type functionality built into numpy. Commented May 27, 2015 at 17:19
  • 2
    As @JoeKington mentions, fillna in pandas will do this. The Cython source is the pad_2d_inplace_ function here, specifically the inner loop at the bottom. The code is exactly what you have written in your example. Commented May 27, 2015 at 18:00
  • 1
    @JoeKington Thanks! nice feature! still I prefer to avoid depending on pandas for this... Commented May 28, 2015 at 14:00
  • You could also use pandas.groupby() in theory - though why bother since you have plenty of other solutions :) Commented May 19, 2022 at 11:00

4 Answers 4

Reset to default
33

Here's a solution using np.maximum.accumulate:

def fill_zeros_with_last(arr):
    prev = np.arange(len(arr))
    prev[arr == 0] = 0
    prev = np.maximum.accumulate(prev)
    return arr[prev]

We construct an array prev which has the same length as arr, and such that prev[i] is the index of the last non-zero entry before the i-th entry of arr. For example, if:

>>> arr = np.array([1, 0, 0, 2, 0, 4, 6, 8, 0, 0, 0, 0, 2])

Then prev looks like:

array([ 0,  0,  0,  3,  3,  5,  6,  7,  7,  7,  7,  7, 12])

Then we just index into arr with prev and we obtain our result. A test:

>>> arr = np.array([1, 0, 0, 2, 0, 4, 6, 8, 0, 0, 0, 0, 2])
>>> fill_zeros_with_last(arr)
array([1, 1, 1, 2, 2, 4, 6, 8, 8, 8, 8, 8, 2])

Note: Be careful to understand what this does when the first entry of your array is zero:

>>> fill_zeros_with_last(np.array([0,0,1,0,0]))
array([0, 0, 1, 1, 1])
Improve this answer
Sign up to request clarification or add additional context in comments.

2 Comments

Your answer is probably faster than mine - except in cases where my in-place operation might save time.
I tried a similar approach myself, but still this one is at least as nice and faster. :) Thanks!
5

Inspired by jme's answer here and by Bas Swinckels' (in the linked question) I came up with a different combination of numpy functions:

def fill_zeros_with_last(arr, initial=0):
     ind = np.nonzero(arr)[0]
     cnt = np.cumsum(np.array(arr, dtype=bool))
     return np.where(cnt, arr[ind[cnt-1]], initial)

I think it's succinct and also works, so I'm posting it here for the record. Still, jme's is also succinct and easy to follow and seems to be faster, so I'm accepting it :-)

Improve this answer

1 Comment

Very nice... there is also a potential solution using np.repeat, but it seems to be a little trickier to get the boundaries right. It's probably not quite as fast as either of ours, too.
1

If the 0s only come in strings of 1, this use of nonzero might work:

In [266]: arr=np.array([1,0,2,3,0,4,0,5])
In [267]: I=np.nonzero(arr==0)[0]
In [268]: arr[I] = arr[I-1]
In [269]: arr
Out[269]: array([1, 1, 2, 3, 3, 4, 4, 5])

I can handle your arr by applying this repeatedly until I is empty.

In [286]: arr = np.array([1, 0, 0, 2, 0, 4, 6, 8, 0, 0, 0, 0, 2])

In [287]: while True:
   .....:     I=np.nonzero(arr==0)[0]
   .....:     if len(I)==0: break
   .....:     arr[I] = arr[I-1]
   .....:     

In [288]: arr
Out[288]: array([1, 1, 1, 2, 2, 4, 6, 8, 8, 8, 8, 8, 2])

If the strings of 0s are long it might be better to look for those strings and handle them as a block. But if most strings are short, this repeated application may be the fastest route.

Improve this answer

1 Comment

Unfortunately, I do expect potentially many consecutive 0. I thought about something this, but the for loop just didn't convinced me... :-/
0 
import numpy as np
a = np.array([1, 0, 0, 2, 0, 4, 6, 8, 0, 0, 0, 0, 2])

a = np.asarray(a)
mask_non_zero = a != 0 
indices = np.arange(len(a))

forward_fill_indices = np.maximum.accumulate(np.where(mask_non_zero,indices,-1))
print(forward_fill_indices)#[ 0  0  0  3  3  5  6  7  7  7  7  7 12]

res =a[forward_fill_indices]
print(res)#[1 1 1 2 2 4 6 8 8 8 8 8 2]
Improve this answer

Comments

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.