Numpy Array Concatenation Multiplies array... why?

Question

I have been desperately trying to figure out why an array concatenation leads to the array being multiplied by 2 on the first iteration...

This is my code:

def run(filename, max_iter = 1e6):
    data, fm = data_extraction(filename)

    large_output = None

    iteration_count = 0    

    while iteration_count < max_iter:
        iteration_count += 1
        print iteration_count
        results = calculations(data, (0.9,1.1))

        if large_output == None:
            large_output = results[:,1] #stores the energy array
            print large_output
        else:
            large_output = np.c_[ large_output, results[:,1]]
            #large_output = np.vstack([large_output, power_and_energy_var[:,1]])
            print large_output

and this is the console output for the print statement and 3 iterations only:

1
[  3.59891391e+01   5.75841568e+00   ]
2
[[  7.22402719e+01   3.62511328e+01]
 [  1.16726670e+01   5.91425129e+00]]
3
[[  7.22402719e+01   3.62511328e+01   3.70141435e+01]
 [  1.16726670e+01   5.91425129e+00   6.02176042e+00]]

As you can see 7.22402719e+01 is about twice 3.59891391e+01, yet it doesn't occur for the successive iterations...

I have no idea why this is happening. I have tried everything I could possibly think off:

1) check what exactly was being executed using print statements 2) reloaded the kernel to erase any lingering variables 3) used np.vstack instead of np.c_ (same error)

Any help would be much welcome !!

large_output = results[:,1] is taking a view of part of results. If the array it's taking a view of gets changed, the view will reflect those changes. The concatenation (while slow, and a bad idea) probably has nothing to do with your problem. — user2357112
– user2357112, Commented Jun 7, 2017 at 17:13
Aren't one of large_output or results[:,1] empty at the first iteration? The output suggests the stacked array is 1D there and in succesives ones 2D. — Divakar
– Divakar, Commented Jun 7, 2017 at 17:13
@Divakar : no the large_output gets filled up during the first iteration as it is None. And the results are not empty. — Sorade
– Sorade, Commented Jun 7, 2017 at 17:18
@user2357112 : I'm not sure I understand the first part of your comment sorry. Would you mind explaining it again ? What alternative to the concatenation would you suggest ? — Sorade
– Sorade, Commented Jun 7, 2017 at 17:19
Here's a quick intro to views and copies, and as for the concatenation, the appropriate decision would depend on what your code is actually doing. — user2357112
– user2357112, Commented Jun 7, 2017 at 17:22

hpaulj · Accepted Answer · 2017-06-07 17:49:26Z

Replicating your concatenation with a simple results array:

In [229]: results = np.arange(12).reshape(2,6)
In [230]: out = results[:,1]
In [231]: out = np.c_[out, results[:,2]]
In [232]: out = np.c_[out, results[:,3]]
In [233]: out
Out[233]: 
array([[1, 2, 3],
       [7, 8, 9]])

Even though the initial out is 1d, the subsequent ones simply concatenate columns

In [234]: results[:,1]
Out[234]: array([1, 7])

So any funny business between cnt 1 and 2 is the result of unknown behavior in data and calculations. It's not a problem with the np.c_ concatenation.

That said, the suggestion to build the list first is a good one

In [237]: out = []
In [238]: out.append(results[:,1])
In [239]: out.append(results[:,2])
In [240]: out.append(results[:,3])
In [241]: out
Out[241]: [array([1, 7]), array([2, 8]), array([3, 9])]
In [242]: np.array(out)
Out[242]: 
array([[1, 7],
       [2, 8],
       [3, 9]])

Though it probably won't bypass any funny business in creating results.

Sorade · Accepted Answer · 2017-06-08 08:24:16Z

0

Despite all the very good comments and suggestions proposed in comments and answers, the answer was much simpler than that (it usually is when I spend a long time banging my head on the computer).

large_output = results[:,1]

should be explicitly expressed as an array:

large_output = np.array(results[:,1])

answered Jun 8, 2017 at 8:24

Sorade

9351 gold badge15 silver badges32 bronze badges

Collectives™ on Stack Overflow

Numpy Array Concatenation Multiplies array... why?

2 Answers 2

Comments

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Related