Appending an empty list to a numpy array changes its dtype

Question

I have a numpy array of integers. In my code I need to append some other integers from a list, which works fine and gives me back an array of dtype int64 as expected. But it may happen that the list of integers to append is empty. In that case, numpy returns an array of float64 values. Exemplary code below:

import numpy as np

a = np.arange(10, dtype='int64')

np.append(a, [10])  # dtype is int64
# array([ 0,  1,  2,  3,  4,  5,  6,  7,  8,  9, 10])

np.append(a, [])    # dtype is float64
# array([0., 1., 2., 3., 4., 5., 6., 7., 8., 9.])

Is this expected behaviour? If so, what is the rationale behind this? Could this even be a bug?

The documentation for np.append states that the return value is

A copy of arr with values appended to axis.

Since there are no values to append, shouldn't it just return a copy of the array?

(Numpy version: 1.22.4, Python version: 3.8.0)

Ultimately concatenate is not joining lists. It is joining arrays. If one or more of the arguments is a list, it is first converted into an array. — hpaulj
– hpaulj, Commented Jun 10, 2022 at 1:35

Marco Bonelli · Accepted Answer · 2022-06-09 23:48:43Z

2

The default dtype for a numpy array constructed from an empty list is float64:

>>> np.array([])
array([], dtype=float64)

A float64 dtype will "win" over the int64 dtype and promote everything to float64, because converting the other way around would cause loss of precision (i.e. truncate any float64). Of course this is an extreme case because there is no value to truncate in an empty array, but the check is only done on the dtype. More info on this is in the doc for numpy.result_type().

In fact:

>>> a = np.array(dtype='int64')
>>> a
array([], dtype=int64)

>>> b = np.array([])
>>> b
array([], dtype=float64)

>>> np.result_type(a, b)
dtype('float64')

The np.promote_types() function is used to determine the type to promote to:

>>> np.promote_types('int64', 'float64')
dtype('float64')

See also: How the dtype of numpy array is calculated internally?

edited Jun 9, 2022 at 23:48

answered Jun 9, 2022 at 23:39

Marco Bonelli

71.1k21 gold badges129 silver badges154 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

Salvatore · Accepted Answer · 2022-06-09 23:34:36Z

2

You can cast your array when appending and specify the type you want:

import numpy as np

a = np.arange(10, dtype="int64")
# array([0, 1, 2, 3, 4, 5, 6, 7, 8, 9], dtype=int64)

np.append(a, [])
array([0., 1., 2., 3., 4., 5., 6., 7., 8., 9.], dtype=float64)

np.append(a, np.array([], dtype=a.dtype))
# array([0, 1, 2, 3, 4, 5, 6, 7, 8, 9], dtype=int64)

answered Jun 9, 2022 at 23:34

Salvatore

12.6k8 gold badges47 silver badges89 bronze badges

Comments

Woody1193 · Accepted Answer · 2022-06-09 23:38:31Z

2

In the source code for numpy.append, we can see that it calls numpy.concatenate. Looking at the documentation for this method, we see that it accepts two type-related arguments: dtype and casting, the default of which is None. Since numpy.append does not provide values for either of these, the defaults are assumed. The default for casting is same-kind, which allows for safe casting within a kind, as described here. Since all of this is done in C, my guess is that concatenate tried to guess the type of the array you provided. Since it was empty, however, no type could be assumed so it took the widest possible type that would fit both inputs and assigned that to the output.

If you want to avoid this, you should probably append using numpy arrays instead of Python lists:

np.append(a, np.array([], dtype = 'int64'))

answered Jun 9, 2022 at 23:38

Woody1193

8,2029 gold badges65 silver badges129 bronze badges

2 Comments

hpaulj Over a year ago

np.array([]) is float dtype.

Woody1193 Over a year ago

@hpaulj Yes, I saw that in Marco Bonelli's answer.

Kyle Drover · Accepted Answer · 2022-06-09 23:27:24Z

1

Can you avoid appending the the empty list with some sort of if statement?

I think its expected behaviour as [] is still a value that gets appended. Obviously [] can't be an integer so it probably assumes its a float (default casting).

answered Jun 9, 2022 at 23:27

Kyle Drover

113 bronze badges

2 Comments

asdf Over a year ago

Yes, that's what I did now. Default casting makes sense, but I still think it's kind of counter-intuitive. Since nothing is appended, it should not change the original array, right?

hpaulj Over a year ago

I disagree with the review. This does provide an answer, just as good as the others. It is appending an array with shape (0,). That is not nothing. If I concatenate lists, [1,2,3]+[], the result is [1,2,3], but it isn't handled as a special case.

Collectives™ on Stack Overflow

Appending an empty list to a numpy array changes its dtype

4 Answers 4

Comments

Comments

2 Comments

2 Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

4 Answers 4

Comments

Comments

2 Comments

2 Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related