Numpy For Large Data Sets [closed]

Question

Closed. This question needs details or clarity. It is not currently accepting answers.

Want to improve this question? As written, this question is lacking some of the information it needs to be answered. If the author adds details in comments, consider editing them into the question. Once there's sufficient detail to answer, vote to reopen the question.

Closed 6 years ago.

Improve this question

For generating a probability density function of some cases, maybe 1 million observations are considered. When I work with numpy array, I was encountered by size limit 32.

Is it too few ?

In this case, how can we store more than 32 elements without distributing the elements into different columns and maybe arrays in arrays ?

import numpy
my_list = []
for i in range(0, 100):
    my_list.append(i)

np_arr = numpy.ndarray(np_arr) # ValueError: sequence too large; cannot be greater than 32

I have no idea what you're asking. What are "elements" here? — roganjosh
– roganjosh, Commented Dec 30, 2018 at 23:40
Any interpretation I can make of the question suggests that no such limit exists — roganjosh
– roganjosh, Commented Dec 30, 2018 at 23:42
How are you creating the array? The only limitation to Numpy array size is your hardware resources. You should have no problem with 1M elements. np.zeros(1e6) will created an array with a million zeros. Give it a try. — kcw78
– kcw78, Commented Dec 31, 2018 at 2:14

RuthC · Accepted Answer · 2018-12-31 17:05:21Z

2

When you create an array with numpy.ndarray, the first argument is the shape of the array. Interpreting that list as a shape would indeed give a huge array. If you just want to turn the list into an array, you want numpy.array:

import numpy
my_list = []
for i in range(0, 100):
    my_list.append(i)

np_arr = numpy.array(my_list)

answered Dec 31, 2018 at 17:05

RuthC

4,3192 gold badges21 silver badges39 bronze badges

Sign up to request clarification or add additional context in comments.

4 Comments

kcw78 Over a year ago

You don't need to create a list...just use arange; np_arr = numpy.arange(100) (for int32) or np_arr = numpy.arange(100.0) (for float64).

RuthC Over a year ago

@kcw78 I believe the OP's error is coming up because they are using the wrong function to convert a list into an array. My answer aims to show the right way to make that conversion.

kcw78 Over a year ago

Good point. It wasn't clear why the OP used a list to create an array. If he really needs to convert a list, then he should follow your steps. If he just needs an array with [0, 1, 2, 3, 4 ...100], he can use arange(). Will leave that for OP to figure out.

RuthC Over a year ago

I assumed that the "real" code has a more complicated list and that range was just a convenient way to make a minimal reproducible example. I could be completely wrong of course!

Collectives™ on Stack Overflow

Numpy For Large Data Sets [closed]

1 Answer 1

4 Comments

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

4 Comments

Linked

Related