how to extend RAM for Python out of memory problem?

Question

I am running big dataset for machine learning and my laptop memory is limited (8GB)

and Python is giving me this error

MemoryError: Unable to allocate 1.34 GiB for an array with shape (49998, 3607) and data type float64

when i run this line

df_features = df_features.pivot(index='RepID', columns='Code', values='Frequency')

How can I use USB stick as a RAM or how to use HDD as RAM with python?

If you have a swap partition mounted your OS should use that for swapping. Keep in mind that swapping to HDD is very, very slow, and likely even slower to a thumbdrive. If you want to use hard disk as memory, you should use a much faster SSD. Though it is preferable to just extend your RAM, if possible. — Jan Christoph Terasa
– Jan Christoph Terasa, Commented Jun 19, 2020 at 10:19
Another approach is to reduce the size of the arrays or the resolution of the data you are dealing with. — Jan Christoph Terasa
– Jan Christoph Terasa, Commented Jun 19, 2020 at 10:21
@JanChristophTerasa I checked Virtual Memory size in my computer, it says "Automatically Managed" that means it is already active? then why Python does not use it? — asmgx
– asmgx, Commented Jun 19, 2020 at 10:29
I do not know what that means. Each OS will have a different way to set up a swap drive, I think dealing with that here is out of the scope of SO. — Jan Christoph Terasa
– Jan Christoph Terasa, Commented Jun 19, 2020 at 10:35
@JanChristophTerasa Thanks, looks like it is working with swap partition :) — asmgx
– asmgx, Commented Jun 19, 2020 at 10:36

Jan Christoph Terasa · Accepted Answer · 2020-06-19 10:42:54Z

1

If you have a swap partition mounted your OS should use that for swapping. Keep in mind that swapping to HDD is very, very slow, and likely even slower to a thumbdrive. If you want to use hard disk as memory, you should use a much faster SSD. Though it is preferable to just extend your RAM, if possible.

Another approach is to reduce the size of the arrays or the resolution of the data you are dealing with.

answered Jun 19, 2020 at 10:42

Jan Christoph Terasa

6,00327 silver badges35 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

how to extend RAM for Python out of memory problem?

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related