Memory allocation error in sklearn random forest classification python

Question

I am trying to run sklearn random forest classification on 2,79,900 instances having 5 attributes and 1 class. But i am getting memory allocation error while trying to run the classification at the fit line, it is not able to train the classifier itself. Any suggestions on how to resolve this issue?

The data a is

x,y, day, week, Accuracy

x and y are the coordinates day is which day of the month (1-30) the week is which day of the week (1-7) and accuracy is an integer

code:

import csv
import numpy as np
from sklearn.ensemble import RandomForestClassifier


with open("time_data.csv", "rb") as infile:
    re1 = csv.reader(infile)
    result=[]
    ##next(reader, None)
    ##for row in reader:
    for row in re1:
        result.append(row[8])

    trainclass = result[:251900]
    testclass = result[251901:279953]


with open("time_data.csv", "rb") as infile:
    re = csv.reader(infile)
    coords = [(float(d[1]), float(d[2]), float(d[3]), float(d[4]), float(d[5])) for d in re if len(d) > 0]
    train = coords[:251900]
    test = coords[251901:279953]

print "Done splitting data into test and train data"

clf = RandomForestClassifier(n_estimators=500,max_features="log2", min_samples_split=3, min_samples_leaf=2)
clf.fit(train,trainclass)

print "Done training"
score = clf.score(test,testclass)
print "Done Testing"
print score

Error:

line 366, in fit
    builder.build(self.tree_, X, y, sample_weight, X_idx_sorted)
  File "sklearn/tree/_tree.pyx", line 145, in sklearn.tree._tree.DepthFirstTreeBuilder.build
  File "sklearn/tree/_tree.pyx", line 244, in sklearn.tree._tree.DepthFirstTreeBuilder.build
  File "sklearn/tree/_tree.pyx", line 735, in sklearn.tree._tree.Tree._add_node
  File "sklearn/tree/_tree.pyx", line 707, in sklearn.tree._tree.Tree._resize_c
  File "sklearn/tree/_utils.pyx", line 39, in sklearn.tree._utils.safe_realloc
MemoryError: could not allocate 10206838784 bytes

jubueche · Accepted Answer · 2018-11-28 19:17:46Z

1

From the scikit-learn doc.: "The default values for the parameters controlling the size of the trees (e.g. max_depth, min_samples_leaf, etc.) lead to fully grown and unpruned trees which can potentially be very large on some data sets. To reduce memory consumption, the complexity and size of the trees should be controlled by setting those parameter values."

I would try to adjust these parameters then. Also, you can try a mem. profiler or try to run it on GoogleCollaborator if your machine has too few RAM.

answered Nov 28, 2018 at 19:17

jubueche

7936 silver badges25 bronze badges

Sign up to request clarification or add additional context in comments.

5 Comments

Labeo Over a year ago

I tried with max_features="log2", min_samples_split=3, min_samples_leaf=2 as my parameter but still I am facing same issue I might try max depth. I am having a 16GB ram

Labeo Over a year ago

And depending on the number of features I have the depth cant be large enough right?

jubueche Over a year ago

I would definitely set a max_depth. Decision trees overfit drastically at a high depth. Usually a depth of 6 is sufficient, but this of course depends on your model.

Labeo Over a year ago

Is it possible to run in chunks of data as i tried to run on 25000 points it ran. I assume it wont as at the end the data is same

jubueche Over a year ago

I think you can do that. But you can not train two models on different chunks of data since the results will be different. Have you tried max_depth?

Ankit · Accepted Answer · 2020-09-02 01:20:17Z

1

Please try Google Colaboratory. You can connect with the localhost or hosted runtime. It worked for me for n_estimators=10000.

answered Sep 2, 2020 at 1:20

Ankit

114 bronze badges

Comments

Zhendong Cao · Accepted Answer · 2020-01-10 17:05:44Z

0

I met with the same MemoryErr recently. But I fixed it by reducing the training data size instead of modifying my model parameters. My OOB value was 0.98 meaning that the model is very less likely overfit.

answered Jan 10, 2020 at 17:05

Zhendong Cao

1691 silver badge5 bronze badges

Collectives™ on Stack Overflow

Memory allocation error in sklearn random forest classification python

3 Answers 3

5 Comments

Comments

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

3 Answers 3

5 Comments

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Related