How to tune parameters in Random Forest, using Scikit Learn?

The crucial parts are usually three elements:

• number of estimators - usually bigger the forest the better, there is small chance of overfitting here
• max depth of each tree (default none, leading to full tree) - reduction of the maximum depth helps fighting with overfitting
• max features per split (default sqrt(d)) - you might one to play around a bit as it significantly alters behaviour of the whole tree. sqrt heuristic is usually a good starting point but an actual sweet spot might be somewhere else

This wonderful article has a detailed explanation of tunable parameters, how to track performance vs speed trade-off, some practical tips, and how to perform grid-search.

n_estimators is good one as others said. It is also good at dealing with the overfitting when increasing it.

But I think min_sample_split is also helpful when dealing with overfitting occurred in a small-sample but big-features dataset.

The most impactful parameters to tune in RandomForestClassifier for identifying feature importance and improving model generalization are:

• n_estimators
  The number of decision trees in the forest. More trees can improve accuracy but increase training time. 100 or more is typically good. Higher numbers tend to reduce variance and gives more reliable importance rankings—especially useful when working with noisy or high-dimensional data.

• max_features
  One of the most important parameters to tune. It determines how many features are considered at each split. The default 'sqrt' (for classification) is a good starting point, but trying values like 'log2' or even a float (e.g. 0.4) can change which features are favored. This randomness is what makes Random Forests different from Bagged Trees:

  The key difference:

  Bagged Trees consider all features at each split.
  Random Forests consider a random subset of features at each split.

• max_depth
  Limits how deep each tree can grow. Shallower trees can reduce overfitting. Try setting it to something like 10 or 30 instead of using the default (None).

Other useful parameters to consider:

• min_samples_split and min_samples_leaf
  Help regularize the tree. Increasing these prevents the model from learning tiny, overfitted patterns.

• bootstrap=True
  This is the default for Random Forests. Create multiple bootstrap datasets by randomly drawing samples with replacement from the original dataset. The image below shows what sampling with replacement looks like. If False, the whole dataset is used. You can try setting it to False just for comparison.

How to tune

You can use GridSearchCV or RandomizedSearchCV. A minimal tuning grid:

param_grid = {
    'n_estimators': [100, 200, 300],
    'max_features': ['sqrt', 'log2', 0.4],
    'max_depth': [None, 10, 30],
}

### Understanding Feature Importance

It is worth noting that Random Forests provide multiple ways to evaluate feature importance:

- **1. Mean Decrease in Impurity (MDI)**  
  Also known as Gini importance, this sums the reduction in impurity brought by each feature across all trees. It can run fast, but it can bias toward high-cardinality features (features with many unique values).

```python
importances = reg.feature_importances_
feature_names = X.columns
sorted_idx = np.argsort(importances)[::-1]

for i in sorted_idx:
    print(f"{feature_names[i]}: {importances[i]:.3f}")

• 2. Permutation Importance
  Measures the drop in model performance when a feature's values are randomly shuffled. This method captures interactions and correlation, making it more reliable (though slower).

from sklearn.inspection import permutation_importance

perm_importance = permutation_importance(reg, X_test, y_test, n_repeats=10, random_state=0)
sorted_idx = perm_importance.importances_mean.argsort()[::-1]

for i in sorted_idx:
    print(f"{X.columns[i]}: {perm_importance.importances_mean[i]:.3f}")

Images and code taken from: https://towardsdatascience.com/understanding-random-forest-using-python-scikit-learn/