Random forests use multiple trees, reduce variance and allow for more exploration of feature combinations. Wouldn’t it be great if we could use random forests for feature importance? Let’s now watch the following video to understand the notion of variable importance in random forests.
The importance of features in random forests, sometimes called ‘Gini importance’ or ‘mean decrease impurity’, is defined as the total decrease in node impurity (it is weighted by the probability of reaching that node (which is approximated by the proportion of samples reaching that node)) averaged over all the trees of the ensemble.
For each variable, the sum of the Gini decreases across every tree of the forest and is accumulated every time that variable is chosen to split a node. The sum is divided by the number of trees in the forest to give an average.