Course Content

Ensemble Learning

1. Basic Principles of Building Ensemble Models

What is Ensemble of Models?Bagging Models Boosting Models Stacking Models

2. Commonly Used Bagging Models

Bagging Classifier Challenge: Solving Task Using Bagging Classifier Bagging Regressor Challenge: Solving Task Using Bagging Regressor Random Forest Challenge: Determining Feature Importances Using Random Forest ExtraTrees

3. Commonly Used Boosting Models

AdaBoost Classifier Challenge: Solving Task Using AdaBoost Classifier Challenge: Solving Task Using AdaBoost Regressor Gradient Boosting XGBoost Challenge: Solving Task Using XGBoost

4. Commonly Used Stacking Models

Stacking Classifier Challenge: Solving Task Using Stacking Classifier Challenge: Solving Task Using Stacking Regressor Using Ensembles As Base Models Course Summary

Ensemble Learning

Random Forest

Random Forest is a bagging ensemble algorithm that is used for both classification and regression tasks. The basic idea behind Random Forest is to create a "forest" of decision trees, where each tree is trained on a different subset of the data and provides its own prediction.

How does Random Forest works?

Bootstrapping and Data Subset: Each tree in the forest is trained using a random subset drawn from the original dataset via bootstrapping. This process involves selecting data points with replacement, creating diverse subsets for each tree.
Decision Tree Construction: These subsets build individual decision trees. Data is recursively divided using features and thresholds, forming binary splits that lead to leaf nodes containing predictions.
Random Feature Selection: Within each tree, only a random subset of features is considered for creating splits. This randomness prevents single features from overpowering predictions and enhances tree diversity.
Prediction Aggregation: After training, each tree predicts for data points. For classification, we use hard or soft voting to create a prediction; for regression, predictions are averaged to provide the final outcome.

We can notice a rather interesting feature of a random tree: each base model is trained not only on a random subset of the training set, but also on a random subset of features. As a result, we get more independent base models and, as a result, more accurate final predictions.

Example

Let's solve classification task using Random Forest on Iris dataset:

Code Description

Import Libraries:

Import necessary libraries and modules from scikit-learn:
- load_iris: Used to load the Iris dataset.
- train_test_split: Used to split the dataset into training and testing sets.
- RandomForestClassifier: The classifier we'll be using, which is part of the ensemble module.
- f1_score: The function to calculate the F1 score for model evaluation.

Load and Prepare Data:

- Load the Iris dataset using load_iris.
- Extract the features into X and the target variable into y.

Train-Test Split:

-split the data into training and testing sets using train_test_split.
test_size=0.2 specifies that 20% of the data will be used for testing.

Create and Train the Random Forest Classifier:

- Create an instance of RandomForestClassifier with n_estimators=100 (number of trees in the forest) and n_jobs=-1 (to train the model using all processors in parallel).
- Train the classifier using the training data (features and target) with the .fit() method.

Make Predictions:

- Use the trained classifier to make predictions on the test data (X_test).
- Store the predicted labels in y_pred.

Calculate F1 Score:

- Calculate the F1 score using the f1_score() function.
- The average='weighted' parameter indicates that the F1 score sho

You can find the official documentation with all the necessary information about implementing this model in Python on the official website. Go here if needed.

What model is used as a base model in Random Forest?

Select the correct answer

Everything was clear?

Section 2. Chapter 5