Course Content
Ensemble Learning
Ensemble Learning
1. Basic Principles of Building Ensemble Models
Using Ensembles As Base Models
Utilizing ensemble models as base models within a stacking ensemble framework is a sophisticated approach in machine learning that offers several advantages and disadvantages. Here, we delve into the details of this method.
Pros of Using Ensembles as Base Models in Stacking Ensembles:
- Enhanced Predictive Performance: Ensemble models, such as Random Forests, Gradient Boosting, or AdaBoost, are renowned for their capacity to enhance predictive accuracy. By integrating ensemble models as base learners, stacking can harness the collective predictive power of these models, often resulting in superior performance compared to individual models;
- Diverse Modeling Approaches: Ensembles introduce diversity into the base models of a stacking ensemble. Different ensemble techniques have distinct strengths and weaknesses. Combining them broadens the range of modeling approaches employed, making the ensemble more adept at handling diverse data patterns and complexities;
- Robustness Against Overfitting: Stacking with ensemble models can mitigate overfitting. Ensembles excel at reducing the impact of noise or outliers in the data, thereby enhancing the stacking ensemble's ability to generalize well to unseen data.
Cons of Using Ensembles as Base Models in Stacking Ensembles:
- Increased Model Complexity: Employing ensemble models as base learners introduces complexity to the stacking ensemble. Managing multiple ensemble models necessitates careful consideration of hyperparameter tuning, training times, and computational resources;
- Computational Overhead: Ensembles, especially deep or large ensembles, can be computationally intensive to train and evaluate. This can result in longer training times and may not be suitable for real-time or resource-constrained applications;
- Interpretability Concerns: Ensembles, particularly deep ensembles, are frequently considered less interpretable than individual models. This reduced interpretability can hinder understanding model predictions, potentially limiting the model's utility in certain domains where interpretability is crucial.
Example
We will use the 'steel-plates-fault' dataset and try to create Stacking Classificator with ensembles as base models:
from sklearn.datasets import fetch_openml from sklearn.model_selection import train_test_split from sklearn.ensemble import RandomForestClassifier, AdaBoostClassifier from sklearn.neural_network import MLPClassifier from sklearn.ensemble import StackingClassifier from sklearn.metrics import f1_score import warnings warnings.filterwarnings('ignore') # Step 1: Load the dataset using fetch_openml dataset = fetch_openml("steel-plates-fault", version=1) # Extract features (X) and target variable (y) X = dataset.data y = dataset.target # Step 2: Split the data into training and testing sets X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42) # Step 3: Create base models rf_model = RandomForestClassifier(n_estimators=100, random_state=42) ada_model = AdaBoostClassifier(n_estimators=50, random_state=42) # Step 4: Create the meta model (neural network) meta_model = MLPClassifier(hidden_layer_sizes=(100, 50), max_iter=500) # Step 5: Create the stacking classifier base_models = [('rf', rf_model), ('ada', ada_model)] stacking_classifier = StackingClassifier(estimators=base_models, final_estimator=meta_model) # Step 6: Train the stacking classifier stacking_classifier.fit(X_train, y_train) # Step 7: Make predictions y_pred = stacking_classifier.predict(X_test) # Step 8: Evaluate the results using F1 score f1 = f1_score(y_test, y_pred, average='weighted') print('F1 Score:', f1)
Everything was clear?
Thanks for your feedback!
Section 4. Chapter 4
