What is Pipeline

In the previous section, three preprocessing steps were completed: imputing, encoding, and scaling.

The preprocessing steps were applied one by one, transforming specific columns and merging them back into the X array. This approach can be cumbersome, particularly with OneHotEncoder, which alters the number of columns.

Another drawback is that any new data used for prediction must go through the same sequence of transformations, requiring the entire process to be repeated.

The Pipeline class in Scikit-learn simplifies this by combining all transformations into a single workflow, making it easier to apply preprocessing consistently to both training data and new instances.

A Pipeline serves as a container for a sequence of transformers, and eventually, an estimator. When you invoke the .fit_transform() method on a Pipeline, it sequentially applies the .fit_transform() method of each transformer to the data.

# Create a pipeline with three steps: imputation, one-hot encoding, and scaling
pipeline = Pipeline([
    ('imputer', SimpleImputer(strategy='most_frequent')),  # Step 1: Impute missing values
    ('encoder', OneHotEncoder()),                         # Step 2: Convert categorical data
    ('scaler', StandardScaler())                          # Step 3: Scale the data
])

# Fit and transform the data using the pipeline
X_transformed = pipeline.fit_transform(X)

This streamlined approach means you only need to call .fit_transform() once on the training set and subsequently use the .transform() method to process new instances.