Data Preprocessing

1. Brief Introduction

Different types of data? How to work with them? If your eyes are wide open, don't worry, let's start with a brief overview of the pandas library and learn how to work with it in the future.

Data Types

Data Processing Methods

Dataset: Test and Training

Deleting an "Extra" Data

Changing the Data Type

2. Processing Quantitative Data

This chapter discusses in detail how to work with quantitative data, what methods it is processed with, how data scaling and normalization differ, and much more.

Data Scaling

Data Scaling vs Data Normalization

Removing Outliers

Removing Missing Values

Data Augmentation: Synthetic Data

3. Processing Categorical Data

Is categorical data as simple as you think it is? Find out what is the complexity of processing and working with it.

Methods for Encoding the Categorical Data

One-Hot Encoding

Ordinal Encoding

Label Encoding of the Target Variable

Challenge

4. Time Series Data Processing

Time series data processing is the process of handling, analyzing, and preparing data that is presented as a sequence of temporally ordered values. Find out what steps it includes in this section.

Train/Test Split & Cross Validation

Challenge

5. Feature Engineering

Did you know that you can extract even more values from your data and create more informative features? In this section, you will learn how to work with feature engineering.

Technique Idea

Realization

Feature Extraction from Text

Feature Extraction from Images

Feature Extraction from Time Series

Challenge

6. Moving on to Tasks

You have reached the end of this course. Let's test your knowledge! There are 3 tasks for you to solve.

Challenge 1

Challenge 2

Challenge 3

Stationarity

One of the main steps is the process of converting a non-stationary time series into a stationary one by eliminating the trend, seasonality, and other factors that affect the change in the statistical properties of the series over time. A transformed stationary time series can be more predictable and easier to analyze than a non-stationary series. There are various methods of transforming data to stationary:

Differencing

Differencing - calculating the difference between the time series's current and previous value of the time series. But how to choose the order of differencing? If the first differences fail to revolve around a constant mean and variance, then we find the second differencing using the values of the first differencing. You can repeat this until you get a stationary series.

You can also plot the differenced series and check to see if there is a constant mean and variance to determine whether or not the series is sufficiently differenced.

Decomposition

Decomposition - breaking down the time series into its trend, seasonality, and random noise components.

Box-Cox transformation

Box-Cox transformation is a method that generalizes the natural logarithm transform and converts non-normal data to more normal distribution.

Outlier removal

Outlier removal - a method that removes outliers from the non-stationary time series, which helps improve its stationarity.

In the example below, we will consider how to implement the transformation of data into stationary data using the decomposition method:

You can look at the plots below. The first - is the original dataset, and the second - is after the differencing method has been applied.

Everything was clear?

Section 4. Chapter 3

Course Content

Data Preprocessing

1. Brief Introduction

Data Types Data Processing Methods Dataset: Test and Training Deleting an "Extra" Data Changing the Data Type

2. Processing Quantitative Data

Data Scaling Data Scaling vs Data Normalization Removing Outliers Removing Missing Values Data Augmentation: Synthetic Data

3. Processing Categorical Data

Methods for Encoding the Categorical Data One-Hot Encoding Ordinal Encoding Label Encoding of the Target Variable Challenge

4. Time Series Data Processing

Data Type Conversion Data Cleaning Stationarity Denoising Train/Test Split & Cross Validation Challenge

5. Feature Engineering

Technique Idea Realization Feature Extraction from Text Feature Extraction from Images Feature Extraction from Time Series Challenge

6. Moving on to Tasks

Challenge 1 Challenge 2 Challenge 3

Data Preprocessing

Stationarity

Differencing

You can also plot the differenced series and check to see if there is a constant mean and variance to determine whether or not the series is sufficiently differenced.

Decomposition

Decomposition - breaking down the time series into its trend, seasonality, and random noise components.

Box-Cox transformation

Box-Cox transformation is a method that generalizes the natural logarithm transform and converts non-normal data to more normal distribution.

Outlier removal

Outlier removal - a method that removes outliers from the non-stationary time series, which helps improve its stationarity.

In the example below, we will consider how to implement the transformation of data into stationary data using the decomposition method:

You can look at the plots below. The first - is the original dataset, and the second - is after the differencing method has been applied.

Everything was clear?

Section 4. Chapter 3