Course Content

Data Preprocessing

1. Brief Introduction

Data Types Data Processing Methods Dataset: Test and Training Deleting an "Extra" Data Changing the Data Type

2. Processing Quantitative Data

Data Scaling Data Scaling vs Data Normalization Removing Outliers Removing Missing Values Data Augmentation: Synthetic Data

3. Processing Categorical Data

Methods for Encoding the Categorical Data One-Hot Encoding Ordinal Encoding Label Encoding of the Target Variable Challenge

4. Time Series Data Processing

Data Type Conversion Data Cleaning Stationarity Denoising Train/Test Split & Cross Validation Challenge

5. Feature Engineering

Technique Idea Realization Feature Extraction from Text Feature Extraction from Images Feature Extraction from Time Series Challenge

6. Moving on to Tasks

Challenge 1 Challenge 2 Challenge 3

Denoising

Noise in time series data refers to the random fluctuations or errors present in the data that can obscure or distort the underlying patterns and trends. It can arise from various sources, such as measurement errors, environmental factors, or sampling variations. Denoising techniques remove unwanted noise from the data to better understand and analyze the true signal or underlying behavior.

The goal of denoising is to improve the data quality and make extracting meaningful information from the time series easier.

There are several methods for denoising in time series data processing, including:

Moving average - this method involves taking a rolling average of the time series data to smooth out the noise.
Wavelet transform - this method involves transforming the time series data into wavelet coefficients and removing coefficients associated with noise.
Singular spectrum analysis - this method involves decomposing the time series data into several components, including trend, periodicity, and noise, and then reconstructing the time series without the noise component.
Kalman filter - this method involves modeling the time series data using a dynamic system and then using a filter to estimate the true state of the system by removing the noise.

We use the moving average method to get rid of noise in the data:


              12345678910
            
import pandas as pd

# Read the dataset
dataset = pd.read_csv('https://codefinity-content-media.s3.eu-west-1.amazonaws.com/9c23bf60-276c-4989-a9d7-3091716b4507/datasets/data_w_noise.csv')

# Calculate the 3-point moving average
dataset['NValueY'] = dataset['ValueY'].rolling(window=6).mean()

# Print the dataset
print(dataset.dropna())

You can look at the images below and see that this method has reduced the noise in the data. The smoother the graph, the less noise it has!

Task

Swipe to show code editor

Read the 'denoising.csv' dataset and use the moving average method to remove the noise with window size equal to 3.

Switch to desktop for real-world practiceContinue from where you are using one of the options below

Everything was clear?

Thanks for your feedback!

Section 4. Chapter 4

Denoising

The goal of denoising is to improve the data quality and make extracting meaningful information from the time series easier.

There are several methods for denoising in time series data processing, including:

Moving average - this method involves taking a rolling average of the time series data to smooth out the noise.
Wavelet transform - this method involves transforming the time series data into wavelet coefficients and removing coefficients associated with noise.
Singular spectrum analysis - this method involves decomposing the time series data into several components, including trend, periodicity, and noise, and then reconstructing the time series without the noise component.
Kalman filter - this method involves modeling the time series data using a dynamic system and then using a filter to estimate the true state of the system by removing the noise.

We use the moving average method to get rid of noise in the data:


              12345678910
            
import pandas as pd

# Read the dataset
dataset = pd.read_csv('https://codefinity-content-media.s3.eu-west-1.amazonaws.com/9c23bf60-276c-4989-a9d7-3091716b4507/datasets/data_w_noise.csv')

# Calculate the 3-point moving average
dataset['NValueY'] = dataset['ValueY'].rolling(window=6).mean()

# Print the dataset
print(dataset.dropna())

You can look at the images below and see that this method has reduced the noise in the data. The smoother the graph, the less noise it has!

Task

Swipe to show code editor

Read the 'denoising.csv' dataset and use the moving average method to remove the noise with window size equal to 3.

Switch to desktop for real-world practiceContinue from where you are using one of the options below

Everything was clear?

Thanks for your feedback!

Section 4. Chapter 4

Denoising

The goal of denoising is to improve the data quality and make extracting meaningful information from the time series easier.

There are several methods for denoising in time series data processing, including:

Moving average - this method involves taking a rolling average of the time series data to smooth out the noise.
Wavelet transform - this method involves transforming the time series data into wavelet coefficients and removing coefficients associated with noise.
Singular spectrum analysis - this method involves decomposing the time series data into several components, including trend, periodicity, and noise, and then reconstructing the time series without the noise component.
Kalman filter - this method involves modeling the time series data using a dynamic system and then using a filter to estimate the true state of the system by removing the noise.

We use the moving average method to get rid of noise in the data:


              12345678910
            
import pandas as pd

# Read the dataset
dataset = pd.read_csv('https://codefinity-content-media.s3.eu-west-1.amazonaws.com/9c23bf60-276c-4989-a9d7-3091716b4507/datasets/data_w_noise.csv')

# Calculate the 3-point moving average
dataset['NValueY'] = dataset['ValueY'].rolling(window=6).mean()

# Print the dataset
print(dataset.dropna())

You can look at the images below and see that this method has reduced the noise in the data. The smoother the graph, the less noise it has!

Task

Swipe to show code editor

Read the 'denoising.csv' dataset and use the moving average method to remove the noise with window size equal to 3.

Switch to desktop for real-world practiceContinue from where you are using one of the options below

Everything was clear?

Thanks for your feedback!

The goal of denoising is to improve the data quality and make extracting meaningful information from the time series easier.

There are several methods for denoising in time series data processing, including:

Moving average - this method involves taking a rolling average of the time series data to smooth out the noise.
Wavelet transform - this method involves transforming the time series data into wavelet coefficients and removing coefficients associated with noise.
Singular spectrum analysis - this method involves decomposing the time series data into several components, including trend, periodicity, and noise, and then reconstructing the time series without the noise component.
Kalman filter - this method involves modeling the time series data using a dynamic system and then using a filter to estimate the true state of the system by removing the noise.

We use the moving average method to get rid of noise in the data:


              12345678910
            
import pandas as pd

# Read the dataset
dataset = pd.read_csv('https://codefinity-content-media.s3.eu-west-1.amazonaws.com/9c23bf60-276c-4989-a9d7-3091716b4507/datasets/data_w_noise.csv')

# Calculate the 3-point moving average
dataset['NValueY'] = dataset['ValueY'].rolling(window=6).mean()

# Print the dataset
print(dataset.dropna())

You can look at the images below and see that this method has reduced the noise in the data. The smoother the graph, the less noise it has!

Task

Swipe to show code editor

Read the 'denoising.csv' dataset and use the moving average method to remove the noise with window size equal to 3.

Switch to desktop for real-world practiceContinue from where you are using one of the options below

Section 4. Chapter 4

Switch to desktop for real-world practiceContinue from where you are using one of the options below