Data Anomaly Detection

1. What is Anomaly Detection?

Let's delve into the concept of data anomalies, their classification, and their impact on decision-making. Throughout this exploration, we will gain a comprehensive understanding of these topics, accompanied by real-life examples to illustrate their significance.

General Information

Types of Anomalies

How Outliers Influence On Prediction Results

What Should We Do With Detected Outliers

2. Statistical Methods in Anomaly Detection

Statistical methods play a crucial role in various domains, including finance, cybersecurity, manufacturing, and healthcare, where detecting anomalies can prevent fraud, ensure data quality, and improve overall system reliability. These methods are particularly valuable when dealing with large datasets and complex systems, as they provide a systematic and quantitative approach to anomaly detection.

Rule-based Approach

Challenge: Rule-based Approach

1.5 IQR Rule

3-Sigma Rule

Median Absolute Deviation

Challenge: Outlier Detection Using MAD Rule

3. Machine Learning Techniques

Now, let's explore how machine learning methods can effectively address anomalies. We will examine various techniques, including clustering, regularization, dropout, and more, that are employed for this purpose.

Clustering

Challenge: Using DBSCAN Clustering to Detect Outliers

Regularisation

Challenge: Solving Task Using Regularisation

Autoencoders

Challenge: Outlier Detection Using MAD Rule

Завдання

Now, you will use the MAD rule to detect outliers in the California Housing Dataset. It contains various features related to housing characteristics in different districts in California.

In this task, we will detect outliers in the column MedInc, which stands for Median Income.

Your task is to:

Fill in all gaps in mad() function to calculate Mean Absolute Deviation.
Calculate the threshold using value 3 as a threshold value.
Specify the rule to detect outliers that will be stored in the outliers variable.

Все було зрозуміло?

Секція 2. Розділ 6

Зміст курсу

Data Anomaly Detection

1. What is Anomaly Detection?

General Information Types of Anomalies How Outliers Influence On Prediction Results What Should We Do With Detected Outliers

2. Statistical Methods in Anomaly Detection

Rule-based Approach Challenge: Rule-based Approach 1.5 IQR Rule 3-Sigma Rule Median Absolute Deviation Challenge: Outlier Detection Using MAD Rule

3. Machine Learning Techniques

Clustering Challenge: Using DBSCAN Clustering to Detect Outliers Regularisation Challenge: Solving Task Using Regularisation Autoencoders

Data Anomaly Detection

Challenge: Outlier Detection Using MAD Rule

Завдання

Now, you will use the MAD rule to detect outliers in the California Housing Dataset. It contains various features related to housing characteristics in different districts in California.

In this task, we will detect outliers in the column MedInc, which stands for Median Income.

Your task is to:

Fill in all gaps in mad() function to calculate Mean Absolute Deviation.
Calculate the threshold using value 3 as a threshold value.
Specify the rule to detect outliers that will be stored in the outliers variable.

Все було зрозуміло?

Секція 2. Розділ 6