Data Anomaly Detection

Data Anomaly Detection

0%

1. What is Anomaly Detection?

Let's delve into the concept of data anomalies, their classification, and their impact on decision-making. Throughout this exploration, we will gain a comprehensive understanding of these topics, accompanied by real-life examples to illustrate their significance.

General Information

Types of Anomalies

How Outliers Influence On Prediction Results

What Should We Do With Detected Outliers

2. Statistical Methods in Anomaly Detection

Statistical methods play a crucial role in various domains, including finance, cybersecurity, manufacturing, and healthcare, where detecting anomalies can prevent fraud, ensure data quality, and improve overall system reliability. These methods are particularly valuable when dealing with large datasets and complex systems, as they provide a systematic and quantitative approach to anomaly detection.

Rule-based Approach

Challenge: Rule-based Approach

Median Absolute Deviation

Challenge: Outlier Detection Using MAD Rule

3. Machine Learning Techniques

Now, let's explore how machine learning methods can effectively address anomalies. We will examine various techniques, including clustering, regularization, dropout, and more, that are employed for this purpose.

Challenge: Using DBSCAN Clustering to Detect Outliers

Challenge: Solving Task Using Regularisation

Зміст курсу

Data Anomaly Detection

Data Anomaly Detection

1. What is Anomaly Detection?

General Information Types of Anomalies How Outliers Influence On Prediction Results What Should We Do With Detected Outliers

2. Statistical Methods in Anomaly Detection

Rule-based Approach Challenge: Rule-based Approach 1.5 IQR Rule 3-Sigma Rule Median Absolute Deviation Challenge: Outlier Detection Using MAD Rule

3. Machine Learning Techniques

Clustering Challenge: Using DBSCAN Clustering to Detect Outliers Regularisation Challenge: Solving Task Using Regularisation Autoencoders

Median Absolute Deviation

The MAD (Median Absolute Deviation) rule is a statistical outlier detection method that uses the median and the median absolute deviation as robust estimators to identify outliers in a dataset.

It is particularly useful when dealing with data that may not follow a normal distribution or when there are potential outliers that can significantly impact the mean and standard deviation.

How to use MAD rule

Calculate the Median: Compute the median of the dataset, which is the middle value when the data is sorted;
Calculate the Median Absolute Deviation (MAD): For each data point, find the absolute difference between the data point and the median. The MAD is the median of these absolute differences;
Define a Threshold: Choose a threshold value (usually a constant, e.g., 2 or 3 times the MAD) to determine how far a data point can deviate from the median before being considered an outlier;
Identify Outliers: Any data point that has an absolute difference from the median greater than the threshold is considered an outlier.

Note

Mathematically, the absolute difference between two values, A and B, is denoted as |A - B|, where "|" represents the absolute value function. This function returns the positive value of the difference between A and B.

MAD rule implementation

MAD vs 1.5 IQR rule

Все було зрозуміло?

Дякуємо за ваш відгук!

Секція 2. Розділ 5

some-alt