Summary
This chapter explains how to detect null or missing values in tabular data structures by using methods that return boolean masks indicating where entries are absent.

General domain of usage
data analysis

DataFrames often contain **missing values**, represented as `None` or `NaN`. When working with DataFrames, it's essential to identify these missing values because they can distort calculations, lead to inaccurate analyses, and compromise the reliability of results. 

Addressing them ensures **data integrity** and improves the performance of tasks like **statistical analysis** and **machine learning**. For this purpose, pandas offers specific methods.

The first of these is `isna()`, which returns a **boolean DataFrame**. In this context, a `True` value indicates a missing value within the DataFrame, while a `False` value suggests the value is present.

For clarity, apply this method to the `animals` DataFrame. The `isna()` method returns a `DataFrame` of `True`/`False` values, where each `True` indicates a missing value in the `animals` DataFrame.


import pandas as pd
import numpy as np

animals_data = {'animal': [np.nan, 'Dog', np.nan, 'Cat','Parrot', None],
'name': ['Dolly', None, 'Erin', 'Kelly', None, 'Odie']}
animals = pd.DataFrame(animals_data)
# Find missing values
missing_values = animals.isna()
print(missing_values)

The second method is `isnull()`. It behaves identically to the previous one, with no discernible difference between them.

import unittest
import importlib
import pandas as pd


def get_first_differing_row(expected_df, actual_df):
    expected_list = expected_df.astype(str).apply(tuple, axis=1)
    actual_list = actual_df.astype(str).apply(tuple, axis=1)

    for i, (row1, row2) in enumerate(zip(expected_list, actual_list)):
        if row1 != row2:
            return i, expected_df.iloc[i].values, actual_df.iloc[i].values

    if len(expected_df) > len(actual_df):
        return len(actual_df), expected_df.iloc[len(actual_df)].values, None
    else:
        return len(expected_df), None, actual_df.iloc[len(expected_df)].values


def _dynamic_test(test_case, condition, success_message, failure_message):
    if condition:
        test_case._testMethodName = success_message
        test_case.assertTrue(True, success_message)
    else:
        test_case._testMethodName = failure_message
        test_case.fail(failure_message)


class TestUserCode(unittest.TestCase):

    def test_missing_values_is_declared(self):
        import user_code
        _dynamic_test(
            self,
            hasattr(user_code, 'missing_values'),
            "The `missing_values` variable is declared.",
            "Expected `missing_values` to be declared."
        )

    def test_missing_values_is_dataframe(self):
        import user_code

        try:
            condition = isinstance(user_code.missing_values, pd.DataFrame)
            failure_message = f"Expected `missing_values` to be a `DataFrame`, but got `{type(user_code.missing_values).__name__}`."
        except AttributeError:
            condition = False
            failure_message = "The `missing_values` variable is not declared."

        _dynamic_test(
            self,
            condition,
            "The variable `missing_values` is a `DataFrame`.",
            failure_message
        )

    def test_missing_values_has_correct_values(self):
        import user_code
        wine_data = pd.read_csv(
            'https://codefinity-content-media.s3.eu-west-1.amazonaws.com/a43d24b6-df61-4e11-9c90-5b36552b3437/wine_with_nan.csv')
        expected_df = wine_data.isnull()

        variable = 'missing_values'
        actual_df = getattr(user_code, variable, None)
        condition = False
        if actual_df is None:
            failure_message = f"The `{variable}` variable is not declared."
        elif not isinstance(actual_df, pd.DataFrame):
            failure_message = f"`{variable}` is not a `DataFrame`."
        elif actual_df.empty:
            failure_message = f"`{variable}` is empty."
        elif actual_df.equals(expected_df):
            condition = True
            failure_message = None
        else:
            idx, expected_row, actual_row = get_first_differing_row(expected_df, actual_df)
            failure_message = f"Expected `{variable}` to contain `{expected_row}` at row {idx}, but got `{actual_row}`."

        _dynamic_test(
            self,
            condition,
            f"`{variable}` correctly identifies missing values.",
            failure_message
        )


if __name__ == '__main__':
    unittest.main()

test_main.py

Pandas is an extremely user-friendly library for data analysis. It's also designed to handle large datasets, using data structures like DataFrame and Series. This makes it an invaluable tool for Data Science. In this guide, you'll get acquainted with a range of statistical functions, including how to find correlations, modes, medians, and maximum and minimum values within a dataset. You'll also learn how to handle missing values and manipulate specific values, as well as how to remove them.

Dive into the fundamentals of the pandas library. Learn to create and manipulate Series and DataFrames, explore their structure, and build confidence in performing essential data operations.

Master how to load and manage data from CSV and TXT files. Gain practical experience importing datasets and preparing them for further analysis using pandas tools.

Learn to explore, clean, and summarize data effectively. Understand how to handle missing values, extract insights, and perform basic statistical operations with pandas.

Finding Null Values

解答