Pandas First Steps. Advanced Techniques in Pandas

Pandas is an open-source Python library for high-performance data manipulation and analysis. It excels with structured data like tables and time series, offering Series (1D labeled arrays) and DataFrame (2D labeled data) for potent cleaning, transformation, and analysis.

Why do we need Pandas?

Pandas is widely used in data science, data analysis, and machine learning tasks due to its numerous benefits:

Efficient data manipulation: provides vectorized operations, significantly speeding up data processing;
Easy data handling: offers intuitive data structures and functions that make data loading, cleaning, and transformation simple and straightforward;
Data alignment: automatically aligns data based on the labels, making it easy to combine datasets and perform operations on data with different shapes;
Handling missing data: provides various methods to handle missing data, making data cleaning more manageable;
Time series functionality: has excellent support for working with time-series data, including resampling, shifting, and rolling window operations.
Integration with other libraries: seamlessly integrates with other popular Python libraries, such as NumPy, Matplotlib, and Scikit-learn, making it a core component of the data science ecosystem.

Why is this course included in the track?

Pandas is vital for data scientists, streamlining data tasks for faster manipulation, exploration, and analysis. It frees time for insights and modeling, reducing data handling complexities.

Why do we need Pandas if we already know Numpy?

numpy and pandas are vital in Python's data science world, serving distinct roles yet complementing each other seamlessly. Pandas extends essential functions: versatile data structures, cleaning, exploration, time series analysis, and loading. Together, they excel: NumPy for numerical work and arrays, Pandas for structured data handling and analysis, and a dynamic duo for data scientists.

Example

pandas is very effective when working with data of different formats and performing exploratory data analysis (EDA).

Let's look at an example:


              123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051
            
import pandas as pd
import matplotlib.pyplot as plt
import seaborn as sns

# Task: Perform EDA on the Boston Housing dataset using pandas

# Step 1: Load the dataset using pandas
url = 'https://codefinity-content-media.s3.eu-west-1.amazonaws.com/Tracks_Intro_Course/BostonHousing.csv'
df = pd.read_csv(url)

# Step 2: Explore the dataset

# Check the summary statistics of the dataset
print(df.describe())

# Check the data types and missing values in each column
print(df.info())

# Step 3: Perform Data Visualization

# Plot the distribution of the target variable (median house value)
plt.figure(figsize=(8, 6))
sns.histplot(df['medv'], kde=True)
plt.xlabel('Median House Value')
plt.ylabel('Count')
plt.title('Distribution of Median House Value')
plt.show()

# Correlation heatmap to visualize relationships between variables
plt.figure(figsize=(10, 8))
sns.heatmap(df.corr(), annot=True, cmap='coolwarm', fmt=".2f", linewidths=0.5)
plt.title('Correlation Heatmap')
plt.show()

# Step 4: Analyze Relationships

# Scatter plot to explore the relationship between 'rm' (average number of rooms) and 'medv'
plt.figure(figsize=(8, 6))
sns.scatterplot(data=df, x='rm', y='medv', alpha=0.5)
plt.xlabel('Average Number of Rooms')
plt.ylabel('Median House Value')
plt.title('Relationship between Average Number of Rooms and Median House Value')
plt.show()

# Box plot to compare the median house values for different neighborhoods ('rad')
plt.figure(figsize=(10, 6))
sns.boxplot(data=df, x='rad', y='medv')
plt.xlabel('Radial Access to Highways')
plt.ylabel('Median House Value')
plt.title('Comparison of Median House Value for Different Radial Access to Highways')
plt.show()

Everything was clear?

Thanks for your feedback!

Section 1. Chapter 3