Filtering and Transforming Data
In automation engineering, you often need to clean and filter data before performing any analysis or reporting. Common scenarios include removing sensor readings that fall outside of expected ranges, filtering out faulty device logs, or selecting only the rows that meet specific criteria for further processing. For example, if you are monitoring temperature sensors, you may want to remove any readings that are unrealistically high or low, as these could indicate sensor malfunctions or data corruption.
12345678910111213141516import pandas as pd # Example data: temperature readings from sensors data = { "sensor_id": [101, 102, 103, 104, 105], "temperature": [72, 150, 68, -20, 85] } df = pd.DataFrame(data) # Define acceptable temperature range min_temp = 60 max_temp = 100 # Filter out rows where temperature is outside acceptable limits filtered_df = df[(df["temperature"] >= min_temp) & (df["temperature"] <= max_temp)] print(filtered_df)
This code demonstrates how to use boolean indexing to filter data in pandas. Boolean indexing allows you to select rows in a DataFrame based on the result of a condition or set of conditions. By combining multiple conditions with logical operators like & (and) or | (or), you can build powerful filters that automate quality checks, data cleaning, and error detection. This approach is fast, concise, and easy to adapt to different automation tasks, making it an essential tool for any automation engineer.
123456# Suppose you need to calibrate sensor readings by adding an offset calibration_offset = 2.5 # Add a new column with calibrated values df["temperature_calibrated"] = df["temperature"] + calibration_offset print(df)
1. What is boolean indexing in pandas?
2. Why might you add a new column to a DataFrame?
3. Fill in the blank: 'df[df["value"] > 10]' filters rows where value is ___ 10.
Thanks for your feedback!
Ask AI
Ask AI
Ask anything or try one of the suggested questions to begin our chat
Awesome!
Completion rate improved to 4.76Filtering and Transforming Data
Swipe to show menu
In automation engineering, you often need to clean and filter data before performing any analysis or reporting. Common scenarios include removing sensor readings that fall outside of expected ranges, filtering out faulty device logs, or selecting only the rows that meet specific criteria for further processing. For example, if you are monitoring temperature sensors, you may want to remove any readings that are unrealistically high or low, as these could indicate sensor malfunctions or data corruption.
12345678910111213141516import pandas as pd # Example data: temperature readings from sensors data = { "sensor_id": [101, 102, 103, 104, 105], "temperature": [72, 150, 68, -20, 85] } df = pd.DataFrame(data) # Define acceptable temperature range min_temp = 60 max_temp = 100 # Filter out rows where temperature is outside acceptable limits filtered_df = df[(df["temperature"] >= min_temp) & (df["temperature"] <= max_temp)] print(filtered_df)
This code demonstrates how to use boolean indexing to filter data in pandas. Boolean indexing allows you to select rows in a DataFrame based on the result of a condition or set of conditions. By combining multiple conditions with logical operators like & (and) or | (or), you can build powerful filters that automate quality checks, data cleaning, and error detection. This approach is fast, concise, and easy to adapt to different automation tasks, making it an essential tool for any automation engineer.
123456# Suppose you need to calibrate sensor readings by adding an offset calibration_offset = 2.5 # Add a new column with calibrated values df["temperature_calibrated"] = df["temperature"] + calibration_offset print(df)
1. What is boolean indexing in pandas?
2. Why might you add a new column to a DataFrame?
3. Fill in the blank: 'df[df["value"] > 10]' filters rows where value is ___ 10.
Thanks for your feedback!
