Course Content

Optimization Techniques in Python

1. Understanding and Measuring Performance

Introduction to Python Performance Timing and Benchmarking Basics Measuring Function Performance Challenge: Implementing Benchmarking

2. Efficient Use of Data Structures

Lists and NumPy Arrays Sets and Tuples Challenge: Choosing Optimal Data Structures Using the collections Module Challenge: Handling Customer Requests

3. Enhancing Performance with Built-in Tools

Leveraging map() and List Comprehensions Handling Large Files Maximizing Sorting Efficiency Efficient String Operations

Maximizing Sorting Efficiency

Built-in Sorting

Whenever you need to sort a list, except for some rare special cases, it's almost always best to use one of its two highly optimized sorting tools: the sorted() function or the sort() method. Both are implemented in C and use Timsort, a hybrid algorithm that combines merge sort and insertion sort for efficiency.

sorted() is ideal for general-purpose sorting when you need to sort any iterable without altering the original data. On the other hand, sort() is best suited for lists when in-place modification is acceptable.

Both methods are efficient, but list.sort() may be only slightly faster for very large lists as it avoids creating a new list. However, use sorted() if you need to keep the original list intact.

Partial Sorting with heapq

For cases where you only need the smallest or largest elements of a dataset, sorting the entire data is unnecessary. The heapq module provides efficient methods like heapq.nsmallest() and heapq.nlargest() to extract these elements without fully sorting the iterable, making it faster and more memory-efficient.

Let's compare the performance of the sorted() function and the heapq.nsmallest() function for retrieving the 10 smallest numbers from a list:


              1234567891011121314151617181920212223
            
import heapq
import os
decorators = os.system('wget https://codefinity-content-media-v2.s3.eu-west-1.amazonaws.com/courses/8d21890f-d960-4129-bc88-096e24211d53/section_1/chapter_3/decorators.py 2>/dev/null')
from decorators import timeit_decorator
import random

# Generate a large list of random integers
numbers = [random.randint(1, 1000000) for _ in range(1000000)]

@timeit_decorator(number=10)
def partial_sort_heapq():
    return heapq.nsmallest(10, numbers)

@timeit_decorator(number=10)
def partial_sort_sorted():
    return sorted(numbers)[:10]

# Compare performance
heapq_result = partial_sort_heapq()
sorted_result = partial_sort_sorted()

# Ensure both methods give the same result
print(heapq_result == sorted_result)

As you can see, in our particular example heapq.nsmallest() is roughly 10 times faster.

However, if the number of largest or smallest elements (n) you want to retrieve is close to the total number of elements in the list, heapq is often slower than using the sorted() function or the .sort() method.

For example, let's now retrieve 100000 smallest elements of the list:


              1234567891011121314151617181920212223
            
import heapq
import os
decorators = os.system('wget https://codefinity-content-media-v2.s3.eu-west-1.amazonaws.com/courses/8d21890f-d960-4129-bc88-096e24211d53/section_1/chapter_3/decorators.py 2>/dev/null')
from decorators import timeit_decorator
import random

# Generate a large list of random integers
numbers = [random.randint(1, 1000000) for _ in range(1000000)]

@timeit_decorator(number=10)
def partial_sort_heapq():
    return heapq.nsmallest(100000, numbers)

@timeit_decorator(number=10)
def partial_sort_sorted():
    return sorted(numbers)[:100000]

# Compare performance
heapq_result = partial_sort_heapq()
sorted_result = partial_sort_sorted()

# Ensure both methods give the same result
print(heapq_result == sorted_result)

The sorted() function in this case clearly outperforms heapq.

1. You need to sort an entire list of numbers **while keeping the original list intact**. Which sorting function/method should you use?

2. You are reviewing a dataset of **500,000 sales records**. To identify the **20 highest revenue-generating transactions**, which approach is likely to be faster and more memory-efficient?

Everything was clear?

Thanks for your feedback!

Section 3. Chapter 3