Lernen Thread Pools and Process Pools | Advanced Patterns and Best Practices

Swipe um das Menü anzuzeigen

When you need to run many tasks concurrently, creating and managing each thread or process manually quickly becomes cumbersome and error-prone. This is where the concept of pools comes in. A pool is a collection of worker threads or processes that are managed for you, allowing you to submit tasks without worrying about the low-level details of starting, stopping, or reusing workers. By using pools, you can efficiently distribute work, limit resource usage, and write cleaner, more maintainable code. Python provides the concurrent.futures module, which includes ThreadPoolExecutor for threads and ProcessPoolExecutor for processes. These executors make it easy to submit tasks and collect results asynchronously.


              1234567891011121314151617
            
import concurrent.futures
import time

def square(n):
    time.sleep(1)
    return n * n

numbers = [1, 2, 3, 4, 5]
results = []

with concurrent.futures.ThreadPoolExecutor(max_workers=3) as executor:
    future_to_num = {executor.submit(square, num): num for num in numbers}
    for future in concurrent.futures.as_completed(future_to_num):
        num = future_to_num[future]
        result = future.result()
        results.append((num, result))
        print(f"Square of {num} is {result}")

How ThreadPoolExecutor and futures.as_completed Work

The ThreadPoolExecutor is a class in the concurrent.futures module that manages a pool of worker threads. When you use a ThreadPoolExecutor, you specify how many threads you want in the pool by setting the max_workers parameter. You then submit tasks to the executor, and it schedules them to run in the available threads.

When you call executor.submit(function, arg), the function and its arguments are scheduled to run in a worker thread. This returns a future object, which acts as a placeholder for the result of the computation. You can use the future to check if the task is done, retrieve the result, or handle exceptions.

The concurrent.futures.as_completed() function allows you to process results as soon as each task finishes, no matter the order you submitted them. You pass it a list of futures, and it yields each future as soon as its associated task is complete. This is especially useful when tasks take varying amounts of time, so you can handle results immediately rather than waiting for all tasks to finish.

Key points:

ThreadPoolExecutor creates and manages a pool of threads for you;
submit() schedules a function to run and returns a future object;
Futures represent the eventual result of an asynchronous operation;
as_completed() lets you process results as soon as each task finishes, in the order they complete.

By combining these tools, you can efficiently submit many tasks, process their results as they are ready, and avoid manual thread management.


              1234567891011121314151617
            
import concurrent.futures
import time

def square(n):
    time.sleep(1)
    return n * n

numbers = [1, 2, 3, 4, 5]
results = []

with concurrent.futures.ThreadPoolExecutor(max_workers=3) as executor:
    future_to_num = {executor.submit(square, num): num for num in numbers}
    for future in concurrent.futures.as_completed(future_to_num):
        num = future_to_num[future]
        result = future.result()
        results.append((num, result))
        print(f"Square of {num} is {result}")

War alles klar?

Danke für Ihr Feedback!

Abschnitt 4. Kapitel 1

Fragen Sie AI

Fragen Sie alles oder probieren Sie eine der vorgeschlagenen Fragen, um unser Gespräch zu beginnen

Abschnitt 4. Kapitel 1