Lære Thread Pools and Process Pools | Advanced Patterns and Best Practices

Sveip for å vise menyen

When you need to run many tasks concurrently, creating and managing each thread or process manually quickly becomes cumbersome and error-prone. This is where the concept of pools comes in. A pool is a collection of worker threads or processes that are managed for you, allowing you to submit tasks without worrying about the low-level details of starting, stopping, or reusing workers. By using pools, you can efficiently distribute work, limit resource usage, and write cleaner, more maintainable code. Python provides the concurrent.futures module, which includes ThreadPoolExecutor for threads and ProcessPoolExecutor for processes. These executors make it easy to submit tasks and collect results asynchronously.


              1234567891011121314151617
            
import concurrent.futures
import time

def square(n):
    time.sleep(1)
    return n * n

numbers = [1, 2, 3, 4, 5]
results = []

with concurrent.futures.ThreadPoolExecutor(max_workers=3) as executor:
    future_to_num = {executor.submit(square, num): num for num in numbers}
    for future in concurrent.futures.as_completed(future_to_num):
        num = future_to_num[future]
        result = future.result()
        results.append((num, result))
        print(f"Square of {num} is {result}")

How ThreadPoolExecutor and futures.as_completed Work

The ThreadPoolExecutor is a class in the concurrent.futures module that manages a pool of worker threads. When you use a ThreadPoolExecutor, you specify how many threads you want in the pool by setting the max_workers parameter. You then submit tasks to the executor, and it schedules them to run in the available threads.

When you call executor.submit(function, arg), the function and its arguments are scheduled to run in a worker thread. This returns a future object, which acts as a placeholder for the result of the computation. You can use the future to check if the task is done, retrieve the result, or handle exceptions.

The concurrent.futures.as_completed() function allows you to process results as soon as each task finishes, no matter the order you submitted them. You pass it a list of futures, and it yields each future as soon as its associated task is complete. This is especially useful when tasks take varying amounts of time, so you can handle results immediately rather than waiting for all tasks to finish.

Key points:

ThreadPoolExecutor creates and manages a pool of threads for you;
submit() schedules a function to run and returns a future object;
Futures represent the eventual result of an asynchronous operation;
as_completed() lets you process results as soon as each task finishes, in the order they complete.

By combining these tools, you can efficiently submit many tasks, process their results as they are ready, and avoid manual thread management.


              1234567891011121314151617
            
import concurrent.futures
import time

def square(n):
    time.sleep(1)
    return n * n

numbers = [1, 2, 3, 4, 5]
results = []

with concurrent.futures.ThreadPoolExecutor(max_workers=3) as executor:
    future_to_num = {executor.submit(square, num): num for num in numbers}
    for future in concurrent.futures.as_completed(future_to_num):
        num = future_to_num[future]
        result = future.result()
        results.append((num, result))
        print(f"Square of {num} is {result}")

Alt var klart?

Takk for tilbakemeldingene dine!

Seksjon 4. Kapittel 1

Spør AI

Spør om hva du vil, eller prøv ett av de foreslåtte spørsmålene for å starte chatten vår

Seksjon 4. Kapittel 1