Impara Bayesian Optimization Overview | Advanced and Bayesian Tuning Techniques

Bayesian optimization is an efficient method for hyperparameter tuning that uses probability and previous results to guide the search for the best values. Instead of randomly trying settings or checking every possibility, it builds a model to predict which hyperparameter choices are most promising. This approach focuses your search on areas likely to give better results, helping you find strong hyperparameters with fewer attempts.

Definition

Acquisition functions guide Bayesian optimization by scoring which hyperparameters to try next using the current probabilistic model. Popular choices include:

Expected Improvement (EI): $EI(x) = E[max(0, f(x) - f(x^+))]$
Probability of Improvement (PI): $PI(x) = P(f(x) > f(x^+) + ξ)$
Upper Confidence Bound (UCB): $UCB(x) = μ(x) + κ·σ(x)$

Here, $f(x^+)$ is the best observed value, $μ(x)$ and $σ(x)$ are the model's predicted mean and standard deviation, and $ξ$ , $κ$ are tunable parameters.


              1234567891011121314151617181920212223242526272829303132333435363738
            
# Conceptual workflow for Bayesian optimization (no external libraries)
import numpy as np

# Suppose we are tuning a single hyperparameter x in [0, 10]
space = np.linspace(0, 10, 100)

def objective_function(x):
    # Simulate a function to optimize (e.g., validation error)
    return (x - 3) ** 2 + np.sin(x) * 2

# Step 1: Initialize with a few random samples
history = []
for x in np.random.uniform(0, 10, 3):
    y = objective_function(x)
    history.append((x, y))

# Step 2: Fit a simple model to predict objective as a function of x
# (Here, we use a moving average just for illustration)
def predict(x, history):
    # Use the closest measured value as prediction
    closest = min(history, key=lambda h: abs(h[0] - x))
    return closest[1]

# Step 3: Acquisition - choose next x to evaluate
def acquisition(space, history):
    # Pick the point with lowest predicted objective
    predictions = [predict(x, history) for x in space]
    return space[np.argmin(predictions)]

# Step 4: Iterate - update history with new evaluation
for _ in range(5):
    x_next = acquisition(space, history)
    y_next = objective_function(x_next)
    history.append((x_next, y_next))

# Step 5: Report best found
best_x, best_y = min(history, key=lambda h: h[1])
print(f"Best x: {best_x:.2f}, Best objective: {best_y:.2f}")

Tutto è chiaro?

Grazie per i tuoi commenti!

Sezione 3. Capitolo 1

Chieda ad AI

Chieda pure quello che desidera o provi una delle domande suggerite per iniziare la nostra conversazione

Awesome!

Completion rate improved to 9.09

Scorri per mostrare il menu