Lära Bayesian Optimization Overview | Advanced and Bayesian Tuning Techniques

Bayesian optimization is an efficient method for hyperparameter tuning that uses probability and previous results to guide the search for the best values. Instead of randomly trying settings or checking every possibility, it builds a model to predict which hyperparameter choices are most promising. This approach focuses your search on areas likely to give better results, helping you find strong hyperparameters with fewer attempts.

Definition

Acquisition functions guide Bayesian optimization by scoring which hyperparameters to try next using the current probabilistic model. Popular choices include:

Expected Improvement (EI): $EI(x) = E[max(0, f(x) - f(x^+))]$
Probability of Improvement (PI): $PI(x) = P(f(x) > f(x^+) + ξ)$
Upper Confidence Bound (UCB): $UCB(x) = μ(x) + κ·σ(x)$

Here, $f(x^+)$ is the best observed value, $μ(x)$ and $σ(x)$ are the model's predicted mean and standard deviation, and $ξ$ , $κ$ are tunable parameters.


              1234567891011121314151617181920212223242526272829303132333435363738
            
# Conceptual workflow for Bayesian optimization (no external libraries)
import numpy as np

# Suppose we are tuning a single hyperparameter x in [0, 10]
space = np.linspace(0, 10, 100)

def objective_function(x):
    # Simulate a function to optimize (e.g., validation error)
    return (x - 3) ** 2 + np.sin(x) * 2

# Step 1: Initialize with a few random samples
history = []
for x in np.random.uniform(0, 10, 3):
    y = objective_function(x)
    history.append((x, y))

# Step 2: Fit a simple model to predict objective as a function of x
# (Here, we use a moving average just for illustration)
def predict(x, history):
    # Use the closest measured value as prediction
    closest = min(history, key=lambda h: abs(h[0] - x))
    return closest[1]

# Step 3: Acquisition - choose next x to evaluate
def acquisition(space, history):
    # Pick the point with lowest predicted objective
    predictions = [predict(x, history) for x in space]
    return space[np.argmin(predictions)]

# Step 4: Iterate - update history with new evaluation
for _ in range(5):
    x_next = acquisition(space, history)
    y_next = objective_function(x_next)
    history.append((x_next, y_next))

# Step 5: Report best found
best_x, best_y = min(history, key=lambda h: h[1])
print(f"Best x: {best_x:.2f}, Best objective: {best_y:.2f}")

Var allt tydligt?

Tack för dina kommentarer!

Avsnitt 3. Kapitel 1

Fråga AI

Fråga vad du vill eller prova någon av de föreslagna frågorna för att starta vårt samtal

Svep för att visa menyn