Apprendre Gradient Descent: The Core Optimizer | Foundations of Neural Network Optimization

At the heart of neural network optimization lies gradient descent, a method designed to minimize a loss function by iteratively updating the network's parameters. The mathematical foundation of gradient descent is rooted in calculus: you compute the gradient (or derivative) of the loss with respect to each parameter, which points in the direction of steepest ascent. By moving in the opposite direction of this gradient, you descend toward a local minimum. The update rule for each parameter $w$ at iteration $t$ is:

w_{t+1} = w_t - \eta \cdot \frac{\partial L}{\partial w_t}

Here, $w_t$ is the current parameter value, $\eta$ is the learning rate, and $\frac{\raisebox{1pt}{$\partial L$}}{\raisebox{-1pt}{$\partial w_t$}}$ is the gradient of the loss $L$ with respect to $w_t$ . The learning rate controls the step size for each update, determining how far you move in the direction opposite the gradient. This process is repeated until the loss function converges to a minimum, ideally resulting in a well-trained neural network.


              12345678910111213141516171819202122232425262728293031323334353637383940
            
import numpy as np
import matplotlib.pyplot as plt

# Generate toy data: y = 2x + 1 with noise
np.random.seed(42)
X = np.linspace(0, 1, 100)
y = 2 * X + 1 + np.random.normal(0, 0.1, size=X.shape)

learning_rates = [0.01, 0.1, 0.5]
epochs = 100
loss_histories = []
labels = []

for lr in learning_rates:
    # Initialize weights and bias for each run
    w = np.random.randn()
    b = np.random.randn()
    loss_history = []
    for epoch in range(epochs):
        y_pred = w * X + b
        loss = np.mean((y - y_pred) ** 2)
        loss_history.append(loss)
        dw = -2 * np.mean((y - y_pred) * X)
        db = -2 * np.mean(y - y_pred)
        w = w - lr * dw
        b = b - lr * db
    loss_histories.append(loss_history)
    labels.append(f"learning rate = {lr}")

# Plot training loss for each learning rate
plt.figure(figsize=(8, 4))
for loss_history, label in zip(loss_histories, labels):
    plt.plot(loss_history, label=label)
plt.title('Training Loss for Different Learning Rates')
plt.xlabel('Epoch')
plt.ylabel('Mean Squared Error Loss')
plt.legend()
plt.grid(True)
plt.tight_layout()
plt.show()

The learning rate is a critical hyperparameter in gradient descent. If you set the learning rate too high, you risk overshooting the minimum, causing the loss to diverge or oscillate without settling. If it is too low, convergence becomes painfully slow, and you may get stuck in local minima or plateaus. By experimenting with the code sample above, you can observe how modifying the learning_rate variable affects the speed and stability of optimization. A moderate learning rate ensures smooth and efficient convergence, while extreme values can lead to suboptimal or unstable training outcomes.

Tout était clair ?

Merci pour vos commentaires !

Section 1. Chapitre 2

Demandez à l'IA

Posez n'importe quelle question ou essayez l'une des questions suggérées pour commencer notre discussion

Suggested prompts:

Can you explain how the gradients are calculated in the code sample?

What happens if I use a learning rate outside the range shown in the example?

How do I choose an appropriate learning rate for my own neural network?

Glissez pour afficher le menu

w_{t+1} = w_t - \eta \cdot \frac{\partial L}{\partial w_t}


              12345678910111213141516171819202122232425262728293031323334353637383940
            
import numpy as np
import matplotlib.pyplot as plt

# Generate toy data: y = 2x + 1 with noise
np.random.seed(42)
X = np.linspace(0, 1, 100)
y = 2 * X + 1 + np.random.normal(0, 0.1, size=X.shape)

learning_rates = [0.01, 0.1, 0.5]
epochs = 100
loss_histories = []
labels = []

for lr in learning_rates:
    # Initialize weights and bias for each run
    w = np.random.randn()
    b = np.random.randn()
    loss_history = []
    for epoch in range(epochs):
        y_pred = w * X + b
        loss = np.mean((y - y_pred) ** 2)
        loss_history.append(loss)
        dw = -2 * np.mean((y - y_pred) * X)
        db = -2 * np.mean(y - y_pred)
        w = w - lr * dw
        b = b - lr * db
    loss_histories.append(loss_history)
    labels.append(f"learning rate = {lr}")

# Plot training loss for each learning rate
plt.figure(figsize=(8, 4))
for loss_history, label in zip(loss_histories, labels):
    plt.plot(loss_history, label=label)
plt.title('Training Loss for Different Learning Rates')
plt.xlabel('Epoch')
plt.ylabel('Mean Squared Error Loss')
plt.legend()
plt.grid(True)
plt.tight_layout()
plt.show()

Tout était clair ?

Merci pour vos commentaires !

Section 1. Chapitre 2