Explore Optimization
Find better parameters, step by step.
Loss landscape
PausedDarker = lower loss · rings = equal loss
f(x,y) = 0.2u² + 8v²; u = (x+y)/√2, v = (x−y)/√2
Five paths. Same start.
| Optimizer | Loss | Steps | Status |
|---|---|---|---|
| Gradient descent | 1.94e+1 | 0 | Ready |
| Momentum | 1.94e+1 | 0 | Ready |
| Nesterov | 1.94e+1 | 0 | Ready |
| RMSProp | 1.94e+1 | 0 | Ready |
| Adam | 1.94e+1 | 0 | Ready |
Paths outside the viewport keep updating. Divergence stops a method; it is never clipped into a false solution. All methods stop at 500 steps or gradient norm below 10⁻⁶.
Inspect current parameters and gradients
| Optimizer | x | y | ∂f/∂x | ∂f/∂y |
|---|---|---|---|---|
| Gradient descent | -1.4000 | 0.80000 | -17.720 | 17.480 |
| Momentum | -1.4000 | 0.80000 | -17.720 | 17.480 |
| Nesterov | -1.4000 | 0.80000 | -17.720 | 17.480 |
| RMSProp | -1.4000 | 0.80000 | -17.720 | 17.480 |
| Adam | -1.4000 | 0.80000 | -17.720 | 17.480 |
Why the same learning rate behaves differently ↗
Gradient descent follows the current gradient. Momentum carries velocity; Nesterov measures the gradient at a look-ahead position. RMSProp rescales coordinates by a moving average of squared gradients (decay 0.9). Adam combines moving averages of the gradient and squared gradient, with bias correction (β₂ = 0.999). The shared learning rate makes behavior easy to compare; it is not a benchmark of optimally tuned methods. These are exact gradients of two-variable functions, not minibatch training.
Source / Adam paper ↗Try overshooting, valleys, and local minima ↗
Start with the narrow ravine and raise the learning rate: some paths bounce or diverge. Try the Rosenbrock valley to see slow progress around a bend. In the double well, different starting points reach different minima; starting exactly at the central saddle gives zero gradient and stops every method. “Converged” means a small gradient, not proof of a global minimum. Ripples adds local minima. The loss curve uses log(1 + loss), and each surface uses its own contour levels.