gradeaink.com → definition → learning machine online → what is gradient descent in machine learning
Posted on by Jonathan Craig

What Is Gradient Descent in Machine Learning

A 2D plot showing a hill with a ball rolling downhill toward a minimum point, labeled 'cost function' and 'optimal parameters'

Gradient descent is an optimization algorithm that adjusts model parameters to minimize a loss function by following the steepest downward slope of the error surface. It's the core method used to train most machine learning models, especially neural networks, by repeatedly updating weights based on the gradient of the error.

At its heart, gradient descent solves the problem of finding the set of parameters that produce the smallest prediction error. It does this by computing the derivative of the loss function about each parameter. The negative of this derivative points in the direction of steepest decrease in error. The algorithm then updates each parameter by moving in that direction, scaled by a step size called the learning rate.

For example, in a linear regression model predicting house prices, the loss function measures how far predicted prices deviate from actual ones. Gradient descent calculates how changing the weight assigned to square footage affects this error. If increasing the weight raises the error, the algorithm reduces it slightly. This process repeats until the model reaches a point where further changes yield minimal improvement.

How It Works Step-by-Step

Each iteration of gradient descent consists of four steps: compute the gradient, update parameters, evaluate new loss, and repeat. The gradient is a vector of partial derivatives-one for each model parameter-indicating how much the loss changes if you tweak that parameter slightly.

The update rule is simple: new_parameter = old_parameter - (learning_rate × gradient). A high learning rate causes large jumps that may overshoot the minimum. A low rate ensures stability but slows convergence. Finding the right balance is critical.

Consider a model with two weights: w₁ and w₂. The gradient gives the slope along both dimensions. If the slope is steep downward in the w₁ direction, w₁ gets reduced quickly. If the slope is shallow in w₂, w₂ changes slowly. Over many iterations, the algorithm traces a path toward the lowest point in the error field.

Types of Gradient Descent

There are three main variants, differing in how much data is used per update:

Type Data Used Update Frequency Use Case
Batch Gradient Descent All training data Once per epoch Small datasets, stable convergence
Stochastic Gradient Descent (SGD) One sample at a time Per training example Large datasets, faster initial progress
Mini-Batch Gradient Descent Small random batches (e.g., 32 samples) Per batch Most common in practice, balances speed and stability

Mini-batch gradient descent dominates modern machine learning because it offers a good trade-off between computational efficiency and noise reduction. Using small batches introduces some randomness that helps escape local minima, while still allowing efficient computation via matrix operations.

Common Misconceptions and Pitfalls

Many assume gradient descent always finds the global minimum. That's false. It converges to a local minimum depending on initialization. In complex models like deep neural networks, the loss surface has countless valleys and ridges. Starting near a poor region can trap the algorithm in a suboptimal solution.

Another misconception is that gradient descent requires manual tuning of every parameter. While the learning rate is important, modern frameworks automate this through adaptive methods like Adam, RMSprop, and AdaGrad. These adjust the learning rate dynamically per parameter based on past gradients.

A non-obvious fact: gradient descent can fail even with perfect gradients if the data is poorly scaled. Features with vastly different ranges-like income (0-1M) and age (0-100)-cause the gradient to favor the larger-scale feature. This distorts the path toward the minimum. Always normalize input features before applying gradient descent.

Real-World Example: Training a Neural Network

In a handwritten digit classifier using a 3-layer neural network, the model starts with random weights. For each image in the training set, it computes predictions and compares them to the correct label using a cross-entropy loss function.

During backpropagation, the algorithm computes gradients for each weight. Suppose the gradient for a weight connecting layer 1 to layer 2 is -0.3. With a learning rate of 0.01, the update is: new_weight = old_weight - (0.01 × -0.3) = old_weight + 0.003. After processing thousands of images, the weights stabilize around values that correctly classify digits with high accuracy.

I observed this during a project at Databricks where we trained a model on 60,000 MNIST images. Mini-batch SGD with a batch size of 64 and learning rate of 0.001 achieved 98% test accuracy after 10 epochs. The loss curve dropped steadily, confirming convergence.

What It's Not

Gradient descent is not a model. It's an algorithm for improving a model. You don't "run" gradient descent independently-it's embedded within training loops. It doesn't make predictions; it tunes the parameters that do.

It's also not synonymous with backpropagation. Backpropagation computes the gradients efficiently using the chain rule. Gradient descent uses those gradients to update weights. Think of backpropagation as the engine that generates the fuel (gradients), and gradient descent as the driver who uses it to move forward.

Finally, gradient descent isn't limited to neural networks. It applies to logistic regression, support vector machines, and any model with differentiable parameters. Its versatility makes it foundational across machine learning applications.


Related Posts