Underfitting vs Overfitting

Underfitting and overfitting are two fundamental problems in machine learning. Both occur when a model fails to generalize well, but they happen for opposite reasons. An underfitted model is too simple to capture important patterns in the data, while an overfitted model learns the training data too closely and captures details that do not generalize.

The goal of model training is to find a useful balance between these extremes. A well-generalized model should learn meaningful relationships in the training data while still performing well on new, unseen examples.

Understanding the difference between underfitting and overfitting helps when choosing model complexity, training duration, regularization, features, and evaluation strategies.

What Is Underfitting?

Underfitting occurs when a machine learning model is too simple or insufficiently trained to capture important patterns in the data. Because the model has not learned enough useful structure, it performs poorly not only on unseen data but often on the training data as well.

For example, suppose a dataset contains a complex nonlinear relationship between several variables and the target. If a model assumes a simple linear relationship, it may be unable to represent the underlying pattern. The resulting predictions can be inaccurate even for examples that were included in training.

Underfitting is therefore often associated with high training error and high validation or test error.

What Is Overfitting?

Overfitting occurs when a model learns the training data too specifically. Instead of learning only general patterns, the model can also learn noise, unusual examples, and accidental relationships that are specific to the training dataset.

An overfitted model can achieve very low training error while performing substantially worse on validation or test data.

For example, a decision tree can be grown so deeply that it creates highly specific rules for individual training examples. The tree may classify the training dataset almost perfectly but fail to make accurate predictions for new examples.

Underfitting vs Overfitting at a Glance

The most important distinction is the relationship between training and unseen-data performance. Underfitting usually produces poor performance everywhere, while overfitting produces a large gap between training performance and validation or test performance.

A Simple Example

Imagine that you are trying to predict house prices from features such as area, location, number of rooms, and age.

An extremely simple model might use only one feature and assume that price increases at a constant rate with area. If the real relationship depends on location, property condition, and other factors, the model may underfit.

At the other extreme, a highly complex model could learn very specific combinations of features that occur only in the training dataset. It might reproduce the training prices extremely accurately but make poor predictions for new houses. That would be overfitting.

A better model captures important relationships without memorizing the particular examples used during training.

How to Recognize Underfitting

Underfitting can often be detected by comparing training and validation performance. If both are poor, the model may not have enough capacity or may not have been trained effectively.

Underfitting should not be diagnosed solely from the model's complexity. A model can underfit because of poor features, insufficient training, excessive regularization, an inappropriate algorithm, or other problems.

How to Recognize Overfitting

Overfitting is commonly identified when training performance is substantially better than validation performance.

For models trained over multiple epochs, another common sign is that training loss continues to decrease while validation loss stops decreasing and begins to increase.

This pattern indicates that the model is becoming increasingly specialized to the training data rather than continuing to improve its ability to generalize.

Training and Validation Curves

Training curves are particularly useful for understanding the difference between underfitting and overfitting.

In an underfitting situation, training and validation loss can both remain relatively high. The model has not learned enough to perform well even on the data it was trained on.

With a well-fitting model, training loss decreases and validation loss also decreases before reaching a stable low level.

With overfitting, training loss can continue decreasing while validation loss starts increasing. The point at which validation performance was best can become a useful stopping point for training.

What Causes Underfitting?

• The model is too simple for the problem.
• The model has insufficient capacity.
• The model has not been trained for long enough.
• Regularization is too strong.
• Important features are missing.
• The chosen algorithm is poorly suited to the problem.
• The input representation does not capture useful information.

Underfitting can therefore be caused by more than simply choosing a model that is too small. Data representation and training configuration can be equally important.

What Causes Overfitting?

• The model has excessive capacity relative to the available data.
• The training dataset is too small.
• The data contains noise or incorrect labels.
• The model is trained for too many epochs or iterations.
• Regularization is insufficient.
• The feature set contains irrelevant or overly specific information.
• The training data is not representative of the data encountered in production.

Model Complexity

Model complexity is one of the main factors involved in the underfitting-overfitting trade-off.

If a model is too simple, it may lack the capacity to represent important relationships. Increasing its complexity can reduce underfitting.

If the model becomes excessively flexible, it can begin fitting noise and training-specific details. Reducing complexity or adding appropriate regularization can then improve generalization.

The optimal complexity depends on the dataset and problem. There is no universal model size that guarantees good generalization.

The Bias-Variance Trade-Off

The difference between underfitting and overfitting is closely related to the bias-variance trade-off.

High bias generally means that a model makes overly strong simplifying assumptions. Such a model may fail to capture important patterns and therefore underfit.

High variance means that model predictions can be highly sensitive to the particular training data. Such a model may fit the training examples extremely closely and therefore overfit.

The goal is not to eliminate bias or variance completely. Practical machine learning involves finding a balance that produces good performance on unseen data.

How to Fix Underfitting

When a model underfits, the general strategy is to increase its ability to learn useful patterns or improve the information available to it.

• Use a more expressive model.
• Increase model capacity where appropriate.
• Add informative features.
• Improve feature representation.
• Reduce excessive regularization.
• Train for more epochs when training has not converged.
• Adjust optimization settings.
• Use a model architecture better suited to the problem.

For example, a decision tree with a very small maximum depth may underfit. Allowing additional depth can improve performance if the extra complexity captures real structure rather than noise.

How to Fix Overfitting

When a model overfits, the goal is generally to improve generalization by reducing unnecessary specialization to the training data.

• Collect more representative training data.
• Reduce unnecessary model complexity.
• Apply regularization.
• Use early stopping.
• Use data augmentation where appropriate.
• Remove irrelevant features.
• Use cross-validation when appropriate.
• Reduce excessive training.

Regularization

Regularization adds constraints or penalties that discourage unnecessarily complex solutions. It can be used to reduce overfitting while preserving enough model capacity to learn useful patterns.

Common examples include L1 and L2 regularization. These methods modify the training objective by adding a penalty related to model parameters.

Total loss = prediction loss + regularization penalty

If regularization is too weak, the model may still overfit. If it is too strong, the model can become excessively constrained and start underfitting.

Early Stopping

Early stopping is particularly useful when a model initially improves on both training and validation data but eventually begins overfitting.

Instead of training until the training loss reaches its lowest possible value, training can stop when validation performance stops improving. The best model checkpoint can then be restored.

More Data

Increasing the amount of high-quality training data can reduce overfitting because the model has more examples from which to learn general patterns.

The additional data should be diverse and representative of the real problem. Simply duplicating existing examples does not provide the same benefit as collecting genuinely informative new examples.

Data Augmentation

Data augmentation creates additional training examples by applying valid transformations to existing data. It is commonly used for tasks such as image, audio, and text processing.

For example, image models may be trained with appropriately cropped, rotated, or otherwise transformed images. This can expose the model to more variation and reduce its reliance on specific details of the original training examples.

Changing the Model When It Underfits

If both training and validation performance are poor, increasing model capacity can be a reasonable next step. This could mean adding layers to a neural network, increasing tree depth, adding useful features, or choosing a more expressive algorithm.

However, increasing complexity should be done gradually. A model that jumps from strong underfitting to excessive capacity can quickly become harder to train and more prone to overfitting.

Changing the Model When It Overfits

If training performance is excellent but validation performance is significantly worse, simply increasing model complexity is usually not the first solution.

Instead, consider stronger regularization, more representative data, early stopping, feature selection, data augmentation, or a simpler model.

Underfitting and Overfitting During Neural Network Training

Neural networks can move from underfitting toward good generalization and eventually toward overfitting as training progresses.

Early in training, the network may not have learned enough useful representations. As training continues, both training and validation performance can improve. Eventually, the model may begin specializing in training-specific details.

This is why monitoring validation metrics during training is important rather than evaluating the model only after the final epoch.

Underfitting and Overfitting in Decision Trees

Decision trees make the trade-off particularly easy to visualize. A tree with very few levels may not contain enough rules to represent the data and can underfit.

As more levels and splits are added, the tree can represent increasingly complex relationships. At some point, it may begin creating rules that describe individual training examples rather than general patterns.

Maximum depth, minimum samples per leaf, and pruning are examples of techniques that can help control this behavior.

Why Training Error Alone Is Not Enough

Training error tells you how well the model fits the examples it was given. It does not directly tell you how well the model will perform on new data.

A model with extremely low training error may be excellent, overfitted, or both depending on its validation and test performance.

This is why a separate validation set or an appropriate cross-validation strategy is essential during model development.

The Role of the Test Set

The test set should generally remain separate from the model-development process. It provides an estimate of how the final selected model performs on unseen data.

Repeatedly adjusting the model based on test-set performance can effectively turn the test set into another validation set. This can make the final performance estimate overly optimistic.

A Practical Diagnostic Process

When a model performs poorly, the following process can help determine whether underfitting or overfitting is involved.

• Measure performance on the training data.
• Measure performance on validation data.
• Compare the two results.
• Inspect training and validation curves when available.
• If both performances are poor, investigate underfitting.
• If training performance is much better than validation performance, investigate overfitting.
• Change one or a small number of training factors at a time.
• Evaluate the final model on an independent test set.

Common Mistakes

• Assuming a more complex model is always better.
• Assuming lower training loss automatically means a better model.
• Using the test set to tune hyperparameters.
• Increasing regularization without checking whether the model is already underfitting.
• Adding data without checking its quality and representativeness.
• Ignoring the difference between training and validation performance.
• Diagnosing overfitting or underfitting from a single metric.

Underfitting vs Overfitting: Quick Decision Guide

Frequently Asked Questions

Conclusion

Underfitting and overfitting are opposite problems that occur when a machine learning model fails to generalize effectively. Underfitting means the model has not learned enough of the underlying structure, while overfitting means it has learned the training data too specifically.

The most useful diagnostic is to compare training and validation performance. Poor results on both datasets often indicate underfitting, while excellent training performance combined with substantially worse validation performance is a common sign of overfitting.

The goal is to find an appropriate balance between model capacity and generalization. Increasing complexity, reducing regularization, and training longer can help an underfitted model, while regularization, early stopping, more representative data, and reduced complexity can help an overfitted model.