What Is Transfer Learning and Why Does It Matter?

This technique uses an existing model trained on one problem as the base for another similar problem. The model is fine-tuned rather than trained from scratch, which saves data, compute, and time.

The process of fine-tuning is by far the most frequently used form of performing transfer learning. You start with a pretrained model and keep training it using task-specific data so that it adapts the learned knowledge to the new task, rather than starting from random weights.

Only when no relevant pretrained model exists, or when your domain is so different that general knowledge does not transfer. In almost all other situations, adapting a pretrained model is faster, cheaper, and more accurate than training from scratch.

It is when a model becomes overtrained to an extent where the beneficial knowledge acquired during pretraining is stripped away in the process of adapting to a narrow task. Parameter-efficient methods and careful training help avoid it by changing the model less.

This is particularly effective if the new task is related to the original one. In general, the further apart the two domains are, the less useful the prior knowledge gained through pretraining, and eventually it makes sense to train from scratch or select a more relevant base model.