What Is Self-Supervised Learning and Why Does It Matter?

It involves training the model by getting it to complete the missing parts of unlabeled data. You cover up some of the data and get the model to guess the covered-up part using the rest, so the data provides its own labels and no human annotation is needed.

Unlike unsupervised learning, which detects patterns in the dataset without any targeted outcome, self-supervised learning uses the same type of data but designs a specific prediction task by generating its own labels from the input, giving it a stronger, more directed learning signal.

The classic task involves predicting the subsequent token based on the previous tokens. Repeated many times over a huge corpus of text, this makes the model understand grammar, facts, and reasoning, which is how large language models become so capable through pretraining.

No labeling by humans involved. The labels are intrinsic to the data because the right answer is simply what is left out. That is why it can learn from nearly unlimited unlabeled data at no extra cost of labeling.

Not generally. This approach develops broad capability at a low cost from raw data during the pretraining phase, after which supervised fine-tuning or preference-based training tailors the model to the particular tasks you care about.