In this article, we will discuss the major contributions of the paper “Big Self-Supervised Models are Strong Semi-Supervised Learners” which is simply termed as SimCLRv2, an improved version of SimCLR.
Major Changes Made to SimCLR
- Larger ResNet models: The largest model used in SimCLR was ResNet50(x4). In SimCLRv2, the authors pre-trained ResNet152(x3) with and without selective kernel (+SK) convolutional layers.
- Increase capacity of Projection layer: The capacity of the non-linear network g(·) (a.k.a. projection head) is also increased, by making it deeper. Furthermore, instead of throwing away g(·) entirely after pretraining as in SimCLR, the authors fine-tune from a middle layer. This small change yields a significant improvement for both linear evaluation and fine-tuning with only a few labelled examples.
- Memory Bank: The authors also incorporate the memory mechanism from MoCo, which designates a memory network (with a moving average of weights for stabilization) whose output will be buffered as negative examples. However, this addition does not improve the performance much (~1%) in linear evaluation.
Key Findings
- Bigger self-supervised models are more label efficient: Empirical results suggest that for semi-supervised learning (via the task-agnostic use of unlabeled data), the fewer the labels, the more it is possible to benefit from a bigger model. Bigger self-supervised models are more label efficient, performing significantly better when fine-tuned on only a few labelled examples, even though they have more capacity to potentially overfit.
- Extra capacity not always necessary: Although big models are important for learning general (visual) representations, the extra capacity may not be necessary when a specific target task is concerned. Therefore, with the task-specific use of unlabeled data, the predictive performance of the model can be further improved and transferred into a smaller network.
- Bigger/Deeper Projection Heads Improve Representation Learning: A deeper projection head not only improves the representation quality measured by linear evaluation but also improves semi-supervised performance when fine-tuning from a middle layer of the projection head.
- Distillation Using Unlabeled Data Improves Semi-Supervised Learning: Distillation with unlabeled examples improves fine-tuned models in two ways: (1) when the student model has a smaller architecture than the teacher model, it improves the model efficiency by transferring task-specific knowledge to a student model, (2) even when the student model has the same architecture as the teacher model (excluding the projection head after ResNet encoder), self-distillation can still meaningfully improve the semi-supervised learning performance. To obtain the best performance for smaller ResNets, the big model is self-distilled before distilling it into smaller models.
Experimental Results
Implementations & Repositories
- Tensorflow:
- Keras
- PyTorch:
Please add in the comments, if the reader feels that any point is not mentioned in this article.
Originally published on Medium.