Probing Few-Shot Generalization with Attributes

Many of the existing few-shot learning approaches include a stage of supervised representation learning. Two classes of approaches are frequently employed:

• Episodic meta-learning approaches train directly from a set of few-shot episodes using episodic labels. This class of methods can be naturally applied to our learning setting.

• Supervised classification approaches train a network to directly classify a set of training classes using absolute labels, and at test time, the embedding network is transferred to solve the test task by training another classifier on top. If absolute attributes are provided to the learner, then one natural approach is to instead train an attribute classifier with multiple binary outputs. After the attribute classifier network is learned, we can then transfer the representations to recognize test attributes. We denote this method as Supervised Attributes (SA).

As supervised representation learning may not generalize to novel attributes, we also consider unsupervised representation learning as another option. We chose SimCLR (Chen et al., 2020) as a representative from this category due to its empirical success. In general, contrastive learning approaches aim to build invariant representations between a pair of inputs $\{x,x^{\prime }\}$ that are produced by applying random data augmentations (e.g. cropping) to an input image. It is likely to preserve more general semantic features since all attributes are useful towards identifying another random crop of the same image. We first obtain the embedding output $h$ from the CNN, and then following (Chen et al., 2020), we project $h$ to $z$ using a multi-layered perceptron (MLP): $h=CNN\left(x\right),z={MLP}_1\left(h\right)$. With a batch of image pairs denoted by $\left\{x_i\right\},\left\{x_i^{\prime }\right\}$, we can obtain their features $\left\{z_i\right\},\left\{z_i^{\prime }\right\}$, and the contrastive loss function is defined similar to the cross entropy function:

where $\tau$ is a temperature parameter. We denote Unsupervised representation learning as U.

For unsupervised learning, we also consider adding a subsequent stage of supervised fine-tuning to utilize attribute labels from the training set. Note that fine-tuning here is different from fine-tuning in regular few-shot learning as it is not fine-tuning on test episodes but rather on the original training set. To prevent overwriting the representations and making them overly sensitive to training attributes, we add another projection MLP that learns more specific representations for finetuning on training attributes: $g={MLP}_2\left(h\right).$ Here, we again consider using two different modes of supervision: 1) the FSAL binary episodic labels, or 2) the underlying absolute attribute labels:

• [leftmargin=*]

• Unsupervised-then-FineTune-on-Episodes (UFTE). We adopt the Prototypical Networks (Snell et al., 2017) formulation, where the network solves a learning episode of $N$ positive and negative support examples by using prototypes $p$: $p^{+}=\frac{1}{N}{\sum }_i{g}_i^{+};p^{-}=\frac{1}{N}{\sum }_i{g}_i^{-}.$ With query example $g^q$, we can make a binary prediction: ${\hat{y}}^q=\frac{\exp (-d(g^q,p^{+}\left)\right)}{\exp (-d(g^q,p^{+}\left)\right)+\exp (-d(g^q,p^{-}\left)\right)},$ where $d$ is some dissimilarity score, e.g. Euclidean distance or cosine dissimilarity, and the training objective is to minimize the classification loss between the prediction ${\hat{y}}^q$ and the label $y^q$:

  $$L_{2E}$$

  $$=\sum _j-y_j\log {\hat{y}}_j^q-(1-y_j^q)\log (1-{\hat{y}}_j^q),$$

  (4)

  where $j$ is the index of query examples.

• Unsupervised-then-FineTune-on-Attributes (UFTA). With persistent attribute information, we can train a linear classifier with sigmoid activation to directly predict the absolute attribute labels $a$: $\hat{a}=W_{A}g+b_A$, with the loss being

  $$L_{2A}$$

  $$=\sum _k-a_k\log {\hat{a}}_k-(1-a_k)\log (1-{\hat{a}}_k),$$

  (5)

  where $k$ is the index of attributes.

For each episode, we randomly select one or two attributes and look for positive examples belonging to these attributes simultaneously. We also sample an equal number of negative examples that don’t match the selected attributes. This will construct a support set of positive and negative samples, and then we repeat the same process for the corresponding query set as well. Sample episodes are shown in Figure 1. Additional episodes are shown in the Appendix.

For Celeb-A and Zappos, images were cropped and resized to 84$\times$84. We used ResNet-12 (He et al., 2016; Oreshkin et al., 2018) as the CNN backbone. The projection MLPs have 512-512-128 units. We train SimCLR entirely on Celeb-A/Zappos images, i.e. not using pre-trained ImageNet checkpoints for fair comparison. For ImageNet-with-Attributes, we utilize the off-the-self SimCLR model from ImageNet-1k, which has access to more unlabeled images. The image dimensions are 224$\times$224. Additional details are in the Appendix.

To understand and interpret the decision made by few-shot linear classifiers, we visualize the classifier weights by using CAM (Zhou et al., 2016), and plot the heatmap over the 11$\times$11 spatial feature map in Figure 2. SA sometimes shows incorrect localization as it is not trained to classify those novel test attributes. SA* shows bigger but less precise heatmaps since the training objective encourages the propagation of attribute information spatially. In contrast, UFTA produces accurate and localized heatmaps that pinpoint the location of the attributes (e.g. mustache or cheekbone); this is impressive since no labeled information concerning these attributes was available during representation pre-training and finetuning. This result supports the hypothesis that local features can be good descriptors that match different views of the same instance during contrastive learning, and finetuning further establishes a positive transfer between training and test attributes.

Our few-shot attribute learning episodes can be ambiguous. For example, by presenting only a smiling face with eyeglasses in the support set, it is unclear whether the positive set is determined by “smiling” or “wearing eyeglasses”. Figure 3 show several approaches evaluated using LR with varying numbers of support examples per class in Celeb-A and Zappos-50K episodes, respectively. The oracle GT gradually approached 100% accuracy as the number of shots approached 20. This demonstrates that FSAL tasks potentially require more support examples than standard FSL to resolve ambiguity. Again here, UFTA and UFTE consistently outperformed U, SA, and ID across different number of shots. Figure 4 shows the correlation between readout performance of attributes and few-shot learning accuracy, using UFTA. With a larger number of shots, there is a higher correlation between the two, but there is still a large amount of variance that is due to the ambiguity of the task itself. More details are included in the Appendix.

Table 5 studies the effect of the projection MLP for attribute classification finetuning. Adding MLP projection layers was found to be beneficial for unsupervised learning in prior work (Chen et al., 2020). Here we found that adding MLP layers is also critical in supervised finetuning. Finetuning directly on the backbone (depth=0), and keeping the MLP during test (Discard=no) both led to significant drop in performance. In the Appendix, we also report on studies of the effect of adding the L1 regularizer on LR.

We hypothesize that the weak performance of episodic learners and SA on our benchmarks is because their training objectives essentially encourage ignoring attributes that are not useful at training time, but may still be useful at test time. In Appendix G, we study a synthetic problem to further analyze these generalization issues. We explore training a ProtoNet model on data from a linear generative model, where each FSAL episode presents ambiguity in identifying the relevant attributes. In this setting, the network is forced to discard information that is useful for test tasks to solve the training tasks effectively, and thus fails to generalize.

We aim to investigate the question of why unsupervised pretraining and supervised finetuning produce better performance, and whether the performance difference is caused by the closeness between training and test attributes. More concretely, we aim to predict the transferability between training and test splits by analyzing the training vs. test attributes. Each image has a complete attribute vector, describing the values of each attribute in the image. Some of these attributes are in the training set, and others in the test set. To quantify the transferability, we leverage the idea of mutual information. In particular, we learn a logistic regression model that takes the training attribute vector in a particular image as input and predicts the value of one of the test attributes in that image. Each logistic regression model will generate an AUC score on held-out images, and we average them across the relevant test attributes in each episode, and we define this AUC score as the “transferability score.” Our hypothesis is that more mutual information between the attribute label distributions will translate to higher transfer performance.

In Figure 5, we ran experiments using 100 random splits of training and test attributes. The results verify our hypothesis. We see positive correlation between the transfer performance and our transferability score: When subtracting U as a baseline, both UFTA and UFTE models get better when there is higher transferability (subtraction reduces the effect of per-episode variability). The same conclusion can be drawn when we subtract SA* from SA. By plotting the relation between U and SA, we show that supervised learning is more helpful when there is higher transferability in the label space whereas self-supervised learning is more flexible at adapting to novel target tasks.

To summarize, our empirical evidence suggests that unsupervised representation learning is superior to supervised methods in terms of retaining information relevant to the test attributes. Other methods, such as supervised representation learning or episodic training, tend to effectively ignore attributes that were not used for labelling during training. Moreover, supervised finetuning of the representations is helpful when the transferability between test and train attributes is high, but less so otherwise, and supervised pretraining alone is harmful to novel attribute generalization for most train vs. test attribute pairs.