Incremental Few-Shot Learning with Attention Attractor Networks

The outline of our meta-learning approach to incremental few-shot learning is: (1) We learn a fixed feature representation and a classifier on a set of base classes; (2) In each training and testing episode we train a novel-class classifier with our meta-learned regularizer; (3) We optimize our meta-learned regularizer on combined novel and base classes classification, adapting it to perform well in conjunction with the base classifier. Details of these stages follow.

Directly learning the few-shot episode, e.g., by setting $R(W_b,\theta )$ to be zero or simple weight decay, can cause catastrophic forgetting on the base classes. This is because $W_b$ which is trained to maximize the correct novel class probability can dominate the base classes in the joint prediction. In this section, we introduce the Attention Attractor Network to address this problem. The key feature of our attractor network is the regularization term $R(W_b,\theta )$:

where $u_{k^{\prime }}$ is the so-called attractor and $W_{b,k^{\prime }}$ is the $k^{\prime }$-th column of $W_b$. This sum of squared Mahalanobis distances from the attractors adds a bias to the learning signal arriving solely from novel classes. Note that for a classifier such as an MLP, one can extend this regularization term in a layer-wise manner. Specifically, one can have separate attractors per layer, and the number of attractors equals the number of output dimension of that layer.

To ensure that the model performs well on base classes, the attractors $u_{k^{\prime }}$ must contain some information about examples from base classes. Since we can not directly access these base examples, we propose to use the slow weights to encode such information. Specifically, each base class has a learned attractor vector $U_k$ stored in the memory matrix $U=[U_1,...,U_K]$. It is computed as, $U_k=f_{\varphi }\left(W_{a,k}\right)$, where $f$ is a MLP of which the learnable parameters are $\varphi$. For each novel class $k^{\prime }$ its classifier is regularized towards its attractor $u_{k^{\prime }}$ which is a weighted sum of $U_k$ vectors. Intuitively the weighting is an attention mechanism where each novel class attends to the base classes according to the level of interference, i.e. how prediction of new class $k^{\prime }$ causes the forgetting of base class $k$.

For each class in the support set, we compute the cosine similarity between the average representation of the class and base weights $W_a$ then normalize using a softmax function

where $A$ is the cosine similarity function, $h_j$ are the representations of the inputs in the support set $S_b$ and $\tau$ is a learnable temperature scalar. $a_{k^{\prime },k}$ encodes a normalized pairwise attention matrix between the novel classes and the base classes. The attention vector is then used to compute a linear weighted sum of entries in the memory matrix $U$, $u_{k^{\prime }}={\sum }_k{a}_{k^{\prime },k}{U}_k+U_0$, where $U_0$ is an embedding vector and serves as a bias for the attractor.

Our design takes inspiration from attractor networks (Mozer, 2009; Zemel and Mozer, 2001), where for each base class one learns an “attractor” that stores the relevant memory regarding that class. We call our full model “dynamic attractors” as they may vary with each episode even after meta-learning. In contrast if we only have the bias term $U_0$, i.e. a single attractor which is shared by all novel classes, it will not change after meta-learning from one episode to the other. We call this model variant the “static attractor”.

In summary, our meta parameters $\theta$ include $\varphi$, $U_0$, $\gamma$ and $\tau$, which is on the same scale as as the number of paramters in $W_a$. It is important to note that $R(W_b,\theta )$ is convex w.r.t. $W_b$. Therefore, if we use the LR model as the classifier, the overall training objective on episodes in Eq. (1) is convex which implies that the optimum $W_b^{\ast }(\theta ,S_b)$ is guaranteed to be unique and achievable. Here we emphasize that the optimal parameters $W_b^{\ast }$ are functions of parameters $\theta$ and few-shot samples $S_b$.

During meta-learning, $\theta$ are updated to minimize an expected loss of the query set $Q_{a+b}$ which contains both base and novel classes, averaging over all few-shot learning episodes,

where the predicted class is .

As there is no closed-form solution to the episodic objective (the optimization problem in Eq. 1), in each episode we need to minimize $L^S$ to obtain $W_b^{\ast }$ through an iterative optimizer. The question is how to efficiently compute $\frac{\partial W_b^{\ast }}{\partial \theta }$, i.e., back-propagating through the optimization. One option is to unroll the iterative optimization process in the computation graph and use back-propagation through time (BPTT) (Werbos, 1990). However, the number of iterations for a gradient-based optimizer to converge can be on the order of thousands, and BPTT can be computationally prohibitive. Another way is to use the truncated BPTT (Williams and Peng, 1990) (T-BPTT) which optimizes for $T$ steps of gradient-based optimization, and is commonly used in meta-learning problems. However, when $T$ is small the training objective could be significantly biased.

Alternatively, the recurrent back-propagation (RBP) algorithm (Almeida, 1987; Pineda, 1987; Liao et al., 2018) allows us to back-propagate through the fixed point efficiently without unrolling the computation graph and storing intermediate activations. Consider a vanilla gradient descent process on $W_b$ with step size $\alpha$. The difference between two steps $\Phi$ can be written as $\Phi \left(W_b^{\left(t\right)}\right)=W_b^{\left(t\right)}-F\left(W_b^{\left(t\right)}\right)$, where $F\left(W_b^{\left(t\right)}\right)=W_b^{(t+1)}=W_b^{\left(t\right)}-\alpha \nabla L^S\left(W_b^{\left(t\right)}\right)$. Since $\Phi \left(W_b^{\ast }\right(\theta \left)\right)$ is identically zero as a function of $\theta$, using the implicit function theorem we have $\frac{\partial W_b^{\ast }}{\partial \theta }=(I-J_{F,W_b^{\ast }}^{\top }{)}^{-1}\frac{}{\partial F}\partial \theta$, where $J_{F,W_b^{\ast }}$ denotes the Jacobian matrix of the mapping $F$ evaluated at $W_b^{\ast }$. Algorithm 1 outlines the key steps for learning the episodic objective using RBP in the incremental few-shot learning setting. Note that the RBP algorithm implicitly inverts $(I-J^{\top })$ by computing the matrix inverse vector product, and has the same time complexity compared to truncated BPTT given the same number of unrolled steps, but meanwhile RBP does not have to store intermediate activations.

• mini-ImageNet Proposed by Vinyals et al. (2016), mini-ImageNet contains 100 object classes and 60,000 images. We used the splits proposed by Ravi and Larochelle (2017), where training, validation, and testing have 64, 16 and 20 classes respectively.

• tiered-ImageNet Proposed by Ren et al. (2018), tiered-ImageNet is a larger subset of ILSVRC-12. It features a categorical split among training, validation, and testing subsets. The categorical split means that classes that belong to the same high-level category, e.g. “working dog” and ”terrier” or some other dog breed, are not split between training, validation and test. This is a harder task, but one that more strictly evaluates generalization to new classes. It is also an order of magnitude larger than mini-ImageNet.

We use a standard ResNet backbone (He et al., 2016) to learn the feature representation through supervised training. For mini-ImageNet experiments, we follow Mishra et al. (2018) and use a modified version of ResNet-10. For tiered-ImageNet, we use the standard ResNet-18 (He et al., 2016), but replace all batch normalization (Ioffe and Szegedy, 2015) layers with group normalization (Wu and He, 2018), as there is a large distributional shift from training to testing in tiered-ImageNet due to categorical splits. We used standard data augmentation, with random crops and horizonal flips. We use the same pretrained checkpoint as the starting point for meta-learning.

In the meta-learning stage as well as the final evaluation, we sample a few-shot episode from the $D_b$, together with a regular mini-batch from the $D_a$. The base class images are added to the query set of the few-shot episode. The base and novel classes are maintained in equal proportion in our experiments. For all the experiments, we consider 5-way classification with 1 or 5 support examples (i.e. shots). In the experiments, we use a query set of size 25$\times$2 =50.

We use L-BFGS (Zhu et al., 1997) to solve the inner loop of our models to make sure $W_b$ converges. We use the ADAM (Kingma and Ba, 2015) optimizer for meta-learning with a learning rate of 1e-3, which decays by a factor of $10$ after 4,000 steps, for a total of 8,000 steps. We fix recurrent backpropagation to 20 iterations and $ϵ=0.1$.

We study two variants of the classifier network. The first is a logistic regression model with a single weight matrix $W_b$. The second is a 2-layer fully connected MLP model with 40 hidden units in the middle and $\tanh$ non-linearity. To make training more efficient, we also add a shortcut connection in our MLP, which directly links the input to the output. In the second stage of training, we keep all backbone weights frozen and only train the meta-parameters $\theta$.

We consider the following evaluation metrics: 1) overall accuracy on individual query sets and the joint query set (“Base”, “Novel”, and “Both”); and 2) decrease in performance caused by joint prediction within the base and novel classes, considered separately (“${\Delta }_a$” and “${\Delta }_b$”). Finally we take the average $\Delta =\frac{1}{2}({\Delta }_a+{\Delta }_b)$ as a key measure of the overall decrease in accuracy.

We implemented and compared to three methods. First, we adapted Prototypical Networks (Snell et al., 2017) to incremental few-shot settings. For each base class we store a base representation, which is the average representation (prototype) over all images belonging to the base class. During the few-shot learning stage, we again average the representation of the few-shot classes and add them to the bank of base representations. Finally, we retrieve the nearest neighbor by comparing the representation of a test image with entries in the representation store. In summary, both $W_a$ and $W_b$ are stored as the average representation of all images seen so far that belong to a certain class. We also compare to the following methods:

• Weights Imprinting (“Imprint”) (Qi et al., 2018): the base weights $W_a$ are learned regularly through supervised pre-training, and $W_b$ are computed using prototypical averaging.

• Learning without Forgetting (“LwoF”) (Gidaris and Komodakis, 2018): Similar to (Qi et al., 2018), $W_b$ are computed using prototypical averaging. In addition, $W_a$ is finetuned during episodic meta-learning. We implemented the most advanced variants proposed in the paper, which involves a class-wise attention mechanism. This model is the previous state-of-the-art method on incremental few-shot learning, and has better performance compared to other low-shot models (Wang et al., 2018; Hariharan and Girshick, 2017).

We first evaluate our vanilla approach on the standard few-shot classification benchmark where no base classes are present in the query set. Our vanilla model consists of a pretrained CNN and a single-layer logistic regression with weight decay learned from scratch; this model performs on-par with other competitive meta-learning approaches (1-shot 55.40 $\pm$ 0.51, 5-shot 70.17 $\pm$ 0.46). Note that our model uses the same backbone architecture as (Mishra et al., 2018) and (Gidaris and Komodakis, 2018), and is directly comparable with their results. Similar findings of strong results using simple logistic regression on few-shot classification benchmarks are also recently reported in (Chen et al., 2019). Our full model has similar performance as the vanilla model on pure few-shot benchmarks, and the full table is available in Supp. Materials.

Next, we compare our models to other methods on incremental few-shot learning benchmarks in Tables 3 and 3. On both benchmarks, our best performing model shows a significant margin over the prior works that predict the prototype representation without using an iterative optimization (Snell et al., 2017; Qi et al., 2018; Gidaris and Komodakis, 2018).

To understand the effectiveness of each part of the proposed model, we consider the following variants:

• Vanilla (“LR, MLP”) optimizes a logistic regression or an MLP network at each few-shot episode, with a weight decay regularizer.

• Static attractor (“+S”) learns a fixed attractor center $u$ and attractor slope $\gamma$ for all classes.

• Attention attractor (“+A”) learns the full attention attractor model. For MLP models, the weights below the final layer are controlled by attractors predicted by the average representation across all the episodes. $f_{\varphi }$ is an MLP with one hidden layer of 50 units.

Tables 5 and 5 shows the ablation experiment results. In all cases, the learned regularization function shows better performance than a manually set weight decay constant on the classifier network, in terms of both jointly predicting base and novel classes, as well as less degradation from individual prediction. On mini-ImageNet, our attention attractors have a clear advantage over static attractors.

Formulating the classifier as an MLP network is slightly better than the linear models in our experiments. Although the final performance is similar, our RBP-based algorithm have the flexibility of adding the fast episodic model with more capacity. Unlike Bertinetto et al. (2018), we do not rely on an analytic form of the gradients of the optimization process.