SketchEmbedNet: Learning Novel Concepts by Imitating Drawings

The pen-states predictions $\{s_1^{\prime },s_2^{\prime },s_3^{\prime }\}$ are optimized as a simple 3-way classification with the softmax cross-entropy loss,

The stroke loss maximizes the log-likelihood of the spatial offsets of each ground truth stroke ${\Delta }_t$ given the mixture density distribution $p_t$ at each timestep:

While pixel-level reconstruction objectives are common in generative models (Kingma and Welling, 2014; Vincent et al., 2010; Gregor et al., 2015), they do not exist for sketching models. However, they still represent a meaningful form of generative supervision, promoting visual similarity in the generated result. To enable this loss, we developed a novel rasterization function $f_{\text{raster}}$ that produces a pixel image from our stroke parameterization of sketch drawings. $f_{\text{raster}}$ transforms the stroke sequence $y$ by viewing it as a set of 2D line segments $(l_0,l_1),(l_1,l_2)\dots (l_{T-1},l_T)$ where $l_t={\sum }_{\tau =0}^t{\Delta }_{\tau }$. Then, for any arbitrary canvas size we can scale the line segments, compute the distance from every pixel on the canvas to each segment and assign a pixel intensity that is inverse to the shortest distance.

To compute the loss, we apply $f_{\text{raster}}$ and a Gaussian blurring filter $g_{\text{blur}}(\cdot )$ to both our prediction $y^{\prime }$ and ground truth $y$ then compute the binary cross-entropy loss. The Gaussian blur is used to reduce the strictness of our pixel-wise loss.

We find that $\alpha$ (in Equation 2) is an important hyperparameter that impacts both the learned embedding space and SketchEmbedNet. A curriculum training schedule is used, increasing $\alpha$ to prioritize $L_{\text{pixel}}$ relative to $L_{\text{stroke}}$ as training progresses; this makes intuitive sense as a single drawing can be produced by many stroke sequences but learning to draw in a fixed manner is easier. While $L_{\text{pen}}$ promotes reproducing a specific drawing sequence, $L_{\text{pixel}}$ only requires that the generated drawing visually matches the image. Like a human, the model should learn to follow one drawing style (à la paint-by-numbers) before learning to draw freely.

We consider the following baselines to compare with SketchEmbedNet.

• Contrastive is similar to the search embedding of Ribeiro et al. (2020); a metric learning baseline that matches CNN image embeddings with corresponding RNN sketch embeddings. Our baseline is trained using the InfoNCE loss (van den Oord et al., 2018).

• Conv-VAE (Kingma and Welling, 2014) performs pixel-level representation learning without motor program information.

• Pix2Pix (Isola et al., 2017) is a generative adversarial approach that performs image to sketch domain transfer but is supervised by sketch images and not the sequential motor program.

Note that Contrastive is an important comparison for SketchEmbedNet as it also uses the motor-program sequence when training on sketch-image pairs.

SketchEmbedNet is trained for 300k iterations with batch size of 256 for Quickdraw and 64 for Sketchy due to memory constraints. Initial learning rate is 1e-3 decaying by $0.85$ every 15k steps. We use the Adam (Kingma and Ba, 2015) optimizer and clip gradient values to $1.0$. Latent space $\text{dim}\left(z\right)=256$, RNN output size is $1024$, and hypernetwork embedding is 64. Mixture count is $M=30$ and Gaussian blur from $L_{\text{pixel}}$ uses $\sigma =2.0$.

Conv4 encoder is identical to  Vinyals et al. (2016) and the ResNet-12 encoder uses $4$ blocks of 64-128-256-512 filters with ReLU activations. $\alpha$ is set to $0$ and increases by $0.05$ every 10k training steps with an empirically obtained cap at ${\alpha }_{\text{max}}=0.50$ for Quickdraw and ${\alpha }_{\text{max}}=0.75$ for Sketchy. See Appendix B for additional details.

The results on Omniglot (Lake et al., 2015) using the split from Vinyals et al. (2016) are reported in Table 1. SketchEmbedNet obtains the highest classification accuracy when training on the Omniglot dataset. The Conv-VAE and as well as the Contrastive model are outperformed by existing unsupervised methods but not by a huge margin.¹¹1We do not include the Pix2Pix baseline here as the input and output images are the same. When training on the Quickdraw dataset SketchEmbedNet sees a substantial accuracy increase and exceeds the classification accuracy of the supervised MAML approach. While our model has arguably more supervision information than the unsupervised methods, our performance gains relative to the Contrastive baseline shows that this does not fully explain the results. Furthermore, our method transfers well from Quickdraw to Omniglot without ever seeing a single Omniglot character.

The results on mini-ImageNet (Vinyals et al., 2016) using the split from Ravi and Larochelle (2017) are reported in Table 2. SketchEmbedNet outperforms existing unsupervised few-shot classification approaches. We report results using both Conv4 and ResNet12 backbones; the latter allows more learning capacity for the drawing imitation task, and consistently achieves better performance. Unlike on the Omniglot benchmark, Contrastive and Conv-VAE perform poorly compared to existing methods, whereas SketchEmbedNet scales well to natural images and again outperforms other unsupervised few-shot learning methods, and even matches the performance of a supervised ProtoNet on 5-way 50-shot (71.99 vs. 72.04). This suggests that forcing the model to generate sketches yields more informative representations.

Effect of pixel-loss weighting. We ablate pixel loss coefficient ${\alpha }_{\text{max}}$ to quantify its impact on the observed representation, using the Omniglot task (Table 3). There is a substantial improvement in few-shot classification when ${\alpha }_{\text{max}}$ is non-zero. ${\alpha }_{\text{max}}$= 0.50 achieves the best results for Quickdraw, while it trends downwards when ${\alpha }_{\text{max}}$ approaches to 1.0. mini-ImageNet performs best at ${\alpha }_{\text{max}}=0.75$ Over-emphasizing the pixel-loss while using teacher forcing causes the model to create sketches by using many strokes, and does not generalize to true autoregressive generation.

The training data consists of 256 labelled examples for each of the 300 training classes. New example generalization is evaluated in 300-way classification on unseen examples of training classes. Novel class generalization is evaluated on 45-way classification of unseen Quickdraw classes. The results are presented in Table 4(a). SketchEmbedNet obtains the best classification performance. The Contrastive method also performs well, demonstrating the informativeness of sketch supervision. Note that while Contrastive performs well on training classes, it performs worse on unseen classes. The few-shot benchmarks in Tables 1, 2 suggest our generative objective is more suitable for novel class generalization. Unlike in the few-shot tasks, a Random CNN performs very poorly likely because the linear classification head lacks the capacity to discriminate the random embeddings.

Since there are not enough examples or classes to test unseen classes within Sketchy, we evaluate model generalization on 1000-way classification of ImageNet-1K (ILSVRC2012), and the validation accuracy is presented in Table 4(b). It is important to note that all the methods shown here only have access to a maximum of 125 Sketchy classes during training, resized down to 84$\times$84, with a max of 100 unique photos per class, and thus they are not directly comparable to current state-of-the-art methods trained on ImageNet. SketchEmbedNet once again obtains the best performance, not only relative to the image-based baselines, Random CNN, Conv-VAE and Pix2Pix, but also to the Contrastive learning model, which like SketchEmbedNet utilizes the sketch information during training. While Contrastive is competitive in Quickdraw classification, it does not maintain this performance on more difficult tasks with natural images, much like in the few-shot natural image setting. Unlike in Quickdraw classification where pretraining is effective, all 3 pixel-based methods perform similarly poorly.

To test component-level awareness, we construct image examples containing different arrangements of multiple objects in image space. We then embed these examples and project into 2D space using UMAP (McInnes et al., 2018) to visualize their organization. The leftmost panel of Figure 4 exhibits a numerosity relation with Quickdraw classes containing duplicated components; snowmen with circles and televisions with squares. The next two panels of Figure 4 contain examples with a placement and containment relation. SketchEmbedding representations are the most distinguishable and are easily separable. The pixel-based Conv-VAE is the least distinguishable, while the Contrastive model performs well in the containment case but poorly in the other two. As these image components are drawn contiguous through time and separated by lifted pen states, SketchEmbedNet learns to group the input pixels together as abstract elements to be drawn together.

We examine how the underlying variables of distance, angle or size are captured by the studied embedding functions. We construct and embed examples changing each of the variables of interest. The embeddings are again projected into 2D by the UMAP (McInnes et al., 2018) algorithm in Figure 5. After projection, SketchEmbedNet recovers the variable of interest as an approximately linear manifold in 2D space; the Contrastive embedding produces similar results, while the pixel-based Conv-VAE is more clustered and non-linear. This shows that relating images to sketch motor programs encourages the system to learn the spatial relationships between components, since it needs to produce the $\Delta x$ and $\Delta y$ values to satisfy the training objective.

Finally, we explore the use of SketchEmbeddings for composing embedded concepts. In natural language literature, vector algebra such as “king” - “man” + “woman” = “queen” (Mikolov et al., 2013) shows linear compositionality in the concept space of word embedding. It has also been demonstrated in human face images and vector graphics (Bojanowski et al., 2018; Shen et al., 2020; Carlier et al., 2020). Here we try to explore such concept compositionality property in sketch image understanding as well. We embed examples of simple shapes such as a square or circle as well as more complex examples like a snowman or mail envelope and perform arithmetic in the latent space. Surprisingly, upon decoding the SketchEmbedding vectors we recover intuitive sketch generations. For example, if we subtract the embedding of a circle from snowman and add a square, then the resultant vector gets decoded into an image of a stack of boxes. We present examples in Figure 6. By contrast, the Conv-VAE does not produce sensible decodings on this task.

Quantative assessment of generated images is often challenging and per-pixel metrics like in  Reed et al. (2018); Rezende et al. (2016) may penalize generative variation that still preserves meaning. We train ResNet classifiers for an Inception Score (Salimans et al., 2016) inspired metric. One classifier is trained on 45 (“seen”) Quickdraw training classes and the other on 45 held out (“unseen”) classes that were not encountered during model training. Samples generated by a sketching model are rendered, then classified; we report each classifier’s accuracy on these examples compared to its training accuracy in Table 5. SketchEmbedNet produces more recognizable sketches than a Conv-VAE model when generating examples of both seen and unseen object classes.

In addition to the Inception-score (Salimans et al., 2016) inspired metric, we also qualitatively assess the generations of SketchEmbedNet on unseen datasets. One-shot generations are sampled from Omniglot (Lake et al., 2015) and are visually compared with other few- and one-shot generation methods (Rezende et al., 2016; Reed et al., 2018) (Figure 8).

None of the models have seen any examples from the character class or parent alphabet. Furthermore, SketchEmbedNet was not trained on any Omniglot data. Visually, our generated images better resemble the support examples and have generative variance that better preserves class semantics. Generations in pixel space may disrupt strokes and alter the character to human perception. This is especially true for written characters as they are frequently defined by a specific set of strokes instead of blurry clusters of pixels.

While having a generative objective is useful for representation learning (we see that SketchEmbedNet outperform our Contrastive representations), it is insufficient to guarantee an informative embedding for other tasks. The Conv-VAE generations perform slightly worse on the recognizability task in Table 5, while being significantly worse in our previous classification tasks in Tables 1, 2 and 4.

This suggests that the output domain has an impact on the learned representation. The increased componential and spatial awareness from generating sketches (as in Section 4.4) makes SketchEmbeddings better for downstream classification tasks by better capturing the visual shape in images.