Self-Supervised Learning of Video Representations From a Child’s Perspective

We use vision transformers (ViTs) as our model architecture Dosovitskiy et al. (2020). This choice is effectively dictated by our choice of SSL algorithm, as described below. We use a large 633M parameter model (ViT-H/14) in all our experiments. We temporally subsample the videos at a rate of 3.75 frames/second and feed the model input clips consisting of 16 consecutive frames with a spatial resolution of 224$\times$224 pixels. Each modeled clip is thus roughly 4.3 seconds long. These clips are divided into 2$\times$14$\times$14 three-dimensional boxes or “patches” (i.e. 2 frames in the temporal dimension, 14$\times$14 pixels in the spatial dimensions). The patches are then linearly projected onto a common patch embedding space and separable (and learnable) spatial and temporal position embeddings are added to the patch embedding of each patch, helping to identify its spatial and temporal position. The rest of the architecture is a standard transformer model operating on the flattened patch representations. Since this is a standard architecture, we refer the reader to Feichtenhofer et al. (2022) for further details.

We use spatiotemporal masked autoencoders (MAEs) as our SSL algorithm of choice Feichtenhofer et al. (2022), which is a straightforward extension of the image-based MAEs He et al. (2022) to video data. The basic idea in spatiotemporal MAEs is to randomly mask out a large proportion (90%) of the three-dimensional “patches” described above and to predict these masked patches from high-level representations of the visible patches (Figure 1). An MAE consists of an encoder and a decoder, both vanilla transformer models. The encoder only processes the visible patches and its output is passed through the decoder, which is typically much smaller than the encoder, to predict the values of the masked patches in the pixel space. In addition to being a highly efficient, generic, state-of-the-art SSL algorithm for video representation learning, MAEs also have the advantage of requiring very minimal data augmentation (we only use random resized crops and horizontal flips in the spatial domain). This is relevant for our purposes, because SSL algorithms that require heavy data augmentation strategies make the input less “human-like”.

Once the MAE models are pretrained with the child headcam data (or with other reference data), we evaluate the quality of the learned video representations through a number of downstream tasks. As is standard with MAEs, we use supervised finetuning in order to evaluate the models in downstream tasks. However, to ensure that the representations are learned mostly through SSL (and not through supervised finetuning) and also in the interest of psychological plausibility, we adopt a few-shot finetuning setting, where we only use a small number of labeled examples to finetune the models.

For video-based evaluation, we consider two fine-grained action recognition tasks: Kinetics-700 and Something-Something-V2 (SSV2) Goyal et al. (2017). Kinetics-700 consists of short YouTube clips of 700 fine-grained action categories, whereas SSV2 contains short clips of people performing 174 different classes of fine-grained actions. The main difference between Kinetics-700 and SSV2 is that in SSV2, people perform instructed actions, where the objects involved in the action are specified and carefully varied across clips to abstract the action concept itself as a template. The videos in Kinetics-700, on the other hand, are “in the wild” clips downloaded from YouTube, so they are much less controlled with respect to potential shortcuts or biases in the data.

For both tasks, we consider 10-shot and 50-shot supervised finetuning scenarios, using 10 and 50 training examples per class, respectively, to finetune the entire model (but with layer-wise learning rate decay as a regularizer so that top layers are modified more than bottom layers). In Kinetics-700, the 10-shot setting uses 17 hours of video and the 50-shot setting uses 87 hours of video in total for finetuning. In SSV2, the 10-shot setting uses roughly 2 hours of video and the 50-shot setting uses roughly 9 hours of video in total for finetuning. Thus, in every case, the amount of supervised video data used for finetuning is significantly less than the amount of video data used for SSL during pretraining (cf. 194 hours of video from child S in SAYCam).

Figure 2a-b show the results for the SSV2 and Kinetics action recognition tasks, respectively, in both 10-shot and 50-shot supervised finetuning conditions. We observe a substantial pretraining benefit in all cases: e.g. compare the models pretrained on SAYCam or Kinetics-700 with reference models trained from scratch on the downstream task without any pretraining (Figure 2). Models pretrained with the full Kinetics-700 dataset outperform all other models, but video pretraining with SAYCam also seems to be remarkably effective for downstream action recognition tasks. Comparing the model pretrained on child S with the model pretrained on the length-matched Kinetics-200h dataset, in particular, we find that they perform comparably in all cases, despite qualitative differences between the spatiotemporal characteristics of these two datasets.

Surprisingly, the model pretrained on the whole SAYCam dataset (SAY) does not perform better than the model pretrained on child S only. We conjecture that the presence of a number of low-quality, underexposed clips in recordings from the other two children (A and Y) may have affected the quality of the representations learned by the spatiotemporal MAE algorithm when trained with these data. We leave a more complete investigation of this finding to future work.

To investigate the extent to which these action recognition tasks genuinely require temporal information, as opposed to being susceptible to simpler image-based strategies such as recognizing the typical objects or scenes in individual frames that tend to co-occur with the given action categories, we next created a purely image-based reference model that did not use any temporal information (S-Img). This model was pretrained on the headcam data from child S using an image-based MAE He et al. (2022) with 80% masking ratio and then finetuned on the downstream task, again in a purely image-based fashion (the model was evaluated by presenting the evaluation clips to the model frame by frame and averaging the predictions). This image-based reference model performed substantially worse than its video-based counterpart in SSV2 (cf. S vs. S-Img in Figure 2a), suggesting that this task genuinely requires learning the temporal regularities corresponding to different actions. However, the relative performance of S-Img was much better in the Kinetics-700 evaluation tasks (cf. S vs. S-Img in Figure 2b), suggesting that it may be possible to achieve high accuracies in this task by exploiting frame-based regularities and without using any temporal information at all. This result also illustrates the difficulty of creating datasets with “in the wild” internet data that are robust to shortcut learning strategies Geirhos et al. (2020).

We next isolated 10 action categories in SSV2 that could be considered “developmentally realistic” in the sense that a 30-month old child would be expected to understand them (the cutoff age for SAYCam is 31 months). We manually selected these categories from SSV2 as follows: we only considered simple, general action categories that involve at most two objects, thus excluding more detailed action categories describing a specific mode of action, for instance, or more complex categories that involve more than two objects. For each of the remaining categories, we then checked the “item trajectory” of the basal verb describing the action on Wordbank Frank et al. (2017) and included only those that are produced by most children by 30 months of age (i.e. “proportion of children producing” $>$ 0.5). The final list of these developmentally realistic action categories can be found in the legend of Figure 3.

Figure 3 shows t-SNE embeddings of these categories obtained from three different models: (i) a model pretrained on child S, but not finetuned on SSV2 at all (0-shot); (ii) a model pretrained on child S and finetuned on 10-shot SSV2; and (iii) a model pretrained on child S and finetuned on 50-shot SSV2. The 0-shot embeddings do not contain much category structure, attesting to the difficulty of this action recognition task. However, category-related semantic structure immediately begins to emerge as the model is finetuned with even a small amount of labeled examples (10-shot), and improves further with more labeled examples (50-shot).

Interestingly, we find that the model performs worse on this developmentally realistic subset of categories compared to the remaining categories: 56% vs. 64% top-5 validation accuracy in the 50-shot condition (chance: 3%). This could be because these are broader, more general categories compared to the rest. Indeed, among the categories with the highest accuracy in the 50-shot condition are: turning the camera left while filming something (97%), lifting a surface with something on it until it starts sliding down (90%), trying to pour something into something but missing so it spills next to it (88%), pouring something into something until it overflows (88%), pulling two ends of something so that it gets stretched (87%), twisting (wringing) something wet until water comes out (83%), and pretending to sprinkle air onto something (81%). These are typically more specific, more detailed action categories than the categories in our developmentally realistic subset.

Although SAYCam is currently the largest developmentally realistic, longitudinal video dataset of its kind, the amount of data available from each child in SAYCam is quite limited compared to the amount of visual experience a child actually receives while growing up. For example, we have 194 hours of video available from child S, which corresponds to roughly 8 days (or equivalent to 16 days of visual experience if we factor in 12 hours/day of sleep). This is two orders of magnitude smaller than the amount of visual experience a child receives by the age of 4-5. This invites the natural question: what would happen if we had developmentally more realistic amounts of data, e.g. two orders of magnitude more data?

To address this question, we performed a data size scaling experiment. We systematically varied the amount of video data from child S used for SSL pretraining: specifically, we trained spatiotemporal MAE models on all 194 hours of data (100%), on 19.4 hours of data (10%), on 1.94 hours of data (1%), and finally on 0.194 hours of data (0.1%) from S, thus covering a four orders of magnitude range in data size. We then evaluated these models on the downstream action recognition tasks and, using a simple log-linear scaling function, extrapolated their performance a few orders of magnitude beyond the amount of data we currently have from child S.

The results of this scaling experiment are shown in Figure 4. We estimate substantial improvements in action recognition (on average, over $30\%$ absolute improvement in accuracy in the SSV2 50-shot task and $\sim 20\%$ improvement in the Kinetics-700 50-shot task) with a two orders of magnitude increase in data size alone without any other changes. Further improvements may be possible by scaling up the input and model sizes as well Orhan (2023).

We next devised a simple qualitative test to probe the models’ understanding of the spatiotemporal dynamics over short video clips. In this test, we mask out the 4 central frames in the middle of a clip consisting of 16 consecutive frames. We then ask the model to “fill in” this central part, given the rest of the clip. To make sure that the behavior we observe is due to the out-of-distribution (ood) generalization capacity of the model and not simply due to memorization or iid generalization, we use the model trained on child S and evaluate it with clips from the other two children in SAYCam (A and Y). Note that this task is doubly challenging for the model: (i) the model was not trained with this particular masking strategy during pretraining (it was instead trained with random spatiotemporal masking as in Figure 1) and (ii) the model did not see any of these clips during training.

Despite these challenges, the model was often able to provide plausible completions of the clips. Visual inspection of the model’s completions suggests that in many cases the model does not just utilize simplistic strategies such as copying the nearest visible frames, but rather generates plausible, novel completions that indicate some understanding of the movement of objects and other visual features across the image over the course of a clip. Figure 5 presents two qualitative examples illustrating this point.

Finally, we sought to compare the object representations that emerge in video models trained with the spatiotemporal MAE algorithm with those learned by image-based MAEs trained on the same data. We conjectured that since spatiotemporal MAEs learn to track visual representations over time, this could lead to more accurate or more robust object representations. To test this hypothesis, we evaluated both spatiotemporal MAEs and image-based MAEs trained with the headcam data from child S on two downstream visual object recognition tasks: ImageNet Russakovsky et al. (2015) and out-of-distribution (OOD) ImageNet Geirhos et al. (2021). ImageNet is a standard benchmark for real-world visual object recognition and consists of a large collection of high-quality, photographic images belonging to 1000 different object categories. OOD ImageNet is an out-of-distribution version of ImageNet, which consists of 17 different evaluation tasks each generated by applying a different type of perturbation or transformation to images from the ImageNet validation set (e.g. changing the colors of the image, taking the silhouettes of the objects, stylizing the image, adding different types of noise to the image). Together, these two benchmarks evaluate the accuracy and robustness of real-word object recognition.

We again adopt a few-shot supervised finetuning setup for our evaluations. Specifically, we finetune both video-based and image-based MAEs with 2% of the training data from ImageNet, which corresponds to 25-26 images per class. To finetune the video model on image data, we simply repeat each image 16 times before feeding it to the model, thus effectively creating a static video clip with 16 frames. This allows us to use the pretrained video model without any alterations in the architecture.

The results are shown in Figure 6a-b. On ImageNet, the model pretrained with video-based SSL on the headcam data from child S outperforms the model pretrained with image-based SSL on the same data (S-Vid: 69% vs. S-Img: 57%). On OOD ImageNet, the video-based model again outperforms the image-based model (S-Vid: 34% vs. S-Img: 30%), suggesting that the object representations learned by the video model are not only more accurate for fine-grained recognition, but also more robust to perturbations and transformations than those learned by the image-based model. The performance difference between the video model and the image-based model was particularly salient on the stylized (+12% in favor of the video model), sketch (+9%), and silhouette (+4%) subtasks in OOD ImageNet, suggesting a qualitatively different type of representation that may be more shape-sensitive and less texture-sensitive than in the image-based model. The respective attention maps from the two models seem to confirm this suggestion (Figure 6c). We leave a more complete investigation of the possible reasons behind the superior object representations that seem to emerge with video-based SSL to future work (e.g. one possible factor is the difference in masking ratio per frame between image-based vs. video-based SSL: 80% vs. 90%).

A recent work also reported more robust and more human-aligned object representations with video-based pretraining compared to image-based pretraining Parthasarathy et al. (2023). However, that work did not use the same datasets for video-based and image-based pretraining. In contrast, by using the exact same data (headcam videos from child S) and the same model architecture (ViT-H/14) for both video-based and image-based pretraining, we can isolate the effect of video-based vs. image-based pretraining objectives on the quality of the learned object representations.