Jekyll2022-06-03T13:39:45+00:00https://yassouali.github.io//ml-blog/atom.xmlYassineA(nother) Machine Learning BlogYassineouali.yasine@gmail.comCVPR 2021: An Overview2021-07-02T00:00:00+00:002021-07-02T00:00:00+00:00https://yassouali.github.io//ml-blog/cvpr2021<p>The 2021 CVPR conference, one of the main computer vision and machine learning conferences, concluded its second 100% virtual version
last week with a record of papers presented at the main conference. Of about 7500 submissions, 5900 made it to the decision making process
and 1660 papers (vs 1467 papers last year) were accepted with an acceptance rate of 23.7% (vs 22.1% last year).
Such a huge (and growing) number of papers can be a bit overwhelming, so to get a feel of the general trends at the conference this year, I will present in this blog post a quick look of the conference by summarizing some papers (& listing some) that seemed interesting to me.</p>
<p>First, let’s start with some useful links:</p>
<ul>
<li>Papers: <a href="https://openaccess.thecvf.com/CVPR2021?day=all">CVPR2021 open access</a></li>
<li>Workshops: <a href="http://cvpr2021.thecvf.com/workshops-schedule">CVPR2021 workshops</a></li>
<li>Tutorials: <a href="http://cvpr2021.thecvf.com/program">CVPR2021 tutorials</a></li>
<li>Presentations: <a href="https://crossminds.ai/search/?keyword=CVPR%202021&filter=">Crossminds</a></li>
<li>Papers search interface: <a href="https://blog.kitware.com/demos/cvpr-2021-papers/?filter=authors&search=">blog.kitware.com</a> & <a href="https://public.tableau.com/views/CVPR2021/Dashboard1?:language=en-US&:display_count=n&:origin=viz_share_link:showVizHome=no">public.tableau.com</a></li>
<li>Awards: <a href="http://cvpr2021.thecvf.com/node/329">CVPR2021 paper awards</a></li>
<li>Papers digest: <a href="https://www.paperdigest.org/2021/06/cvpr-2021-highlights/">CVPR2021 Paper Digest</a></li>
<li>Papers & code: <a href="https://github.com/amusi/CVPR2021-Papers-with-Code">CVPR2021 paper & code</a></li>
</ul>
<p><em>Note: This post is not an objective representation of the papers and subjects presented in CVPR 2021, it is just a personnel overview of what I found interesting. Any feedback is welcomed!</em></p>
<h2 id="table-of-contents">Table of Contents</h2>
<ul>
<li><a href="#cvpr-2021-in-numbers">CVPR 2021 in numbers</a></li>
<li><a href="#recognition-detection--tracking">Recognition, Detection & Tracking</a></li>
<li><a href="#model-architectures--learning-methods">Model Architectures & Learning Methods</a></li>
<li><a href="#3d-computer-vision">3D Computer Vision</a></li>
<li><a href="#image-and-video-synthesis">Image and Video Synthesis</a></li>
<li><a href="#scene-analysis--understanding">Scene Analysis & Understanding</a></li>
<li><a href="#representation--adversarial-learning">Representation & Adversarial Learning</a></li>
<li><a href="#transfer-low-shot-semi--unsupervised-learning">Transfer, Low-shot, Semi & Unsupervised Learning</a></li>
<li><a href="#computational-photography">Computational Photography</a></li>
<li><a href="#other">Other Subjects</a></li>
</ul>
<h1 id="cvpr-2021-in-numbers">CVPR 2021 in numbers</h1>
<p>A portion of the statistics presented in this section are taken from <a href="https://github.com/hoya012/CVPR-2021-Paper-Statistics">this</a> github repo & this <a href="https://public.tableau.com/views/CVPR2021/Dashboard1?:language=en-US&:display_count=n&:origin=viz_share_link:showVizHome=no">public tableau gallery</a>.</p>
<figure style="width: 60%" class="align-center">
<img src="https://yassouali.github.io//ml-blog/images/CVPR21/acceptance_rate.png" />
<figcaption>
Image source: <a href="https://github.com/hoya012/CVPR-2021-Paper-Statistics">github</a>
</figcaption>
</figure>
<p>The trends of earlier years continued with a notable increase in authors and number of submitted papers, joined by a rising the number of reviewers and area chairs to accommodate this expansion.</p>
<figure style="width: 60%" class="align-center">
<img src="https://yassouali.github.io//ml-blog/images/CVPR21/authors_by_country.png" />
</figure>
<p>Similar to the last two years, China is the first contributor to CVPR in terms of accepted papers, followed by the USA, Korea, UK and Germany.</p>
<figure style="width: 90%" class="align-center">
<img src="https://yassouali.github.io//ml-blog/images/CVPR21/subjects.png" />
<figcaption>
Image source: <a href="https://public.tableau.com/views/CVPR2021/Dashboard1?:language=en-US&:display_count=n&:origin=viz_share_link:showVizHome=nos">public tableau gallery</a>
</figcaption>
</figure>
<p>As expected, the majority of the accepted papers focus on topics related to learning, recognition, detection, and understanding. However, the topic of the year is 3D computer vision with more than 200 papers focusing on this subject alone, followed by deep & representation
learning, image synthesis, and computation photography. There is also a notable increase in papers related to explainable AI and medical & biological imaging.</p>
<h1 id="recognition-detection--tracking">Recognition, Detection & Tracking</h1>
<h4 id="task-programming-learning-data-efficient-behavior-representations-paper">Task Programming: Learning Data Efficient Behavior Representations (<a href="https://arxiv.org/abs/2011.13917">paper</a>)</h4>
<figure style="width: 100%" class="align-center">
<img src="https://yassouali.github.io//ml-blog/images/CVPR21/1.png" />
<figcaption>Image source: <a href="https://arxiv.org/abs/2011.13917">Sun et al.</a></figcaption>
</figure>
<p>In behavioral analysis, the location and pose of agents is first extracted from each frame of a behavior video, and then the labels are generated for a given set of behaviors of interest on a frame-by-frame basis based on the pose and movements of the agents as depicted in the figure above.
However, to predict the behaviors frame-by-frame in the second step, we need to train behavior detection models which are data intensive and require specialized domain knowledge and high-frequency temporal annotations. This paper studies two alternative ways to better use domain experts instead of a simple increase in the number of annotations: (1) self-supervised learning and (2) creating task programs by domain experts which are engineered decoder tasks for representation learning.</p>
<figure style="width: 100%" class="align-center">
<img src="https://yassouali.github.io//ml-blog/images/CVPR21/2.png" />
<figcaption>Image source: <a href="https://arxiv.org/abs/2011.13917">Sun et al.</a></figcaption>
</figure>
<p>Trajectory Embedding for Behavior Analysis (TREBA) uses these two ideas to learn task-relevant low-dimensional representations of pose trajectories, this is done
by training a network to jointly reconstruct the input trajectory and predict the expert-programmed decoder tasks (see figure above). For the first task, given a set
of unlabelled trajectories, where each trajectory is a sequence of states (eg, location or pose of the agents), the history of the agent stages is
encoded using an RNN encoder, and the RNN the decoder (ie, a <a href="https://arxiv.org/abs/1806.02813">trajectory variational autoencoder or TVAE</a>) then predicts the next states.
As for the second task, we first need to create decoder tasks for trajectory self-supervised learning by domain experts (which is called the process of Task Programing).
First, the experts find the attributes from the trajectory data that are useful to detect the agents behaviors of interest, then write a program
to compute these attributes (eg, distance or angle between two interacting mice) based on the trajectory data using systems like <a href="https://www.biorxiv.org/content/10.1101/2020.07.26.222299v1">MARS</a> or <a href="https://www.biorxiv.org/content/10.1101/2020.04.19.049452v2">SimBA</a>. These programs are finally used to generate training
data for self-supervised multi-task learning.</p>
<h4 id="permute-quantize-and-fine-tune-efficient-compression-of-neural-networks-paper">Permute, Quantize, and Fine-tune: Efficient Compression of Neural Networks (<a href="https://arxiv.org/abs/2010.15703">paper</a>)</h4>
<p>One of the main challenges of deploying deep nets on mobile and low-power computational platforms for large scale usage is their large memory and computational requirements.
Luckily, such deep nets are often overparameterized, living some room for compression to reduce their memory and computational demands wit a minimal accuracy hit.
One way to compress deep nets is scalar quantization, which compresses each parameter individually, but the compression rates are still limited by the number of parameters.
Another approach is vector quantization which compresses multiple parameters into a single code, thus exploiting redundancies among groups of network parameters, but
finding the parameters to group can be challenging (eg, in fully connected layers for example, there is no notion of spatial dimensions to group the parameters by).</p>
<figure style="width: 100%" class="align-center">
<img src="https://yassouali.github.io//ml-blog/images/CVPR21/3.png" />
<figcaption>Image source: <a href="https://arxiv.org/abs/2010.15703">Martinez et al.</a></figcaption>
</figure>
<p>The paper proposes Permute, Quantize, and Fine-tune (PQF), a method that (1) searches for permutations of the network weights that yield functionally equivalent, yet an easier-to-quantize network, (2) quantizes the permuted weights, and finally, (3) fine tunes the permuted-and-quantized network to recover the accuracy of the original uncompressed network. Since the second step consists splitting each weight matrix \(W\) into subvectors (say \(d\) elements per subvector) which are then compressed individually, where each subvector is approximated by a smaller code or centroid, the objective of the first step is to find a permutation \(P\) of the weight matrix such that the constructed subvectors are easier to quantize into codes (see figure above). The optimization of \(P\) is done by minimizing the determinant of the covariance of the resulting subvectors (see the paper for why they do this). Finally, a fine tuning of the compressed network is conducted to remove the accumulated errors in the activations after the quantization which degrade the performances.</p>
<h4 id="towards-open-world-object-detection-paper">Towards Open World Object Detection (<a href="https://arxiv.org/abs/2103.02603">paper</a>)</h4>
<p>In this work, the authors propose a novel computer vision problem called <em>Open World Object Detection</em>, where a model is trained to: 1) identify objects that have not been introduced to it as <em>unknown</em> without explicit supervision to do so (ie, open set learning), and 2) incrementally learn these identified unknown categories without forgetting previously learned classes when the corresponding labels are progressively received (ie, incremental and continual learning). In <a href="https://ieeexplore.ieee.org/document/6365193">open set</a> recognition, the objective is to identify the new instances not seen during training as unknowns, while <a href="https://arxiv.org/abs/1412.5687">open world</a> recognition extends this framework by requiring the classifier to recognize the newly identified unknown classes.
However, adapting the open set and open world methods from recognition to detection is not trivial since the object detector is explicitly trained to detect the unknown classes as background, making the task of detecting unknown classes harder.</p>
<figure style="width: 100%" class="align-center">
<img src="https://yassouali.github.io//ml-blog/images/CVPR21/4.png" />
<figcaption>Image source: <a href="https://arxiv.org/abs/2103.02603">Joseph et al.</a></figcaption>
</figure>
<p>To solve this problem, the paper proposes ORE, Open World Object Detector, that learns a clear discrimination between classes in the latent space. This way, (1) the task of detecting an unknown instance as a novelty can be reduced to comparing its representation with the representations of the known instances (ie, open set detection), and also (2), it facilitates learning feature representations for the new class instances without overlapping with the previous classes (ie, incremental and continual learning, thus extending open set to open world detection). To have such a learning behavior, a contrastive clustering objective is introduced in order to force instances of same class to remain close-by, while instances of dissimilar classes would be pushed far apart. This is done by minimizing the distances between the class prototypes and the class representations, where the instances of class unknown are instances with high objectness score, but do not overlap with a ground-truth objects.
The classification head of the trained detector is then transformed into an <a href="http://yann.lecun.com/exdb/publis/pdf/lecun-06.pdf">energy function</a> to label an instance as unknown or not.</p>
<h4 id="learning-calibrated-medical-image-segmentation-via-multi-rater-agreement-modeling-paper">Learning Calibrated Medical Image Segmentation via Multi-Rater Agreement Modeling (<a href="https://openaccess.thecvf.com/content/CVPR2021/html/Ji_Learning_Calibrated_Medical_Image_Segmentation_via_Multi-Rater_Agreement_Modeling_CVPR_2021_paper.html">paper</a>)</h4>
<p>For a standard vision task, it is a common practice to adopt the ground-truth labels obtained via either the majority-vote or by simply one annotation from a preferred rater
as the single source of the training data. However, in medical images, the typical practice consists of collecting multiple annotations, each from a different clinical expert or rater, in the expectation that possible diagnostic errors could be mitigated. In this case, using standard training procedure of other vision tasks will overlook the rich information of agreement or disagreement ingrained in the raw multi-rater annotations available in medical image analysis.
To take this into consideration, the paper proposes MR-Net to explicitly model the multi-rater agreement or disagreement.</p>
<figure style="width: 100%" class="align-center">
<img src="https://yassouali.github.io//ml-blog/images/CVPR21/5.png" />
<figcaption>Image source: <a href="https://openaccess.thecvf.com/content/CVPR2021/html/Ji_Learning_Calibrated_Medical_Image_Segmentation_via_Multi-Rater_Agreement_Modeling_CVPR_2021_paper.html">Ji et al.</a></figcaption>
</figure>
<p>MRNet contains a coarse to fine two-stage processing pipeline (figure above, right). The first stage consists of a U-Net encoder with a ResNet34 backbone pretrained on ImageNet. Together with an Expertise-aware Inferring Module (EIM), inserted at the bottleneck layer to embed the expertise information of individual raters, named expertness vector, into the extracted high-level semantic features of the network. The outputs are then passed to the U-Net decoder to give a coarse prediction. The second stage then refines the coarse predictions using two modules. The first one is Multi-rater Reconstruction Module (MRM) that reconstructs the raw multi-rater’s grading, the reconstructions are then used to estimate the pixel-wise uncertainty map that represents the inter-observer variability across different regions. Finally, a Multi-rater Perception Module (MPM) with a soft attention mechanism utilizes the produced uncertainty map to refine the coarse predictions of the first stage and predict the final fine segmentation maps (see section 3 of the paper for details about each component).</p>
<h4 id="re-labeling-imagenet-from-single-to-multi-labels-from-global-to-localized-paper">Re-Labeling ImageNet: From Single to Multi-Labels, From Global to Localized (<a href="https://arxiv.org/abs/2101.05022">paper</a>)</h4>
<p>One of the flaws of ImageNet is the presence of a significant level of label noise, where many instances contain multiple classes while having a single-label ground-truth. This mismatch between the single label annotations of ImageNet and the multi-label nature of its images becomes even more problematic with random-crop training, where the input may contain an entirely different class than the ground-truth.
To aviate this problem, the paper proposes to generate the multi-labels using a strong image classifier trained on an extra source of data,
in addition to leveraging pixel-wise multi-label predictions before the final pooling layer as a complementary location-specific supervision signal.</p>
<figure style="width: 100%" class="align-center">
<img src="https://yassouali.github.io//ml-blog/images/CVPR21/6.png" />
<figcaption>Image source: <a href="https://arxiv.org/abs/2101.05022">Yun et al.</a></figcaption>
</figure>
<p>Re-labeling ImageNet consists of using a machine annotator, which is a classifier trained on a super-ImageNet scale datasets (like <a href="https://arxiv.org/abs/1707.02968">JFT-300M</a> and <a href="https://arxiv.org/abs/1805.00932">InstagramNet-1B</a>) and fine-tunned on ImageNet. While being trained on single-label classification, such classifiers are still
capable of multi-label predictions for images with multiple categories, making them suitable for relabeling ImageNet. In addition to generating the multi-label classes, Re-labeling also leverages the location-specific labels, which are the spatial features before the global pooling layer weighted by the weights of the last fully-connected layers. The model is then trained on both the original ImageNet labels and the generated location-specific labels. For the second part, the paper proposes LabelPooling which conducts a regional pooling (<a href="https://arxiv.org/abs/1703.06870">RoI Align</a>) operation on the label map corresponding to the coordinates of the random crop
(see figure above, right), which are passed to the machine annotator (pretrained then finetuned classifier) to generate the multi-class labels used for the multi-label loss.</p>
<h4 id="other-papers-to-check-out">Other papers to check out</h4>
<ul>
<li><a href="https://arxiv.org/abs/2103.09460">You Only Look One-Level Feature</a>.</li>
<li><a href="https://arxiv.org/abs/2103.16483">Benchmarking Representation Learning for Natural World Image Collections</a>.</li>
<li><a href="https://arxiv.org/abs/2101.01909">Line Segment Detection Using Transformers Without Edges</a>.</li>
<li><a href="https://arxiv.org/abs/2103.11511">MoViNets: Mobile Video Networks for Efficient Video Recognition</a>.</li>
<li><a href="https://arxiv.org/abs/2103.11624">Multimodal Motion Prediction With Stacked Transformer</a>s.</li>
<li><a href="https://arxiv.org/abs/2105.11595">SiamMOT: Siamese Multi-Object Tracking</a>.</li>
</ul>
<h1 id="model-architectures--learning-methods">Model Architectures & Learning Methods</h1>
<h4 id="pre-trained-image-processing-transformer-paper">Pre-Trained Image Processing Transformer (<a href="https://arxiv.org/abs/2012.00364">paper</a>)</h4>
<p>This paper presents Image Processing Transformer (IPT), a transformer model for low-level computer vision tasks (mainly denoising, super-resolution and deraining).
IPT has 4 main blocks. First, multiple heads for extracting features from the inputs, which are the corrupted images such as images with noise and low-resolution images. Each head is a small convolutional net (3 layers) that outputs \(C\) dimensional feature maps of the same spatial dimensions as the input. The standard encoder-decoder transformer blocks for feature refinement and information recovery, with the introduction of task embedding as input seeds into the decoder. Finally, multiple tails that map the transformer’s output features back into the image space.</p>
<figure style="width: 100%" class="align-center">
<img src="https://yassouali.github.io//ml-blog/images/CVPR21/32.png" />
<figcaption>Image source: <a href="https://arxiv.org/abs/2012.00364">Chen et al.</a></figcaption>
</figure>
<p>For Pre-training, IPT is trained on corrupted ImageNet inputs. For a given image, first, a corruption function is applied (such as bicubic degradation to generate low-resolution images for super-resolution, or adding Gaussian noise for denoising), the model is then train with an L1 loss between the reconstructed image from the output of IPT and the clean image. Additionally, in order to make IPT applicable to other low-level tasks outside of the introduced corruptions, they also train with a contrastive loss where the model is trained to maximize the similarity between the features of the patches coming from the same input image. After pre-training, the model can then be fine-tuned on the low level task of choice.</p>
<h4 id="repvgg-making-vgg-style-convnets-great-again-paper">RepVGG: Making VGG-Style ConvNets Great Again (<a href="https://arxiv.org/abs/2101.03697">paper</a>)</h4>
<p>The popular CNN architectures, while delivering good results, still have some drawbacks:
1) The complicated multi-branch designs, such as residual connections in ResNet and branch-concatenation in Inception, making the models more difficult to implement/customize, and slowing down the inference and reduce the memory utilization. 2) Some components such as depthwise convs in MobileNets and channel shuffle in ShuffleNets increase the memory access cost and lack support in various devices.
This paper takes a step back in time, and proposes RepVGG, a network with VGG-like design, where at inference-time, the network is composed of nothing but a stack of 3x3 convolutions and ReLUs.</p>
<figure style="width: 100%" class="align-center">
<img src="https://yassouali.github.io//ml-blog/images/CVPR21/33.png" />
<figcaption>Image source: <a href="https://arxiv.org/abs/2101.03697">Ding et al.</a></figcaption>
</figure>
<p>While plain CNNs have many strengths, they possess one fatal weakness: poor performance :). So in the design of RepVGG, there needs to be a multi-branch architecture introduced in training time, and at inference time, RepVGG can then fall back in the plain one branch architecture for efficiency. In order to do this, the design of RepVGG is based on what the paper calls <em>Structural Re-param</em>, which describes how to convert a trained multi-branch block into a single 3x3 conv layer for inference.
For a single block with three branches, a 3x3 conv, a 1x1 conv and identity (can be viewed as 1x1 conv with an identity kernel), each one followed by a batch norm, where the output of the block is the sum of the outputs of the three branches. At inference time, RepVGG first converts each conv & the following batch norm of each branch into a single conv with a bias vector. Then, the remaining 3 convs are combined into a single 3x3 conv layer by adding up the biases and adding up the kernels (where the center of 3x3 conv gets the single value in the 1x1 kernel).</p>
<figure style="width: 50%" class="align-center">
<img src="https://yassouali.github.io//ml-blog/images/CVPR21/34.png" />
<figcaption>Image source: <a href="https://arxiv.org/abs/2101.03697">Ding et al.</a></figcaption>
</figure>
<h4 id="bottleneck-transformers-for-visual-recognition-paper">Bottleneck Transformers for Visual Recognition (<a href="https://arxiv.org/abs/2101.11605">paper</a>)</h4>
<figure style="width: 50%" class="align-center">
<img src="https://yassouali.github.io//ml-blog/images/CVPR21/36.png" />
<figcaption>Image source: <a href="https://arxiv.org/abs/2101.11605">Srinivas et al.</a></figcaption>
</figure>
<p>BoTNet, or Bottleneck Transformer, consists of a simple adjustment of the ResNet architecture to incorporate self-attention.
This is done by just replacing the spatial convolutions with global self-attention in the final three bottleneck blocks of a ResNet and no other changes. With such an adjudgment, the resulting BoTNet improves upon the ResNet baselines significantly on instance segmentation and object detection while also reducing the parameters, with minimal overhead in latency.</p>
<figure style="width: 100%" class="align-center">
<img src="https://yassouali.github.io//ml-blog/images/CVPR21/35.png" />
<figcaption>Image source: <a href="https://arxiv.org/abs/2101.11605">Srinivas et al.</a></figcaption>
</figure>
<h4 id="scaling-local-self-attention-for-parameter-efficient-visual-backbones-paper">Scaling Local Self-Attention for Parameter Efficient Visual Backbones (<a href="https://arxiv.org/abs/2103.12731">paper</a>)</h4>
<p>While self-attention models have recently been shown to provide notable improvements on accuracy-parameter trade-offs compared to baseline convolutional models such as ResNet-50, they are still not on par with the high-performing convolutional models
such as <a href="https://arxiv.org/abs/1905.11946">EfficientNet</a> (<a href="https://arxiv.org/pdf/2104.00298.pdf">V2</a>).
In order to close this gap, this paper proposes a new self-attention model family called HaloNets, which is based
on a more efficient implementation of self-attention, that improves the speed, memory usage, and accuracy of these models.</p>
<figure style="width: 100%" class="align-center">
<img src="https://yassouali.github.io//ml-blog/images/CVPR21/37.png" />
<figcaption>Image source: <a href="https://arxiv.org/abs/2103.12731">Vaswani et al.</a></figcaption>
</figure>
<p>Global self-attention, in which all locations attend to each other, is too expensive for most image scales due to the quadratic computation cost with respect to \(k\) (the kernel size). Thus, self-attention based models like <a href="https://arxiv.org/abs/1906.05909">SASA</a>
use a local form of self-attention that aggregates the information around each pixel similar to convolutions. However, to do this,
we need to extract local 2D grids around each pixel, and such an operation can be quite expensive both computationally and memory wise, since for each pixel, we need to fetch \(k^2\) pixels, and this operation contains a lot of duplicates since two neighboring pixel share
most of their neighbors (\(k \times (k-1)\) out of \(k^2\)). To solve this, HaloNets use blocked local self-attention (see figure above), where the local neighborhood for a block of pixels is extracted once together, instead of extracting separate neighborhoods per pixel. This operation consist of first dividing the input tensor into non-overlapping blocks, where each block behaves as a group of query pixels, then a shared neighborhood is constructed around each block, which is used to compute the keys and values and finally the outputs. This way, we only compute one neighborhood per block instead of one neighborhood per pixel (see section 2.2 for more details). After defining such an operation, HaloNets is designed based on a similar architecture to ResNets with blocked local self-attention instead of convolutions.</p>
<h4 id="involution-inverting-the-inherence-of-convolution-for-visual-recognition-paper">Involution: Inverting the Inherence of Convolution for Visual Recognition (<a href="https://arxiv.org/abs/2103.06255">paper</a>)</h4>
<p>Convolution kernels have two main properties, spatial-agnostic, where the same kernel is applied to all of the spatial location of the input volume, and channel-specific, where many kernels is applied over the same input, resulting in an output with multiple channels in order to collect diverse information. These two properties result in an enhanced efficiency and makes the convolution operation translation equivalence. However, with such a design comes some limitations, such as a constrained receptive field where a single conv operation can’t capture long range spatial interactions and a possible inter-channel redundancy inside the convolution filters.</p>
<figure style="width: 100%" class="align-center">
<img src="https://yassouali.github.io//ml-blog/images/CVPR21/38.png" />
<figcaption>Image source: <a href="https://arxiv.org/abs/2103.06255">Li et al.</a></figcaption>
</figure>
<p>This paper takes a contrarian approach, and proposes an operation called <em>involution</em> that switches the two properties of convolutions,
resulting in a spatial-specific and channel-agnostic operation. To have a spatial-specific operation,
the involution kernel belonging to a specific spatial location is to be generated solely conditioned on the incoming feature vector at the corresponding location itself. This is implemented in two steps, first a 1x1 conv, where the output channels correspond to the size of the kernel (\(C = k^2\)) is first applied, generating spatial location specific kernel weights (each spatial location will have its own specific conv kernel). The output is then reshaped from one vector per spatial location into a spatial kernel, and then applied over the input volume followed by an aggregation operation (average pooling) over \(k \times k\) volume per spatial location.
As for channel-agnostic, it can be obtained by simply sharing the convolution over all channels. This operation is then used to design RedNet, a ResNet style model with involutions.</p>
<h4 id="on-feature-normalization-and-data-augmentation-paper">On Feature Normalization and Data Augmentation (<a href="https://arxiv.org/abs/2002.11102">paper</a>)</h4>
<p>The usage of normalization techniques such as bach norm in recognition models have become a standard practice, where the moments (ie, mean and standard deviation) of latent features are often removed when training image recognition models, which helps increase their stability and reduce the training time. However, such moments can play a much more central role in some vision tasks like image generation, where they capture style and shape information of an image and can be instrumental in the generation process. In this context, this paper proposes <em>Moment Exchange</em> or MoEx, an implicit data augmentation method that encourages the recognition model to utilize the moment information for better performances and better robustness.</p>
<figure style="width: 80%" class="align-center">
<img src="https://yassouali.github.io//ml-blog/images/CVPR21/39.png" />
<figcaption>Image source: <a href="https://arxiv.org/abs/2002.11102">Li et al.</a></figcaption>
</figure>
<p>MoEx is an operation applied in the feature space in order to systematically regulate how much attention a network pays to the signal in the feature moments. As illustrated above, given an input, the mean and variance across channels are first extracted after the first layer. Then, instead of removing them, they are swapped with the moments of an other image that are extracted in the same manner. This results in a set of features that contain information about both images, and then the model is then trained to predict an interpolation of the labels of the two inputs. This way, the model is pushed to focus on two different signals for classification,
the normalized feature of the first image and the moments of the second.</p>
<ul>
<li><a href="https://arxiv.org/abs/1811.10725">MIST: Multiple Instance Spatial Transformer</a></li>
<li><a href="https://arxiv.org/abs/2012.07177">Simple Copy-Paste Is a Strong Data Augmentation Method for Instance Segmentation</a></li>
<li><a href="https://arxiv.org/abs/2104.14107">Decoupled Dynamic Filter Networks</a></li>
<li><a href="https://arxiv.org/abs/2104.11487">Skip-Convolutions for Efficient Video Processing</a></li>
<li><a href="https://arxiv.org/abs/2104.09052">Metadata Normalization</a></li>
</ul>
<h1 id="3d-computer-vision">3D Computer Vision</h1>
<h4 id="mp3-a-unified-model-to-map-perceive-predict-and-plan-paper">MP3: A Unified Model to Map, Perceive, Predict and Plan (<a href="https://arxiv.org/abs/2101.06806">paper</a>)</h4>
<p>Most modern self-driving stacks require up-to-date high-definition maps that contain rich semantic information necessary for driving, such as the topology and location of the lanes, crosswalks, traffic lights, intersections as well as the traffic rules for each lane. While such maps greatly facilitate the perception and motion forecasting tasks, as the online inference process has to mainly focus on dynamic objects (eg, vehicles, pedestrians, cyclists), scaling them is hard given their complexity and cost, and given that even very small errors in the mapping might result in fatal mistakes. This motivates the development of mapless technology, which can serve as a fail-safe alternative in the case of localization failures or outdated maps, and potentially unlock self-driving at scale at a much lower cost.</p>
<figure style="width: 60%" class="align-center">
<img src="https://yassouali.github.io//ml-blog/images/CVPR21/10.png" />
<figcaption>Image source: <a href="https://arxiv.org/abs/2101.06806">Casas et al.</a></figcaption>
</figure>
<p>However, with a mapless approach comes a number of challenges: (1) The sole source of training signal is the controls of an expert driver (such steering and acceleration), without providing intermediate interpretable representations that can help explain the self-driving vehicle decisions. (2) Without any mechanism to inject structure and prior knowledge, such an approach can be very brittle to distributional shift such missing a lane. To address these issues, the paper presents MP3, an end-to-end approach to mapless driving that is interpretable, does not incur any information loss, and reasons about uncertainty in the intermediate representations.</p>
<figure style="width: 100%" class="align-center">
<img src="https://yassouali.github.io//ml-blog/images/CVPR21/11.png" />
<figcaption>Image source: <a href="https://arxiv.org/abs/2101.06806">Casas et al.</a></figcaption>
</figure>
<p>The MP3 model takes as input a high-level goal, a history of LiDAR point clouds to extract rich geometric and semantic features from the scene over time (see <a href="https://arxiv.org/abs/2012.12395">this</a> for more details), and odometry data to compensate for the vehicle’s motion.
The inputs are then processed using a backbone network and fed into a set of probabilistic spatial layers to model the static and dynamic parts of the environment.
The static environment is represented by a planning-centric online map which captures information about which areas are drivable and which ones are reachable given the traffic rules. The dynamic actors are captured in a novel occupancy flow that provides occupancy and velocity estimates over time. The motion planning module then leverages these representations
to retrieve dynamically feasible trajectories, predicts a spatial mask over the map to estimate the route given an abstract goal, and leverages the online map and occupancy flow directly as cost functions for explainable, safe plans.</p>
<h4 id="multi-modal-fusion-transformer-for-end-to-end-autonomous-driving-paper">Multi-Modal Fusion Transformer for End-to-End Autonomous Driving (<a href="https://arxiv.org/abs/2104.09224">paper</a>)</h4>
<p>In a standard driving situation such as the one depicted in the left image bellow, the vehicle must capture the global context of the scene involving the interaction between the traffic light (yellow) and the vehicles (red) for a safe navigation. To do this, the different modalities (ie, the LiDAR point cloud and the camera view) must be fused together in order to obtain such a global view. This raises the following questions: how to fuse such multi-modal representations? to what extent each modality should be processed independently before fusion? and how can such fusion be conduced?. The paper proposes a Modal Fusion Transformer (TransFuser), a transformer-based model designed to integrate both LiDAT and camera views with global attention, thus capturing the necessary 3D context for safe navigation.</p>
<figure style="width: 100%" class="align-center">
<img src="https://yassouali.github.io//ml-blog/images/CVPR21/9.png" />
<figcaption>Image source: <a href="https://arxiv.org/abs/2104.09224">Prakash et al.</a></figcaption>
</figure>
<p>The paper considers the task of point-to-point navigation in an urban setting, where the goal is to complete a given route while safely reacting to other dynamic agents and following traffic rules. The TransFuser is trained using an L1 loss between the predicted trajectories and the correct trajectories using a dataset consisting of high-dimensional observations of the environment, and the corresponding expert trajectory.
TransFuser takes as input RGB images and <a href="https://arxiv.org/abs/1905.01296">LiDAR BEV</a> representations and uses several transformer modules to fuse the intermediate feature maps between both modalities. The fusion is applied at multiple resolutions and throughout the feature extractor resulting in a 512-dimensional feature vector output from both the image and LiDAR BEV stream, which is the desired compact representation of the environment that encodes the global context of the 3D scene. This compact representation can then be used as input to an auto-regressive (a GRU) prediction network that outputs the trajectories in the form of waypoints in vehicle’s coordinate frame.</p>
<h4 id="neural-lumigraph-rendering-paper">Neural Lumigraph Rendering (<a href="https://arxiv.org/abs/2103.11571">paper</a>)</h4>
<p>The recent <a href="https://arxiv.org/abs/2004.03805">neural rendering</a> techniques are capable of generating photorealistic image quality for novel view synthesis and 3D shape estimation from 2D images. But they are either slow to train and/or require a considerable rendering time time for high image resolutions. <a href="https://arxiv.org/abs/2003.08934">NeRF</a> for instance does not offer real-time rendering due to the use of neural scene representation and <a href="http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.128.3394">volume rendering</a>.
To overcome this, the paper proposes the use SDF-based sinusoidal representation network (<a href="https://arxiv.org/abs/2006.09661">SIREN</a>) to implicitly model the surface of objects (ie, implicitly defining an object or a scene using a neural network and training directly with 3D data), which can be extracted using the <a href="https://dl.acm.org/doi/10.1145/37401.37422">marching cubes algorithm</a> and exported into traditional mesh-based representations for real-time rendering.</p>
<figure style="width: 100%" class="align-center">
<img src="https://yassouali.github.io//ml-blog/images/CVPR21/7.png" />
<figcaption>Image source: <a href="https://arxiv.org/abs/2103.11571">Kellnhofer et al.</a></figcaption>
</figure>
<p>However, given the high capacity of SIRENs, they are prone to overfitting, making them incapable of rendering new views that are interpolations of views encountered during training. To solve this, the authors propose a novel smoothness loss function that maintains SIREN’s high-capacity encoding in the image domain while constraining it in the angular domain to prevent overfitting on these views. The SIREN based network can then be trained using a sparse set of multi-view 2D images while providing a high-quality 3D surface that can be directly exported for real-time rendering algorithms at test time.</p>
<h4 id="nerv-neural-reflectance-and-visibility-fields-for-relighting-and-view-synthesis-paper">NeRV: Neural Reflectance and Visibility Fields for Relighting and View Synthesis (<a href="https://arxiv.org/abs/2012.03927">paper</a>)</h4>
<p>NeRV extends <a href="https://arxiv.org/abs/2003.08934">NeRF</a> to arbitrary lighting conditions, taking as input a set of images of a scene illuminated by unconstrained lighting, and producing a 3D representation that can be rendered from novel viewpoints under novel and unobserved lighting conditions.</p>
<figure style="width: 100%" class="align-center">
<img src="https://yassouali.github.io//ml-blog/images/CVPR21/8.png" />
<figcaption>Image source: <a href="https://arxiv.org/abs/2012.03927">Srinivasan et al.</a></figcaption>
</figure>
<p>Given that NeRF models just the amount of outgoing light from a location, the fact that this out going light is itself the result of interactions between incoming light and the material properties of an underlying surface is ignored, so rendering viewpoints
under novel lighting conditions is not possible. A naive way (figure above, right) to solve this is to query NeRF’s MLP for the volume density at samples along the camera ray to determine the amount of light reflected at each location that reaches the camera. Then for each location,
query the MLP for the volume density at points between the location and every light source. This procedure is clearly too computationally infeasible. The paper solve this by proposing NeRV, a method to train a NeRF-like model that can simulate realistic environment lighting and global illumination by using an additional MLP as a lookup table into a visibility field during rendering. As a result, the training consists of jointly optimizing the visibility MLP (or reflectance MLP) for estimating the light surface visibility at a given 3D position along, alongside the NeRF’s MLP (also called the shape MLP) for volumetric representation. At test time, the rendering is then conducted along the ray by querying the shape and reflectance MLPs for the volume densities, surface normals, and <a href="https://en.wikipedia.org/wiki/Bidirectional_reflectance_distribution_function">BRDF</a> parameters at each point.</p>
<h4 id="neural-body-implicit-neural-representations-with-structured-latent-codes-for-novel-view-synthesis-of-dynamic-humans-paper">Neural Body: Implicit Neural Representations With Structured Latent Codes for Novel View Synthesis of Dynamic Humans (<a href="https://arxiv.org/abs/2012.15838">paper</a>)</h4>
<figure style="width: 100%" class="align-center">
<img src="https://yassouali.github.io//ml-blog/images/CVPR21/12.png" />
<figcaption>Image source: <a href="https://arxiv.org/abs/2012.15838">Peng et al.</a></figcaption>
</figure>
<p>Given a very sparse set of camera views (say 3 or 4 views as depicted above), learning implicit neural representations of 3D scenes becomes infeasible. To solve this, the paper proposes Neural Body, a method that leverages observations over video frames order to learn
a new human body representation that is consistent over the different frames, and shares the same set of latent codes anchored to a deformable mesh, so that the observations across frames can be naturally integrated.</p>
<figure style="width: 100%" class="align-center">
<img src="https://yassouali.github.io//ml-blog/images/CVPR21/13.png" />
<figcaption>Image source: <a href="https://arxiv.org/abs/2012.15838">Peng et al.</a></figcaption>
</figure>
<p>First, a set of latent codes are defined and anchored to the vertices of the <a href="https://smpl.is.tue.mpg.de/">SMLP</a> deformable human model, so that the spatial location of each vertex varies with the human pose. Then, the 3D representation at a given frame is estimated from sparse camera views, and then applied to the code location based on the obtained 3D human pose. Finally, the network is trained to regress the density and color for any 3D point based on these latent codes. Both the latent codes and the network are jointly learned from images of all video frames during the reconstruction process.</p>
<h4 id="pixelnerf-neural-radiance-fields-from-one-or-few-images-paper">pixelNeRF: Neural Radiance Fields From One or Few Images (<a href="https://arxiv.org/abs/2012.02190">paper</a>)</h4>
<p>The <a href="https://arxiv.org/abs/2003.08934">NeRF</a> framework consists of optimizing the representation of every scene independently, requiring many views per scene and a significant compute time. pixelNeRF adapts NeRF in order to be trained across multiple scenes jointly to learn a scene prior, enabling it to perform novel view synthesis in a feed-forward manner from a sparse set of views (as few as one). This is done by
using spatial features of a given image produced by a CNN, which are aligned to each pixel as an input, and then passes through NeRF model for training. This simple image conditioning allows the model to be trained on a set of multi-view images, where it can learn scene priors to perform view synthesis.</p>
<figure style="width: 100%" class="align-center">
<img src="https://yassouali.github.io//ml-blog/images/CVPR21/14.png" />
<figcaption>Image source: <a href="https://arxiv.org/abs/2012.02190">Yu et al.</a></figcaption>
</figure>
<p>pixelNeRF consists of two components: a fully-convolutional image encoder, which encodes the input image into a pixel-aligned feature grid, and a NeRF network which outputs color and density given a spatial location and its corresponding encoded feature.
When multiple input views are available, each view is encoded into the feature grid, the multiple features are processed in parallel and then aggregated into the final color and opacity.</p>
<h4 id="other-papers-to-check-out-1">Other papers to check out</h4>
<ul>
<li><a href="https://arxiv.org/abs/2012.03065">Dynamic Neural Radiance Fields for Monocular 4D Facial Avatar Reconstruction</a></li>
<li><a href="https://arxiv.org/abs/2008.02268">NeRF in the Wild: Neural Radiance Fields for Unconstrained Photo Collections</a></li>
<li><a href="https://arxiv.org/abs/2011.13961">D-NeRF: Neural Radiance Fields for Dynamic Scenes</a></li>
<li><a href="https://arxiv.org/abs/2012.09365">Learning to recover 3D Scene Shape from a single image</a></li>
<li><a href="https://arxiv.org/abs/2104.00681">NeuralRecon: Real-Time Coherent 3D Reconstruction From Monocular Video</a></li>
<li><a href="https://arxiv.org/abs/2012.00230">Point2Skeleton: Learning Skeletal Representations from Point Clouds</a></li>
<li><a href="https://arxiv.org/abs/2103.03319">Learning High Fidelity Depths of Dressed Humans by Watching Social Media Dance Videos</a></li>
<li><a href="https://arxiv.org/abs/2103.05606">NeX: Real-Time View Synthesis With Neural Basis Expansion</a></li>
<li><a href="https://arxiv.org/abs/2103.06422">Holistic 3D Scene Understanding From a Single Image With Implicit Representation</a></li>
<li><a href="https://arxiv.org/abs/2012.02206">Scan2Cap: Context-aware Dense Captioning in RGB-D Scans</a></li>
<li><a href="https://arxiv.org/abs/2012.01451">Neural Deformation Graphs for Globally-consistent Non-rigid Reconstruction</a></li>
<li><a href="https://arxiv.org/abs/2012.09165">Exploring Data-Efficient 3D Scene Understanding with Contrastive Scene Contexts</a></li>
<li><a href="https://arxiv.org/abs/2012.01203">Learning Delaunay Surface Elements for Mesh Reconstruction</a></li>
<li><a href="https://arxiv.org/abs/2103.01261">A Deep Emulator for Secondary Motion of 3D Characters</a></li>
<li><a href="https://arxiv.org/abs/2103.00762">NeuTex: Neural Texture Mapping for Volumetric Neural Rendering</a></li>
<li><a href="https://arxiv.org/abs/2011.12490">DeRF: Decomposed Radiance Fields</a></li>
<li><a href="https://arxiv.org/abs/2102.13090">IBRNet: Learning Multi-View Image-Based Rendering</a></li>
<li><a href="https://arxiv.org/abs/2012.02189">Learned Initializations for Optimizing Coordinate-Based Neural Representations</a></li>
<li><a href="https://arxiv.org/abs/2012.01714">AutoInt: Automatic Integration for Fast Neural Volume Rendering</a></li>
<li><a href="https://arxiv.org/abs/2103.03319">Learning High Fidelity Depths of Dressed Humans by Watching Social Media Dance Videos</a></li>
<li><a href="https://arxiv.org/abs/2103.09213">Back to the Feature: Learning Robust Camera Localization From Pixels To Pose</a></li>
<li><a href="https://arxiv.org/abs/2106.03336">Wide-Baseline Relative Camera Pose Estimation With Directional Learning</a></li>
<li><a href="https://arxiv.org/abs/2004.07484">Pulsar: Efficient Sphere-Based Neural Rendering</a></li>
<li><a href="https://arxiv.org/abs/2101.10994">Neural Geometric Level of Detail: Real-Time Rendering With Implicit 3D Shapes</a></li>
<li><a href="https://arxiv.org/abs/2101.06720">Deep Multi-Task Learning for Joint Localization, Perception, and Prediction</a></li>
<li><a href="https://arxiv.org/abs/2104.11224">KeypointDeformer: Unsupervised 3D Keypoint Discovery for Shape Control</a></li>
<li><a href="https://arxiv.org/abs/2102.06195">Shelf-Supervised Mesh Prediction in the Wild</a></li>
<li><a href="https://arxiv.org/abs/2104.00680">LoFTR: Detector-Free Local Feature Matching With Transformers</a></li>
<li><a href="https://arxiv.org/abs/2104.06397">Shape and Material Capture at Home</a></li>
<li><a href="https://arxiv.org/abs/2006.14660">SPSG: Self-Supervised Photometric Scene Generation From RGB-D Scans</a></li>
<li><a href="https://arxiv.org/abs/2104.08278">Fusing the Old with the New: Learning Relative Camera Pose with Geometry-Guided Uncertainty</a></li>
<li><a href="https://arxiv.org/abs/2105.02047">Cuboids Revisited: Learning Robust 3D Shape Fitting to Single RGB Images</a></li>
<li><a href="https://arxiv.org/abs/2011.11814">MonoRec: Semi-Supervised Dense Reconstruction in Dynamic Environments From a Single Moving Camera</a></li>
<li><a href="https://arxiv.org/abs/2012.09159">DECOR-GAN: 3D Shape Detailization by Conditional Refinement</a></li>
</ul>
<h1 id="image-and-video-synthesis">Image and Video Synthesis</h1>
<h4 id="giraffe-representing-scenes-as-compositional-generative-neural-feature-fields-paper">GIRAFFE: Representing Scenes As Compositional Generative Neural Feature Fields (<a href="https://arxiv.org/abs/2011.12100">paper</a>)</h4>
<p>While GaNs are capable of generating photorealistic and diverse high resolutions images, having a fine-grained control over the factors of variation in the data and the compositionality of the generated scenes is sill limited, operating only in 2D and ignoring the three-dimensional nature of the underlying scenes.</p>
<figure style="width: 80%" class="align-center">
<img src="https://yassouali.github.io//ml-blog/images/CVPR21/15.png" />
<figcaption>Image source: <a href="XXXX">Niemeyer et al.</a></figcaption>
</figure>
<p>GIRAFFE proposes to represent scenes as compositional generative neural feature fields. During training, instead of having a single latent code as in the standard GaN setting, GIRAFFE randomly generated a set of <em>Shape</em> and <em>Appearance</em> codes for each object (& background) in the scene, which are used to generated Feature Fields. Then a second set of latent codes are generated, this time representing the pose transformation, which are applied to the generated Feature standard to obtain the Posed Feature Fields. Finally, the final Posed Feature Fields are aggregated into a single scene representation, and given a camera pose, are used as input to the neural rendering network to generate a 2D image. The generated and real 2D images are then passed to the discriminator to compute the adversarial loss. The whole model is trained end-to-end, and at test time, the composition of the generated images can be controlled using the Shape & Appearance and Pose latent codes.</p>
<h4 id="geosim-realistic-video-simulation-via-geometry-aware-composition-for-self-driving-paper">GeoSim: Realistic Video Simulation via Geometry-Aware Composition for Self-Driving (<a href="https://arxiv.org/abs/2101.06543">paper</a>)</h4>
<figure style="width: 100%" class="align-center">
<img src="https://yassouali.github.io//ml-blog/images/CVPR21/16.png" />
<figcaption>Image source: <a href="https://arxiv.org/abs/2101.06543">Chen et al.</a></figcaption>
</figure>
<p>The ability to simulate/enhance various real world scenarios is an important yet challenging open problem, especially for safety-critical domains such as self-driving where producing visually appealing and realistic results requires a physics-based rendering, which is very costly. As a, alternative, GeoSim exploits the recent advances in image synthesis by combining data driven approaches (such as generative modeling) and computer graphics to insert dynamic objects into existing videos, while maintaining high visual quality through physically grounded simulation (see the example above).</p>
<figure style="width: 100%" class="align-center">
<img src="https://yassouali.github.io//ml-blog/images/CVPR21/17.png" />
<figcaption>Image source: <a href="https://arxiv.org/abs/2101.06543">Chen et al.</a></figcaption>
</figure>
<p>The first step of GeoSim is to create a large 3D assets (vehicles of different types and shapes) with accurate pose, shape and texture. Instead of using artists to create these assets, GeoSim leverages publicly available datasets to construct 3D assets of the objects. This is done using a learning-based, multi-view, multi-sensor reconstruction approach that leverages the 3D bounding boxes and trained in a self-supervised manner so that there is an agreement between the predicted 3D shape and that of the camera and LiDAR observations.
GeoSim then exploits the 3D scene layout from high-definition maps and LiDAR data to add these learned 3D assets in plausible locations and make them behave realistically by considering the full scene. Finally, using this new 3D scene, GeoSim performs image-based rendering to properly handle occlusions, and neural network-based image in-painting to ensure the inserted object seamlessly blends in by filling holes, adjusting color inconsistencies due to lighting changes, and removing sharp boundaries.
Check out <a href="https://tmux.top/publication/geosim/">the paper’s website</a> for some results.</p>
<h4 id="taming-transformers-for-high-resolution-image-synthesis-paper">Taming Transformers for High-Resolution Image Synthesis (<a href="https://arxiv.org/abs/2012.09841">paper</a>)</h4>
<p>The recently introduced vision transformers (such as <a href="https://arxiv.org/abs/2010.11929">ViT</a>) demonstrated that they can perform on par with
CNNs, and given enough training data, they tend to learn convolutional structures. This raises an obvious question, do we have to relearn such an inductive bias
from scratch each time we train a vision model?. This paper proposes to merge both CNNs and transformers into a single framework for image synthesis. Thus leveraging the
efficiency of CNNs and their inductive image biases while still retaining the flexibility of transformers.</p>
<figure style="width: 90%" class="align-center">
<img src="https://yassouali.github.io//ml-blog/images/CVPR21/18.png" />
<figcaption>Image source: <a href="https://arxiv.org/abs/2012.09841">Esser et al.</a></figcaption>
</figure>
<p>As depicted above, the proposed framework consists of CNN encoder-decoder network trained adversarially for <a href="https://arxiv.org/abs/1711.00937">Neural Discrete Representation Learning</a>, and then a transformer that operates over the discrete representations in an autoregressive manner. More specifically, the training consist of two stages. First, the encoder, the decoder and the discriminator, all CNN-based, are trained using a reconstruction loss, <a href="https://arxiv.org/abs/1801.03924">a perceptual loss</a>, a commitment loss (ie, used to refine the codebook, see <a href="https://arxiv.org/abs/1711.00937">VQVAE</a> for more details), and an adversarial loss. At the end of training, we end-up with a learned codebook where each spatial location in the input image can be represented by an index in the codebook. By using such a formulation, the input image can be viewed as a sequence of codebook indices, and such a sequence can be used to train the auto-regressive transformer in the second step of the training process. Starting from the top left corner, at each time step, the transformer is tasked with predicting the next codebook index, and in order to reduce the computation, the input to the transformer is restricted into a sliding window without a significant loss in performance. Finally, at test time, we can use the trained transformer to generate large sequences without any restrictions (and with any type of conditioning, see section 3.2 of the paper), which correspond to very large images. The predicted indices are then used to fetch the discrete representations from the codebook, which are then passed to the decoder to synthesis an image.</p>
<h4 id="rethinking-and-improving-the-robustness-of-image-style-transferpaper">Rethinking and Improving the Robustness of Image Style Transfer(<a href="https://arxiv.org/abs/2104.05623">Paper</a>)</h4>
<p>The objective of image style transfer is to map the content of a given image into the style of a different one, and in such a task, VGG network has demonstrated
a remarkable ability to capture the visual style of an image. However, when such a network is replaced with a more modern, and a better performing network such as
ResNet, the stylization performance degrades significantly as shown in the figure bellow.</p>
<figure style="width: 50%" class="align-center">
<img src="https://yassouali.github.io//ml-blog/images/CVPR21/19.png" />
<figcaption>Image source: <a href="https://arxiv.org/abs/2104.05623">Wang et al.</a></figcaption>
</figure>
<p>In this paper, the authors investigate the root cause of this behavior, and find that residual connections, which represent the main architectural difference between VGG and ResNet, produce peaky feature maps of small entropy, which are not suitable for style transfer. To improve the stylization of the ResNet architecture, the authors then propose a simple yet effective solution based on a softmax transformation of the feature activations that enhances their entropy. This method, dubbed Stylization With Activation smoothinG (SWAG), consists of adding a softmax-based smoothing transformation to all of the activations in order to push the model to produce smoother activations, thus reducing large peaks and increasing small values, creating a more uniform distribution.</p>
<h4 id="learning-continuous-image-representation-with-local-implicit-image-function-paper">Learning Continuous Image Representation With Local Implicit Image Function (<a href="https://arxiv.org/abs/2012.09161">Paper</a>)</h4>
<p>This paper proposes Local Implicit Image Function (LIIF) for representing natural and complex images in a continuous manner. With LIIF, an image is represented as a set of latent codes distributed in spatial dimensions. Given a coordinate, the decoding function takes the coordinate information and queries the local latent codes around it, and
then predicts the RGB value at the given coordinate as an output. Since the LIIF representation is continuous, we can query an arbitrary high target resolution up to x30 higher than the resolution encountered during training.</p>
<figure style="width: 100%" class="align-center">
<img src="https://yassouali.github.io//ml-blog/images/CVPR21/20.png" />
<figcaption>Image source: <a href="https://arxiv.org/abs/2012.09161">Chen et al.</a></figcaption>
</figure>
<p>The proposed framework consists of an encoder that produces 2D feature maps given an input image, where the feature maps are evenly distributed in the 2D space of the continuous image domain, and each feature at a given spatial location is called a latent code. Then, the decoder takes as input a 2D coordinate in the image domain in addition to a weighted average of the 4 nearest latent codes of the chosen 2D coordinate and outputs the RGB values. Now, in order to train both the encoder and the decoder jointly using self-supervision, this is done by taking a training image, randomly down-sampling it, the encoder then encodes the down-sampled image, while the decoder is queried to produced the RGB values of the original image, which is used as ground-truth.</p>
<h4 id="other-papers-to-check-out-2">Other papers to check out</h4>
<ul>
<li><a href="https://arxiv.org/abs/2104.14551">Ensembling with Deep Generative Views</a></li>
<li><a href="https://arxiv.org/abs/2007.08211">SSN: Soft Shadow Network for Image Compositing</a></li>
<li><a href="https://arxiv.org/abs/2012.02992">Spatially-Adaptive Pixelwise Networks for Fast Image Translation</a></li>
<li><a href="https://arxiv.org/abs/2104.15060">DriveGAN: Towards a Controllable High-Quality Neural Simulation</a></li>
<li><a href="https://arxiv.org/abs/2104.11280">Motion Representations for Articulated Animation</a></li>
<li><a href="https://arxiv.org/abs/2101.12195">Playable Video Generation</a></li>
<li><a href="https://arxiv.org/abs/2103.16183">Repopulating Street Scenes</a></li>
<li><a href="https://arxiv.org/abs/2011.15128">Animating Pictures With Eulerian Motion Fields</a></li>
<li><a href="https://arxiv.org/abs/2007.06600">Closed-Form Factorization of Latent Semantics in GANs</a></li>
<li><a href="https://arxiv.org/abs/2011.08114">Stylized Neural Painting</a></li>
<li><a href="https://arxiv.org/abs/2011.13775">Image Generators With Conditionally-Independent Pixel Synthesis</a></li>
<li><a href="https://arxiv.org/abs/2101.04702">Cross-Modal Contrastive Learning for Text-to-Image Generation </a></li>
<li><a href="https://arxiv.org/abs/2011.10063">Dual Contradistinctive Generative Autoencoder</a></li>
<li><a href="https://arxiv.org/abs/2011.12950">Space-Time Neural Irradiance Fields for Free-Viewpoint Video</a></li>
<li><a href="https://arxiv.org/abs/2012.05217">Positional Encoding As Spatial Inductive Bias in GANs</a></li>
<li><a href="https://arxiv.org/abs/2104.03310">Regularizing Generative Adversarial Networks Under Limited Data</a></li>
<li><a href="https://arxiv.org/abs/2104.02416">Variational Transformer Networks for Layout Generation</a></li>
<li><a href="https://arxiv.org/abs/2104.02495">Deep Animation Video Interpolation in the Wild</a></li>
<li><a href="https://arxiv.org/abs/2011.07233">Stable View Synthesis</a></li>
<li><a href="https://arxiv.org/abs/2011.13786">Navigating the GAN Parameter Space for Semantic Image Editing</a></li>
<li><a href="https://arxiv.org/abs/2105.04551">Stochastic Image-to-Video Synthesis Using cINNs</a></li>
<li><a href="https://arxiv.org/abs/2104.14754">Exploiting Spatial Dimensions of Latent in GAN for Real-Time Image Editing</a></li>
<li><a href="https://arxiv.org/abs/2106.05375">Plan2Scene: Converting Floorplans to 3D Scenes</a></li>
<li><a href="https://arxiv.org/abs/2101.06541">SceneGen: Learning to Generate Realistic Traffic Scenes</a></li>
<li><a href="https://openaccess.thecvf.com/content/CVPR2021/html/Bowen_OCONet_Image_Extrapolation_by_Object_Completion_CVPR_2021_paper.html">OCONet: Image Extrapolation by Object Completion</a></li>
<li><a href="https://arxiv.org/abs/2103.03243">Anycost GANs for Interactive Image Synthesis and Editing</a></li>
<li><a href="https://arxiv.org/abs/2011.12799">StyleSpace Analysis: Disentangled Controls for StyleGAN Image Generation</a></li>
<li><a href="https://arxiv.org/abs/2008.00951">Encoding in Style: A StyleGAN Encoder for Image-to-Image Translation</a></li>
<li><a href="https://arxiv.org/abs/2103.17185">Rethinking Style Transfer: From Pixels to Parameterized Brushstrokes</a></li>
<li><a href="https://arxiv.org/abs/2011.15126">One-Shot Free-View Neural Talking-Head Synthesis for Video Conferencing</a></li>
<li><a href="https://arxiv.org/abs/2103.00430">Training Generative Adversarial Networks in One Stage</a></li>
<li><a href="https://arxiv.org/abs/2007.10379">Generative Hierarchical Features From Synthesizing Images</a></li>
</ul>
<h1 id="scene-analysis--understanding">Scene Analysis & Understanding</h1>
<h4 id="rethinking-semantic-segmentation-from-a-sequence-to-sequence-perspective-with-transformers-paper">Rethinking Semantic Segmentation From a Sequence-to-Sequence Perspective With Transformers (<a href="https://arxiv.org/abs/2012.15840">Paper</a>)</h4>
<p>The proposed SEgmentation TRansformer (SETR) is based on an alternative formulation of semantic segmentation as a sequence-to-sequence task.
This way, instead of using the standard encoder-decoder architecture, such a formulation gives us the possibility to employ a pure transformer without convolution and resolution reduction for pixel-level classification, since it is in line with the way transformers operate over
the inputs and how they produce the predictions. In addition to leveraging the capability of a transformer layer to model the global context which important for sematic segmentation in order to obtain coherent masks.</p>
<figure style="width: 100%" class="align-center">
<img src="https://yassouali.github.io//ml-blog/images/CVPR21/21.png" />
<figcaption>Image source: <a href="https://arxiv.org/abs/2012.15840">Zheng et al.</a></figcaption>
</figure>
<p>SETR (figure above, a) treats an input image as a sequence of image patches, where each image is first decomposed into a fixed-sized patches. Then, each path is flattened into a vector of pixel values and passed through a linear layer, outputting the patch embedding. These patch embedding are passed as a sequence to the transformer-encoder (ie, 24 transformer layers) with the global self-attention in order to discriminative features tailored for the segmentation task. The produced representations are then reshaped from 2D shape (number of patches x embedding dimensionality) back into the standard 3D features map shape (H x W x embedding dimensionality). The reshaped features are then passed to the decoder to predict the final per pixel classification at the original input size. Here, SETR proposes 3 types of decoders: (1) Naive upsampling: a 2-layer network followed by bilinear upsampling, (2) Progressive UPsampling: alternates between conv layers and a single 2x bilinear upsampling operation (figure above, b). (3) Multi-Level feature Aggregation: apply many levels of conv layers & 4x by bilinear upsampling over the encoder’s output, merge them and apply a final upsampling (figure above, c).</p>
<h4 id="max-deeplab-end-to-end-panoptic-segmentation-with-mask-transformers-paper">MaX-DeepLab: End-to-End Panoptic Segmentation With Mask Transformers (<a href="https://arxiv.org/abs/2012.00759">Paper</a>)</h4>
<p>MaX-DeepLab is an end-to-end model for panoptic segmentation without any hand-designed components such as box detection, non-maximum suppression, or thing-stuff merging. MaX-DeepLab directly predicts class-labeled masks with a mask transformer, and is trained with a panoptic quality inspired loss via bipartite matching.</p>
<figure style="width: 100%" class="align-center">
<img src="https://yassouali.github.io//ml-blog/images/CVPR21/22.png" />
<figcaption>Image source: <a href="https://arxiv.org/abs/2012.00759">Wang et al.</a></figcaption>
</figure>
<p>Building upon the recent transformer-based end-to-end object detectors such as <a href="https://arxiv.org/abs/2005.12872">DERT</a> or <a href="https://arxiv.org/abs/2010.04159">Deformable DETR</a>, MaX-DeepLab directly outputs non-overlapping masks and their corresponding semantic labels with a mask transformer. The model is then trained with a novel panoptic quality style loss (see section 3.2). This loss measures the similarity between ground-truth and predicted class-labeled masks as the multiplication of their mask similarity and their class similarity, and since the MaX-DeepLab outputs a larger number of masks than the ground-truths, a one-to-one matching is applied (as in DERT) before computing the loss.</p>
<p>As for the architecture, MaX-DeepLab integrates transformer blocks (called dual-path transformer) along a given CNN backbone in a dual-path fashion, with bidirectional communication blocks between the two paths. Each 2D pixel-based CNN is augmented with 1D global memory (of the same size as the number of predictions) with different types of attention. Specifically, a dual-path transformer takes as input the 2D CNN features and 1D memory, and applies 4 types of attention: 1) pixel-to-pixel over the CNN features, and since the attention over spatial dimensions is expensive, they use <a href="https://arxiv.org/abs/2003.07853">axial-attention</a>.
1) memory-to-memory updating the memory features with global context, and then the cross-attention, 3) pixel-to-memory and 4) memory-to-pixel attention, where each time the query of one is applied to the keys and values of the other to update either the pixel or memory features conditioned on the other.</p>
<h4 id="binary-ttc-a-temporal-geofence-for-autonomous-navigation-paper">Binary TTC: A Temporal Geofence for Autonomous Navigation (<a href="https://arxiv.org/abs/2101.04777">Paper</a>)</h4>
<p>Path planning, whether for robotics or automotive applications, requires accurate perception, and one of main task used to acquire such an accurate perception is depth information. To infer depth, the most popular strategy is to use LiDAR which is capable of estimating depth, but only at sparse locations and can be quite expensive. Another alternative is to only use monocular cameras (this is the Tesla approach)
to construct the optical flow between consecutive frames, which carries information on the scene’s depth, while greatly reducing the acquisition and maintenance costs. But this approach also has its drawback since it can only be estimated reliably in constrained and simple scenes. In this context, and since the objective behind the perception is to inform decisions, Binary TTC proposes to replace learning to infer depth with a new and simpler task that can be directly used to inform planning decisions.</p>
<figure style="width: 100%" class="align-center">
<img src="https://yassouali.github.io//ml-blog/images/CVPR21/23.png" />
<figcaption>Image source: <a href="https://arxiv.org/abs/2101.04777">Badki et al.</a></figcaption>
</figure>
<p>The proposed Binary TTC task is based on the concept of <a href="https://pubmed.ncbi.nlm.nih.gov/1834797/">time-to-contact (TTC)</a>, which is the time for an object to collide with the camera plane under the current velocity conditions, and consists of predicting a binary classification map, where the objects that will collide with the camera plane within a given time are assigned labels of 1.
More specifically, a binary classification network is trained to detect the objects that will collide with the camera plane for a given time interval. To train the network, the labels are generated using two images of a given dynamic scene, then, the sizes of each moving object in the two images are compared (after a scaling is applied to take into account the chosen time of collision). If the size becomes larger from the first to the second image, this means that the object is getting closer to the camera plane and can be labeled as 1.</p>
<h4 id="boosting-monocular-depth-estimation-models-to-high-resolution-via-content-adaptive-multi-resolution-merging-paper">Boosting Monocular Depth Estimation Models to High-Resolution via Content-Adaptive Multi-Resolution Merging (<a href="https://arxiv.org/abs/2105.14021">Paper</a>)</h4>
<figure style="width: 100%" class="align-center">
<img src="https://yassouali.github.io//ml-blog/images/CVPR21/24.png" />
<figcaption>Image source: <a href="https://arxiv.org/abs/2105.14021">Miangoleh et al.</a></figcaption>
</figure>
<p>The paper starts with a analysis of the behavior of standard monocular depth estimation models (they use <a href="https://arxiv.org/abs/1907.01341">MiDaS</a>) when fed images at different resolutions. With small resolutions, the estimations lack many high-frequency details while generating a consistent overall structure of the scene. As the input resolution gets higher, more details are generated in the result but with inconsistencies in the scene structure characterized by gradual shifts in depth between image regions.</p>
<p>Based on the above observation, the paper proposes to merge depth estimates at different resolution with an image-to-image translation network that merges these estimates into a final prediction.
In order to determine the input resolutions which to be merged later, the authors propose to base this selection on the number of pixels that do not have any contextual cues nearby. These regions with the lowest contextual cue density will dictate the maximum resolution
that can be used for an image. Additionally, in order to also benefit from higher-resolution estimations to generate more high-frequency details, a patch based selection method is used to find regions with higher contextual cue density that requires more high-frequency details, which are then merged together for the final results.</p>
<p>Check out the authors <a href="https://www.youtube.com/watch?v=lDeI17pHlqo">video</a> for a brief but great explanation of the work.</p>
<h4 id="polygonal-building-extraction-by-frame-field-learning-paper">Polygonal Building Extraction by Frame Field Learning (<a href="https://arxiv.org/abs/2004.14875">Paper</a>)</h4>
<p>For the task of building segmentation where the objective is to output a polygon for each building in a given aerial photo, the existing approach are either based on vector representations, directly predicting vector polygons, or two step approach that first produce probability a map, which is then followed by polygon simplification. However, both approaches are either hard to train or contain many steps before the final output.</p>
<figure style="width: 100%" class="align-center">
<img src="https://yassouali.github.io//ml-blog/images/CVPR21/25.png" />
<figcaption>Image source: <a href="https://arxiv.org/abs/2004.14875">Girard et al.</a></figcaption>
</figure>
<p>For an end-to-end method that is easy to optimize, the authors propose to build on the semantic segmentation methods, where in addition to predicting a segmentation maps corresponding to the buildings in the image, they also task the model with predicting a frame field as a geometric prior to constrain the segmentation maps to have sharpe corners.</p>
<h4 id="other-papers-to-check-out-3">Other papers to check out</h4>
<ul>
<li><a href="https://arxiv.org/abs/2012.05258">VIP-DeepLab: Learning Visual Perception With Depth-Aware Video Panoptic Segmentation </a></li>
<li><a href="https://arxiv.org/abs/2105.08336">Exemplar-Based Open-Set Panoptic Segmentation Network</a></li>
<li><a href="https://arxiv.org/abs/2104.14540">The Temporal Opportunist: Self-Supervised Multi-Frame Monocular Depth</a></li>
<li><a href="https://arxiv.org/abs/2012.05901">Robust Consistent Video Depth Estimation</a></li>
<li><a href="https://openaccess.thecvf.com/content/CVPR2021/html/Qiu_Scene_Essence_CVPR_2021_paper.html">Scene Essence </a></li>
<li><a href="https://arxiv.org/abs/2104.05833">Semantic Segmentation With Generative Models: Semi-Supervised Learning and Strong Out-of-Domain Generalization</a></li>
<li><a href="https://arxiv.org/abs/2103.04379">Repurposing GANs for One-Shot Semantic Part Segmentation</a></li>
<li><a href="https://arxiv.org/abs/2103.12340">Deep Occlusion-Aware Instance Segmentation With Overlapping BiLayers</a></li>
<li><a href="https://arxiv.org/abs/2012.07287">Information-Theoretic Segmentation by Inpainting Error Maximization</a></li>
<li><a href="https://arxiv.org/abs/2106.02022">Single Image Depth Prediction With Wavelet Decomposition</a></li>
<li><a href="https://arxiv.org/abs/2012.09365">Learning to Recover 3D Scene Shape from a Single Image</a></li>
</ul>
<h1 id="representation--adversarial-learning">Representation & Adversarial Learning</h1>
<h4 id="exploring-simple-siamese-representation-learning-paper">Exploring Simple Siamese Representation Learning (<a href="https://arxiv.org/abs/2011.10566">Paper</a>)</h4>
<p>In the recent contrastive learning methods used for learning useful visual representation in an unsupervised manner, a model
is trained to map similar input images close by in the embedding space. Such methods, such <a href="https://arxiv.org/abs/1911.05722">MoCo</a>, <a href="https://arxiv.org/abs/2002.05709">SimCLR</a>, or <a href="https://arxiv.org/abs/2006.07733">BYOL</a>, add some additional conditions to the similarity maximization objective to avoid collapsing solutions, such as negative sample pairs, large batches, or momentum encoders.</p>
<figure style="width: 100%" class="align-center">
<img src="https://yassouali.github.io//ml-blog/images/CVPR21/26.png" />
<figcaption>Image source: <a href="https://arxiv.org/abs/2011.10566">Chen et al.</a></figcaption>
</figure>
<p>This paper investigates the usage of SimSiam, a simple Siamese based setup where such conditions are not necessary. Such an architecture consists of an encoder and a projector/predictor, which are then trained to maximize the similarity between a two given features (one after projection and one without) corresponding to two augmented version of an input image with a stop gradient operator applied to the second non projected output. The obtained results show that this simple approach gives are similar to other more elaborate approaches, indicating that the Siamese architecture may be an essential reason for the common success of the other related contrastive methods, see section 5 for more details.</p>
<h4 id="where-and-what-examining-interpretable-disentangled-representations-paper">Where and What? Examining Interpretable Disentangled Representations (<a href="https://arxiv.org/abs/2104.05622">Paper</a>)</h4>
<p>In disentangled representation learning, the objective is produce a representation of an input, where each dimension captures variations with
a semantic meaning. One of the main limitations of existing work is their inability to differentiate between entangled and disentangled representations in the solution pool.</p>
<figure style="width: 100%" class="align-center">
<img src="https://yassouali.github.io//ml-blog/images/CVPR21/27.png" />
<figcaption>Image source: <a href="https://arxiv.org/abs/2104.05622">Zhu et al.</a></figcaption>
</figure>
<p>To solve this non-uniqueness problem, the paper defines disentanglement from three perspectives: informativeness, independence, and <em>interpretability</em>. By adding interpretability condition to the produced representations, we force the model to only produce disentangled representations that do correspond to human-defined concepts. Now, the question becomes how can we enforce such an interpretability constraint in the representations without supervision?.
To solve this, the authors propose to exploit two hypotheses about interpretability to learn disentangled representations. The first one is Spatial Constriction (SC): a representation is usually interpretable if we can consistently tell where the controlled variations are in an image. The second hypothesis is Perceptual Simplicity: an interpretable code usually corresponds to a concept consisting of perceptually simple variations.</p>
<p>Based on these two hypothesis, a new model is introduced, where the Spatial Constriction is enforced with a SC module that restricts the impact of each latent code to specific areas on feature maps during generation. As for Perceptual Simplicity, the model is trained with a loss that encourages the model to embed simple data variations along each latent dimension.</p>
<h4 id="audio-visual-instance-discrimination-with-cross-modal-agreement-paper">Audio-Visual Instance Discrimination with Cross-Modal Agreement (<a href="https://arxiv.org/abs/2004.12943">Paper</a>)</h4>
<p>The paper presents a for cross-modal discrimination for self-supervised learning in order to learn audio-visual representations from video and audio. The proposed contrastive learning frame-work consists of contrasting video representations against multiple audios representations at once (and vice versa), thus the cross-modal nature of the method.</p>
<figure style="width: 100%" class="align-center">
<img src="https://yassouali.github.io//ml-blog/images/CVPR21/28.png" />
<figcaption>Image source: <a href="https://arxiv.org/abs/2004.12943">Morgado et al.</a></figcaption>
</figure>
<p>The proposed approach learns both a cross-modal similarity metric by grouping video and audio instances that co-occur over multiple instances, in addition to optimizing for visual similarity rather than just cross-modal similarity.</p>
<h4 id="up-detr-unsupervised-pre-training-for-object-detection-with-transformers-paper">UP-DETR: Unsupervised Pre-Training for Object Detection With Transformers (<a href="https://arxiv.org/abs/2011.09094">Paper</a>)</h4>
<p>While <a href="https://arxiv.org/abs/2005.12872">DETR</a> proposed a simple end-to-end object detector, thus removing all hand-designed components, it still comes with some training and optimization challenges, requiring large-scale training data and long training times (up to 500 epochs). UP-DETR proposes a new unsupervised pre-training tasks in order to reduce the amount of training time and data required, where
DETR is first pre-trained on a pretext task designed specifically for object detection as a desired down-stream task, then fine-tunned for object detection.</p>
<figure style="width: 100%" class="align-center">
<img src="https://yassouali.github.io//ml-blog/images/CVPR21/29.png" />
<figcaption>Image source: <a href="https://arxiv.org/abs/2011.09094">Dai et al.</a></figcaption>
</figure>
<p>Unsupervised Pre-training DETR (UP-DETR) defines random query patch detection as a pretext task to pre-train DETR in a self-supervised manner.
For this task, a set of patches are first cropped from the input image at random, the model is then trained to predict the bounding boxes of these patches.
The objective of this pre-training stage is to equip the model with better localization while maintaining its classification features. So to avoid suppressing the learned
classification features of the pre-trained backbone, UP-DETR freezes the backbone and adds a patch feature reconstruction loss.
Additionally, in order to specify the query patch the model needs to detect, the CNN features of the patch itself are added to the object queries before feeding them
to the decoder. For multi-patch detection, the patches are grouped into sets of patches and each set is assigned to a given object query, and the attention aggregation is applied with a masking operation so that each prediction is not dependent on the rest (see section 3.2).</p>
<h4 id="fast-end-to-end-learning-on-protein-surfaces-paper">Fast End-to-End Learning on Protein Surfaces (<a href="https://www.biorxiv.org/content/10.1101/2020.12.28.424589v1">Paper</a>)</h4>
<p>Chemically, proteins are composed of a sequence of amino acids which determines the structure (called fold) of the protein, and this structure in turn
determines the function the protein will have. The task of either structure prediction from a sequence of amino acids (what AlphaFold does) or the protein design from its structure, are both major unsolved problems in structural biology. Another challenging problem, the task of interest in this paper, is the study of the interactions of a given molecule with other molecules given their composition in order to identify interaction patterns on protein surfaces.</p>
<figure style="width: 65%" class="align-center">
<img src="https://yassouali.github.io//ml-blog/images/CVPR21/30.png" />
<figcaption>Image source: <a href="https://www.biorxiv.org/content/10.1101/2020.12.28.424589v1">Sverrisson et al.</a></figcaption>
</figure>
<p>In this context, the paper proposes dMaSIF (differentiable molecular surface interaction fingerprinting), a new and computationally efficient deep learning approach
that operates directly on the large set of atoms that compose the protein. It first generates a point cloud representation for the protein surface, learns task-specific geometric and chemical features on the surface point cloud, and finally, applies a new convolutional operator that approximates geodesic coordinates in the tangent space (please see the paper since I’m way over my head here).</p>
<h4 id="natural-adversarial-examples-paper">Natural Adversarial Examples (<a href="https://arxiv.org/abs/1907.07174">Paper</a>)</h4>
<p>ImageNet test examples tend to be simple, clear, close-up images, so the current test set may be too easy and may not represent harder images encountered in the real world, and a large capacity model can leverage spurious cues or shortcuts to solve the ImageNet classification problem.
To counteract this, the paper proposes two hard ImageNet test sets: <em>ImageNet-A</em> and <em>ImageNet-O</em> of natural adversarial examples with adversarial filtration.</p>
<figure style="width: 100%" class="align-center">
<img src="https://yassouali.github.io//ml-blog/images/CVPR21/31.png" />
<figcaption>Image source: <a href="https://arxiv.org/abs/1907.07174">Hendrycks et al.</a></figcaption>
</figure>
<p><em>ImageNet-A</em> consists of real-world adversarially filtered images that fool current ImageNet classifiers.
Starting from a large set of images related to an ImageNet class, adversarially filtered examples are found by removing the samples that are correctly classified by ResNet-50 trained on ImageNet. <em>ImageNet-O</em> on the other hand in is a dataset of adversarially filtered examples for ImageNet out-of-distribution detectors.
Starting from ImageNet-22K, first, all of the examples that belong to ImageNet-1K classes are removed. Then, using a ResNet-50, the only maintained examples are the ones that were classified by the model into ImageNet-1K classes with high confidence.</p>
<ul>
<li><a href="https://arxiv.org/abs/2008.03800">Spatiotemporal Contrastive Video Representation Learning</a></li>
<li><a href="https://arxiv.org/abs/2106.01899">Adversarially Adaptive Normalization for Single Domain Generalization</a></li>
<li><a href="https://arxiv.org/abs/2103.03114">Self-Supervised Geometric Perception</a></li>
<li><a href="https://arxiv.org/abs/2104.04015">CutPaste: Self-Supervised Learning for Anomaly Detection and Localization</a></li>
<li><a href="https://arxiv.org/abs/2005.07289">Taskology: Utilizing Task Relations at Scale</a></li>
<li><a href="https://arxiv.org/abs/2105.01879">MOS: Towards Scaling Out-of-Distribution Detection for Large Semantic Space</a></li>
</ul>
<h1 id="transfer-low-shot-semi--unsupervised-learning">Transfer, Low-shot, Semi & Unsupervised Learning</h1>
<h4 id="datasetgan-efficient-labeled-data-factory-with-minimal-human-effort-paper">DatasetGAN: Efficient Labeled Data Factory With Minimal Human Effort (<a href="https://arxiv.org/abs/2104.06490">Paper</a>)</h4>
<p>DatasetGAN is a method that generates massive datasets of high-quality semantically segmented images with minimal human effort.
Based on the observation that GaNs are capable of acquiring rich semantic knowledge in order to render diverse and realistic examples of objects, DatasetGAN exploits the feature space of a trained GAN and trains a shallow decoder to produce a pixel-level labeling, where such a decoder is first trained on a very small number of labeled examples, and can then be used to labeled an infinite amount of synthetic images. The generated dataset can be used to train a model in an semi-supervised manner on the synthetic dataset, which
can then be tested on real world images.</p>
<figure style="width: 100%" class="align-center">
<img src="https://yassouali.github.io//ml-blog/images/CVPR21/40.png" />
<figcaption>Image source: <a href="https://arxiv.org/abs/2104.064907">Zhang et al.</a></figcaption>
</figure>
<p>The architecture of DatasetGAN consist of two models, a StyleGAN that generates synthetic image, in addition to Style-Interpreter in the form of an ensemble of three-layer MLP classifiers, where each classifier takes as inputs feature maps from StyleGAN (outputs of AdaIN layers), upsamples them to the image resolutions, and predicts the pixel-level labels. The final prediction is the aggregation of the predictions of all of the MLP classifiers, which then trained with a small number of finely annotated examples and used for labeling the synthetic images.</p>
<h4 id="ranking-neural-checkpoints-paper">Ranking Neural Checkpoints (<a href="https://arxiv.org/abs/2011.11200">Paper</a>)</h4>
<p>During a given deep learning experiment, it is common practice to collect many checkpoint, which are different versions of the final model at different training iterations. This paper is concerned with ranking such checkpoints to find the models that best transfers to the downstream task of interest.</p>
<p>More specifically, given a number of pretrained neural nets, called checkpoints \(\mathcal{C}\). The objective is to find the best checkpoint over a distribution of downstream tasks \(\mathcal{T}\). Each task consists of training and testing sets, and an evaluation procedure \(\mathbf{G}\) that adjusts the pretrained model by adding the task specific head, finetunes the model on the training set with a hyperparameter sweep under a given computation contraint, and retuns a performance measure \(\mathbf{R}\).
The objective of the paper is to find the best measure \(\mathcal{M}\) to correctly rank the checkpoints.</p>
\[\mathbf{R}^{*} \leftarrow \underset{\mathbf{R} \in \mathcal{R}}{\arg \max } \mathbb{E}_{t \sim \mathcal{T}} \mathcal{M}\left(\mathbf{R}_{t}, \mathbf{G}_{t}\right)\]
<p>The paper proposes a measure called NLEEP, an extention of <a href="https://arxiv.org/abs/2002.12462">LEEP</a> that evaluates the degree of transferability of the learned representations from source to target data without training. LEEP consists of computing the empirical conditional distribution of target labels given dummy source labels to measure the degree of transferability. NLEEP simply replaces the softmax classifier used for generating the dummy source labels with a Gaussian mixture model for more reliable class assignments. This way, we can evaluate the checkpoints on downstrem tasks while reducing cost of the evaluation procedure since LEEP does not require fine-tuning on the target data.</p>
<h4 id="other-papers-to-check-out-4">Other papers to check out</h4>
<ul>
<li><a href="https://arxiv.org/abs/2104.00303">MeanShift++: Extremely Fast Mode-Seeking With Applications to Segmentation and Object Tracking</a></li>
<li><a href="https://arxiv.org/abs/2003.10580">Meta Pseudo Labels</a></li>
<li><a href="https://arxiv.org/abs/2104.01893">Adaptive Prototype Learning and Allocation for Few-Shot Segmentation</a></li>
<li><a href="https://arxiv.org/abs/2103.07600">Student-Teacher Learning From Clean Inputs to Noisy Inputs</a></li>
<li><a href="https://arxiv.org/abs/2102.01987">Learning Graph Embeddings for Compositional Zero-Shot Learning</a></li>
</ul>
<h1 id="computational-photography">Computational Photography</h1>
<h4 id="real-time-high-resolution-background-matting-paper">Real-Time High-Resolution Background Matting (<a href="https://arxiv.org/abs/2012.07810">Paper</a>)</h4>
<figure style="width: 100%" class="align-center">
<img src="https://yassouali.github.io//ml-blog/images/CVPR21/42.png" />
<figcaption>Image source: <a href="https://arxiv.org/abs/2012.07810">Lin el al.</a></figcaption>
</figure>
<p>While many tools now provide background replacement functionality, they often yield artifacts at the boundaries, particularly in areas where there is fine detail like hair or glasses. On the other hand, the traditional image matting methods
provide much higher quality results, but do not run in real-time, at high resolutions, and frequently require manual input.</p>
<figure style="width: 100%" class="align-center">
<img src="https://yassouali.github.io//ml-blog/images/CVPR21/41.png" />
<figcaption>Image source: <a href="https://arxiv.org/abs/2012.07810">Lin el al.</a></figcaption>
</figure>
<p>This paper proposes a real-time and high-resolution background matting method capable of processing 4K (3840x2160) images at 30fps and HD (1920x1080) images at 60fps. To acheive this, the model needs to be trained on large volumes of images with high-quality alpha mattes to generalize. To this end, the paper introduces two datasets with high-resolution alpha mattes and foreground layers extracted with chroma-key software. The model is then trained on these datases to learn strong priors, then fine-tunned on public dataset to learn fine-grained details. As for the network design, the model contains two processing paths; a base network that predicts the alpha matte and foreground layer at lower resolution, along with an error prediction map which specifies areas that may need high-resolution refinement. Based on this maps, a refinement network then takes the low-resolution result and the original image to generate high-resolution output, but only at select regions for efficency.</p>
<h4 id="im2vec-synthesizing-vector-graphics-without-vector-supervision-paper">Im2Vec: Synthesizing Vector Graphics without Vector Supervision (<a href="https://arxiv.org/abs/2102.02798">Paper</a>)</h4>
<p>Despite the large amount of generative methods for images, there a limited amount of approaches that operate directly on
vector graphics and require direct supervison. To solve this, the paper proposes Im2Vec, a new neural network that generates complex vector graphics with varying topologies, and only requires indirect supervision from readily-available training imges with no vector counterparts.</p>
<figure style="width: 100%" class="align-center">
<img src="https://yassouali.github.io//ml-blog/images/CVPR21/43.png" />
<figcaption>Image source: <a href="https://arxiv.org/abs/2102.02798">Reddy et al.</a></figcaption>
</figure>
<p>Im2Vec consists of a standard encode-decoder architecture. Given an input image, the encoder maps it into a latent variable, which is then decoded into a vector graphic structure. The decoder on the other hand is designed so that it can generate complex graphics (see section 3 of the paper for more details).</p>
<h4 id="other-papers-to-check-out-5">Other papers to check out</h4>
<ul>
<li><a href="https://arxiv.org/abs/2104.00059">Passive Inter-Photon Imaging</a></li>
<li><a href="https://arxiv.org/abs/2103.02376">Event-Based Synthetic Aperture Imaging With a Hybrid Network</a></li>
<li><a href="https://arxiv.org/abs/2105.06070">GAN Prior Embedded Network for Blind Face Restoration in the Wild</a></li>
<li><a href="https://arxiv.org/abs/2012.06722">Mask Guided Matting via Progressive Refinement Network</a></li>
</ul>
<h1 id="other">Other</h1>
<h4 id="biometrics-face-gesture-and-body-pose">Biometrics, Face, Gesture and Body Pose</h4>
<ul>
<li><a href="https://arxiv.org/abs/2103.06871">SMPLicit: Topology-Aware Generative Model for Clothed People</a></li>
<li><a href="https://arxiv.org/abs/2104.03176">On Self-Contact and Human Pose</a></li>
<li><a href="https://arxiv.org/abs/2007.12287">Body2Hands: Learning To Infer 3D Hands From Conversational Gesture Body Dynamics</a></li>
<li><a href="https://arxiv.org/abs/2105.02465">PoseAug: A Differentiable Pose Augmentation Framework for 3D Human Pose Estimation</a></li>
<li><a href="https://arxiv.org/abs/2012.15370">OSTeC: One-Shot Texture Completion</a></li>
<li><a href="https://arxiv.org/abs/2104.03313">SCANimate: Weakly Supervised Learning of Skinned Clothed Avatar Networks</a></li>
<li><a href="https://arxiv.org/abs/2104.13682">HOTR: End-to-End Human-Object Interaction Detection with Transformers</a></li>
<li><a href="https://arxiv.org/abs/2105.09396">Birds of a Feather: Capturing Avian Shape Models From Images</a></li>
</ul>
<h4 id="vision--language">Vision & Language</h4>
<ul>
<li><a href="https://arxiv.org/abs/2102.06183">Less is More: ClipBERT for Video-and-Language Learning via Sparse Sampling</a></li>
<li><a href="https://arxiv.org/abs/2104.12836">Multimodal Contrastive Training for Visual Representation Learning</a></li>
<li><a href="https://arxiv.org/abs/2101.07396">ArtEmis: Affective Language for Visual Art</a></li>
<li><a href="https://arxiv.org/abs/2006.06666">VirTex: Learning Visual Representations From Textual Annotations</a></li>
<li><a href="https://arxiv.org/abs/2106.13156">Learning by Planning: Language-Guided Global Image Editing</a></li>
</ul>
<h4 id="datasets">Datasets</h4>
<ul>
<li><a href="https://arxiv.org/abs/2102.08981">Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Concepts</a></li>
<li><a href="https://arxiv.org/abs/2011.11015">Enriching ImageNet With Human Similarity Judgments and Psychological Embeddings</a></li>
<li><a href="https://arxiv.org/abs/2104.12690">Towards Good Practices for Efficiently Annotating Large-Scale Image Classification Datasets</a></li>
<li><a href="https://arxiv.org/abs/2105.08612">SAIL-VOS 3D: A Synthetic Dataset and Baselines for Object Detection and 3D Mesh Reconstruction From Video Data</a></li>
</ul>
<h4 id="explainable-ai--privacy">Explainable AI & Privacy</h4>
<ul>
<li><a href="https://arxiv.org/abs/2006.06634">Privacy-Preserving Image Features via Adversarial Affine Subspace Embeddings</a></li>
<li><a href="https://arxiv.org/abs/2012.09838">Transformer Interpretability Beyond Attention Visualization</a></li>
<li><a href="https://arxiv.org/abs/2006.03204">Black-box Explanation of Object Detectors via Saliency Maps</a></li>
</ul>
<h4 id="video-analysis-and-understanding">Video Analysis and Understanding</h4>
<ul>
<li><a href="https://arxiv.org/abs/2104.10386">Guided Interactive Video Object Segmentation Using Reliability-Based Attention Maps</a></li>
<li><a href="https://arxiv.org/abs/2103.07941">Modular Interactive Video Object Segmentation: Interaction-to-Mask, Propagation and Difference-Aware Fusion</a></li>
<li><a href="https://arxiv.org/abs/2105.06993">Omnimatte: Associating Objects and Their Effects in Video </a></li>
<li><a href="https://arxiv.org/abs/2101.08833">SSTVOS: Sparse Spatiotemporal Transformers for Video Object Segmentation</a></li>
</ul>Yassineouali.yasine@gmail.comThe 2021 CVPR conference concluded last week, with 1660 papers, 30 tutorials, 83 workshops. In this blog post, we'll take a quick look into the emerging trends at the conference by going through a portion of the accepted papers.ECCV 2020: Some Highlights2020-09-02T00:00:00+00:002020-09-02T00:00:00+00:00https://yassouali.github.io//ml-blog/eccv2020<p>The 2020 European Conference on Computer Vision took place online, from 23 to 28 August, and consisted of
1360 papers, divided into 104 orals, 160 spotlights and the rest of 1096 papers as posters.
In addition to 45 workshops and 16 tutorials. As it is the case in recent years with ML and CV conferences, the huge number of papers can be overwhelming at times. Similar to my <a href="https://yassouali.github.io/ml-blog/cvpr2020/">CVPR2020 post</a>, to get a grasp of the general trends of the conference this year, I will present in this blog post a sort of a snapshot of the conference by summarizing some papers (& listing some) that grabbed my attention.</p>
<p>First, some useful links:</p>
<ul>
<li>All of the papers can be found here: <a href="https://www.ecva.net/papers.php">ECCV Conference Papers</a></li>
<li>A list of available presentations on YT: <a href="https://crossminds.ai/category/eccv%202020/">Crossminds ECCV</a>. In addition to this <a href="https://www.youtube.com/playlist?list=PL6liSIqFR4BXnfg7-HM5-f7LGEKL1EDYb">YT playlist</a>.</li>
<li>One sentence description of all ECCV-2020 papers: <a href="https://www.paperdigest.org/2020/08/eccv-2020-highlights/">ECCV Paper Digest</a></li>
<li>ECCV virtual website: <a href="https://papers.eccv2020.eu/paper/160/">ECCV papers and presentations</a></li>
</ul>
<p><em>Disclaimer: This post is not a representation of the papers and subjects presented in ECCV 2020; it is just a personnel overview of what I found interesting. Any feedback is welcomed!</em></p>
<ul>
<li><a href="#general-statistics">General Statistics</a></li>
<li><a href="#recognition-detection-segmentation-and-pose-estimation">Recognition, Detection, Segmentation and Pose Estimation</a></li>
<li><a href="#semi-supervised-unsupervised-transfer-representation--few-shot-learning">Semi-Supervised, Unsupervised, Transfer, Representation & Few-Shot Learning</a></li>
<li><a href="#3d-computer-vision--robotics">3D Computer Vision & Robotics</a></li>
<li><a href="#image-and-video-synthesis">Image and Video Synthesis</a></li>
<li><a href="#vision-and-language">Vision and Language</a></li>
<li><a href="#the-rest">The Rest</a></li>
</ul>
<h1 id="general-statistics">General Statistics</h1>
<p>The statistics presented in this section are taken from the official Opening & Awards presentation. Let’s start
by some general statistics:</p>
<figure style="width: 65%" class="align-center">
<img src="https://yassouali.github.io//ml-blog/images/ECCV20/growth.png" />
<figcaption>Image source: Official Opening & Awards presentation.</figcaption>
</figure>
<figure style="width: 65%" class="align-center">
<img src="https://yassouali.github.io//ml-blog/images/ECCV20/acceptance.png" />
<figcaption>Image source: Official Opening & Awards presentation.</figcaption>
</figure>
<figure style="width: 70%" class="align-center">
<img src="https://yassouali.github.io//ml-blog/images/ECCV20/growth_review.png" />
<figcaption>Image source: Official Opening & Awards presentation.</figcaption>
</figure>
<p>The trends of earlier years continued with more than 200% increase in submitted papers compared to the 2018 conference, and with a similar number of papers to CVPR 2020. As expected, this increase is joined by a corresponding increase in the number of reviewers and area chairs to accommodate this expansion.</p>
<figure style="width: 100%" class="align-center">
<img src="https://yassouali.github.io//ml-blog/images/ECCV20/areas.png" />
<figcaption>Image source: Official Opening & Awards presentation.</figcaption>
</figure>
<p>As expected, the majority of the accepted papers focus on topics related to deep learning, recognition, detection, and understanding. Similar to CVPR 2020, we see an increasing interest in growing areas such as label-efficient methods (e.g., unsupervised learning) and low-level vision.</p>
<figure style="width: 100%" class="align-center">
<img src="https://yassouali.github.io//ml-blog/images/ECCV20/institutions.png" />
<figcaption>Image source: Official Opening & Awards presentation.</figcaption>
</figure>
<p>In terms of institutions; similar to ICML this year, Google takes the lead with 180 authors, followed by The Chinese University of Hong Kong with 140 authors and Peking University with 110 authors.</p>
<p>In the next sections, we’ll present some paper summaries by subject.</p>
<h1 id="recognition-detection-segmentation-and-pose-estimation">Recognition, Detection, Segmentation and Pose Estimation</h1>
<h4 id="end-to-end-object-detection-with-transformers-paper">End-to-End Object Detection with Transformers (<a href="https://arxiv.org/abs/2005.12872">paper</a>)</h4>
<p>The task of object detection consists of localizing and classifying objects visible given an input image.
The popular framework for object detection consist of pre-defining a set of boxes (ie., a set of geometric priors like <a href="https://arxiv.org/abs/1708.02002">anchors</a> or <a href="https://arxiv.org/abs/1506.01497">region proposals</a>), which are first classified, followed by a regression step to the adjust the dimensions of the predefined box, and then a post-processing step to remove duplicate predictions. However, this approach requires selecting a subset of candidate boxes to classify, and is not typically end-to-end differentiable.
In this paper, the authors propose <a href="https://github.com/facebookresearch/detr">DETR</a> (<strong>DE</strong>tection <strong>TR</strong>ansformer), an end-to-end
fully differentiable approach with no geometric priors. Bellow is a comparison of DETR and Faster R-CNN pipelines (image taken
from the authors presentation), highlighting the holistic nature of the approach.</p>
<figure style="width: 80%" class="align-center">
<img src="https://yassouali.github.io//ml-blog/images/ECCV20/detr_compare.png" />
<figcaption>Image source: <a href="https://arxiv.org/abs/2005.12872">Carion et al.</a></figcaption>
</figure>
<p>DETR is based on the encoder-decoder transformer architecture. The model consists of three components: the CNN feature extractor,
the encoder, and the decoder. A given image is first passed through the feature extractor to get image features. Then, positional encodings generated using sinusoids at different frequencies are added to the features to retain the 2D structure of the image. The resulting features are then passed through the transformer encoder to aggregate information across features and separate the object instances. For decoding, object queries are passed to the decoder with the encoded feature producing the output feature vectors. These queries are a fixed set of learned embeddings called object queries, which are randomly initialized embeddings that are learned during training then fixed during evaluation, and their number defines an upper bound on the number of objects the model can detect. Finally, the output feature vectors are fed through a (shared) fully connected layer to predict the class and bounding box for each query. To compute the loss and train the model, the outputs are matched with the ground truths with a one-to-one matching using the <a href="https://en.wikipedia.org/wiki/Hungarian_algorithm">Hungarian algorithm</a>.</p>
<figure style="width: 100%" class="align-center">
<img src="https://yassouali.github.io//ml-blog/images/ECCV20/detr.png" />
<figcaption>Image source: <a href="https://arxiv.org/abs/2005.12872">Carion et al.</a></figcaption>
</figure>
<h4 id="mutualnet-adaptive-convnet-via-mutual-learning-from-network-width-and-resolution-paper">MutualNet: Adaptive ConvNet via Mutual Learning from Network Width and Resolution (<a href="https://arxiv.org/abs/1909.12978">paper</a>)</h4>
<p>Traditional neural network can only be used if a specific amount of compute is available, and if the resource constraints are not met, the model becomes unusable. However, this can greatly limit the usage of the models in real applications. For example,
if the model is used for in-phone inference, the computational constrains are always changing depending on the load and the
phone’s battery charge. A simple solution is to keep several models of different sizes on the device, and use the
one with the corresponding constrains each time, but this requires a large amount of memory and cannot be scaled to
different constraints. Recent methods like <a href="https://arxiv.org/abs/1812.08928">S-Net</a> and <a href="[Universally Slimmable Networks and Improved Training Techniques](https://arxiv.org/abs/1903.05134)">US-Net</a> sample sub-networks during training so the model can be used at different width during deployment. But the performance drops dramatically with very low constraints.</p>
<figure style="width: 100%" class="align-center">
<img src="https://yassouali.github.io//ml-blog/images/ECCV20/MutualNet.png" />
<figcaption>Image source: <a href="https://arxiv.org/abs/1909.12978">Yang et al.</a></figcaption>
</figure>
<p>This paper proposes to leverage both the network scale and the input scale to find a good trade-off between the accuracy and
the computational efficiency. As illustrated above, for a given training iteration, four sub-networks are sampled, a full one
and three sub-networks with varying widths. The full network is trained on the original size of the image with the ground-truth labels using the standard cross-entropy loss, while the rest of the sub-networks are trained with randomly down-scaled version of the input image using KL divergence loss between their outputs and the output of the full network (ie., a distillation loss).
This way, each sub-network will be able to learn multi-scale representations from both the input scale and the network scale. During deployment, and given a specific resource constraint, the optimal combination of network scale and input scale can
be chosen for inference.</p>
<h4 id="gradient-centralization-a-new-optimization-technique-for-deep-neural-networks-paper">Gradient Centralization: A New Optimization Technique for Deep Neural Networks (<a href="https://arxiv.org/abs/2004.01461">paper</a>)</h4>
<p>Using second order statistics such as mean and variance during optimization to perform some form standardization of the activations or network’s weight, such as Batch norm or weight norm, have become an important component of neural network training. So, instead of operating on the weights or the activations with additional normalization modules, Gradient Centralization (GC) operates directly on gradients by centralizing the gradient vectors to have zero mean, which can smooth and accelerate the training process of neural networks and even improve the model generalization performance.</p>
<figure style="width: 80%" class="align-center">
<img src="https://yassouali.github.io//ml-blog/images/ECCV20/GC.png" />
<figcaption>Image source: <a href="https://arxiv.org/abs/2004.01461">Yong et al.</a></figcaption>
</figure>
<p>The GC operator, given the computed gradients, first computes the mean of the gradient vectors as illustrated above, then removes the
mean of them. Formally, for a weight vector \(\mathbf{w}_i\) whose gradient is \(\nabla_{\mathbf{w}_{i}} \mathcal{L}(i=1,2, \ldots, N)\), the GC operator \(\Phi_{G C}\) is defined as:</p>
\[\Phi_{G C}\left(\nabla_{\mathbf{w}_{i}} \mathcal{L}\right)=\nabla_{\mathbf{w}_{i}} \mathcal{L}-\frac{1}{M} \sum_{j=1}^{M} \nabla_{w_{i, j}} \mathcal{L}\]
<h3 id="smooth-ap-smoothing-the-path-towards-large-scale-image-retrieval-paper">Smooth-AP: Smoothing the Path Towards Large-Scale Image Retrieval (<a href="https://arxiv.org/abs/2007.12163">paper</a>)</h3>
<p>In image retrieval, the objective is to retrieve images of the same class as the query image from a large collection of
images. This tasks differs from classification where the classes encountered during testing were already seen during training, in image retrieval, we might get an image with a novel class and we need to fetch similar images, ie., an open set problem.
The general pipeline of image retrieval consists of extracting embeddings for the query image, and also embeddings for all of image collection using a CNN feature extractor, compute the cosine similarity score between each pair, then rank the images in the collection based on such a similarity.
The feature extractor is then trained to have a good ranking. The ranking performance is measured using <a href="https://en.wikipedia.org/wiki/Evaluation_measures_(information_retrieval)#Average_precision">Average Precision</a> (AP), computing the sum of the rank of each positive over its rank on the whole image collection. However, computing the ranking of a given image consists of a thresholding operation using a <a href="https://en.wikipedia.org/wiki/Heaviside_step_function">Heaviside step function</a>, making it non-differentiable, so we cannot train the model end-to-end to directly optimize the ranking.</p>
<figure style="width: 100%" class="align-center">
<img src="https://yassouali.github.io//ml-blog/images/ECCV20/smooth_ap.png" />
<figcaption>Image source: <a href="https://arxiv.org/abs/2007.12163">Brown et al.</a></figcaption>
</figure>
<p>To solve this, the authors proposed to replace the Heaviside step function with a smooth temperature controlled sigmoid, making
the ranking differentiable and useable as a loss function for end-to-end training. Compared to the triplet loss, the smooth-Ap loss optimizes a ranking loss, while the triplet loss is a surrogate loss that indirectly optimizes for a good ranking.</p>
<h4 id="hybrid-models-for-open-set-recognition-paper">Hybrid Models for Open Set Recognition (<a href="https://arxiv.org/abs/2003.12506">paper</a>)</h4>
<p>Existing image classification methods are often based on a closed-set assumption, ie., the training set covers
all possible classes that may appear in the testing phase. But this assumption is clearly unrealistic, given that
even with large scale datasets such as ImageNet with 1K classes, it is impossible to cover all possible real-world classes.
This where open-set classification comes, and tries to solves this by assuming that the test set contains
both known and unknown classes.</p>
<figure style="width: 100%" class="align-center">
<img src="https://yassouali.github.io//ml-blog/images/ECCV20/hybrid_model.png" />
<figcaption>Image source: <a href="https://arxiv.org/abs/2003.12506">Zhang et al.</a></figcaption>
</figure>
<p>In this paper, the authors use Flow-based model to tackle the problem of open-set classification. Flow-based are able to fit a probability distribution to training samples in an unsupervised manner via maximum likelihood estimation. The flow model can then be used to predict the probability density of each example. When the probability density of an input sample is large, it is likely to be part of the training distribution with a known class, and outliers will have a small density value. While previous methods stacked a classifier
on top of the flow model, the authors propose to learn a joint embedding for both the flow model and the classifier since the embedding space learned from only flow-based model may not have sufficient discriminative features for effective classification. As illustrated above,
during training, images are mapped into a latent feature space by the encoder, then the encoded features are fed into both the classifier trained with a cross-entropy loss, and the flow model for density estimation. The whole architecture is trained in an end-to-end manner. For testing, the \(\log p(x)\) of each image is computed and then compared with the lowest \(\log p(x)\) taken over the training set. If it is greater than the threshold, it is sent to the classifier to identify its specific known class, otherwise it is rejected as an unknown sample.</p>
<h4 id="conditional-convolutions-for-instance-segmentation-paper">Conditional Convolutions for Instance Segmentation (<a href="https://arxiv.org/abs/2003.05664">paper</a>)</h4>
<p>Instance segmentation remains as one of the challenging tasks in computer vision, requiring a per-pixel mask and a class label for each visible object in a given image. The dominant approach is <a href="https://arxiv.org/abs/1703.06870">Mask R-CNN</a> which consists of two steps, first, the object detector Faster R-CNN predicts a bounding box for each instance. Then, for each detected instance, the regions of interest are cropped from the output feature maps using ROI Align, resized to the same resolution, and are then fed into a mask head which is a small fully convolutional network used to predict the segmentation mask. However, the authors point out the following limitation with such an architecture; (1) the ROI Align might fetch irrelevant features belonging to the background or to other instances, (2) the resizing operation restricts the resolution of the instance segmentation, and (3) the mask head requires a stack of 3x3 convolutions to induce a large enough receptive field to predict the mask, which considerably increases the computational requirements of the mask head.</p>
<figure style="width: 80%" class="align-center">
<img src="https://yassouali.github.io//ml-blog/images/ECCV20/condinst.png" />
<figcaption>Image source: <a href="https://arxiv.org/abs/2003.05664">Tian et al.</a></figcaption>
</figure>
<p>In this paper, the authors propose to adapt FCNs used for semantic segmentation for instance segmentation. For effective instance segmentation, FCNs require two type of information, appearance information to categorize objects and location information to distinguish multiple objects belonging to the same category. The proposed network, called CondInst (conditional convolutions for instance segmentation), is a network based on <a href="https://arxiv.org/abs/1904.04971">CondConv</a> and <a href="https://arxiv.org/abs/1609.09106">HyperNetworks</a>, where for each instance, a sub-network will generate the mask FCN head’s weights conditioned on the center area of each instance, which are then used to predict the mask of the given instance. Specifically, as shown above, the network consists of multiple heads applied at multiple scales of the feature map. Each head predicts the class of a given instance at pre-defined positions, and the network’s weights to be used by the mask FCN head. Then the mask prediction is done using the parameters produced by each head.</p>
<h4 id="multitask-learning-strengthens-adversarial-robustness-paper">Multitask Learning Strengthens Adversarial Robustness (<a href="https://arxiv.org/abs/2007.07236">paper</a>)</h4>
<p>One of the main limitations of deep neural networks is their vulnerability to adversarial attacks, where very small and invisible perturbations are injected into the input, resulting in the wrong outputs, even if the appearance of the input remains the same. In recent years, the adversarial robustness of deep nets was rigorously investigated at different stages of the pipeline, from the input data (eg., using unlabeled data and adversarial training) to the model itself using regularization (eg., <a href="https://arxiv.org/abs/1704.08847">Parseval Networks</a>), but the outputs of the model are still not utilized to
improve the robustness of the model. In this paper, the authors investigate the effect of having multiple outputs for multi-task
learning on the robustness of the learned model, such a setting is useful since a growing number of machine learning applications call for
models capable of solving multiple tasks at once.</p>
<figure style="width: 80%" class="align-center">
<img src="https://yassouali.github.io//ml-blog/images/ECCV20/multitask_robustness.png" />
<figcaption>Image source: <a href="https://arxiv.org/abs/2007.07236">Mao et al.</a></figcaption>
</figure>
<p>Using p-norm ball bounded attack, where the adversarial perturbation is found within a p-norm ball for a given radius of a given input example. Then, the vulnerability computed as the total loss change. The authors showed an improved robustness when training on a pair of tasks (eg., two tasks are chosen from: segmentation, depth, normals, reshading, input reconstruction, 2d and 3d keypoints…).
The improved robustness is observed on single tasks attacks (ie., the perturbation is computed using one output) and multi tasks attacks (ie., the maximal perturbation of all the perturbations computed using all of outputs). The authors also theoretically showed that such a multi task robustness is only obtained if the tasks are correlated.</p>
<h4 id="dynamic-group-convolution-for-accelerating-convolutional-neural-networks-paper">Dynamic Group Convolution for Accelerating Convolutional Neural Networks (<a href="https://arxiv.org/abs/2007.04242">paper</a>)</h4>
<p>Group convolutions were first introduced in AlexNet to accelerate training, and subsequently adapted for efficient
CNNs such as <a href="https://arxiv.org/abs/1704.04861">MobileNet</a> and <a href="https://arxiv.org/abs/1707.01083">Shufflenet</a>. They consist of equally splitting the input and output channels in a convolution layer into mutually exclusive sections or groups while performing a normal convolution operation within each individual groups. So for \(G\)
groups, the computation is reduced by \(G\) times. However, the authors argue that they introduce two key limitations:
(1) they weaken the representation capability of the normal convolution by introducing sparse neuron connections,
and (2) they have fixed channel division regardless of the properties of each input.</p>
<figure style="width: 100%" class="align-center">
<img src="https://yassouali.github.io//ml-blog/images/ECCV20/DGC.png" />
<figcaption>Image source: <a href="https://arxiv.org/abs/2007.04242">Su et al.</a></figcaption>
</figure>
<p>In order to adaptively select the most related input channels for each group while keeping the full structure of the original networks, the authors propose dynamic group convolution (DGC).
DCG consists of two heads, in each head, there is a saliency score generator that assigns an importance score to each
channel. Using these scores, the channels with low importance scores are pruned. Then the normal convolution is conducted based on the selected subset of input channels generating the output channels in each head. Finally, the output channels from different heads are concatenated and shuffled.</p>
<h4 id="disentangled-non-local-neural-networks-paper">Disentangled Non-local Neural Networks (<a href="https://arxiv.org/abs/2006.06668">paper</a>)</h4>
<p>The <a href="https://arxiv.org/abs/1711.07971">non-local block</a> models long-range dependency between pixels using the attention mechanism, and has been widely used for numerous visual recognition tasks, such as object detection, semantic segmentation, and video action recognition.</p>
<figure style="width: 100%" class="align-center">
<img src="https://yassouali.github.io//ml-blog/images/ECCV20/disentangledNL.png" />
<figcaption>Image source: <a href="https://arxiv.org/abs/2006.06668">Yin et al.</a></figcaption>
</figure>
<p>In this paper, the authors try to better understand the non-local block, find its limitations, and propose an improved version. First, they start by reformulating the similarity between a pixel \(i\) (referred to as a key pixel) to pixel \(j\) (referred to as a query pixel) as the sum of two term: a pairwise term, which is a whitened dot product term representing the pure pairwise relation between query and key pixels, and a unary term, representing where a given key pixel has the same impact on all query pixels. Then, to understand the impact of each term, they train using either one, and find that pair-wise term is responsible for the category information, while the unary is responsible for boundary information. However, by analyzing the gradient of the non-local block, when the two are combined in the normal attention operator, their gradients are multiplied, so if the gradients of one of the two term is zero, the non-zero gradients of the other wont have any contribution. To solve this, the authors proposed a disentangled version of the non-local, where each term is optimized separately.</p>
<h4 id="hard-negative-examples-are-hard-but-useful-paper">Hard negative examples are hard, but useful (<a href="https://arxiv.org/abs/2007.12749">paper</a>)</h4>
<p>Deep metric learning optimizes an embedding function that maps semantically similar images to relatively nearby locations and maps semantically dissimilar images to distant locations. A popular way to learn the mapping is to define a loss function based on triplets of images: an anchor image, a positive image from the same class, and a negative image from a different class. The model is then penalized when the anchor is mapped closer to the negative image than it is to the positive image. However, during optimization, most triplet candidates already have the anchor much closer to the positive than the negative making them redundant. On the other hand, optimizing with the hardest negative examples leads to bad local minima in the early phase of the training. This is because in this case, the anchor-negative similarity is larger than the anchor-positive similarity as measured by the cosine similarity, ie., dot product between normalized feature vectors.</p>
<figure style="width: 70%" class="align-center">
<img src="https://yassouali.github.io//ml-blog/images/ECCV20/hard_negatives.png" />
<figcaption>Image source: <a href="https://arxiv.org/abs/2007.12749">Xuan et al.</a></figcaption>
</figure>
<p>The authors show that such problems with the usage of hard-negatives come from the standard implementation of the triplet loss. Specifically, (1) if the normalization is not considered during
the gradient computation, a large part of the gradient is lost, and (2) if two images of different classes are close by in the embedding space, the gradient of the loss might pull them closer instead of pushing them away. To solve this, instead of pulling the anchor-positive pair together to be tightly clustered as done in the standard triplet loss, the authors propose to avoid updating the anchor-positive pairs resulting in less tight clusters for a class of instances. This way the network focuses only on directly pushing apart the hard negative examples away from the anchor.</p>
<h4 id="volumetric-transformer-networks-paper">Volumetric Transformer Networks (<a href="https://arxiv.org/abs/2007.09433">paper</a>)</h4>
<p>One of the keys behind the success CNNs is their ability to learn discriminative feature representations of semantic object parts, which are very useful for computer vision tasks. However, CNNs still lacks the ability to handle various spatial variations, such as scale, view point and intra-class variations. Recent methods, such as <a href="https://arxiv.org/abs/1506.02025">spatial transformer networks</a> (STNs), try to suppress such variations by first wrapping the feature maps of spatially different images to a standard canonical configuration, then train classifiers on such standard features. But such methods apply the same wrapping to all the feature channels, which does not take into consideration the fact that the individual feature channels can represent different semantic parts, which may require different spatial transformations with respect to the canonical configuration.</p>
<figure style="width: 100%" class="align-center">
<img src="https://yassouali.github.io//ml-blog/images/ECCV20/VTN.png" />
<figcaption>Image source: <a href="https://arxiv.org/abs/2007.09433">Kim et al.</a></figcaption>
</figure>
<p>To solve this, the paper introduces Volumetric transformer network (VTN) shown above, a learnable module that predicts per channel and per spatial location wrapping transforms, which are used reconfigure the intermediate CNN features into a spatially agnostic and standard representations. VTN is an encoder-decoder network with modules dedicated to letting the information flow across the feature channels to account for the dependencies between the semantic parts.</p>
<h4 id="faster-autoaugment-learning-augmentation-strategies-using-backpropagation-paper">Faster AutoAugment: Learning Augmentation Strategies Using Backpropagation (<a href="https://arxiv.org/abs/1911.06987">paper</a>)</h4>
<p>Data augmentations (DA) have become a important and indispensable component of deep learning methods, and recent works
(eg., <a href="https://arxiv.org/abs/1805.09501">AutoAugment</a>, <a href="https://arxiv.org/abs/1905.00397">Fast AutoAugment</a> and <a href="https://arxiv.org/abs/1909.13719">RandAugment</a>) showed that augmentation strategies found by search algorithms outperform standard augmentations.
With a pre-defined set of possible transformations, such as geometric transformations like rotation or color enhancing transformations like solarization, the objective is to find the optimal data augmentation parameters, ie., the magnitude of the augmentation, the probability of applying it, and the number of transformations to combine as illustrated in the left figure below.
The optimal strategy is learned with a double optimization loop, so that the validation error of a given CNN trained with a given strategy is minimized. However, such an optimization method suffers from a large search space of possible policies, requiring sophisticated search strategies, and a single iteration of policy optimization requires the full training of the CNN. To solve this, the authors
propose to find the optimal strategy using density matching of original and augmented images with gradient based optimization.</p>
<figure style="width: 100%" class="align-center">
<img src="https://yassouali.github.io//ml-blog/images/ECCV20/faster_aug.png" />
<figcaption>Image source: <a href="https://arxiv.org/abs/1911.06987">Hataya et al.</a></figcaption>
</figure>
<p>By viewing DA as a way to fill missing points of original data, the objective then is to minimize the distance between the distributions of augmented data and the original data using adversarial learning, and in order to learn the optimal
augmentation strategy, the policy needs to be differentiable with respect to the parameters of the transformations.
For the probability of applying a given augmentation, the authors use a stochastic binary variable sampled from
a Bernoulli distribution, and optimized using the <a href="https://francisbach.com/the-gumbel-trick/">Gumbel trick</a>, while
the magnitude is approximate with a straight-through estimator and the combination are learned as a combination
of one-hot vectors.</p>
<h4 id="other-papers">Other Papers</h4>
<ul>
<li><a href="https://arxiv.org/abs/2003.08983">Metric learning: cross-entropy vs. pairwise losses</a></li>
<li><a href="https://arxiv.org/abs/2002.10120">Semantic Flow for Fast and Accurate Scene Parsing</a></li>
<li><a href="https://arxiv.org/abs/1909.11065">Object-Contextual Representations for Semantic Segmentation</a></li>
<li><a href="https://arxiv.org/abs/2001.01536">Learning From Multiple Experts: Self-paced Knowledge Distillation for Long-tailed Classification</a></li>
<li><a href="http://www.ecva.net/papers/eccv_2020/papers_ECCV/papers/123700664.pdf">Feature Normalized Knowledge Distillation for Image Classification</a></li>
<li><a href="http://www.ecva.net/papers/eccv_2020/papers_ECCV/papers/123550630.pdf">Mixup Networks for Sample Interpolation via Cooperative Barycenter Learning</a></li>
<li><a href="https://arxiv.org/abs/2007.09271">OnlineAugment: Online Data Augmentation with Less Domain Knowledge</a></li>
<li><a href="https://arxiv.org/abs/2007.09654">Distribution-Balanced Loss for Multi-Label Classification in Long-Tailed Datasets</a></li>
<li><a href="https://arxiv.org/abs/2004.13458">DiVA: Diverse Visual Feature Aggregation for Deep Metric Learning</a></li>
<li><a href="https://arxiv.org/abs/1911.10782">Estimating People Flows to Better Count Them in Crowded Scenes</a></li>
<li><a href="https://arxiv.org/abs/1912.11474">SoundSpaces: Audio-Visual Navigation in 3D Environments</a></li>
<li><a href="https://arxiv.org/abs/2003.08866">Spatially Adaptive Inference with Stochastic Feature Sampling and Interpolation</a></li>
<li><a href="https://arxiv.org/abs/2003.03780">DADA: Differentiable Automatic Data Augmentation</a></li>
<li><a href="https://arxiv.org/abs/2003.08979">URIE: Universal Image Enhancement for Visual Recognition in the Wild</a></li>
<li><a href="https://arxiv.org/abs/2007.11056">BorderDet: Border Feature for Dense Object Detection</a></li>
<li><a href="https://arxiv.org/abs/2008.08115">TIDE: A General Toolbox for Understanding Errors in Object Detection</a></li>
<li><a href="https://arxiv.org/abs/2007.09336">AABO: Adaptive Anchor Box Optimization for Object Detection via Bayesian Sub-sampling</a></li>
<li><a href="https://arxiv.org/abs/2007.09584">PIoU Loss: Towards Accurate Oriented Object Detection in Complex Environments</a></li>
<li><a href="http://arxiv.org/abs/2007.05676">Learning Object Depth from Camera Motion and Video Object Segmentation</a></li>
<li><a href="https://arxiv.org/abs/1908.01259">Attentive Normalization</a></li>
<li><a href="[https://www4.comp.polyu.edu.hk/~cslzhang/paper/conf/ECCV20/ECCV_MBN.pdf">Momentum Batch Normalization for Deep Learning with Small Batch Size </a></li>
<li><a href="https://arxiv.org/abs/2001.06057">A Simple Way to Make Neural Networks Robust Against Diverse Image Corruptions</a></li>
</ul>
<h1 id="semi-supervised-unsupervised-transfer-representation--few-shot-learning">Semi-Supervised, Unsupervised, Transfer, Representation & Few-Shot Learning</h1>
<h4 id="big-transfer-bit-general-visual-representation-learning-paper">Big Transfer (BiT): General Visual Representation Learning (<a href="https://arxiv.org/abs/1912.11370">paper</a>)</h4>
<p>In this paper, the authors revisit the simple paradigm of transfer learning: pre-train on a large amount of labeled source data (e.g., <a href="https://arxiv.org/abs/1707.02968">JFT-300M</a> and <a href="https://github.com/dmlc/mxnet-model-gallery/blob/master/imagenet-21k-inception.md">ImageNet-21k</a> datasets), then fine-tune the per-trained weights on the target tasks, reducing both the amount of data needed for target tasks and the fine-tuning time. The proposed framework is BiT (Big Transfer), and consists of a number of components that are necessary to build an effective network capable of leveraging large scale datasets and learning general and transferable representations.</p>
<p>On the (upstream) pre-training side, BiT consists of the following:</p>
<ul>
<li>For very large datasets, the fact that Batch Norm (BN) uses statistics from the training data during testing results in train/test discrepancy, where the training loss is correctly optimized while the validation loss is very unstable. In addition to the sensitivity of BN to the batch size. To solve this, BiT uses both <a href="https://arxiv.org/abs/1803.08494">Group Norm</a> and <a href="https://arxiv.org/abs/1602.07868">Weight Norm</a> instead of Batch Norm.</li>
<li>A small model such as ResNet 50 does not benefit from large scale training data, so the size of the model needs to also be correspondingly scaled up.</li>
</ul>
<p>For (down-stream) target tasks, BiT proposes the following:</p>
<ul>
<li>The usage of standard SGD, withoyt any layer freezing, dropout, L2-regularization or any adaptation gradients. In addition to initializing the last prediction layer to all 0’s.</li>
<li>Instead of resizing all of inputs to a fixed size, eg., 224. During training, the images are resized and cropped to a square with a randomly chosen size, and randomly h-flipped. At test time, the image is resized to a fixed size,</li>
<li>While <a href="https://arxiv.org/abs/1710.09412">mixup</a> is not useful for large scale pre-training given the abundance of data, BiT finds that mixup regularization can be very beneficial for mid-sized dataset used for downstream tasks.</li>
</ul>
<h4 id="learning-visual-representations-with-caption-annotations-paper">Learning Visual Representations with Caption Annotations (<a href="https://arxiv.org/abs/2008.01392">paper</a>)</h4>
<p>Training deep models on large scale annotated dataset results in not only a good performance on the task at hand, but also enables the model
to learn useful representation for downstream tasks. But can we obtain such useful features without such an expensive fine grained annotations?.
This paper investigates weakly-supervised pre-training using noisy labels, which are image captions in this case.</p>
<figure style="width: 80%" class="align-center">
<img src="https://yassouali.github.io//ml-blog/images/ECCV20/ICMLM.png" />
<figcaption>Image source: <a href="https://arxiv.org/abs/2008.01392">Sariyildiz et al.</a></figcaption>
</figure>
<p>With the objective of using a limited set of image-caption pairs to learn visual representations, how can the training objective be formulated to push for an effective
interaction between the images and their captions? Based on masked image modeling used in <a href="https://arxiv.org/abs/1810.04805">BERT</a> which randomly masks 15%
of the input tokens, and the model is then trained to reconstruct the masked input tokens using the encoder part of the <a href="https://arxiv.org/abs/1706.03762">transformer</a> model.
The paper proposed image-conditioned masked language modeling (ICMLM), where the images are leveraged to reconstruct the masked tokens of their corresponding captions.
To solve this objective, the authors proposes two multi modal architectures, (1) ICMLM tfm, where the image is encoded using a CNN, the masked caption using the
BERT model, the caption and image features are then concatenated and passed through a transformer encoder resulting in multi-modal embedding used to predict the masked token.
And (2), ICMLM att+fc, similarity, the caption and image features are first produced, then passed through a pair-wise attention block to aggregate the information between the caption and the image. The resulting features are then pooled and passed through a fully connected layer for masked token prediction.</p>
<h4 id="memory-augmented-dense-predictive-coding-for-video-representation-learning-paper">Memory-augmented Dense Predictive Coding for Video Representation Learning (<a href="https://arxiv.org/abs/2008.01065">paper</a>)</h4>
<p>The recent progress in self-supervised representation learning for images showed impressive results on down-stream tasks. However,
although multi-model representation learning for videos saw similar gains, self-supervision using video streams only, without any other modalities
such as text or audio, is still not as developed. Even if the temporal information of videos provide a free supervisory signal to train a model to predict the future states from the past in self-supervised manner. The task remains hard to solve since the exact future is not deterministic, and at a given time step, there many likely
and plausible hypotheses for future states (eg., when the action is “playing golf”, a future frame could have the hands and golf club in many possible positions).</p>
<figure style="width: 80%" class="align-center">
<img src="https://yassouali.github.io//ml-blog/images/ECCV20/MemDPC.png" />
<figcaption>Image source: <a href="https://arxiv.org/abs/2008.01065">Han et al.</a></figcaption>
</figure>
<p>This paper uses contrastive learning with a memory module to solve the issues with future prediction. To reduce the uncertainty, the model predicts the future
at the feature level, and is trained using a contrastive loss to avoid overstrict constrains. And to deal with multiple hypothesis, a memory module is used to infer multiple future states simultaneously. Given a set of successive frame, a 2d-3d CNN encoder (ie., \(f\)) produces context features and a GRU (ie., \(g\)) aggregates all the past information, which are then used to select slots from the shared memory module. A predicted future state is then produced as a convex combination of the selected memory slots. The predicted future state is then compared with true features vectors of the future states using a contrastive loss. For downstream tasks, the feature produced by \(g\) are pooled and then fed to the classifier.</p>
<h4 id="scan-learning-to-classify-images-without-labels-paper">SCAN: Learning to Classify Images without Labels (<a href="https://arxiv.org/abs/2005.12320">paper</a>)</h4>
<p>To group the unlabeled input images into semantically meaningful clusters, we need to find the solutions using the visual similarities alone. Prior work either, (1) learn
rich features with a self-supervised method, then applies k-means on the features to find the cluster, but this can lead to degeneracy quite easily. (2) end-to-end clustering approaches that either leverage CNNs features for deep clustering, or are based on mutual information maximization. However, the produced clusters depend heavily on the initialization
and is likely to latch into low-level features.</p>
<figure style="width: 90%" class="align-center">
<img src="https://yassouali.github.io//ml-blog/images/ECCV20/SCAN.png" />
<figcaption>Image source: <a href="https://arxiv.org/abs/2005.12320">Gansbeke et al.</a></figcaption>
</figure>
<p>To solve the issues found in prior work, the paper proposes SCAN (semantic clustering by adopting nearest neighbors) consiting of a two step procedure. In a first step, the feature representations are learned through a pretext task, then, to generate the initial cluster, SCAN mines the nearest neighbors of each image based on feature similarity instead of applying K-means. In a second step, the semantically meaningful nearest neighbors as are used as a prior to train the model to classify each image and its mined neighbors together. This is optimized using a loss function that maximizes their dot product after softmax, pushing the network to produce both consistent and discriminative (one-hot) predictions.</p>
<h4 id="gatcluster-self-supervised-gaussian-attention-network-for-image-clustering-paper">GATCluster: Self-Supervised Gaussian-Attention Network for Image Clustering (<a href="https://arxiv.org/abs/2002.11863">paper</a>)</h4>
<p>Clustering consists of separating data into clusters according to sample similarity. Traditional methods use hand-crafted features and domain specific distance
function to measure the similarity, but such hand crafted feature are very limited in expressiveness. Subsequent work leveraged deep representations with clustering algorithms, but the performance of deep clustering still suffers when the input data is complex. For an effective clustering, in terms of the features, they must contain both high-level
discriminative features, and capture object semantics. In terms of the clustering step, trivial solutions such as assigning all samples to a single or few clusters
must be avoided, and the clustering needs to be efficient to be applied to large-sized images.</p>
<p>The paper proposes GATCluster, which directly outputs semantic cluster labels without further post-processing, where the learned features are one-hot encoded vectors to guarantee the avoidance of trivial solutions. GATCluster is trained in an unsupervised manner with four self-learning tasks under the constraints of transformation invariance, separability maximization, entropy analysis, and attention mapping.</p>
<h4 id="associative-alignment-for-few-shot-image-classification-paper">Associative Alignment for Few-shot Image Classification (<a href="https://arxiv.org/abs/1912.05094">paper</a>)</h4>
<p>In few-shot image classification, the objective is to produce a model that can learn to recognize novel image classes when very few training examples are available. One of the popular approaches is Meta-learning that extracts common knowledge from a large amount of labeled data containing the base classes, and used to train a model. The model is then trained to classify images from novel concepts with only a few examples. The meta objective is to find a good set of initial weights that converge rapidly when trained on the new concepts. Interestingly, recent works demonstrated that standard transfer learning without meta learning, where a feature extractor is first pre-trained on the base classes, then fine-tunes a classifier on top of the pre-trained extractor on the new few examples performs on par with more sophisticated meta-learning strategies. However, the freezing of the extractor during fine-tuning that is necessary to avoid overfilling hinders the performances.</p>
<figure style="width: 90%" class="align-center">
<img src="https://yassouali.github.io//ml-blog/images/ECCV20/AssociativeAlignment.png" />
<figcaption>Image source: <a href="https://arxiv.org/abs/1912.05094">Afrasiyabi et al.</a></figcaption>
</figure>
<p>The paper proposes a two step approach to solve this. First, the feature extractor is used to produce features for the novel examples. The feature of each example is then
mapped to one of the base classes using a similarity metric in the embeddings space. The second step consists of associative alignment, where the feature extractor is fine-tuned so that the embeddings of the novel images are pushed closer to the embeddings of their corresponding bases images. This is done by either centroid alignment where the distance between the center of each base class and the novel classes is reduced, or adversarial alignment where a discriminator pushes the feature extractor to align the base and novel examples in the embedding space.</p>
<h4 id="other-papers-1">Other Papers</h4>
<ul>
<li><a href="https://arxiv.org/abs/2008.11911">Domain Adaptation through Task Distillation</a></li>
<li><a href="https://arxiv.org/abs/2003.12056">Are Labels Necessary for Neural Architecture Search?</a></li>
<li><a href="https://arxiv.org/abs/2008.10599">The Hessian Penalty: A Weak Prior for Unsupervised Disentanglement</a></li>
<li><a href="https://arxiv.org/abs/1906.01526">Cross-Domain Cascaded Deep Translation</a></li>
<li><a href="https://arxiv.org/abs/2007.02454">Self-Challenging Improves Cross-Domain Generalization</a></li>
<li><a href="https://arxiv.org/abs/2007.07695">Label Propagation with Augmented Anchors for UDA</a></li>
<li><a href="https://arxiv.org/abs/2007.07695">Regularization with Latent Space Virtual Adversarial Training</a></li>
<li><a href="https://www.ecva.net/papers/eccv_2020/papers_ECCV/papers/123490494.pdf">Transporting Labels via Hierarchical Optimal Transport for Semi-Supervised Learning</a></li>
<li><a href="https://arxiv.org/abs/2003.12060">Negative Margin Matters: Understanding Margin in Few-shot Classification</a></li>
<li><a href="https://arxiv.org/abs/2007.09584">Rethinking Few-Shot Image Classification: a Good Embedding Is All You Need?</a></li>
<li><a href="https://arxiv.org/abs/1911.10713">Prototype Rectification for Few-Shot Learning</a></li>
</ul>
<h1 id="3d-computer-vision--robotics">3D Computer Vision & Robotics</h1>
<h4 id="nerf-representing-scenes-as-neural-radiance-fields-for-view-synthesis-paper">NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis (<a href="https://arxiv.org/abs/2003.08934">paper</a>)</h4>
<p>3D view synthesis from 2D images is a challenging problem, especially if the input 2D images are sparsely sampled.
The goal is to train a model that takes a set of 2D images of a 3D scene (with the optional camera pose and its intrinsics),
then, using the trained model, we can render novel views of the 3D scene that were not found in the input 2D images.
A successful approach is voxed-based representations that represent the 3D scene on a discredited grid. Anf the 3D <a href="https://en.wikipedia.org/wiki/Voxel">voxel</a> of RGB-alpha grid values is predicted using a 3D CNN. However, such methods are memory inefficient since they scale cubically with the space resolution, can be hard to optimize and are not able to parametrize scene surfaces smoothly.
A recent trend in the computer vision community is to represent a given 3D scene as a continuous function using a fully-connected neural network. So the neural network itself is a compressed representation of the 3D scene, trained using the set of 2D images and then used to render novel views. Still, the existing methods were not able to match existing voxed-based methods.</p>
<figure style="width: 100%" class="align-center">
<img src="https://yassouali.github.io//ml-blog/images/ECCV20/NERF.png" />
<figcaption>Image source: <a href="https://arxiv.org/abs/2003.08934">Mildenhall et al.</a></figcaption>
</figure>
<p>NeRF (neural radiance fields) represents a scene as a continuous 5D function using a fully-connected network of 9 layers and 256 channels, whose input is a single continuous 5D coordinate, ie., 3D spatial locations (\(x\), \(y\), \(z\)) and the viewing directions (\(\theta\), \(\phi\)), and whose output is RGB color and opacity (output density).
To synthesize a given view, the rendering procedure consists of querying 5D coordinates along camera rays and use classic volume rendering techniques to project the output colors and densities into an image. Because volume rendering is naturally differentiable, the only input required to optimize the representation is a set of images with known camera poses. This way, NeRF is able to effectively optimize neural radiance fields to render photorealistic novel views of scenes with complicated geometry and appearance with a simple reconstruction loss between the rendered images and the ground truths, and demonstrates results that outperform prior work on neural rendering and view synthesis.</p>
<h4 id="towards-streaming-perception-paper">Towards Streaming Perception (<a href="https://arxiv.org/abs/2005.10420">paper</a>)</h4>
<p>Practical applications such as self-driving vehicles require fast reaction times similar to that of of humans, which is typically 200 milliseconds. In such settings, low-latency algorithms are required to ensure safe operation. However, even if the latency of computer vision algorithms is often studied, it have been primarily explored only in an offline setting. While Vision-for-online perception imposes quite different latency demands. Because by the time an algorithm finishes processing a particular image frame, say after 200ms, the surrounding world has changed as shown in the figure bellow. This forces the perception to be ultimately predictive of the future, which is a fundamental property of human vision (e.g., as required whenever a baseball player strikes a fast ball).</p>
<figure style="width: 70%" class="align-center">
<img src="https://yassouali.github.io//ml-blog/images/ECCV20/streaming_perception.png" />
<figcaption>Image source: <a href="https://arxiv.org/abs/2005.10420">Li et al.</a></figcaption>
</figure>
<p>To develope better benchmarks that reflects real-world scenarios and make comparing existing method easier. The paper introduces
the objective of streaming perception, ie., real-time online perception, and proposes a new meta-benchmark that systematically converts any image understanding task into a streaming image understand-ing task. This benchmark is built on a key insight: streaming perception requires understanding the state of the world at all time instants. So when a new frame arrives, streaming algorithms must report the state of the world even if they have not done processing the previous frame, forcing them to consider the amount of streaming data that should be ignored while the computation is occurring. Specifically, when comparing the model’s output and the ground truths, the alignment is done using time instead of the input index, so the model needs to give the correct prediction for time step \(t\) before the processing the corresponding input, ie., if the model takes \(\Delta t\) to process the inputs, it can
only use data before \(t - \Delta t\) to predict the output corresponding to the input at time \(t\).</p>
<h4 id="teaching-cameras-to-feel-estimating-tactile-physical-properties-of-surfaces-from-images-paper">Teaching Cameras to Feel: Estimating Tactile Physical Properties of Surfaces From Images (<a href="https://arxiv.org/abs/2004.14487">paper</a>)</h4>
<p>Humans are capable of forming a mental model at a young age that maps the perception of an object with a perceived sense of touch, which is based on previous experiences when interacting with different items. Having autonomous agents equipped with such a mental model can be a very valuable tool when interacting with novel objects, especially when a simple object class is not informative enough to accurately estimate tactile physical properties.</p>
<figure style="width: 90%" class="align-center">
<img src="https://yassouali.github.io//ml-blog/images/ECCV20/camera_feels.png" />
<figcaption>Image source: <a href="https://arxiv.org/abs/2004.14487">Purri et al.</a></figcaption>
</figure>
<p>In order to simulate such a mental model in a more direct manner, the paper proposes to estimate the physical properties directly, allowing attributes of objects to be utilized directly. First, the authors propose a dataset of 400+ surface image sequences and tactile property measurements. Since when estimating surface properties, people often unconsciously move their heads, acquiring multiple views of a surface, the captured images sequences comprise multiple viewing angles for each material surface.
Then, they propose a cross-modal framework for learning the complex mapping for visual cues to the tactile properties.
The training objective of the model is to generate precise tactile properties estimates given vision information. Both visual and tactile information are embedded into a shared latent space through separate encoder networks. A generator function then estimates tactile property values from the embedded visual vector. The discriminator network learns to predict whether a tactile-visual pair is a real or synthetic example. During inference, the encoder-generator pair is used to infer the tactile properties if the input images.</p>
<figure style="width: 80%" class="align-center">
<img src="https://yassouali.github.io//ml-blog/images/ECCV20/camera_feels_model.png" />
<figcaption>Image source: <a href="https://arxiv.org/abs/2004.14487">Purri et al.</a></figcaption>
</figure>
<h4 id="convolutional-occupancy-networks-paper">Convolutional Occupancy Networks (<a href="https://arxiv.org/abs/2003.04618">paper</a>)</h4>
<p>3D reconstruction is an important problem in computer vision with numerous applications. For an ideal representation of 3D geometry we need to be able to, a) encode complex geometries and arbitrary topologies, b) scale to large scenes, c) encapsulate local and global information, and d) be tractable in terms of memory and computation. However, existing representations for 3D reconstruction do not satisfy all of these requirements. While recent implicit neural representation have demonstrated impressive performances
in 3D reconstruction, they suffer from some limitation due to using a simple fully-connected network architecture which does not allow for integrating local information in the observations or incorporating inductive biases such as translational equivariance.</p>
<figure style="width: 90%" class="align-center">
<img src="https://yassouali.github.io//ml-blog/images/ECCV20/CON.png" />
<figcaption>Image source: <a href="https://arxiv.org/abs/2003.04618">Peng et al.</a></figcaption>
</figure>
<p>Convolutional Occupancy Networks uses convolutional encoders with implicit occupancy decoders to incorporates inductive biases and enabling structured reasoning in 3D space. Resulting in a more fine-grained implicit 3D reconstruction of single objects, with the ability to scale to large indoor scenes, and generalizes well from synthetic to real data.</p>
<h4 id="other-papers-2">Other Papers</h4>
<ul>
<li><a href="http://arxiv.org/abs/2008.01295">Tracking Emerges by Looking Around Static Scenes, with Neural 3D Mapping</a></li>
<li><a href="https://www.ecva.net/papers/eccv_2020/papers_ECCV/papers/123460324.pdf">Privacy Preserving Structure-from-Motion</a></li>
<li><a href="http://arxiv.org/abs/2007.07247">Multiview Detection with Feature Perspective Transformation</a></li>
<li><a href="https://arxiv.org/abs/2008.07931">Motion Capture from Internet Videos</a></li>
<li><a href="https://arxiv.org/abs/2003.10432">Atlas: End-to-End 3D Scene Reconstruction from Posed Images</a></li>
<li><a href="https://arxiv.org/abs/2006.12356">Generative Sparse Detection Networks for 3D Single-shot Object Detection</a></li>
<li><a href="https://arxiv.org/abs/2005.02138">PointTriNet: Learned Triangulation of 3D Point Sets</a></li>
<li><a href="https://arxiv.org/abs/2007.10453">Points2Surf: Learning Implicit Surfaces from Point Cloud Patches</a></li>
<li><a href="https://arxiv.org/abs/1912.03310">Geometric Capsule Autoencoders for 3D Point Clouds</a></li>
<li><a href="https://arxiv.org/abs/1803.00092">Deep Feedback Inverse Problem Solver</a></li>
<li><a href="https://arxiv.org/abs/2007.09529">Single View Metrology in the Wild</a></li>
<li><a href="https://arxiv.org/abs/2007.10982">Shape and Viewpoint without Keypoints</a></li>
<li><a href="https://arxiv.org/abs/2003.04232">Hierarchical Kinematic Human Mesh Recovery</a></li>
<li><a href="https://arxiv.org/abs/2007.13666">3D Human Shape and Pose from a Single Low-Resolution Image with Self-Supervised Learning</a></li>
<li><a href="https://arxiv.org/abs/2004.06302">Few-Shot Single-View 3D Object Reconstruction with Compositional Priors</a></li>
<li><a href="https://arxiv.org/abs/1912.03207">NASA: Neural Articulated Shape Approximation</a></li>
<li><a href="https://cse.buffalo.edu/~jsyuan/papers/2020/4836.pdf">Hand-Transformer: Non-Autoregressive Structured Modeling for 3D Hand Pose Estimation</a></li>
<li><a href="https://arxiv.org/abs/2007.15649">Perceiving 3D Human-Object Spatial Arrangements from a Single Image in the Wild</a></li>
</ul>
<h1 id="image-and-video-synthesis">Image and Video Synthesis</h1>
<h4 id="transforming-and-projecting-images-into-class-conditional-generative-networks-paper">Transforming and Projecting Images into Class-conditional Generative Networks (<a href="https://arxiv.org/abs/2005.01703">paper</a>)</h4>
<p>GaNs are capable of generating diverse images from different classes. For instance, <a href="https://arxiv.org/abs/1809.11096">BigGaN</a>, a class conditional GaN,
given a noise vector \(z\) and a class embeddings \(c\), the model is capable of generating a new image from that class. The image can then be manipulated by editing the latent variable of the noise vectors and class embedding. But is the inverse possible?, ie., given an input image, can we find the latent variable \(z\) and the class embedding \(c\) that best matches to the image? This problem remains challenging since many input images cannot be generated by a GaN. Additionally, the objective function have many local minimas, the search algorithms can get stuck in such regions easily.</p>
<figure style="width: 100%" class="align-center">
<img src="https://yassouali.github.io//ml-blog/images/ECCV20/pix2latent.png" />
<figcaption>Image source: <a href="https://arxiv.org/abs/2005.01703">Huh et al.</a></figcaption>
</figure>
<p>To address these problems, the paper proposes pix2latent with two new ideas: estimating input transformations at scale, and using a non-local search algorithm to find better solutions. As illustrated above, given an input image, pix2latent first finds the best transformation so that the transformed input is likely to be generated by a GaN, then the image is projected into the latent space using the proposed BasicCMA optimization method. The obtained latent variables are then edited, projected back into the image space obtaining an edited image, which can then be transformed with the inverse of the initial transformation</p>
<h4 id="contrastive-learning-for-unpaired-image-to-image-translation-paper">Contrastive Learning for Unpaired Image-to-Image Translation (<a href="https://arxiv.org/abs/2007.15651">paper</a>)</h4>
<p>Given two training sets of image pairs of different properties and modes, eg., images of horses and zebras, the objective of
unpaired image-to-image translation is to learn a translation function between the two modes, eg., transform horses to zebras and vice-versa, while retaining the sensible information such as pose or size without having access to a set of one-to-one matches between the two modes.
Existing methods such as <a href="https://arxiv.org/abs/1703.10593">CycleGaN</a> forces the model to have back translated images that are consistent with the original ones. But such methods assume a bijection, which is often too restrictive since a given translated image might have many plausible source images. An ideal loss should be invariant to different styles, but differentiate between sensitive
information.</p>
<figure style="width: 80%" class="align-center">
<img src="https://yassouali.github.io//ml-blog/images/ECCV20/contrastive_img2img.png" />
<figcaption>Image source: <a href="https://arxiv.org/abs/2007.15651">Park et al.</a></figcaption>
</figure>
<p>Contrastive Unpaired Translation (CUT) aims to learn such an embedding space. In addition to the standard GaN loss where the generator is trained to generate realistic translated images while the discriminator tries to differentiate between the translate images and real ones. An additional loss that pushes for similar embeddings between two corresponding patches from the input and translated image in used. Optimized with a contrastive objective which pulls the embeddings of the two corresponding patches, while pushing away the embedding of a give patch and its negatives which are randomly sampled patches (ie., only internal patches from the same input image are used, external ones from other images decrease the performances).</p>
<h4 id="rewriting-a-deep-generative-model-paper">Rewriting a Deep Generative Model (<a href="https://arxiv.org/abs/2007.15646">paper</a>)</h4>
<p>GAN are capable of modeling a rich set of semantic and physical rules about the data distribution, but up to now, it has been obscure how such rules are encoded in the network, or how a rule could be changed. This paper introduces a new problem setting: manipulation of specific rules encoded by a deep generative model. So given a generative model, the objective is to adjust its weights, so that
the new and modified model follows new rules, and generates images that follow the new set of rules as shown bellow.</p>
<figure style="width: 80%" class="align-center">
<img src="https://yassouali.github.io//ml-blog/images/ECCV20/RewritingGaNs.png" />
<figcaption>Image source: <a href="https://arxiv.org/abs/2007.15646">Bau et al.</a></figcaption>
</figure>
<p>By viewing each layer as an associative memory that stores latent rules as a set of key-value relationships over hidden features. The model can be edited by defining a constrained optimization that adds or edits one specific rule within the associative memory while preserving the existing semantic relationships in the model as much as possible. The papers does this directly by measuring and manipulating the model’s internal structure, without requiring any new training data.</p>
<h4 id="learning-stereo-from-single-images-paper">Learning Stereo from Single Images (<a href="https://arxiv.org/abs/2008.01484">paper</a>)</h4>
<p>Given a pair of corresponding images, the goal of stereo matching is to estimate the per-pixel horizontal displacement (i.e. disparity) between the corresponding location of every pixel from the first view to the second, or vice-versa. While fully supervised methods give good results, the precise ground truth disparity between a pair of stereo images is often hard to acquire.
A possible alternative is to train on synthetic data, then fine-tune on a limited amount of real labeled data. But without
a fine-tuning step with enough labels, such model are not capable of generating well to real images.</p>
<figure style="width: 100%" class="align-center">
<img src="https://yassouali.github.io//ml-blog/images/ECCV20/LearningStereo.png" />
<figcaption>Image source: <a href="https://arxiv.org/abs/2008.01484">Watson et al.</a></figcaption>
</figure>
<p>The paper proposed a novel and fully automatic pipeline for generating stereo training data from unstructured collections of single images given a depth-from-color model, requiring no synthetic data or stereo pairs of images to train. Using a depth estimation network. First, a given left input image is converted into a synthesized right image by a forward wrapping operation using the depth disparity. Then, with stereo pairs of images, the stereo network can then be trained in a supervised manner, resulting in a well generalizable model.</p>
<h4 id="what-makes-fake-images-detectable-understanding-properties-that-generalize-paper">What makes fake images detectable? Understanding properties that generalize (<a href="https://arxiv.org/abs/2008.10588">paper</a>)</h4>
<p>Although the the quality of GaN generated images is reaching impressive levels, deep networks trained to detect fake images can still pick up on the subtle artifacts in these generated images, and such trained networks can also find the same artifacts across
many models trained on different dataset and with different methods. This paper aims to visualize and understand which artifacts are shared between models and are easily detectable and transferable across different scenarios.</p>
<figure style="width: 100%" class="align-center">
<img src="https://yassouali.github.io//ml-blog/images/ECCV20/fakeimages.png" />
<figcaption>Image source: <a href="https://arxiv.org/abs/2008.10588">Chai et al.</a></figcaption>
</figure>
<p>Since the global facial structure can vary among different generators and datasets, local patches of the generated images are more stereotyped and may share redundant artifacts. To this end, a fully-convolutional patch-based classifier is used to focus on local patches rather than global structure. The path level classifier can then be used to visualize and categorize the patches that are most indicative of real or fake images across various test datasets. Additionally, the generated image can be manipulated to exaggerate characteristic attributes of fake images.</p>
<h4 id="other-papers-3">Other Papers</h4>
<ul>
<li><a href="https://arxiv.org/abs/2008.05511">Free View Synthesis</a></li>
<li><a href="http://arxiv.org/abs/2007.15068">Unselfie: Translating Selfies to Neutral-pose Portraits in the Wild</a></li>
<li><a href="https://arxiv.org/abs/2007.08509">World-Consistent Video-to-Video Synthesis</a></li>
<li><a href="http://arxiv.org/abs/2007.08513">RetrieveGAN: Image Synthesis via Differentiable Patch Retrieval</a></li>
<li><a href="https://arxiv.org/abs/1912.02401">Generating Videos of Zero-Shot Compositions of Actions and Objects</a></li>
<li><a href="https://arxiv.org/abs/2007.15649">Perceiving 3D Human-Object Spatial Arrangements from a Single Image in the Wild</a></li>
<li><a href="http://arxiv.org/abs/2003.08872">Across Scales & Across Dimensions: Temporal Super-Resolution using Deep Internal Learning</a></li>
<li><a href="https://arxiv.org/abs/2008.09180">Conditional Entropy Coding for Efficient Video Compression</a></li>
<li><a href="https://arxiv.org/abs/2008.10598">Semantic View Synthesis</a></li>
<li><a href="https://arxiv.org/abs/2008.09370">Learning Camera-Aware Noise Models</a></li>
<li><a href="https://arxiv.org/abs/2004.00049">In-Domain GAN Inversion for Real Image Editing</a></li>
</ul>
<h1 id="vision-and-language">Vision and Language</h1>
<h4 id="connecting-vision-and-language-with-localized-narratives-paper">Connecting Vision and Language with Localized Narratives (<a href="https://arxiv.org/abs/1912.03098">paper</a>)</h4>
<p>One of the popular ways for connecting vision and language is image captioning, where each image is paired with human authored textual captions, but this link is only at the full image scale
where the sentences describes the whole image.
To improve this linking, grounded images captioning adds links between specific parts of the image caption and object boxes
in the image. However, the links are still very sparse and the majority of objects and words are not grounded and the annotation process if expensive.</p>
<figure style="width: 80%" class="align-center">
<img src="https://yassouali.github.io//ml-blog/images/ECCV20/LocalizedNarratives.png" />
<figcaption>Image source: <a href="https://arxiv.org/abs/1912.03098">Pont-Tuset et al.</a></figcaption>
</figure>
<p>The paper proposes a new and efficient form of multi-modal image annotations for connecting vision and language called Localized Narratives. Localized Narratives are generated by asking the annotators to describe an image with their voice while simultaneously hovering their mouse over the region they are describing. For instance, as shown in the figure above, the annotator says “woman” while using their mouse to indicate her spatial extent, thus providing visual grounding for this noun. Later they move the mouse from the woman to the balloon following its string, saying “holding”. This provides direct visual grounding of this relation. They also describe attributes like “clear blue sky” and “light blue jeans”. Since voice is synchronized to the mouse pointer, the image location of every single word in the description can be determined. This provides dense visual grounding in the form of a mouse trace segment for each word.
This rich from of annotation with multiple modalities (ie., image, text, speech and grounding) can be used to for different tasks such as text-to-image generation, visual question answering and voice-driven environnement navigation. Or for a more fine-grained
control of tasks, such conditioning captions on specific parts of the image, which can be used by a person with imperfect vision
to get descriptions of specific parts by hovering their finger on the image.</p>
<h4 id="uniter-universal-image-text-representation-learning-paper">UNITER: UNiversal Image-TExt Representation Learning (<a href="https://arxiv.org/abs/1909.11740">paper</a>)</h4>
<p>Most Vision-and-Language (V&L) tasks such Visual Question Answering (VQA) rely on joint multi modal embeddings to bridge the semantic gap between visual and textual clues in images and text. But such representations are usually tailored for specific tasks, and require specific architectures. In order to learn general joint embeddings that
can be used on all of the V&L downstream tasks. The paper introduces UNITER, a large-scale pre-trained model for joint multimodal embedding illustrated bellow. Based on the transformer model, UNITER is pre-trained on 4 tasks: Masked Language Modeling (MLM) conditioned on image, where the randomly masked words are recovered using both image and text features. Masked Region Modeling (MRM) conditioned on text, where the model reconstructs some regions of a given image. Image-Text Matching (ITM), where the model predicts if an image and a text instances are paired or not. And Word-Region Alignment (WRA), where the model learn the optimal alignment between words and images found using optimal transport. To use UNITER on downstream tasks, first they are reformulated as a classification problem, then the added classifier on top of the [CLS] features can be trained using a cross-entropy loss.</p>
<figure style="width: 100%" class="align-center">
<img src="https://yassouali.github.io//ml-blog/images/ECCV20/UNITER.png" />
<figcaption>Image source: <a href="https://arxiv.org/abs/1909.11740">Chen et al.</a></figcaption>
</figure>
<h4 id="learning-to-learn-words-from-visual-scenes-paper">Learning to Learn Words from Visual Scenes (<a href="https://arxiv.org/abs/1911.11237">paper</a>)</h4>
<p>The standard approach in vision and language consists of learning a common embedding space, however this approach is inefficient, often requiring millions of examples to learn, generalizes poorly to the natural compositional structure of language, and the learned embeddings are unable to adapt to novel words at inference time. So instead of learning the word embeddings, this paper proposes to learn the process for acquiring word embeddings.</p>
<figure style="width: 100%" class="align-center">
<img src="https://yassouali.github.io//ml-blog/images/ECCV20/l2lwords.png" />
<figcaption>Image source: <a href="https://arxiv.org/abs/1911.11237">Surís et al.</a></figcaption>
</figure>
<p>The model is based on the transformer model, and at each iteration, the model receives an episode of image and language pairs, and then meta-learns a policy to acquire word representations from the episode. This produces a representation that is able to acquire novel words at inference time as well as more robustly generalize to novel compositions. Specifically, every tasks is formulated as a language acquisition task or an episode, consisting of training examples and a testing examples, where the testing examples evaluates
the language acquired from the training examples. In this figure above for instance, the model needs to acquire the word “chair” from the training samples, a word which it has never seen before. The meta-training is done in the forward pass, where the model needs to point to the correct word, “chair”, in the training example, and a matching loss is used to train the model.
After training on many episodes and tasks, the model is able to adapt very quickly to novel task during inference.</p>
<h4 id="other-papers-4">Other Papers</h4>
<ul>
<li><a href="https://arxiv.org/abs/2006.09920">Contrastive Learning for Weakly Supervised Phrase Grounding</a></li>
<li><a href="https://arxiv.org/abs/2004.02857">Beyond the Nav-Graph: Vision-and-Language Navigation in Continuous Environments</a></li>
<li><a href="http://www.ecva.net/papers/eccv_2020/papers_ECCV/papers/123610052.pdf">Adaptive Text Recognition through Visual Matching</a></li>
</ul>
<h2 id="the-rest">The Rest</h2>
<p>Unfortunately, the number of papers makes the summarization task difficult and time consuming. So for the rest of the papers, I will simply list some papers I came across in case the the reader is interested in the subjects.</p>
<details>
<summary>Click to expand</summary> <br />
<small> <div class="tip">
<p><strong>Deep Learning: Applications, Methodology, and Theory</strong>:</p>
<ul>
<li><a href="https://arxiv.org/abs/2007.09748">A Generic Visualization Approach for Convolutional Neural Networks</a></li>
<li><a href="https://arxiv.org/abs/2003.06696">Spike-FlowNet: Event-based Optical Flow Estimation</a></li>
<li><a href="https://arxiv.org/abs/2003.08505">A Metric Learning Reality Check</a></li>
<li><a href="https://arxiv.org/abs/1912.12773">Learning Predictive Models from Observation and Interaction</a></li>
<li><a href="https://arxiv.org/abs/2008.09269">Beyond Fixed Grid: Learning Geometric Image Representation with a Deformable Grid</a></li>
<li><a href="https://arxiv.org/abs/2008.05441">Stable Low-rank Tensor Decomposition for Compression of Convolutional Neural Network</a></li>
<li><a href="https://arxiv.org/abs/2007.02491">EagleEye: Fast Sub-net Evaluation for Efficient Neural Network Pruning</a></li>
<li><a href="https://arxiv.org/abs/2008.01777">Making Sense of CNNs: Interpreting Deep Representations & Their Invariances with INNs</a></li>
<li><a href="https://arxiv.org/abs/2003.09148">Event-based Asynchronous Sparse Convolutional Networks</a></li>
</ul>
<p><strong>Low level vision, Motion and Tracking</strong>:</p>
<ul>
<li><a href="https://arxiv.org/abs/2003.12039">RAFT: recurrent all pairs field transforms for optical flow</a></li>
<li><a href="https://arxiv.org/abs/2005.01616">VisualEchoes: Spatial Image Representation Learning through Echolocation</a></li>
<li><a href="https://arxiv.org/abs/2008.04237">Self-Supervised Learning of Audio-Visual Objects from Video</a></li>
<li><a href="https://arxiv.org/abs/2004.01177">Tracking Objects as Points</a></li>
</ul>
<p><strong>Face, Gesture, and Body Pose</strong>:</p>
<ul>
<li><a href="https://arxiv.org/abs/2007.12553">Style Transfer for Co-Speech Gesture Animation: A Multi-Speaker Conditional-Mixture Approach</a></li>
<li><a href="https://arxiv.org/abs/2007.09355">Thinking in Frequency: Face Forgery Detection by Mining Frequency-aware Clues</a></li>
<li><a href="https://arxiv.org/abs/2003.09764">Lifespan Age Transformation Synthesis</a></li>
<li><a href="https://arxiv.org/abs/2008.09062">Monocular Expressive Body Regression through Body-Driven Attention</a></li>
<li><a href="https://arxiv.org/abs/2003.08386">DLow: Diversifying Latent Flows for Diverse Human Motion Prediction</a></li>
<li><a href="https://arxiv.org/abs/2008.10174">Fast Bi-layer Neural Synthesis of One-Shot Realistic Head Avatars</a></li>
<li><a href="https://arxiv.org/abs/2008.00418">Blind Face Restoration via Deep Multi-scale Component Dictionaries</a></li>
</ul>
<p><strong>Action Recognition, Understanding</strong>:</p>
<ul>
<li><a href="http://www.ecva.net/papers/eccv_2020/papers_ECCV/papers/-123640494.pdf">RubiksNet: Learnable 3D-Shift for Efficient Video Action Recognition</a></li>
<li><a href="https://arxiv.org/abs/2008.05861">Self-supervised Video Representation Learning by Pace Prediction</a></li>
<li><a href="https://arxiv.org/abs/2007.04515">Aligning Videos in Space and Time</a></li>
<li><a href="https://arxiv.org/abs/-1911.10967">Forecasting Human-Object Interaction: Joint Prediction of Motor Attention and Actions in First Person Video</a></li>
<li><a href="https://arxiv.org/abs/2007.10984">Foley Music: Learning to Generate Music from Videos</a></li>
</ul>
</div></small></details>Yassineouali.yasine@gmail.comThe European Conference on Computer Vision (ECCV) 2020 ended last week. This year's online conference contained 1360 papers, with 104 as orals, 160 as spotlights and the rest as posters. In addition to 45 workshops and 16 tutorials. In this blog post, I'll summarize some papers I've read and list the ones that've caught my attention.CVPR 2020: A Snapshot2020-06-21T00:00:00+00:002020-06-21T00:00:00+00:00https://yassouali.github.io//ml-blog/cvpr2020<p>The first virtual CVPR conference ended, with 1467 papers accepted, 29 tutorials, 64 workshops, and 7.6k virtual attendees. The huge number of papers and the new virtual version made navigating the conference overwhelming (and very slow) at times. To get a grasp of the general trends of the conference this year, I will present in this blog post a sort of a snapshot of the conference by summarizing some papers (& listing some) that grabbed my attention.</p>
<ul>
<li>All of the papers can be found here: <a href="http://openaccess.thecvf.com/CVPR2020.py">CVPR2020 open access</a></li>
<li>CVPR virtual website: <a href="http://cvpr20.com/">CVPR2020 virtual</a></li>
</ul>
<p><em>Disclaimer: This post is not a representation of the papers and subjects presented in CVPR; it is just a personnel overview of what I found interesting. Any feedback is welcomed!</em></p>
<ul>
<li><a href="#cvpr-2020-in-numbers">CVPR 2020 in numbers</a></li>
<li><a href="#recognition-detection-and-segmentation">Recognition, Detection and Segmentation</a></li>
<li><a href="#generative-models-and-image-synthesis">Generative models and image synthesis</a></li>
<li><a href="#representation-learning">Representation Learning</a></li>
<li><a href="#computational-photography">Computational photography</a></li>
<li><a href="#transferlow-shotsemiunsupervised-learning">Transfer/Low-shot/Semi/Unsupervised Learning</a></li>
<li><a href="#vision-and-language">Vision and Language</a></li>
<li><a href="#the-rest">The rest</a></li>
</ul>
<h1 id="cvpr-2020-in-numbers">CVPR 2020 in numbers</h1>
<p>The statistics presented in this section are taken from the official <a href="https://www.youtube.com/watch?v=aHUYXtbwl_8&t=1903s">Opening & Awards</a> presentation. Let’s start
by some general statistics:</p>
<figure style="width: 100%" class="align-center">
<img src="https://yassouali.github.io//ml-blog/images/CVPR20/participants.png" />
<figcaption>Image source: <a href="https://www.youtube.com/watch?v=aHUYXtbwl_8&t=1903s">Opening & Awards presentation.</a></figcaption>
</figure>
<figure style="width: 100%" class="align-center">
<img src="https://yassouali.github.io//ml-blog/images/CVPR20/acceptance.png" />
<figcaption>Image source: <a href="https://www.youtube.com/watch?v=aHUYXtbwl_8&t=1903s">Opening & Awards presentation.</a></figcaption>
</figure>
<p>The trends of earlier years continued with a 20% increase in authors and a 29% increase in submitted papers, joined by rising the number of reviewers and area chairs to accommodate this expansion.</p>
<figure style="width: 100%" class="align-center">
<img src="https://yassouali.github.io//ml-blog/images/CVPR20/authors_distribution.png" />
<figcaption>Image source: <a href="https://www.youtube.com/watch?v=aHUYXtbwl_8&t=1903s">Opening & Awards presentation.</a></figcaption>
</figure>
<p>Similar to last year, China is the first contributor to CVPR in terms of accepted papers with Tsinghua University with the most significant number of authors, followed by the USA as the second contributor by country and Google by organization.</p>
<figure style="width: 100%" class="align-center">
<img src="https://yassouali.github.io//ml-blog/images/CVPR20/subject_areas.png" />
<figcaption>Image source: <a href="https://www.youtube.com/watch?v=aHUYXtbwl_8&t=1903s">Opening & Awards presentation.</a></figcaption>
</figure>
<p>As expected, the majority of the accepted papers focus on topics related to learning, recognition, detection, and understanding. However, there is an increasing interest in relatively new areas such as label-efficient methods (e.g., transfer learning), image synthesis and robotic perception. Some emerging topics like fairness and explain AI are also starting to gather more attention within the computer vision community.</p>
<h1 id="recognition-detection-and-segmentation">Recognition, Detection and Segmentation</h1>
<h4 id="pointrend-image-segmentation-as-rendering-paper">PointRend: Image segmentation as rendering (<a href="https://arxiv.org/abs/1912.08193">paper</a>)</h4>
<p>Image segmentation models, such as <a href="https://arxiv.org/abs/1703.06870">Mask R-CNN</a>, typically operate on regular grids: the input image is a regular grid of pixels, their hidden representations are feature vectors on a regular grid, and their outputs are label maps on a regular grid. However, a regular grid will unnecessarily over sample the smooth areas while simultaneously undersampling object boundaries, often resulting in blurry contours, as illustrated in the right figure below.</p>
<figure style="width: 100%" class="align-center">
<img src="https://yassouali.github.io//ml-blog/images/CVPR20/pointrend.png" />
<figcaption>Image source: <a href="https://arxiv.org/abs/1912.08193">Kirillov et al.</a></figcaption>
</figure>
<p>The paper proposes to view image segmentation as a rendering problem and adapt classical ideas from computer graphics to render high-quality label maps efficiently. This is done using a neural network module called PointRend. PointRend takes as input a given number of CNN feature maps that are defined over regular grids and outputs high-resolution predictions over a finer grid. These fine predictions are only made in carefully selected points, chosen to be near high-frequency areas such as object boundaries where we have uncertain predictions (i.e., similar to <a href="http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.107.3997&rep=rep1&type=pdf">adaptive subdivision</a>), which are then upsampled and a small subhead is used to make the prediction from such point-wise features.</p>
<h4 id="self-training-with-noisy-student-improves-imagenet-classification-paper">Self-training with Noisy Student improves ImageNet classification (<a href="https://arxiv.org/abs/1911.04252">paper</a>)</h4>
<p>Semi-supervised learning methods work quite well in a low-data regime, but with a large number of labeled data, fully-supervised learning still works best.
In this paper, the authors revisit this assumption and show that noisy self-training works well, even when labeled data is abundant.</p>
<figure style="width: 60%" class="align-center">
<img src="https://yassouali.github.io//ml-blog/images/CVPR20/noisy-self-training.png" />
<figcaption>Image source: <a href="https://arxiv.org/abs/1911.04252">Xie et al.</a></figcaption>
</figure>
<p>The method used a large corpus of unlabeled images (i.e., different than ImageNet training set distribution), and consists of three main steps. First, a teacher model is trained on the labeled images, the trained teacher is then used to generate pseudo-labels on the unlabeled images, which are then used to train a student model on the combination of labeled images and pseudo-labeled images, the student model is larger than the teacher (e.g., starting with <a href="https://arxiv.org/pdf/1905.11946.pdf">EfficientNetB0</a> then EfficientNetB3) and is trained with an injected noise (e.g., dropout). The student is then considered as a teacher, and the last two-step are repeated a few times to relabel the unlabeled data and training a new student. The last model achieves SOTA on ImageNet top-1 and shows a higher degree of robustness.</p>
<h4 id="designing-network-design-spaces-paper">Designing network design spaces (<a href="https://arxiv.org/abs/2003.13678">paper</a>)</h4>
<p>Instead of focusing on designing individual network instances, this paper focuses on designing network design spaces that parametrize populations of networks,
in order to find some guiding design principals for fast and simple networks.</p>
<figure style="width: 70%" class="align-center">
<img src="https://yassouali.github.io//ml-blog/images/CVPR20/designspaces.png" />
<figcaption>Image source: <a href="https://arxiv.org/abs/2003.13678">Radosavovic et al.</a></figcaption>
</figure>
<p>The proposed method focuses on finding a good model population instead of good model instances (e.g., natural architecture search). Based on the comparison paradigm of <a href="https://arxiv.org/abs/1905.13214">distribution estimates</a>, the process consists of initializing a design space A, followed by introducing a new design principle to obtain a new and refined design space B, containing simpler and better models. The process is repeated until the resulting population consists of models that are more likely to be robust and generalize well.</p>
<h4 id="efficientdet-scalable-and-efficient-object-detection-paper">EfficientDet: Scalable and Efficient Object Detection (<a href="https://arxiv.org/abs/1911.09070">paper</a>)</h4>
<p>EfficientDet is a model with STOA in object detection, with better efficiency across a wide range of resource constraints.</p>
<figure style="width: 100%" class="align-center">
<img src="https://yassouali.github.io//ml-blog/images/CVPR20/EfficientDet.png" />
<figcaption>Image source: <a href="https://arxiv.org/abs/1911.09070">Tan et al.</a></figcaption>
</figure>
<p>The model’s architecture with an <a href="https://arxiv.org/abs/1905.11946">EfficientNet</a> backbone consists of two new design choices: a bidirectional Feature Pyramid Network (<a href="https://arxiv.org/abs/1612.03144">FPN</a>) with a bidirectional topology, or BiFPN, and using learned weights when merging the features from different scales. Additionally, the network is designed with compound scaling, where the backbone, class/box network and input resolution are jointly adapted to meet a wide spectrum of resource constraints, instead of simply employing bigger backbone networks as done in previous works.</p>
<h4 id="dynamic-convolution-attention-over-convolution-kernels-paper">Dynamic Convolution: Attention Over Convolution Kernels (<a href="https://arxiv.org/abs/1912.03458">paper</a>)</h4>
<p>One of the main problems with light-weight CNNs, such as <a href="https://arxiv.org/abs/1801.04381">MobileNetV2</a>, is their limited representation capability due to the constrained depth (i.e., number of layers) and width (i.e., number of channels) to maintain low computational requirements. In this paper, the authors propose dynamic convolutions to boost the capability of the convolution layers by aggregating the results of multiple parallel convolutions with attention weights, without increasing the computation significantly.</p>
<figure style="width: 70%" class="align-center">
<img src="https://yassouali.github.io//ml-blog/images/CVPR20/dynamicconvs.png" />
<figcaption>Image source: <a href="https://arxiv.org/abs/1912.03458">Chen et al.</a></figcaption>
</figure>
<p>Dynamic convolutions consist of applying K convolution kernels that share the same kernel size and input/output dimensions instead of a single operation, their results are then aggregated using attention weights produced with small attention module. For faster training, the kernel weights are constrained to triangles where each attention weights are in range [0, 1] and their sum equal to one.</p>
<h4 id="polarmask-single-shot-instance-segmentation-with-polar-representation-paper">PolarMask: Single Shot Instance Segmentation with Polar Representation (<a href="https://arxiv.org/abs/1909.13226">paper</a>)</h4>
<p>PolarMask proposes to represent the masks for each detected object in an instance segmentation task using polar coordinates. Polar representation compared to Cartesian representation has many inherent advantages: (1) The origin point of the Polar coordinates can be seen as the center of the object. (2) Starting from the origin point, the contour of the object can be determined only by the distance from the center and the angle. (3) The angle is naturally directional (starting from 0° to 360°) and makes it very convenient to connect the points into a whole contour.</p>
<figure style="width: 100%" class="align-center">
<img src="https://yassouali.github.io//ml-blog/images/CVPR20/polarmask.png" />
<figcaption>Image source: <a href="https://arxiv.org/abs/1909.13226">Xie et al.</a></figcaption>
</figure>
<p>The model is based on <a href="https://arxiv.org/abs/1904.01355">FCOS</a>, where for a given instance, we have three outputs: the classification probabilities over \(k\) classes (e.g., \(k= 80\) on COCO dataset), the center of the object (Polar Centerness), and the distances from the center (Mask Regression). The paper proposes to use \(n = 36\) distances from the center, so the angle between two points in the contour is 10° in this case. Based on these outputs, the extent of each object can be detected easily in a single shot manner without needing a sub-head network for pixel-wise segmentation over each detected object as in <a href="https://arxiv.org/abs/1703.06870">Mask-RCNN</a>.</p>
<h4 id="other-papers">Other papers:</h4>
<ul>
<li><a href="https://arxiv.org/abs/2001.01629">Deep Snake for Real-Time Instance Segmentation</a></li>
<li><a href="https://arxiv.org/abs/2004.13621">Exploring Self-attention for Image Recognition</a></li>
<li><a href="https://arxiv.org/abs/1912.02424">Bridging the Gap Between Anchor-based and Anchor-free Detection</a></li>
<li><a href="https://arxiv.org/abs/1912.05027">SpineNet: Learning Scale-Permuted Backbone for Recognition and Localization</a></li>
<li><a href="https://arxiv.org/abs/2003.14142">Look-into-Object: Self-supervised Structure Modeling for Object Recognition</a></li>
<li><a href="https://arxiv.org/abs/2004.00445">Learning to Cluster Faces via Confidence and Connectivity Estimation</a></li>
<li><a href="https://arxiv.org/abs/2003.11113">PADS: Policy-Adapted Sampling for Visual Similarity Learning</a></li>
<li><a href="https://arxiv.org/abs/2001.07437">Evaluating Weakly Supervised Object Localization Methods Right</a></li>
<li><a href="https://arxiv.org/abs/2001.00309">BlendMask: Top-Down Meets Bottom-Up for Instance Segmentation</a></li>
<li><a href="http://openaccess.thecvf.com/content_CVPR_2020/papers/Liu_Hyperbolic_Visual_Embedding_Learning_for_Zero-Shot_Recognition_CVPR_2020_paper.pdf">Hyperbolic Visual Embedding Learning for Zero-Shot Recognition</a></li>
<li><a href="https://arxiv.org/abs/2005.08104">Single-Stage Semantic Segmentation from Image Labels</a></li>
</ul>
<h1 id="generative-models-and-image-synthesis">Generative models and image synthesis</h1>
<h4 id="learning-physics-guided-face-relighting-under-directional-light-paper">Learning Physics-Guided Face Relighting Under Directional Light (<a href="https://arxiv.org/abs/1906.03355">paper</a>)</h4>
<p>Relighting involves adjusting the lighting of an unseen source image with its corresponding directional light, towards the new desired directional light. The previous works give good results but are limited to smooth lighting and do not model non-diffuse effects such as cast shadows and specularities.</p>
<p>To be able to create precise and believable relighting results and generalizes to complex illumination conditions and challenging poses, the authors propose
an end-to-end deep learning architecture that both delights and relights an image of a human face. This is done in two stages, as shown below.</p>
<figure style="width: 100%" class="align-center">
<img src="https://yassouali.github.io//ml-blog/images/CVPR20/relighting.png" />
<figcaption>Image source: <a href="https://arxiv.org/abs/1906.03355">Nestmeyer et al.</a></figcaption>
</figure>
<p>The first stage consists of predicting the <a href="https://en.wikipedia.org/wiki/Albedo">albedo</a> and <a href="https://en.wikipedia.org/wiki/Normal_(geometry)">normals</a> of the input
image using a Unet architecture, the desired directional light is then used with the normals to predict the shading and then the <a href="http://learnwebgl.brown37.net/09_lights/lights_diffuse.html">diffuse relighting</a>. The outputs of the first stage are used in the second stage to predict the correct shading. The whole model is trained end-to-end with a generative adversarial network (GaN) loss similar to the one used in the <a href="https://arxiv.org/abs/1611.07004">pix2pix</a> paper.</p>
<h4 id="synsin-end-to-end-view-synthesis-from-a-single-image-paper">SynSin: End-to-End View Synthesis From a Single Image (<a href="https://arxiv.org/abs/1912.08804">paper</a>)</h4>
<p>The goal of view synthesis is to generate new views of a scene given one or more images. But this can be challenging, requiring an understanding of the 3D scene from images. To overcome this, current methods rely on multiple images, train on ground-truth depth, or are limited to synthetic data. The authors propose a novel end-to-end model for view synthesis from a single image at test time while being trained on real images without any ground-truth 3D information (e.g., depth).</p>
<figure style="width: 100%" class="align-center">
<img src="https://yassouali.github.io//ml-blog/images/CVPR20/synsin.png" />
<figcaption>Image source: <a href="https://arxiv.org/abs/1912.08804">Wiles et al.</a></figcaption>
</figure>
<p>SynSin takes an input image, the target image, and the desired relative pose (i.e., the desired rotation and translation). The input image is first passed through a feature network to embed it into a feature space at each pixel location, followed by depth prediction at each pixel via a depth regressor. Based on the features and the depth information, a <a href="https://en.wikipedia.org/wiki/Point_cloud">point cloud</a> representation is created, the relative pose (i.e., applying rotation and translation) is then used to render the features at the new view with a fully differentiable neural point cloud renderer. However, the projected features might have some artifacts (e.g., some unseen parts of the image are now visible in the new view, and need to be rendered), in order to fix this, a generator is used to fill the missing regions. The whole model is then trained end-to-end with: an L2 loss, a discriminator loss, and a <a href="https://arxiv.org/abs/1603.08155">perceptual loss</a>, without requiring any depth information. At test time, the network takes an image and the target relative pose and outputs the image with the desired view.</p>
<h4 id="novel-view-synthesis-of-dynamic-scenes-with-globally-coherent-depths-from-a-monocular-camera-paper">Novel View Synthesis of Dynamic Scenes with Globally Coherent Depths from a Monocular Camera (<a href="https://arxiv.org/abs/2004.01294">paper</a>)</h4>
<p>The objective in this paper is to synthesize an image from arbitrary views and times given a collection of images of a dynamic scene, i.e., a series of images captured by a single monocular camera from many the locations (image bellow, left). The method can produce a novel view from an arbitrary location within the original range of locations (image bellow, middle), and can also produce the dynamic content that appeared across any views in different times (image bellow, right). This is done using a single camera, without requiring a multiview system or human-specific priors as previous methods.</p>
<figure style="width: 100%" class="align-center">
<img src="https://yassouali.github.io//ml-blog/images/CVPR20/dynamic_scenes_sythesis.png" />
<figcaption>Image source: <a href="https://arxiv.org/abs/2004.01294">Yoon et al.</a></figcaption>
</figure>
<p>The authors combine the depth from multiview stereo (DMV) with the depth from a single view (DSV) using depth fusion network with the help of the input image from the target view, producing a scale-invariant and a complete depth map. With geometrically consistent depths across views, a novel view can be synthesized using a self-supervised rendering network that produces a photorealistic image in the presence of missing data with an adversarial loss and a reconstruction loss.</p>
<figure style="width: 70%" class="align-center">
<img src="https://yassouali.github.io//ml-blog/images/CVPR20/dynamic_scenes_sythesis_net.png" />
<figcaption>Image source: <a href="https://arxiv.org/abs/2004.01294">Yoon et al.</a></figcaption>
</figure>
<h4 id="stefann-scene-text-editor-using-font-adaptive-neural-network-paper">STEFANN: Scene Text Editor using Font Adaptive Neural Network (<a href="https://arxiv.org/abs/1903.01192">paper</a>)</h4>
<p>This paper presents a method to directly modify text in an image at a character level while maintaining the same style.
This is done in two steps. First, a network called FANnet takes as input the source character we would like to modify and outputs the target character while
keeping structural consistency and the style of the source. Second, the coloring network, Colornet, takes the output of the first stage and the source character and colors the target character while reserving visual consistency. After doing this process for each character of the text, the characters are placed in the in-painted background while maintaining the correct spacing between characters. Below are some examples of the results from <a href="https://prasunroy.github.io/stefann/">the project’s webpage</a>.</p>
<figure style="width: 100%" class="align-center">
<img src="https://yassouali.github.io//ml-blog/images/CVPR20/stefann.png" />
<figcaption>Image source: <a href="https://arxiv.org/abs/1903.01192">Roy et al.</a></figcaption>
</figure>
<h4 id="mixnmatch-multifactor-disentanglement-and-encoding-for-conditional-image-generation-paper">MixNMatch: Multifactor Disentanglement and Encoding for Conditional Image Generation (<a href="https://arxiv.org/abs/1911.11758">paper</a>)</h4>
<p>MixNMatch is a conditional GAN capable of disentangling background, object pose, shape, and texture from real images with minimal supervision, i.e., bounding box annotations to model background. A trained model can then be used to arbitrarily combine the our factors to generate new images, including sketch2color, cartoon2img, and img2gif applications.</p>
<figure style="width: 100%" class="align-center">
<img src="https://yassouali.github.io//ml-blog/images/CVPR20/MixNMatch.png" />
<figcaption>Image source: <a href="https://arxiv.org/abs/1911.11758">Li et al.</a></figcaption>
</figure>
<p>Given a collection of images of a single object category, the model is trained to simultaneously encode background, object pose, shape, and texture factors associated with each images into a disentangled latent code space, and then generate real looking image by combining latent factors from the disentangled code space. Four encoders are used to separately encode each latent code. Four different latent codes are then sampled and fed into the <a href="https://arxiv.org/abs/1811.11155">FineGAN</a> generator to hierarchically generate images, the model is then trained with four image-code pair discriminators optimize the encoders and generator to match their joint image-code distributions.</p>
<h4 id="stargan-v2-diverse-image-synthesis-for-multiple-domains-paper">StarGAN v2: Diverse Image Synthesis for Multiple Domains (<a href="https://arxiv.org/abs/1912.01865">paper</a>)</h4>
<p>The main objective in image-to-image translation (i.e., changing some attributes of an image, such as hair color) is to increase the quality and the diversity of the generated images, while maintaining high scalability over multiple domains (i.e., a domain refers to set of images having the same attribute value, like black hair).
Given that existing methods address only one of these issues, resulting in either limited diversity or various models for all domains. StarGAN v2 tries to solves both issues simultaneously, using style codes instead of an explicit domain labels as in the first version of <a href="https://arxiv.org/abs/1711.09020">StarGAN</a>.</p>
<figure style="width: 90%" class="align-center">
<img src="https://yassouali.github.io//ml-blog/images/CVPR20/starganv2.png" />
<figcaption>Image source: <a href="https://arxiv.org/abs/1912.01865">Choi et al.</a></figcaption>
</figure>
<p>The StarGAN v2 model contains four modules: A generator that translates an input image into an output image with the desired domain-specific style code. A latent encoder (or a mapping network) that produces a style code for each domain, one of which is randomly selected during training. A style encoder that extracts the style code of an image, allowing the generator to perform reference-guided image synthesis, and a discriminator that distinguishes between real and fake (R/F) images from multiple domains. All modules except the generator contain multiple output branches, one of which is selected when training the corresponding domain. The model is then trained using an adversarial loss, a style reconstruction to force the generator to utilize the style code when generating the image, a style diversification loss to enable the generator to produce diverse images and a cycle loss to preserve the characteristics of each domain.</p>
<h4 id="gan-compression-efficient-architectures-for-interactive-conditional-gans-paper">GAN Compression: Efficient Architectures for Interactive Conditional GANs (<a href="https://arxiv.org/abs/2003.08936">paper</a>)</h4>
<p>Conditional GANs (cGANs) give the ability to do controllable image synthesis for many computer vision and graphics applications. However,
the computational resources needed for training them are orders of magnitude larger than that of traditional CNNs used for detection and recognition.
For example, GANs require 10x to 500x more computation that image recognition models. To solve this problem, the authors propose a GAN compression approach based on distillation, channel pruning, and neural architecture search (NAS), resulting in a compressed model while maintaining the same performance.</p>
<figure style="width: 90%" class="align-center">
<img src="https://yassouali.github.io//ml-blog/images/CVPR20/gan_compression.png" />
<figcaption>Image source: <a href="https://arxiv.org/abs/2003.08936">Li et al.</a></figcaption>
</figure>
<p>The proposed GAN Compression framework takes a pre-trained generator, considered as a teacher, which is first distilled into a smaller <em>once-for-all</em> student generator that contains all possible channel numbers through weight sharing, where different channel numbers are chosen for the student at each iteration.
Now, in order to choose the correct number of channels of the student for each layer, many sub-generators are extracted from the <em>once-for-all</em> (student) generator and evaluated, creating the candidate generator pool. Finally, the best sub-generator with the desired compression ratio target and performance target (e.g., <a href="https://arxiv.org/abs/1706.08500">FID</a> or mIoU) using <a href="https://arxiv.org/abs/1904.00420">one-shot NAS</a>, the selected generator is then fine-tuned, resulting in the final compressed model.</p>
<h4 id="semantic-pyramid-for-image-generation-paper">Semantic Pyramid for Image Generation (<a href="https://arxiv.org/abs/2003.06221">paper</a>)</h4>
<p>Semantic Pyramid tries to bridge the gap between discriminative and generative models. This is done using a novel GAN-based model that utilizes the space of deep features learned by a pre-trained classification model. Given a set of features extracted from a reference image, the model generates diverse image samples, each with matching features at each semantic level of the classification model.</p>
<figure style="width: 100%" class="align-center">
<img src="https://yassouali.github.io//ml-blog/images/CVPR20/semantic_pyramid.png" />
<figcaption>Image source: <a href="https://arxiv.org/abs/2003.06221">Shocher et al.</a></figcaption>
</figure>
<p>Concretely, given a pretrained classification network, a GAN network is designed with a generator with a similar architecture as the classification network. Each layer of the generator is trained to be conditioned on the previous layers, and the corresponding layers of the classification network. For example, conditioning the generator on the classification features close to the input results in an image similar to the input image of the classification model, with the possibility of exploring the sup-space of such image by sampling different noise vectors. On the other hand, conditioning on deeper layers results in a wider distribution of generated images. The model is trained with an adversarial loss to produce realistic images, a diversity loss to produce diverse images with different noises, and a reconstruction loss to match the features of the generated image to the reference image. Different regions of the image can be conditioned on different semantic levels using a masking operation \(m\), which can be used to semantically modify the image.</p>
<h4 id="analyzing-and-improving-the-image-quality-of-stylegan-paper">Analyzing and Improving the Image Quality of StyleGAN (<a href="https://arxiv.org/abs/1912.04958">paper</a>)</h4>
<p>In the first version of <a href="https://arxiv.org/abs/1812.04948">StyleGAN</a>, the authors proposed an alternative generator architecture, capable of producing high-quality images and capable of separating high-level attributes (e.g., pose and identity when trained on human faces). This new architecture consisted of using a mapping network from the latent space \(\mathcal{Z}\) into an intermediate space \(\mathcal{W}\) to more closely match the distribution of features in the training set, and avoid the forbidden combinations present in \(\mathcal{Z}\). The intermediate latent vector is incorporated into the generator using Adaptive Instance Normalization (<a href="https://arxiv.org/abs/1703.06868">AdaIN</a>) layers while a uniform noise is additively injected before each application of AdaIN, and trained in a <a href="https://arxiv.org/abs/1710.10196">progressive</a> manner. Yielding impressive results in data-driven unconditional generative image modeling. However, the generated images still contain some artifacts, like water-splotches (more details: <a href="http://www.whichfaceisreal.com/learn.html">whichfaceisreal</a>) and unchanged positions of face attributes like eyes.</p>
<figure style="width: 100%" class="align-center">
<img src="https://yassouali.github.io//ml-blog/images/CVPR20/stylegan2.png" />
<figcaption>Image source: <a href="https://arxiv.org/abs/1912.04958">Karras et al.</a></figcaption>
</figure>
<p>First, to avoid the droplet effects, which are results of the AdaIN discarding information in feature maps, AdaIN is replaced with a weight demodulation layer by removing some redundant operations, moving the addition of the noise to be outside of the active area of a style, and adjusting only the standard deviation per feature map. The progressive GAN training is removed to avoid the permanent positions of face attributes based on <a href="https://arxiv.org/abs/1903.06048">MSG-GAN</a>. Finally, StyleGAN2 introduces a new regularization term to the loss to enforce smoother latent space interpolations based on the Jacobian matrix at a single position at the intermediate latent space.</p>
<h4 id="adversarial-latent-auto-encoders-paper">Adversarial Latent Auto-encoders (<a href="https://arxiv.org/abs/2004.04467">paper</a>)</h4>
<p>Auto-Encoders (AE) are characterized by their simplicity and their capability of combining generative and representational properties by learning an encoder-generator map simultaneously. However, they do not have the same generative capabilities as GANs. The proposed Adversarial Latent Autoencoder (ALAE) retain the generative properties of GANs by learning an output data distribution with an adversarial strategy, with AE architecture where the latent distribution is learned from data to improve the disentanglement properties (i.e., the \(\mathcal{W}\) intermediate latent space of StyleGAN).</p>
<figure style="width: 70%" class="align-center">
<img src="https://yassouali.github.io//ml-blog/images/CVPR20/ALAE.png" />
<figcaption>Image source: <a href="https://arxiv.org/abs/1912.04958">Pidhorskyi et al.</a></figcaption>
</figure>
<p>The ALAE architecture decomposes the generator G and the discriminator D in two networks: F, G, and E, D, where the latent spaces between F and G, and between E and D are considered same, and refereed to as the intermediate latent space \(\mathcal{W}\). In this case, the mapping network F is deterministic, while E and G are stochastic depending on an injected noise. The pair of networks (G,E) consist a generator-encoder network that auto-encodes the latent space \(\mathcal{W}\), and trained to minimize the discrepancy \(\Delta\) (e.g., an MSE loss) between the two distributions, i.e., the distribution at the input of G and the distribution of the output of E. As a whole, model is the trained by alternating between optimizing the GAN loss and the discrepancy \(\Delta\).</p>
<h4 id="other-papers-1">Other papers:</h4>
<ul>
<li><a href="https://arxiv.org/abs/1907.10786">Interpreting the Latent Space of GANs for Semantic Face Editing</a></li>
<li><a href="https://arxiv.org/abs/1907.11922">MaskGAN: Towards Diverse and Interactive Facial Image Manipulation</a></li>
<li><a href="https://arxiv.org/abs/2003.12697">Semantically Multi-modal Image Synthesis</a></li>
<li><a href="https://arxiv.org/abs/2003.14401">TransMoMo: Invariance-Driven Unsupervised Video Motion Retargeting</a></li>
<li><a href="https://arxiv.org/abs/2002.11812">Learning to Shadow Hand-drawn Sketches</a></li>
<li><a href="https://arxiv.org/abs/2005.10663">Wish You Were Here: Context-Aware Human Generation</a></li>
<li><a href="https://arxiv.org/abs/2004.12411">Disentangled Image Generation Through Structured Noise Injection</a></li>
<li><a href="https://arxiv.org/abs/1903.06048">MSG-GAN: Multi-Scale Gradients for Generative Adversarial Networks</a></li>
<li><a href="https://arxiv.org/abs/2004.03623">PatchVAE: Learning Local Latent Codes for Recognition</a></li>
<li><a href="https://arxiv.org/abs/2006.10728">Diverse Image Generation via Self-Conditioned GANs</a></li>
<li><a href="https://arxiv.org/abs/1912.05237">Towards Unsupervised Learning of Generative Models for 3D Controllable Image Synthesis</a></li>
</ul>
<h1 id="representation-learning">Representation Learning</h1>
<h4 id="self-supervised-learning-of-pretext-invariant-representations-paper">Self-Supervised Learning of Pretext-Invariant Representations (<a href="https://arxiv.org/abs/1912.01991">paper</a>)</h4>
<p>Existing self-supervised learning methods consist of creating a pretext task, for example, diving the images into nine patches and solving a <a href="https://arxiv.org/abs/1603.09246">jigsaw puzzle</a> on the permuted patches. These pretext tasks involve transforming an image, computing a representation of the transformed image, and predicting properties of transformation from that representation. As a result, the authors argue that the learned representation must covary with the transformation, and as a results, reducing the amount of learned semantic information. To solve this, they propose PIRL (Pretext-Invariant Representation Learning) to learn invariant representations with respect to the transformations and retain more semantic information.</p>
<figure style="width: 100%" class="align-center">
<img src="https://yassouali.github.io//ml-blog/images/CVPR20/PIRL.png" />
<figcaption>Image source: <a href="https://arxiv.org/abs/1912.01991">Misra et al.</a></figcaption>
</figure>
<p>PIRL trains a network that produces image representations that are invariant to image transformations, and this is done by minimizing a <a href="http://yann.lecun.com/exdb/publis/pdf/hadsell-chopra-lecun-06.pdf">contrastive loss</a>, where the model is trained to differentiate a positive sample (i.e., an image and its transformed version) from N corresponding negative samples that are drawn uniformly at random from the dataset excluding the image used for the positive samples.
Using a large number of negative samples is critical for noise contrastive estimation based losses. To this end, PIRL uses a <a href="https://arxiv.org/abs/1805.01978">memory bank</a> containing feature representations for each example, where each representation at a given instance is an exponential moving average of previous representations.</p>
<h4 id="clusterfit-improving-generalization-of-visual-representations-paper">ClusterFit: Improving generalization of visual representations (<a href="https://arxiv.org/abs/1912.03330">paper</a>)</h4>
<p>Weakly-supervised (e.g., hashtag prediction) and self-supervised (e.g., jigsaw puzzle) strategies are becoming increasingly popular for pretraining CNNs for visual downstream tasks. However, the learned representations using such methods may overfit to the pretraining objective given the limited training signal that can be extracted during pretraining, leading to a reduced generalization to downstream tasks.</p>
<figure style="width: 100%" class="align-center">
<img src="https://yassouali.github.io//ml-blog/images/CVPR20/clusterfit.png" />
<figcaption>Image source: <a href="https://arxiv.org/abs/1912.03330">Yan et al.</a></figcaption>
</figure>
<p>The idea of ClusterFit is quite simple, a network is first pretrained using some chosen pretraining task, be it self- or weakly-supervised, this network is then used to extract features for each image to then apply k-means clustering and assign a pseudo-label for each data points. The pseudo-labels can then be used for training a network from scratch, which will be more adapted to downstream tasks, with either linear probing or fine-tunning.</p>
<h4 id="momentum-contrast-for-unsupervised-visual-representation-learning-paper">Momentum contrast for unsupervised visual representation learning (<a href="https://arxiv.org/abs/1911.05722">paper</a>)</h4>
<p>Recent works on unsupervised visual representation learning are based on minimizing the <a href="http://yann.lecun.com/exdb/publis/pdf/hadsell-chopra-lecun-06.pdf">contrastive loss</a>, which can be seen as building dynamic dictionaries, where the <em>keys</em> in the dictionary are sampled from data (e.g., images or patches) and are represented by an encoder network, which is then trained so that a <em>query</em> \(q\) is similar to a given <em>key</em> \(k\) (a positive sample) and dissimilar to the other keys (negative samples) .</p>
<figure style="width: 90%" class="align-center">
<img src="https://yassouali.github.io//ml-blog/images/CVPR20/moco.png" />
<figcaption>Image source: <a href="https://arxiv.org/abs/1911.05722">He et al.</a></figcaption>
</figure>
<p>Momentum Contrast (MoCo) trains an encoder by matching an encoded query \(q\) to a dictionary of encoded keys using a contrastive loss. The dictionary keys are defined on-the-fly by a set of data samples, where the dictionary is built as a queue, with the current mini-batch enqueued and the oldest mini-batch dequeued, decoupling it from the mini-batch size. By using a queue, a large number of negatives can be used even outside of the current mini-batch. Additionally, the keys are encoded by a slowly progressing encoder, i.e., an exponential moving average of the query encoder, this way, the key encoder is slowly changing over time, producing stable predictions during the course of training. An other benefit of the query encoder is that the dequeue keys used as negatives are not too dissimilar to the current prediction of the key encoder, avoiding having a simple matching problem where the negatives are easily distinguishable from the positive sample.</p>
<h4 id="steering-self-supervised-feature-learning-beyond-local-pixel-statistics-paper">Steering Self-Supervised Feature Learning Beyond Local Pixel Statistics (<a href="https://arxiv.org/abs/2004.02331">paper</a>)</h4>
<figure style="width: 40%" class="align-center">
<img src="https://yassouali.github.io//ml-blog/images/CVPR20/LCI.png" />
<figcaption>Image source: <a href="https://arxiv.org/abs/2004.02331">Jenni et al.</a></figcaption>
</figure>
<p>The authors argue that good image representations should capture both local and global image statistics to better generalize to downstream tasks, where local statistics capture the distribution of near by pixels, e.g., texture, and global statistics to capture the distribution of far away pixels and patches, e.g., shape. However, <a href="https://openreview.net/pdf?id=Bygh9j09KX">CNNs are more biased toward local statistics</a>, and need to be explicitly forced to focus on global features for better generalization. To this end, the authors carefully choose a set of image transformations (i.e., warping, local inpainting and rotation) so that the network can not predict the applied transformation while only observing local statistics, forcing the network to focus on global pixel statistics. With the selected transformation, the network is then pretrained using a classification objective to predicted to label corresponding to the applied transformation.</p>
<h4 id="other-papers-2">Other papers:</h4>
<ul>
<li><a href="https://arxiv.org/abs/1912.02783">Self-Supervised Learning of Video-Induced Visual Invariances</a></li>
<li><a href="https://arxiv.org/pdf/2002.10857.pdf">Circle Loss: A Unified Perspective of Pair Similarity Optimization</a></li>
<li><a href="https://arxiv.org/abs/2002.12247">Learning Representations by Predicting Bags of Visual Words</a></li>
</ul>
<h1 id="computational-photography">Computational photography</h1>
<h4 id="learning-to-see-through-obstructions-paper">Learning to See Through Obstructions (<a href="https://arxiv.org/abs/2004.01180">paper</a>)</h4>
<p>The paper propose a learning based approach for removing unwanted obstructions (examples bellow). The method uses a multi-frame obstruction removal algorithm that exploits the advantages of both optimization-based and learning-based methods, alternating between dense motion estimation and background/obstruction layer reconstruction steps in a coarse-to-fine manner. By modeling of the dense motion, detailed content in the respective layers can be progressively recovered, gradually separating the background from the unwanted occlusion layers. The first staeg consists of flow decomposition, followed by two subsequent stages, background and obstruction layer reconstruction stages, and finally optical flow refinement.</p>
<figure style="width: 100%" class="align-center">
<img src="https://yassouali.github.io//ml-blog/images/CVPR20/remove_obstructions.png" />
<figcaption>Image source: <a href="https://arxiv.org/abs/2004.01180">Liu et al.</a></figcaption>
</figure>
<h4 id="background-matting-the-world-is-your-green-screen-paper">Background Matting: The World is Your Green Screen (<a href="https://arxiv.org/abs/2004.00626">paper</a>)</h4>
<p>The process of separating an image into foreground and background, called matting, generally requires a green screen background or a manually created trimap to produce a good <a href="https://en.wikipedia.org/wiki/Matte_(filmmaking)">matte</a>, to then allow placing the extracted foreground in the desired background. In this paper, the authors propose to use a captured background as an estimate of the true background which is then used to solve for the foreground and alpha value (i.e., every pixel in the image is represented as a combination of foreground and background with a weight alpha).</p>
<figure style="width: 100%" class="align-center">
<img src="https://yassouali.github.io//ml-blog/images/CVPR20/bg_matting.png" />
<figcaption>Image source: <a href="https://arxiv.org/abs/2004.00626">Sengupta et al.</a></figcaption>
</figure>
<p>The model takes as input an image or video of a person in front of a static, natural background, plus an image of just the background. A deep matting network then extracts foreground color and alpha at each spatial location for a given input frame, augmented with background, soft segmentation, and optionally nearby video frames, in addition to a discriminator network that guides the training to generate realistic results. The whole model is trained end-to-end using a combination of a supervised and self-supervised adversarial losses.</p>
<h4 id="3d-photography-using-context-aware-layered-depth-inpainting-paper">3D Photography using Context-aware Layered Depth Inpainting (<a href="https://arxiv.org/abs/2004.04727">paper</a>)</h4>
<p>The objective of the paper is to synthesize content in regions occluded in the input image from a single RGB-D image. The proposed method consists of a three steps pipeline. First, given the RGB-D image, a preprocessing step is applied by filtering the depth and color input using a <a href="https://en.wikipedia.org/wiki/Bilateral_filter">bilateral median filter</a>, the raw discontinuities are then detected using disparity thresholds to estimate the depth edges. Followed by a detection of a context/synthesis regions for each detected depth. Given the color, depth and edge information, the last step consist of depth edge inpating guided by color and depth inpating, resulting a new view as seen in the GIF bellow (taken from <a href="https://www.youtube.com/watch?v=pCSI8YKdCPE">authors YT video</a>).</p>
<figure style="width: 60%" class="align-center">
<img src="https://yassouali.github.io//ml-blog/images/CVPR20/rgbd_to_3d.gif" />
<figcaption>Image source: <a href="https://arxiv.org/abs/2004.04727">Shih et al.</a></figcaption>
</figure>
<h4 id="pulse-self-supervised-photo-upsampling-via-latent-space-exploration-of-generative-models-paper">PULSE: Self-Supervised Photo Upsampling via Latent Space Exploration of Generative Models (<a href="https://arxiv.org/abs/2003.03808">paper</a>)</h4>
<p>The goal of single-image super-resolution is to output a corresponding high-resolution (HR) image from a low-resolution (LR) one. Previous methods train with a supervised loss that measures the pixel-wise average distance between the ground-truth HR image and the output of the model. However, multiple HR images that map to the same LR image exist, and such methods try to match the true HR image, outputting a per-pixel average of all the possible HR images that do not contain a lot details in high frequency regions, resulting in a blurry HR output.</p>
<figure style="width: 80%" class="align-center">
<img src="https://yassouali.github.io//ml-blog/images/CVPR20/PULSE.png" />
<figcaption>Image source: <a href="https://arxiv.org/abs/2003.03808">Menon et al.</a></figcaption>
</figure>
<p>PULSE seeks to find only one plausible HR image from the set of possible HR images that down-scale to the same LR input, and can be trained in a self-supervised manner without the need for a labeled dataset, making the method more flexible and not confined to a specific degradation operator. Specifically, instead of starting with the LR image and slowly adding detail, PULSE traverses the high-resolution natural image manifold, searching for images that downscale to the original LR image. This is done by minimizing a distance measure between the down-scaled HR output of the generator, taking as input the LR image, and the LR image itself. Additionally, the search space is restricted to guarantee that outputs of the generator are realistic by using the unit sphere in \(d\) dimensional Euclidean space as the latent space.</p>
<h4 id="other-papers-3">Other papers:</h4>
<ul>
<li><a href="https://arxiv.org/abs/2004.12260">Learning to Autofocus</a></li>
<li><a href="https://arxiv.org/abs/2003.08367">Lighthouse: Predicting Lighting Volumes for Spatially-Coherent Illumination</a></li>
<li><a href="https://arxiv.org/abs/2002.11616">Zooming Slow-Mo: Fast and Accurate One-Stage Space-Time Video Super-Resolution</a></li>
<li><a href="https://arxiv.org/abs/1912.01839">Explorable Super Resolution</a></li>
<li><a href="https://arxiv.org/abs/1908.00620">Deep Optics for Single-shot High-dynamic-range Imaging</a></li>
<li><a href="https://arxiv.org/abs/2001.04642">Seeing the World in a Bag of Chips</a></li>
</ul>
<h1 id="transferlow-shotsemiunsupervised-learning">Transfer/Low-shot/Semi/Unsupervised Learning</h1>
<h4 id="conditional-channel-gated-networks-for-task-aware-continual-learning-paper">Conditional Channel Gated Networks for Task-Aware Continual Learning (<a href="https://arxiv.org/abs/2004.00070">paper</a>)</h4>
<p>In the case where the training examples come in a sequence of sub-tasks, deep nets where gradient-based optimization is required are subject to catastrophic forgetting, where the learned information from previous tasks is lost. Continual learning tries to solve this by allowing the models to protect and preserve the acquired information while still being capable of extracting new information from new tasks. Similar to the gating mechanism in LSTMs/GRUs, the authors propose a channel-gating module where only a subset of the feature maps are selected depending on the current task. This way, the important filters are protected to avoid a loss in performance of the model on previously learned tasks, additionally, by selecting a limited set of kernels to be updated, the model will still have the capacity to learn new tasks.</p>
<figure style="width: 90%" class="align-center">
<img src="https://yassouali.github.io//ml-blog/images/CVPR20/gated_cnns.png" />
<figcaption>Image source: <a href="https://arxiv.org/abs/2004.00070">Abati et al.</a></figcaption>
</figure>
<p>The paper also introduces a task classifier to overcome the need to know which task the model is being applied to at test time, the task classifier is trained to predict the task at train time and selects which CNN features to pass to the fully-connected layers for classification. However, the task classifier is also subject to catastrophic forgetting, and the authors propose to train it with <a href="https://arxiv.org/abs/1706.08840">Episodic memory</a> and <a href="https://arxiv.org/abs/1710.10368">Generative memory</a> to avoid this.</p>
<h4 id="few-shot-learning-via-embedding-adaptation-with-set-to-set-functions-paper">Few-Shot Learning via Embedding Adaptation with Set-to-Set Functions (<a href="https://arxiv.org/abs/1812.03664">paper</a>)</h4>
<p>Few-Shot Learning consists of learning a well-performing model with N-classes, K examples in each class (i.e., referred to as a N-way, K-shot task), but a high-capacity deep net is easily prone to over-fitting with limited training data. Many few-shot learning methods (e.g., <a href="https://arxiv.org/abs/1703.05175">Prototypical Networks</a>) address this by learning an instance embedding function from seen classes during training where there are ample labeled instances, and then apply a simple function to the embeddings of the new instances from unseen classes with limited labels at test time. However, the learned embeddings are task-agnostic given that the learned the embedding function is not optimally discriminative with respect to the unseen classes.</p>
<figure style="width: 100%" class="align-center">
<img src="https://yassouali.github.io//ml-blog/images/CVPR20/settoset.png" />
<figcaption>Image source: <a href="https://arxiv.org/abs/1812.03664">Ye et al.</a></figcaption>
</figure>
<p>The authors propose to adapt the instance embeddings to the target classification task with a set-to-set function, yielding embeddings that are task-specific and are discriminative. To have task-specific embeddings, an additional adaptation step is conducted, where the embedding function is transformed with a set-to-set function that contextualizes over the image instances of a set, to enable strong co-adaptation of each item. The authors tested many set-to-set functions, such as BiLSTMs, Graph Convolutional Networks, and Transformers, and found that Transformers work bets in this case.</p>
<h4 id="towards-discriminability-and-diversity-batch-nuclear-norm-maximization-under-label-insufficient-situations-paper">Towards Discriminability and Diversity: Batch Nuclear-norm Maximization under Label Insufficient Situations (<a href="https://arxiv.org/abs/2003.12237">paper</a>)</h4>
<p>In cases where we are provided with a small labeled set, the performance of deep nets degrades on ambiguous samples as a result of placing the decision boundary close to high-density regions (right figure below). A common solution is to minimize the entropy, but one side effect caused by entropy minimization is the reduction of the prediction diversity, where ambiguous samples are classified as belonging to the most dominant classes, i.e., an increase in <em>discriminability</em> but a reduction in <em>diversity</em>.</p>
<figure style="width: 100%" class="align-center">
<img src="https://yassouali.github.io//ml-blog/images/CVPR20/BNM.png" />
<figcaption>Image source: <a href="https://arxiv.org/abs/2003.12237">Cui et al.</a></figcaption>
</figure>
<p>The paper investigates ways to increase both the discriminability: outputting highly certain predictions, and the diversity: predicting all the categories somewhat equally. By analyzing the rank of the output matrix \(A \in \mathbb{R}^{B \times C}\), with a batch of \(B\) samples and \(C\) classes, the authors find that the prediction discriminability and diversity could be separately measured by the Frobenius-norm and the rank of \(A\), and propose Batch Nuclear-norm Maximization (BNM) and apply it on the output matrix \(A\) to increase the performance in cases where we have a limited amount of labels, such as semi-supervised learning and domain adaptation.</p>
<h4 id="other-papers-4">Other papers:</h4>
<ul>
<li><a href="https://arxiv.org/abs/1910.00701">Distilling Effective Supervision from Severe Label Noise</a></li>
<li><a href="https://arxiv.org/abs/2003.11712">Mask Encoding for Single Shot Instance Segmentation</a></li>
<li><a href="http://www.eecs.ucf.edu/~gqi/publications/CVPR2020_WCP.pdf">WCP: Worst-Case Perturbations for Semi-Supervised Deep Learning</a></li>
<li><a href="https://arxiv.org/abs/1911.11090">Meta-Learning of Neural Architectures for Few-Shot Learning</a></li>
<li><a href="https://arxiv.org/abs/2004.04388">Towards Inheritable Models for Open-Set Domain Adaptation</a></li>
<li><a href="https://arxiv.org/abs/1909.03403">Open Compound Domain Adaptation</a>)</li>
</ul>
<h1 id="vision-and-language">Vision and Language</h1>
<h4 id="12-in-1-multi-task-vision-and-language-representation-learning-paper">12-in-1: Multi-Task Vision and Language Representation Learning (<a href="https://arxiv.org/abs/1912.02315">paper</a>)</h4>
<p>Vision-and-language based methods often focus on a small set of independent tasks that are studied in isolation. However, the authors point that visually-grounded language understanding skills required for success at each of these tasks overlap significantly. To this end, the paper proposes a large-scale, multi-task training regime with a single model trained on 12 datasets from four broad categories of tasks: visual question answering, caption-based image retrieval, grounding referring expressions, and multi-modal verification. Using a single model helps reduce the number of parameters from approximately 3 billion parameters to 270 million while simultaneously improving the performances across tasks.</p>
<figure style="width: 100%" class="align-center">
<img src="https://yassouali.github.io//ml-blog/images/CVPR20/12in1.png" />
<figcaption>Image source: <a href="https://arxiv.org/abs/1912.02315">Lu et al.</a></figcaption>
</figure>
<p>The model is based on <a href="https://arxiv.org/abs/1908.02265">ViLBERT</a>, where each task has a task-specific <em>head</em> network that branches off a common, shared <em>trunk</em> (i.e., ViLBERT model). With 6 task heads, 12 datasets, and over 4.4 million individual training instances, multi-task training of this scale is hard to control. To overcome this, all the models are first pretrained on the same dataset. Then a round-robin batch sampling is used to cycle through each task from the beginning of multi-task training, with an early stopping for stopping a given task where some over-fitting is observed, with the possibility of restarting the training to avoid catastrophic forgetting.</p>
<h4 id="other-papers-5">Other papers:</h4>
<ul>
<li><a href="https://arxiv.org/abs/2003.13830">Sign Language Transformers: Joint End-to-End Sign Language Recognition and Translation</a></li>
<li><a href="http://openaccess.thecvf.com/content_CVPR_2020/papers/Abbasnejad_Counterfactual_Vision_and_Language_Learning_CVPR_2020_paper.pdf">Counterfactual Vision and Language Learning</a></li>
<li><a href="https://arxiv.org/abs/2004.02194">Iterative Context-Aware Graph Inference for Visual Dialog</a></li>
<li><a href="https://arxiv.org/abs/1912.08226">Meshed-Memory Transformer for Image Captioning</a></li>
<li><a href="https://arxiv.org/abs/2003.05078">Visual Grounding in Video for Unsupervised Word Translation</a></li>
<li><a href="https://people.cs.umass.edu/~smaji/papers/phrasecut+supp-cvpr20.pdf">PhraseCut: Language-Based Image Segmentation in the Wild</a></li>
</ul>
<h1 id="the-rest">The rest</h1>
<p>This post turned into a long one very quickly, so in order to avoid ending-up with a 1h long reading session, I will simply list some papers I came across in case the the reader is interested in the subjects.</p>
<details>
<summary>Click to expand</summary> <br />
<small> <div class="tip">
<p><strong>Efficient training & inference</strong>:</p>
<ul>
<li><a href="https://arxiv.org/abs/1912.01106">MnasFPN: Learning Latency-aware Pyramid Architecture for Object Detection on Mobile Devices</a></li>
<li><a href="https://arxiv.org/abs/1911.09723">Fast Sparse ConvNets</a></li>
<li><a href="https://arxiv.org/abs/1911.11907">GhostNet: More Features from Cheap Operations</a></li>
<li><a href="https://arxiv.org/abs/1909.10788">Forward and Backward Information Retention for Accurate Binary Neural Networks</a></li>
<li><a href="https://arxiv.org/abs/2001.06232">Sideways: Depth-Parallel Training of Video Models</a></li>
<li><a href="https://arxiv.org/abs/1906.02256">Butterfly Transform: An Efficient FFT Based Neural Architecture Design</a></li>
</ul>
<p><strong>3D applications and methods</strong>:</p>
<ul>
<li><a href="https://arxiv.org/abs/1911.11763">SuperGlue: Learning Feature Matching with Graph Neural Networks</a></li>
<li><a href="https://arxiv.org/abs/1911.11130">Unsupervised Learning of Probably Symmetric Deformable 3D Objects From Images in the Wild</a></li>
<li><a href="https://arxiv.org/abs/2004.00452">PIFuHD: Multi-Level Pixel-Aligned Implicit Function for High-Resolution 3D Human Digitization</a></li>
<li><a href="https://arxiv.org/abs/1911.06971">BSP-Net: Generating Compact Meshes via Binary Space Partitioning</a></li>
<li><a href="https://arxiv.org/abs/2004.11364">Single-view view synthesis with multiplane images</a></li>
<li><a href="http://openaccess.thecvf.com/content_CVPR_2020/html/Fieraru_Three-Dimensional_Reconstruction_of_Human_Interactions_CVPR_2020_paper.html">Three-Dimensional Reconstruction of Human Interactions</a></li>
<li><a href="https://arxiv.org/abs/1912.02923">Generating 3D People in Scenes Without People</a></li>
<li><a href="https://arxiv.org/abs/2005.08144">High-Dimensional Convolutional Networks for Geometric Pattern Recognition</a></li>
<li><a href="http://openaccess.thecvf.com/content_CVPR_2020/papers/Li_Shape_correspondence_using_anisotropic_Chebyshev_spectral_CNNs_CVPR_2020_paper.pdf">Shape correspondence using anisotropic Chebyshev spectral CNNs</a></li>
</ul>
<p><strong>Face, gesture, and body pose</strong>:</p>
<ul>
<li><a href="https://arxiv.org/abs/1908.10357">HigherHRNet: Scale-Aware Representation Learning for Bottom-Up Human Pose Estimation</a></li>
<li><a href="https://arxiv.org/abs/2003.08325">DeepCap: Monocular Human Performance Capture Using Weak Supervision</a></li>
<li><a href="https://arxiv.org/abs/2003.00080">Transferring Dense Pose to Proximal Animal Classes</a></li>
<li><a href="https://arxiv.org/abs/2006.08586">Coherent Reconstruction of Multiple Humans from a Single Image</a></li>
<li><a href="https://arxiv.org/abs/1912.05656">VIBE: Video Inference for Human Body Pose and Shape Estimation</a></li>
</ul>
<p><strong>Video & Scene analysis and understanding</strong>:</p>
<ul>
<li><a href="https://arxiv.org/abs/2004.02788">Self-Supervised Scene De-occlusion</a></li>
<li><a href="https://arxiv.org/abs/2002.11949">Unbiased Scene Graph Generation from Biased Training</a></li>
<li><a href="http://openaccess.thecvf.com/content_CVPR_2020/papers/Dwibedi_Counting_Out_Time_Class_Agnostic_Video_Repetition_Counting_in_the_CVPR_2020_paper.pdf">Counting Out Time: Class Agnostic Video Repetition Counting in the Wild</a></li>
<li><a href="https://arxiv.org/abs/2004.06376">Footprints and Free Space From a Single Color Image</a></li>
<li><a href="https://arxiv.org/abs/1912.06992">Action Genome: Actions As Compositions of Spatio-Temporal Scene Graphs</a></li>
<li><a href="https://arxiv.org/abs/1912.06430">End-to-End Learning of Visual Representations From Uncurated Instructional Videos</a></li>
</ul>
</div> </small>
</details>Yassineouali.yasine@gmail.comThe first virtual CVPR conference ended, with 1467 papers accepted, 29 tutorials, 64 workshops, and 7k virtual attendees. In this blog post, I present an overview of the conference by summarizing some papers that caught my attention.Deep Semi-Supervised Learning2020-06-11T08:00:00+00:002020-06-11T08:00:00+00:00https://yassouali.github.io//ml-blog/deep-semi-supervised<p><span style="color: #c8634d;">[A more detailed version of this post is available on <a href="https://arxiv.org/abs/2006.05278">arXiv</a>.]</span>
<br />
<span style="color: #c8634d;">[A curated and an up-to-date list of SSL papers is available on <a href="https://github.com/yassouali/awesome-semi-supervised-learning">github</a>.]</span></p>
<p>Deep neural networks demonstrated their ability to provide remarkable performances on certain supervised learning tasks (e.g., image classification) when trained on
extensive collections of labeled data (e.g. ImageNet). However,
creating such large collections of data requires a considerable amount of resources, time, and effort. Such resources may not be available in many practical cases,
limiting the adoption and application of many deep learning (DL) methods.</p>
<p>In a search for more data-efficient DL methods to overcome the need for large
annotated datasets, we see a lot
of research interest in recent years with regards to the application of semi-supervised learning (SSL) to deep neural nets as a possible alternative,
by developing novel methods and adopting existing SSL frameworks for a
deep learning setting.
This post discusses SSL in a deep learning setting and goes through some of the main deep learning SSL methods.</p>
<ul>
<li><a href="#semi-supervised-learning">Semi-supervised Learning</a>
<ul>
<li><a href="#what-is-semi-supervised-learning">What is Semi-supervised Learning?</a></li>
<li><a href="#semi-supervised-learning-methods">Semi-supervised learning methods</a></li>
<li><a href="#main-assumptions-in-ssl">Main Assumptions in SSL</a></li>
</ul>
</li>
<li><a href="#consistency-regularization">Consistency Regularization</a>
<ul>
<li><a href="#ladder-networks">Ladder Networks</a></li>
<li><a href="#π-model">Π-model</a></li>
<li><a href="#temporal-ensembling">Temporal Ensembling</a></li>
<li><a href="#mean-teachers">Mean Teachers</a></li>
<li><a href="#dual-students">Dual Students</a></li>
<li><a href="#virtual-adversarial-training">Virtual Adversarial Training</a></li>
<li><a href="#adversarial-dropout">Adversarial Dropout</a></li>
<li><a href="#interpolation-consistency-training">Interpolation Consistency Training</a></li>
<li><a href="#unsupervised-data-augmentation">Unsupervised Data Augmentation</a></li>
</ul>
</li>
<li><a href="#entropy-minimization">Entropy Minimization</a></li>
<li><a href="#proxy-label-methods">Proxy-label Methods</a>
<ul>
<li><a href="#self-training">Self-training</a></li>
<li><a href="#meta-pseudo-labels">Meta Pseudo Labels</a></li>
<li><a href="#multi-view-training">Multi-view training</a>
<ul>
<li><a href="#co-training">Co-training</a></li>
<li><a href="#tri-training">Tri-Training</a></li>
</ul>
</li>
</ul>
</li>
<li><a href="#holistic-methods">Holistic Methods</a>
<ul>
<li><a href="#mixmatch">MixMatch</a></li>
<li><a href="#remixmatch">ReMixMatch</a></li>
<li><a href="#fixmatch">FixMatch</a></li>
</ul>
</li>
<li><a href="#references">References</a></li>
</ul>
<h1 id="semi-supervised-learning">Semi-supervised Learning</h1>
<h2 id="what-is-semi-supervised-learning">What is Semi-supervised Learning?</h2>
<blockquote>
<p>Semi-supervised learning (SSL) is halfway between supervised and unsupervised learning.
In addition to unlabeled data, the algorithm is provided with some supervision
information – but not necessarily for all examples. Often, this information will
be the targets associated with some of the examples. In this case, the data set \( X=\left(x_{i}\right); i \in [n]\)
can be divided into two parts: the points \( X_{l}:=\left(x_{1}, \dots, x_{l}\right) \), for which labels
\( Y_{l}:=\left(y_{1}, \dots, y_{l}\right) \) are provided, and the points
\( X_{u}:=\left(x_{l+1}, \ldots, x_{l+u}\right) \), the labels of which are
not known.</p>
<footer><strong>Chapelle et al.</strong> —
<a href="http://www.acad.bg/ebook/ml/MITPress- SemiSupervised Learning.pdf">SSL book</a>
</footer>
</blockquote>
<p>As stated in the definition above, in SSL, we are provided with a dataset containing both
labeled and unlabeled examples. The portion of labeled examples is usually quite small
compared to the unlabeled example (e.g., 1 to 10% of the total number of examples). So with a
dataset \(\mathcal{D}\) containing a labeled subset \(\mathcal{D}_l\) and an unlabeled subset
\(\mathcal{D}_u\). The objective, or rather hope, is to leverage the unlabeled
examples to train a better performing model than what can be obtained using only the
labeled portion. And hopefully, get closer to the desired optimal performance, in which
all of the dataset \(\mathcal{D}\) is labeled.</p>
<p>More formally, SSL’s goal is to leverage the unlabeled data \(\mathcal{D}_u\) to produce a prediction function
\(f_{\theta}\) with trainable parameters \(\theta\), that is more accurate than what would have been obtained by only using the labeled data \(\mathcal{D}_l\). For instance, \(\mathcal{D}_u\) might provide us with additional information about the structure of the data distribution \(p(x)\), to better estimate the decision boundary
between the different classes. As shown in Fig. 1 bellow, where the data points with distinct labels are separated with low-density regions, leveraging unlabeled data with a SSL approach can provide us with additional information about the shape of the decision boundary between two classes and reduce the ambiguity present in the supervised case.</p>
<figure style="width: 75%" class="align-center">
<img src="https://yassouali.github.io//ml-blog/images/SSL/cluster_ssl.png" alt="" />
<figcaption>Fig. 1. The decision boundaries obtained on two moons dataset, with a supervised and different SSL approaches, using 6 labeled examples, 3 for each class and the rest of the points as unlabeled data. (Image source: <a href="https://arxiv.org/abs/1804.09170">Oliver et al</a>)
</figcaption>
</figure>
<p>Semi-supervised learning first appeared in the form of self-training, where a model is first trained on labeled data, and then, iteratively, at each training iteration, a portion of the unlabeled data is annotated using the trained model and added to the training set for the next iteration. SSL really took off in the 1970s after its success
with iterative algorithms such as the <a href="https://en.wikipedia.org/wiki/Expectation–maximization_algorithm">expectation-maximization</a> algorithm, using labeled and unlabeled
data to maximize the likelihood of the model. In this post, we are only interested in SSL applied to deep learning. For a detailed review of the field, <a href="http://www.acad.bg/ebook/ml/MITPress- SemiSupervised Learning.pdf">Semi-Supervised Learning Book</a> is a good resource.</p>
<h2 id="semi-supervised-learning-methods">Semi-supervised learning methods</h2>
<p>There have been many SSL methods and approaches that have been introduced over the years, SSL algorithms can be broadly divided into the following categories:</p>
<ul>
<li><strong>Consistency Regularization (Consistency Training).</strong> Based on the assumption that if a realistic perturbation was applied to the unlabeled data points, the prediction should not change significantly. We can then train the model to have a consistent prediction on a given unlabeled example and its perturbed version.</li>
<li><strong>Proxy-label Methods.</strong> Such methods leverage a trained model on the labeled set to produce additional training examples extracted from the unlabeled set based on some heuristic. These approaches can also be referred to as <em>self-teaching</em> or <em>bootstrapping</em> algorithms;
we follow <a href="https://arxiv.org/abs/1804.09530">Ruder et al.</a> and refer to them as proxy-label methods. Some examples of such methods are <em>Self-training</em>, <em>Co-training</em>, and <em>Multi-View Learning</em>.</li>
<li><strong>Generative models.</strong> Similar to the supervised setting, where the learned features on one task can be transferred to other downstream tasks. Generative models that are able to generate images from the data distribution \(p(x)\) must learn transferable features to a supervised task \(p(x | y)\) for a given task with targets \(y\).</li>
<li><strong>Graph-Based Methods.</strong> A labeled and unlabeled data points constitute the nodes of the graph, and the objective is to propagate the labels from the labeled nodes to the unlabeled ones. The similarity of two nodes \(n_i\) and \(n_j\) is reflected by how strong is the edge \(e_{ij}\) between them.</li>
</ul>
<p>In addition to these main categories, there is also some SSL work on <strong>entropy minimization</strong>, where we force the model to make confident predictions by minimizing the entropy of the predictions.
Consistency training can also be considered as a proxy-label method, with a subtle difference where instead of considering
the predictions as ground-truths and compute the cross-entropy loss, we enforce consistency of predictions by minimizing
a given distance between the outputs.</p>
<p>In this post, we will focus more on consistency regularization based approaches, given that they are the most commonly used methods in deep learning, and we will present a brief introduction to the proxy-label, and holistic approaches.</p>
<h2 id="main-assumptions-in-ssl">Main Assumptions in SSL</h2>
<p>The first question we need to answer, is under what assumptions can we apply SSL algorithms? SSL algorithms only work under some conditions, where some assumptions about the structure of the data need to hold. Without such assumptions, it would not be possible to generalize from a finite training set to a set of possibly infinitely many unseen test cases.</p>
<p>The main assumptions in SSL are:</p>
<ul>
<li><strong>The Smoothness Assumption</strong>: <em>If two points \(x_1\), \(x_2\) that reside in a high-density regions are close, then so should be their corresponding outputs \(y_1\), \(y_2\)</em>. Meaning that if two inputs are of the same class and belong to the same cluster, which is a high-density region of the input space, then their corresponding outputs need to be close. The inverse holds; if the two points are separated by a low-density region, the outputs must be distant from each other. This assumption can be quite helpful in a classification task, but not so much for regression.</li>
<li><strong>The Cluster Assumption</strong>: <em>If points are in the same cluster, they are likely to be of the same class.</em> In this special case of the smoothness assumption, we suppose that input data points form clusters, and each cluster corresponds to one of the output classes.
The cluster assumption can also be seen as the low-density separation assumption: <em>the decision boundary should lie in the low-density regions.</em>
The relation between the two assumptions is easy to see, if a given decision boundary lies in a high-density region, it will likely cut a cluster
into two different classes, resulting in samples from different classes belonging to the same cluster, which is a violation of the cluster assumption.
In this case, we can restrict our model to have consistent predictions on the unlabeled data over some small perturbations pushing its decision boundary to low-density regions.</li>
<li><strong>The Manifold Assumption</strong>: <em>The (high-dimensional) data lie (roughly) on a low-dimensional manifold.</em> With high dimensional space, where the volume grows exponentially with the number of dimensions, it can be quite hard to estimate the true data distribution for generative tasks, and for discriminative tasks, the distances are similar regardless of the class type, making classification quite challenging. However, if our input data lies on some lower-dimensional manifold, we can try to find a low dimensional representation using the unlabeled data and then use the labeled data to solve the simplified task.</li>
</ul>
<h1 id="consistency-regularization">Consistency Regularization</h1>
<p>A recent line of works in deep semi-supervised learning utilize the unlabeled data
to enforce the trained model to be in line with the cluster assumption, i.e., the
learned decision boundary must lie in low-density regions. These methods are based
on a simple concept that, if a realistic perturbation was to be applied to an unlabeled
example, the prediction should not change significantly, given that under the
cluster assumption: Data points with distinct labels are separated with low-density regions, so the likelihood of one example switching classes after a perturbation is small (see Figure 1).</p>
<p>More formally, with consistency regularization, we are favoring the functions \(f_\theta\) that give consistent
prediction for similar data points. So rather than minimizing the classification cost at the zero-dimensional data points
of the inputs space, the regularized model minimizes the cost on a manifold around each data point, pushing the
decision boundaries away from the unlabeled data points and smoothing the manifold on which the data
resides (<a href="http://pages.cs.wisc.edu/~jerryzhu/pub/ssl_survey.pdf">Zhu, 2005</a>).
Given an unlabeled data point \(x_u \in \mathcal{D}_u\) and its perturbed version \(\hat{x}_u\),
the objective is to minimize the distance between the two outputs
\(d(f_{\theta}(x_u), f_{\theta}(\hat{x}_u))\). The popular distance measures \(d\) are
mean squared error (MSE), Kullback-Leiber divergence (KL)
and Jensen-Shannon divergence (JS). For two
outputs \(y_u = f_{\theta}(x_u)\) and \(\hat{y}_u = f_{\theta}(\hat{x}_u)\) in the form of a probability distribution over the \(C\)
classes,
and \(m=\frac{1}{2}(f_{\theta}(x_u) + f_{\theta}(\hat{x}_u))\), we can compute these measures as follows:</p>
\[\small d_{\mathrm{MSE}}(y_u, \hat{y}_u)=\frac{1}{C} \sum_{k=1}^{C}(f_{\theta}(x_u)_k -f_{\theta}(\hat{x}_u)_k)^{2}\]
\[\small d_{\mathrm{KL}}(y_u, \hat{y}_u)=\frac{1}{C} \sum_{k=1}^{C} f_{\theta}(x_u)_k \log \frac{f_{\theta}(x_u)_k}{f_{\theta}(\hat{x}_u)_k}\]
\[\small d_{\mathrm{JS}}(y_u, \hat{y}_u)=\frac{1}{2}
d_{\mathrm{KL}}(y_u, m)+\frac{1}{2} \mathrm{d}_{\mathrm{KL}}(\hat{y}_u, m)\]
<p>Note that we can also enforce a consistency over two perturbed versions of \(x_u\),
\(\hat{x}_{u_1}\) and \(\hat{x}_{u_2}\). Now let’s go through the popular consistency regularization methods
in deep learning.</p>
<h2 id="ladder-networks">Ladder Networks</h2>
<p>With the objective to take any well-performing feed-forward network on supervised data and augment it with
additional branches to be able to utilize additional unlabeled data.
<a href="https://arxiv.org/abs/1507.02672">Rasmus et al.</a> proposed to use Ladder
Networks (<a href="https://arxiv.org/abs/1411.7783">Harri Valpola</a>) with an additional
encoder and decoder for SSL.
As illustrated in Figure 2, the network consists of two encoders, a corrupted and clean one, and a decoder.
At each training iteration, the input \(x\) is passed through both encoders. In the corrupted encoder,
Gaussian noise is injected at each layer after batch normalization, producing two outputs, a clean prediction
\(y\) and a prediction based on corrupted activations \(\tilde{y}\). The output \(\tilde{y}\) is then fed into
the decoder to reconstruct the uncorrupted input and the clean hidden activations.
The unsupervised training loss \(\mathcal{L}_u\)
is then computed as the MSE between the activations of the clean encoder \(\mathbf{z}\)
and the reconstructed activations \(\hat{\mathbf{z}}\) (ie., after batch normalization)
in the decoder using the corrupted output \(\tilde{y}\), this is computed over all layers,
from the input to the last layer \(L\), with a weighting \(\lambda_{l}\) for each layer’s contribution loss:</p>
\[\mathcal{L}_u = \frac{1}{|\mathcal{D}|} \sum_{x \in \mathcal{D}} \sum_{l=0}^{L} \lambda_{l}\|\mathbf{z}^{(l)}-\hat{\mathbf{z}}^{(l)}\|^{2}\]
<p>If the input \(x\) is a labeled data point (\(x \in \mathcal{D}_l\)). Then we can add a supervised loss term
to \(\mathcal{L}_u\) to obtain the final loss. Note the supervised cross-entropy \(\mathrm{H}(\tilde{y}, t)\)
loss is computed between the corrupted output \(\tilde{y}\) and the targets \(t\):</p>
\[\mathcal{L} = \mathcal{L}_u + \mathcal{L}_s = \mathcal{L}_u +
\frac{1}{|\mathcal{D}_l|} \sum_{x, t \in \mathcal{D}_l} \mathrm{H}(\tilde{y}, t)\]
<figure style="width: 75%" class="align-center">
<img src="https://yassouali.github.io//ml-blog/images/SSL/ladder_network.png" alt="" />
<figcaption>Fig. 2. An illustration of one forward pass of Ladder Networks, C refers
to the MSE loss between the activations at various layers.
(Image source: <a href="https://arxiv.org/abs/1507.02672">Rasmus et al</a>)
</figcaption>
</figure>
<p>The method can be easily adapted for convolutional neural networks (CNNs)
by replacing the fully-connected layers with
convolution and deconvolution layers for semi-supervised vision tasks.
However, the ladder network is quite heavy computationally, approximately tripling
the computation needed for one training iteration. To mitigate this,
the authors propose a variant of ladder networks called <strong>Γ-Model</strong> where
\(\lambda_{l}=0\) when \(l<L\). In this case, the decoder is omitted, and the unsupervised loss
is computed as the MSE between the two outputs \(y\) and \(\tilde{y}\).</p>
<h2 id="π-model">Π-model</h2>
<p>The <strong>Π-model</strong> (<a href="https://arxiv.org/abs/1610.02242">Laine et al.</a>) is a simplification of the <strong>Γ-Model</strong> of Ladder Networks,
where the corrupted encoder is removed, and the same network is used to get the prediction for both corrupted and uncorrupted inputs.
Specifically, <strong>Π-model</strong> takes advantage of the stochastic nature of the prediction function \(f_ \theta\) in
neural networks due to conventional regularization techniques, such as data augmentation and dropout, that typically don’t alter the model’s predictions.
For any given input \(x\), the objective is to reduce the distances between two predictions of \(f_ \theta\) with \(x\) as input in both forward passes.
Concretely, as illustrated in Figure 3, we would like to minimize \(d(z, \tilde{z})\), where we consider one of the two outputs as a target.
Given the stochastic nature of the predictions function (ie., using dropout as noise source),
the two outputs \(f_\theta(x) = z\) and \(f_\theta(x) = \tilde{z}\) will be distinct. The objective is
to obtain consistent predictions for both of them. In case the input \(x\) is a labeled data point,
we also compute the cross-entropy supervised loss using the provided labels \(y\) and the total loss will be:</p>
\[\mathcal{L} = w \frac{1}{|\mathcal{D}_u|} \sum_{x \in \mathcal{D}_u}
d_{\mathrm{MSE}}(z, \tilde{z}) +
\frac{1}{|\mathcal{D}_l|} \sum_{x, y \in \mathcal{D}_l} \mathrm{H}(y, z)\]
<p>With \(w\) as a weighting function, starting from 0 up to a fixed weight \(\lambda\) (eg., 30) after a given number of epochs (eg., 20% of training time). This way, we avoid using the untrained and random prediction function providing us with unstable predictions at the start of training to extract the training signal from
the unlabeled examples.</p>
<figure style="width: 100%" class="align-center">
<img src="https://yassouali.github.io//ml-blog/images/SSL/pi_model.png" alt="" />
<figcaption>Fig. 3. Loss computation for <b>Π-model</b>, we compute the MSE between the two outputs for the unsupervised loss, and if the input
is a labeled example, we add the supervised loss to the weighted unsupervised loss.
(Image source: <a href="https://arxiv.org/abs/1610.02242">Laine et al</a>)
</figcaption>
</figure>
<h2 id="temporal-ensembling">Temporal Ensembling</h2>
<p>Π-model can be divided into two stages. We first classify all of the training data without updating the weights of the model,
obtaining the predictions \(\tilde{z}\), and in the second stage, we consider the predictions \(\tilde{z}\) as targets for the unsupervised
loss and enforce consistency of predictions by minimizing the distance between the current outputs \(z\) and the outputs of
the first stage \(\tilde{z}\) under different dropout and augmentations.
The problem with this approach is that the targets \(\tilde{z}\) are based on a single evaluation of the network and can
rapidly change, this instability in the targets can lead to instability during training and reduces the amount of training signal that can be extracted from the unlabeled examples. To solve this, <a href="https://arxiv.org/abs/1610.02242">Laine et al.</a> proposed a second
version of Π-model called <strong>Temporal Ensembling</strong>, where the targets \(\tilde{z}\) are the aggregation of all the previous predictions.
This way, during training, we only need a single forward pass to get the current predictions \(z\) and the aggregated targets \(\tilde{z}\),
speeding up the training time by approximately 2x. The training process is illustrated in Figure 4.</p>
<figure style="width: 100%" class="align-center">
<img src="https://yassouali.github.io//ml-blog/images/SSL/temporal_ensembling.png" alt="" />
<figcaption>Fig. 4. Loss computation for <b>Temporal Ensembling</b>, we compute the MSE between the current prediction and
the aggregated target for the unsupervised loss, and if the input is a labeled example, we add the supervised loss to the weighted unsupervised loss.
(Image source: <a href="https://arxiv.org/abs/1610.02242">Laine et al</a>)
</figcaption>
</figure>
<p>For a target \(\tilde{z}\), at each training iteration, the current output \(z\) are accumulated into the <em>ensemble outputs</em> \(\tilde{z}\)
by an exponentially moving average update:</p>
\[\tilde{z} = \alpha \tilde{z}+(1-\alpha) z\]
<p>where \(\alpha\) is a momentum term that controls how far the ensemble reaches into training history. \(\tilde{z}\)
can also be seen as the output of an ensemble network \(f\) from previous training epochs, where the
recent ones have a greater weight than the distant ones.</p>
<p>At the start of training, temporal ensembling reduces to Π-model since the aggregated targets are very noisy,
to overcome this, similar to the bias correction used in Adam optimizer, a training target \(\tilde{z}\) are corrected for the startup bias
at a training step \(t\) as follows:</p>
\[\tilde{z} = (\alpha \tilde{z}+(1-\alpha) z) / (1-\alpha^{t})\]
<p>The loss computation in temporal ensembling remains the same as in Π-model, but with two critical benefits. First, the training is
faster since we only need a single forward pass through the network to obtain \(z\), while maintaining
an exponential moving average (EMA) of label predictions on each training example, and penalizes predictions that are inconsistent with these targets.
Second, the targets are more stable during training, yielding better results.
The downside of such a method is a large amount of memory needed to keep an aggregate of the predictions for all of the training examples,
which can become quite memory intensive for large datasets and dense tasks (e.g., semantic segmentation).</p>
<h2 id="mean-teachers">Mean Teachers</h2>
<p>In the previous approach, the same model plays a dual role as a <em>teacher</em> and a <em>student</em>. Given a set of
unlabeled data, as a teacher, the model generates the targets, which are then used by itself as a student for learning using a consistency loss.
These targets may very well be misclassified, and if the weight of the unsupervised loss outweighs that of the supervised loss,
the model is prevented from learning new information and keeps predicting the same targets, resulting in the form of confirmation bias. To solve this, the quality of the targets must be improved.</p>
<p>The quality of targets can be improved by either: (1) carefully choosing the perturbations instead of simply injecting
additive or multiplicative noise, or (2) carefully choosing the teacher model responsible for generating the targets,
instead of using a replica of the student model.</p>
<p>Π-model and its improved version with Temporal Ensembling provides a better and more stable teacher model by maintaining an EMA of the predictions of each example, which is formed by an ensemble of the model’s current version and those earlier
versions that evaluated the same example. This ensembling improves the quality of the predictions and using them as the
teacher predictions improve results. However, the newly learned information is incorporated into the training at a slow pace, since each target is updated only once during training, and the larger the dataset, the bigger the span between the updates gets. To overcome the limitations of Temporal Ensembling,
<a href="https://arxiv.org/abs/1703.01780">Tarvainen et al.</a> propose to average the model weights instead of its predictions and call this
method Mean Teacher, illustrated in Figure 5.</p>
<figure style="width: 100%" class="align-center">
<img src="https://yassouali.github.io//ml-blog/images/SSL/mean_teacher.png" alt="" />
<figcaption>Fig. 5. The Mean Teacher method. The teacher model, which is an EMA of the student model, is responsible for generating the targets for consistency training. The student model is then trained to minimize the supervised loss over labeled examples and the consistency loss over unlabeled examples. At each training iteration, both models are evaluated with an injected noise (η, η'), and the weights of the teacher model are updated using the current student model to incorporate the learned information at a faster pace.
(Image source: <a href="https://arxiv.org/abs/1703.01780">Tarvainen et al.</a>)
</figcaption>
</figure>
<p>A training iteration of Mean Teacher is very similar to previous methods. The main difference is that were the Π-model uses
the same model as a student and a teacher \(\theta^{\prime}=\theta\), and temporal ensembling approximate a stable teacher \(f_{\theta^{\prime}}\)
as an ensemble function with a weighted average of successive predictions.
Mean Teacher defines the weights \(\theta^{\prime}_t\) of the teacher model \(f_{\theta^{\prime}}\)
at training step \(t\) as the EMA of successive student’s weights \(\theta\) as follows:</p>
\[\theta_{t}^{\prime}=\alpha \theta_{t-1}^{\prime}+(1-\alpha) \theta_{t}\]
<p>In this case, the loss computation is the sum of the supervised and unsupervised loss, where the teacher model is used to obtain the targets
for the unsupervised loss for a given input \(x_i\):</p>
\[\mathcal{L} = w \frac{1}{|\mathcal{D}_u|} \sum_{x \in \mathcal{D}_u}
d_{\mathrm{MSE}}(f_{\theta}(x), f_{\theta^{\prime}}(x)) +
\frac{1}{|\mathcal{D}_l|} \sum_{x, y \in \mathcal{D}_l} \mathrm{H}(y, f_{\theta}(x))\]
<h2 id="dual-students">Dual Students</h2>
<p>One of the main drawbacks of using Mean Teacher, where the teacher’s weights are an EMA of the student’s weights,
is that given a large number of training iterations, the weights of the teacher model will converge to that
of the student model, and any biased and unstable predictions will be carried over to the student.</p>
<p>To solve this, <a href="https://arxiv.org/abs/1909.01804">Ke et al.</a> propose a dual students step-up, where
two student models with different initialization are simultaneously trained, and at a given iteration, one of them
provides the targets for the other. To choose which one, we check for the most stable predictions that satisfy
the following stability conditions:</p>
<ul>
<li>The predictions using two input versions, a clean \(x\) and a perturbed version \(\tilde{x}\) give
the same results: \(f(x) = f(\tilde{x})\).</li>
<li>Both predictions are confident, ie, are far from the decision boundary. This
can be tested by seeing if \(f(x)\) (resp. \(f(\tilde{x})\)) is greater than a confidence threshold \(\epsilon\), such as 0.1.</li>
</ul>
<p>Given two student models,
\(f_{\theta_1}\) and \(f_{\theta_2}\), an unlabeled input \(x_u\) and its perturbed version \(\tilde{x}_u\). We
compute four predictions: \(f_{\theta_1}(x_u), f_{\theta_1}(\tilde{x}_u), f_{\theta_2}(x_u), f_{\theta_2}(\tilde{x}_u)\).
In addition to training each model to minimize both the supervised and unsupervised losses for both models:</p>
\[\mathcal{L}_s = \frac{1}{|\mathcal{D}_l|} \sum_{x_l, y \in \mathcal{D}_l} \mathrm{H}(y, f_{\theta_i}(x_l))\]
\[\mathcal{L}_u = \frac{1}{|\mathcal{D}_u|} \sum_{x_u \in \mathcal{D}_u} d_{\mathrm{MSE}}(f_{\theta_i}(x_u), f_{\theta_i}(\tilde{x}_u))\]
<p>We also force one of the students to have a similar prediction
to its counterpart. To chose which one to update its weights,
we check for the stability constraint for both models. If the predictions one of the models is unstable, we update its weights.
If both are stable, we update the model with the largest variation
\(\mathcal{E}^{i} =\left\|f_{i}(x_u)-f_{i}(\tilde{x}_u)\right\|^{2}\), so the least stable.</p>
<figure style="width: 90%" class="align-center">
<img src="https://yassouali.github.io//ml-blog/images/SSL/dualstudents.png" alt="" />
<figcaption>Fig. 6. Examples of the perturbed inputs for different values of the scaling hyperparameter Ɛ.
(Image source: <a href="https://arxiv.org/abs/1909.01804">Ke et al</a>)
</figcaption>
</figure>
<p>In the end, as depicted in Figure 6, the least stable model is trained with the following loss:</p>
\[\mathcal{L} = \mathcal{L}_s + \lambda_{1} \mathcal{L}_u + \lambda_{2}
\frac{1}{|\mathcal{D}_u|} \sum_{x_u \in \mathcal{D}_u}
d_{\mathrm{MSE}}(f_{\theta_i}(x_u), f_{\theta_j}(x_u))\]
<p>while the stable model is trained using traditional loss for consistency training: \(\lambda_{1} \mathcal{L}_u + \mathcal{L}_s\).</p>
<h2 id="virtual-adversarial-training">Virtual Adversarial Training</h2>
<p>The previous approaches focused on applying random perturbations to each input to generate artificial input points,
encouraging the model to assign similar outputs to the unlabeled data points and their perturbed versions, this way
we push for a smoother output distribution, and as a result, the generalization performance of the model can be improved. Such
random noise and random data augmentation often leaves the predictor particularly vulnerable to small perturbations in a specific direction,
that is, the adversarial direction, which is the direction in the input space in which the label probability \(p(y|x)\) of the model is most sensitive.</p>
<p>To solve this, and inspired by adversarial training (<a href="https://arxiv.org/abs/1412.6572">Goodfellow et al.</a>) that trains the model to assign to each input data a label that is similar to the labels to be assigned to its neighbors in the adversarial direction.
<a href="https://arxiv.org/abs/1704.03976">Miyato et al.</a> propose Virtual Adversarial Training (VAT), a regularization technique that enhances the model’s robustness around each input data point against random and local perturbations. The term “virtual” comes from the fact that the adversarial perturbation is approximated without label information and is hence applicable to semi-supervised learning
to smooth the output distribution.</p>
<p>Concretely, VAT trains the output distribution to be identically smooth around each data
point by selectively smoothing the model in its most adversarial direction.
For a given data point \(x\), we would like to compute the adversarial perturbation \(r_{adv}\) that will alter the model’s predictions the most.
We start by sampling a Gaussian noise \(r\) of the same dimensions as the input \(x\). We then compute its gradients \(grad_r\)
with respect the loss between the two predictions, with and without the injections of the noise \(r\)
(i.e., KL-divergence is used as a distance measure \(d(.,.)\)).
\(r_{adv}\) can then be obtained by normalizing and scaling \(grad_r\) by a hyperparameter \(\epsilon\). This can be written as follows:</p>
\[1) \ \ r \sim \mathcal{N}(0, \frac{\xi}{\sqrt{\operatorname{dim}(x)}} I)\]
\[2) \ \ grad_{r}=\nabla_{r} d_{\mathrm{KL}}(f_{\theta}(x), f_{\theta}(x+r))\]
\[3) \ \ r_{adv}=\epsilon \frac{grad_{r}}{\|grad_{r}\|}\]
<p>Note that the computation above is a single iteration of the approximation of \(r_{adv}\), for a more accurate
approximation, we consider \(r_{adv} = r\) and recompute \(r_{adv}\) following the last two steps.
But in general, given how computationally expensive this computation is, requiring additional forward and backward passes,
we only apply a single power iteration for computing the adversarial perturbation.</p>
<p>With the optimal perturbation \(r_{adv}\), we can then compute the unsupervised loss as the MSE
between the two predictions of the model, with and without the injection of \(r_{adv}\):</p>
\[\mathcal{L}_u = w \frac{1}{|\mathcal{D}_u|} \sum_{x_u \in \mathcal{D}_u}
d_{\mathrm{MSE}}(f_{\theta}(x_u), f_{\theta}(x_u + r_{adv}))\]
<p>For a more stable training, we can use a mean teacher to generate stable targets
by replacing \(f_{\theta}(x_u)\) with \(f_{\theta^{\prime}}(x_u)\), where \(f_{\theta^{\prime}}\)
is an EMA teacher model of the student \(f_{\theta}\).</p>
<figure style="width: 75%" class="align-center">
<img src="https://yassouali.github.io//ml-blog/images/SSL/vat.png" alt="" />
<figcaption>Fig. 7. Examples of the perturbed inputs for different values of the scaling hyperparameter Ɛ.
(Image source: <a href="https://arxiv.org/abs/1704.03976">Miyato et al</a>)
</figcaption>
</figure>
<h2 id="adversarial-dropout">Adversarial Dropout</h2>
<p>Instead of using an additive adversarial noise as VAT, <a href="https://arxiv.org/abs/1707.03631">Park et al.</a>
propose adversarial dropout (AdD), in which dropout masks are adversarially optimized to alter the model’s predictions.
With this type of perturbations, we induce a sparse structure of the neural network,
while the other forms of additive noise do not
make changes to the structure of the neural network directly.</p>
<p>The first step is to find the dropout conditions that is most sensitive to the model’s predictions. In a SSL setting,
where we do not have access to the true labels, we use the model predictions on the unlabeled data points to approximate
the adversarial dropout mast \(\epsilon^{adv}\), which is subject to the boundary condition:
\(\|\epsilon^{adv}-\epsilon\|_{2} \leq \delta H\) with \(H\)
as the dropout layer dimension and a hyperparameter \(\delta\),
which restricts adversarial dropout mask to be infinitesimally different from the random dropout mask \(\epsilon\).
Without this constraint, the adversarial dropout might induce a layer without any connections.
By restricting the adversarial dropout to be similar to the random dropout,
we prevent finding such an irrational layer, which does not support backpropagation.</p>
<p>Similar to VAT, we start from a random dropout mask, we compute a KL-divergence loss
between the outputs with and without dropout, and given the gradients of the loss with respect to the activations
before the dropout layer, we update the random dropout mask in an adversarial manner.
The prediction function \(f_{\theta}\) is divided into two parts, \(f_{\theta_1}\)
and \(f_{\theta_2}\), where \(f_{\theta}(x_i, \epsilon)=f_{\theta_{2}}(f_{\theta_{1}}(x_i) \odot \epsilon)\),
we start by computing an approximation of the jacobian matrix as follows:</p>
\[J(x_i, \epsilon) \approx f_{\theta_{1}}(x_i)\odot
\nabla_{f_{\theta_{1}}(x_i)} d_{\mathrm{KL}}(f_{\theta}(x_i),
f_{\theta}(x_i, \epsilon))\]
<p>Using \(J(x_i, \epsilon)\), we can then update the random dropout mask \(\epsilon\)
to obtain \(\epsilon^{adv}\), so that if \(\epsilon(i) = 0\) and \(J(x_i, \epsilon)(i) > 0\)
or \(\epsilon(i) = 1\) and \(J(x_i, \epsilon)(i) < 0\) at a given position \(i\), we inverse the
value of \(\epsilon\) at that location. Resulting in \(\epsilon^{adv}\), which can then
be used to compute the unsupervised loss:</p>
\[\mathcal{L}_u = w \frac{1}{|\mathcal{D}_u|} \sum_{x_u \in \mathcal{D}_u}
d_{\mathrm{MSE}}(f_{\theta}(x_u), f_{\theta}(x_u, \epsilon^{adv}))\]
<h2 id="interpolation-consistency-training">Interpolation Consistency Training</h2>
<p>As discussed earlier, the random perturbations are inefficient in high dimensions, given that only a limited subset of the
input perturbations are capable of pushing the decision boundary into low-density regions. VAT
and AdD find the adversarial perturbations that will maximize the change in the model’s predictions, which
involve multiple forward and backward passes to compute these perturbations. This additional computation can
be restrictive in many cases and makes such methods less appealing.
As an alternative, <a href="https://arxiv.org/abs/1903.03825">Verma et al.</a> propose Interpolation Consistency Training (ICT) as an
efficient consistency regularization technique for SSL.</p>
<p>Given a mixup operation
\(\operatorname{Mix}_{\lambda}(a, b)=\lambda \cdot a+(1-\lambda) \cdot b\) that outputs an interpolation
between the two inputs with a weight \(\lambda \sim \operatorname{Beta}(\alpha, \alpha)\) for \(\alpha \in(0, \infty)\).
As shown in Figure 8, ICT trains a prediction function \(f_{\theta}\) to provide consistent predictions at different interpolations
of unlabeled data points \(u_i\) and \(u_j\), where the targets are generated using a teacher model \(f_{\theta^{\prime}}\)
which is an EMA of \(f_{\theta}\):</p>
\[f_{\theta}(\operatorname{Mix}_{\lambda}(u_{j}, u_{k})) \approx
\operatorname{Mix}_{\lambda}(f_{\theta^{\prime}}(u_{j}), f_{\theta^{\prime}}(u_{k}))\]
<figure style="width: 90%" class="align-center">
<img src="https://yassouali.github.io//ml-blog/images/SSL/ICT.png" alt="" />
<figcaption>Fig. 8. ICT where a student model is trained to have consistent predictions at different interpolations
of unlabeled data points, where a teacher is used to generated the targets before the mixup operation.
(Image source: <a href="https://arxiv.org/abs/1903.03825">Verma et al</a>)
</figcaption>
</figure>
<p>The unsupervised objective is to have similar values between the student model’s prediction given a mixed input of two unlabeled data points and the mixed outputs of the teacher model.</p>
\[\mathcal{L}_u = w \frac{1}{|\mathcal{D}_u|} \sum_{u_j, u_k \in \mathcal{D}_u}
d_{\mathrm{MSE}}(f_{\theta}(\operatorname{Mix}_{\lambda}(u_{j}, u_{k})),
\operatorname{Mix}_{\lambda}(f_{\theta^{\prime}}(u_{j}), f_{\theta^{\prime}}(u_{k}))\]
<p>The benefit of ICT compared to random noise can be analyzed by considering the mixup operation
as a perturbation applied to a given unlabeled example: \(u_{j}+\delta=\operatorname{Mix}_{\lambda}(u_{j}, u_{k})\),
for a large number of classes and a with a similar distribution of examples per class, it is likely that the pair
of point \(\left(u_{j}, u_{k}\right)\) lie in different clusters and belong to different classes. If one of these two data points
lies in a low-density region, applying an interpolation toward \(u_{k}\) points to a low-density region, which is a good
direction to move the decision boundary toward.</p>
<h2 id="unsupervised-data-augmentation">Unsupervised Data Augmentation</h2>
<p>Unsupervised Data Augmentation (<a href="https://arxiv.org/abs/1904.12848">Xie et al.</a>) uses
advanced data augmentation methods, such as <a href="https://arxiv.org/abs/1805.09501">AutoAugment</a>,
<a href="https://arxiv.org/abs/1909.13719">RandAugment</a> and <a href="https://arxiv.org/abs/1808.09381">Back Translation</a> as perturbations
for consistency training based SSL.
Similar to supervised learning, advanced data augmentation methods can also provide extra advantages
over simple augmentations and random noise
for consistency training, given that
(1) it generates realistic augmented examples, making it safe to encourage the consistency between predictions
on the original and augmented examples. (2) it can generate a diverse set of examples improving the sample efficiency and
(3) it is capable of providing the missing inductive biases for different tasks.</p>
<p>Motivated by these points, <a href="https://arxiv.org/abs/1904.12848">Xie et al.</a> propose to apply the following
augmentations to generate transformed versions of the unlabeled inputs:</p>
<ul>
<li>RandAugment for Image Classification: consists of uniformly sampling from the same set of possible
transformations in PIL, without requiring any labeled data to search to find a good augmentation strategy.</li>
<li>Back-translation for Text Classification: consists of translating an existing example in language A
into another language B, and then translating it back into A to obtain an augmented example.</li>
</ul>
<figure style="width: 90%" class="align-center">
<img src="https://yassouali.github.io//ml-blog/images/SSL/uda.png" alt="" />
<figcaption>Fig. 9. The training procedure in UDA.
(Image source: <a href="https://arxiv.org/abs/1904.12848">Qizhe Xie et al</a>)
</figcaption>
</figure>
<p>After defining the augmentations to be applied during training, the training procedure shown in Figure 9 is quite straight forward.
The objective is to have the correct predictions over the labeled set, and consistency of predictions
on the original and augmented examples from the unlabeled set.</p>
<h1 id="entropy-minimization">Entropy Minimization</h1>
<p>In the previous section, in a setting where the cluster assumption is maintained, we enforce consistency of predictions
to push the decision boundary into low-density regions to avoid classifying
samples from the same cluster with distinct classes, which is a violation of the cluster assumption.
Another way to enforce this is to encourage the network to make confident (low-entropy) predictions on
unlabeled data regardless of the predicted class, discouraging the decision boundary from passing near data points
where it would otherwise be forced to produce low-confidence predictions.
This is done by adding a loss term which minimizes the entropy of the prediction function \(f_\theta(x)\),
e.g., for a categorical output space with \(C\) possible classes, the entropy minimization term
(<a href="http://papers.nips.cc/paper/2740-semi-supervised-learning-by-entropy-minimization.pdf">Grandvalet et al.</a>) is:</p>
\[-\sum_{k=1}^{C} f_{\theta}(x)_{k} \log f_{\theta}(x)_{k}\]
<p>However, with high capacity models such as neural networks,
the model can quickly overfit to low confident data points by simply outputting large logits, resulting in a model
with very confident predictions.
On its own, entropy minimization doesn’t produce competitive results compared to other SSL methods but can produce state-of-the-art results when combined with other SSL approaches.</p>
<h1 id="proxy-label-methods">Proxy-label Methods</h1>
<p>Proxy label methods (<a href="https://arxiv.org/abs/1804.09530">Ruder et al.</a>) are the class of SSL algorithms
that produce proxy labels on unlabeled data, using the prediction function itself or some variant of it without any supervision. These proxy labels
are then used as targets together with the labeled data, providing some additional training information
even if the produced labels are often noisy or weak and do not reflect the ground truth, which can be divided mainly
into two groups: self-training, where the model itself produces the proxy labels; and multi-view learning,
where the proxy labels are produced by models trained on different views of the data.</p>
<h2 id="self-training">Self-training</h2>
<p>In self-training or bootstrapping, the small amount of labeled data \(\mathcal{D}_l\) is first used to train a prediction function
\(f_{\theta}\). The trained model is then used to assign pseudo-labels to the unlabeled data points in \(\mathcal{D}_u\).
Given an output \(f_{\theta}(x_u)\) for an unlabeled data point \(x_u\) in the form of a probability distribution
over the classes, the pair \((x_u, \text{argmax}f_{\theta}(x_u))\) is added to the labeled set if the probability assigned to
its most likely class is higher than a predetermined threshold \(\tau\). The process of training the model using
the augmented labeled set and then set using it to label the remaining of \(\mathcal{D}_u\) is repeated until the model is incapable
of producing confident predictions.</p>
<p><strong>Pseudo-labeling</strong> can also be seen as a special case of self-training, differing
only in the heuristics used to decide which proxy labeled examples to retain, such as using the relative confidence
instead of the absolute confidence, where the top \(n\) unlabeled examples predicted with the highest
confidence after every epoch is added to the labeled training dataset \(\mathcal{D}_l\).</p>
<p>The main downside of such methods is that the model is unable to correct its own mistakes
and any biased and wrong classifications can be quickly amplified resulting in confident but erroneous proxy labels on the
unlabeled data points.</p>
<h2 id="meta-pseudo-labels">Meta Pseudo Labels</h2>
<p>Given how important the heuristics used
to generate the proxy labels, where a proper method could lead to a sizable gain.
<a href="https://arxiv.org/abs/2003.10580">Pham et al.</a> propose to use the student-teacher setting, where the teacher
model is responsible for producing the proxy labels based on an efficient meta-learning algorithm called Meta Pseudo Labels (MPL),
which encourages the teacher to adjust the target distributions
of training examples in a manner that improves the learning of the student model. The teacher is updated by policy gradients computed
by evaluating the student model on a held-out validation set.</p>
<p>A given training step of MPL consists of two phases (Figure 10):</p>
<ul>
<li><strong>Phase 1:</strong> The Student learns from the teacher. In this phase, given a single input example \(x_u\), the teacher \(f_{\theta^{\prime}}\)
produces a target class-distribution to train the student \(f_{\theta}\), where
the pair \((x_u, f_{\theta^{\prime}}(x_u))\) is shown to the student to update its parameters by back-propagating from the cross-entropy loss.</li>
<li><strong>Phase 2:</strong> The Teacher learns from the Student’s Validation Loss. After the student updates its parameters in first step,
its new parameter \(\theta(t+1)\)
are evaluated on an example \((x_{val},y_{val})\) from the held-out validation dataset using the cross-entropy loss. Since the validation loss
depends on \(\theta^{\prime}\) via the first step, this validation cross-entropy loss is also a function of the teacher’s weights \(\theta^{\prime}\).
This dependency allows us to compute the gradients of the validation loss with respect to the teacher’s weights, and then update \(\theta^{\prime}\)
to minimize the validation loss using policy gradients.</li>
</ul>
<figure style="width: 75%" class="align-center">
<img src="https://yassouali.github.io//ml-blog/images/SSL/MPL.png" alt="" />
<figcaption>Fig. 10. The MPL training procedure.
(Image source: <a href="https://arxiv.org/abs/2003.10580">Pham et al</a>)
</figcaption>
</figure>
<p>While the student’s performance allows the teacher to adjust and adapt to the student’s learning state, this signal alone
is not sufficient to train the teacher since when the teacher has observed enough evidence to produce meaningful target
distributions to teach the student, the student might have already entered a lousy region of parameters. To overcome this, the teacher is also trained using the pair of labeled data points from the held-out validation set.</p>
<h2 id="multi-view-training">Multi-view training</h2>
<p>Multi-view training (MVL, <a href="https://www.sciencedirect.com/science/article/abs/pii/S1566253516302032">Zhao et al.</a>)
utilizes multi-view data that are very common in real-world applications, where
different views can be collected by different measuring methods (e.g., color information and texture information for images)
or by creating limited views of the original data. In such a setting, MVL aims to learn a distinct prediction function \(f_{\theta_i}\)
to model a given view \(v_{i}(x)\) of a data point \(x\), and jointly optimize all the functions to improve the generalization performance.
Ideally, the possible views complement each other so that the produced models can collaborate in improving each other’s performance.</p>
<h3 id="co-training">Co-training</h3>
<p>Co-training (<a href="https://www.cs.cmu.edu/~avrim/Papers/cotrain.pdf">Blum et al.</a>) requires that each data point \(x\) can be represented using two conditionally independent views \(v_1(x)\) and \(v_2(x)\), and that each view is sufficient to train a good model.</p>
<p>After training two prediction functions \(f_{\theta_1}\) and \(f_{\theta_2}\) on a specific view on the labeled set \(\mathcal{D}_l\).
We start the proxy labeling procedure, where, at each iteration, an unlabeled data point is added to the training
set of the model \(f_{\theta_i}\) if the other model \(f_{\theta_j}\) outputs a confident prediction with a probability higher
than a threshold \(\tau\). This way, one of the models provides newly labeled examples where the other model is uncertain.
The two views \(v_1(x)\) and \(v_2(x)\) can also be generated using consistency training methods detailed in the previous section,
for example, <a href="https://arxiv.org/abs/1803.05984">Qiao et al.</a> use adversarial perturbations to produce new views for deep co-training
for image classification, where the models are encouraged to have the same predictions on \(\mathcal{D}_l\) but make different
errors when they are exposed to adversarial attacks.</p>
<p><strong>Democratic Co-training</strong> (<a href="http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.76.3152&rep=rep1&type=pdf">Zhou et al.</a>),
an extension of Co-training, consists of replacing the different views of the input data with a number of models with different architectures
and learning algorithms, which are first trained on the labeled examples. The trained models are then used to label a given an example \(x\) if a majority of models confidently agree on the label of an example.</p>
<h3 id="tri-training">Tri-Training</h3>
<p>Tri-training (<a href="http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.487.2431&rep=rep1&type=pdf">Zhou et al.</a>)
tries to reduce the bias of the predictions on unlabeled data produced with self-training
by utilizing the agreement of three independently trained models instead of a single model.
First, the labeled data \(\mathcal{D}_l\) is used to train three prediction functions: \(f_{\theta_1}\), \(f_{\theta_2}\) and \(f_{\theta_3}\).
An unlabeled data point \(x\) is then added to the supervised training set of the function \(f_{\theta_i}\) if the other two models
agree on its predicted label. The training stops if no data points are being added to any of the model’s training sets.</p>
<p>For a stronger heuristic when selecting the prediction to consider as proxy labels,
<strong>Tri-training with disagreement</strong> (<a href="https://www.aclweb.org/anthology/P10-2038.pdf">Søgaard</a>),
in addition to the only considering confident predictions with a probability higher than
a threshold \(\tau\),
only adds a data point \(x\) to the training set of the model \(f_{\theta_i}\)
if the other two models agree, and \(f_{\theta_i}\) disagree on the predicted label.
This way, the training set of a given model is only extended with data points where the model needs
to be strengthened, and the easy examples that can skew the labeled data are avoided.</p>
<p>Using Tri-training with neural networks can be very expensive, requiring predictions for each one of the three models on all the unlabeled data. <a href="https://arxiv.org/abs/1804.09530">Ruder et al.</a> propose to sample a
limited number of unlabeled data points at each training epoch,
the candidate pool size is increased as the training progresses and the models become more accurate.
<strong>Multi-task tri-training</strong> (<a href="https://arxiv.org/abs/1804.09530">Ruder et al.</a>)
can also be used to reduce the time and sample complexity, where all three models
share the same feature-extractor with model-specific classification layers. This way, the models are trained jointly
with an additional orthogonality constraint on two of the three classification layers to be added to loss term, to avoiding
learning similar models and falling back to the standard case of self-training.</p>
<h1 id="holistic-methods">Holistic Methods</h1>
<p>An emerging line of work in SSL is a set of holistic approaches that unify the current dominant approaches in SSL
in a single framework, achieving better performances.</p>
<h2 id="mixmatch">MixMatch</h2>
<p><a href="https://arxiv.org/abs/1905.02249">Berthelot et al.</a> propose a “holistic” approach which gracefully unifies
ideas and components from the dominant paradigms for SSL, resulting in a algorithm that is greater than the sum of its parts
and surpasses the performance of the traditional approaches.</p>
<figure style="width: 75%" class="align-center">
<img src="https://yassouali.github.io//ml-blog/images/SSL/mixmatch.png" alt="" />
<figcaption>Fig. 11. MixMatch. The procedure of label guessing process used in MixMatch, taking as input a batch of unlabeled examples, and
outputting a batch of K augmented version of each input, with a corresponding sharpened proxy labels.
(Image source: <a href="https://arxiv.org/abs/1905.02249">David Berthelot et al.</a>)
</figcaption>
</figure>
<p>MixMatch takes as input a batch from the labeled set \(\mathcal{D}_l\) containing a pair of inputs and their corresponding one-hot targets,
a batch from the unlabeled set \(\mathcal{D}_u\) containing only unlabeled data, and a set of hyperparameters: sharpening softmax temperature \(T\),
number of augmentations \(K\), Beta distribution parameter \(\alpha\) for MixUp. Producing a batch of augmented labeled examples
and a batch of augmented unlabeled examples with their proxy labels. These augmented examples can then be used to
compute the losses and train the model. Specifically, MixMatch consists of the following steps:</p>
<ul>
<li><strong>Step 1: Data Augmentation.</strong> Using a given transformation, a labeled example \(x^l\) from the labeled batch is transformed, generating
its augmented versions \(\tilde{x}^l\). For an unlabeled example \(x^u\),
the augmentation function is applied \(K\) times, resulting in \(K\) augmented
versions of the unlabeled examples {\(\tilde{x}_1^u\), …, \(\tilde{x}_K^u\)}.</li>
<li><strong>Step 2: Label Guessing.</strong> The second step consists of producing proxy labels for the unlabeled examples.
First, we generate the predictions for the \(K\) augmented versions of
each unlabeled example using the predictions function \(f_\theta\). The \(K\) predictions are then averaged together, obtaining
a proxy or a pseudo label \(\hat{y}^u = 1/K \sum_{k=1}^{K}(\hat{y}^u_k)\) for each one of the augmentations of the unlabeled example \(x^u\):
{(\(\tilde{x}_1^u, \hat{y}^u\)), …, (\(\tilde{x}_K^u, \hat{y}^u\))}.</li>
<li><strong>Step 3: Sharpening.</strong> To push the model to produce confident predictions and minimize the entropy of the output distribution, the generated
proxy labels \(\hat{y}^u\) in step 2 in the form of a probability distribution over \(C\) classes are sharpened by adjusting the temperature
of the categorical distribution, computed as follows where \((\hat{y}^u)_k\) refers to the probability of class \(k\) out of \(C\) classes:</li>
</ul>
\[(\hat{y}^u)_k = (\hat{y}^u)_k^{\frac{1}{T}} / \sum_{k=1}^{C} (\hat{y}^u)_k^{\frac{1}{T}}\]
<ul>
<li><strong>Step 4 MixUp.</strong> After the previous step, we created two new augmented batch, a batch \(\mathcal{L}\) of augmented labeled
examples and their target, and a batch \(\mathcal{U}\) of augmented unlabeled examples and their sharpened proxy labels. Note that the size
of \(\mathcal{U}\) is \(K\) times larger than the original batch given that each example \(x_u\) is replaced by its \(K\)
augmented versions. In the last step, we mix these two batches. First, a new batch merging both batches is created
\(\mathcal{W}=\text{Shuffle}(\text{Concat}(\mathcal{L}, \mathcal{U}))\). \(\mathcal{W}\) is then
divided into two batches: \(\mathcal{W}_1\) of the same size as \(\mathcal{L}\) and \(\mathcal{W}_2\) of the same
size as \(\mathcal{L}\). Using a Mixup operation that is slightly adjusted so that the mixed
example is closer the labeled examples, the final step is to create new labeled and unlabeled batches by mixing the produced batches together
using Mixup as follows:</li>
</ul>
\[\mathcal{L}{\prime}=\operatorname{MixUp}(\mathcal{L}, \mathcal{W}_1)\]
\[\mathcal{U}{\prime}=\operatorname{MixUp}(\mathcal{U}, \mathcal{W}_2)\]
<p>After creating two augmented batches \(\mathcal{L}{\prime}\) and \(\mathcal{U}{\prime}\) using MixMatch,
we can then train the model using the standard SSL by computing the CE loss for the supervised loss, and
the consistency loss for the unsupervised loss using the augmented batches as follows:</p>
\[\mathcal{L}_s=\frac{1}{|\mathcal{L}^{\prime}|} \sum_{x_l, y \in \mathcal{L}^{\prime}}
\mathrm{H}(y, f_\theta(x_l)))\]
\[\mathcal{L}_u=w \frac{1}{|\mathcal{U}^{\prime}|} \sum_{x_u, \hat{y} \in \mathcal{U}^{\prime}}
d_{\mathrm{MSE}}(\hat{y}, f_{\theta}(x_u))\]
<h2 id="remixmatch">ReMixMatch</h2>
<p><a href="https://arxiv.org/abs/1911.09785">Berthelot et al.</a> propose to
improve MixMatch by introducing two new techniques: <strong>distribution alignment</strong> and <strong>augmentation anchoring</strong>.
Distribution alignment encourages the marginal distribution of predictions on unlabeled data
to be close to the marginal distribution of ground-truth labels. Augmentation anchoring feeds multiple strongly
augmented versions of the input into the model and encourages each output to be close to the prediction
for a weakly-augmented version of the same input.</p>
<figure style="width: 90%" class="align-center">
<img src="https://yassouali.github.io//ml-blog/images/SSL/remixmatch.png" alt="" />
<figcaption>Fig. 12. ReMixMatch. Distribution alignment adjusts the guessed labels distributions to match the ground-truth class distribution divided by the average model predictions on unlabeled data. Augmentation anchoring uses the prediction obtained using a weakly augmented image as targets for a strongly augmented version of the same image.
(Image source: <a href="https://arxiv.org/abs/1911.09785">Pham et al</a>)
</figcaption>
</figure>
<p><strong>Distribution alignment:</strong> In order to force that the aggregate of predictions on unlabeled data to match
the distribution of the provided labeled data. Over the course of training, a running average \(\tilde{y}\) of the model’s predictions
on unlabeled data is maintained over the last 128 batches. For the marginal class distribution \(p(y)\), it is estimated based on the labeled
examples seen during training. Given a prediction \(f_{\theta}(x_u)\) on the unlabeled example \(x_u\), the output probability distribution
is aligned as follows: \(f_{\theta}(x_u) = \text { Normalize }(f_{\theta}(x_u) \times p(y) / \tilde{y})\).</p>
<p><strong>Augmentation Anchoring:</strong> MixMatch uses a simple flip-and-crop augmentation strategy, ReMixMatch replaces the weak augmentations with strong augmentations
learned using a control theory based strategy following AutoAugment. With such augmentations, the model’s prediction for a weakly augmented unlabeled image is used as the guessed label for many strongly augmented versions of the same image in a standard cross-entropy loss.</p>
<p>For training, MixMatch is applied to the unlabeled and labeled batches, with the application of distribution alignment and replacing the \(K\) weakly
augmented example with a strongly augmented example, in addition to using the weakly augmented examples for predicting proxy labels
for the unlabeled, strongly augmented examples. With two augmented batches \(\mathcal{L}^{\prime}\) and \(\mathcal{U}^{\prime}\), the
supervised and unsupervised losses are computed as follows:</p>
\[\mathcal{L}_s=\frac{1}{|\mathcal{L}^{\prime}|} \sum_{x_l, y \in \mathcal{L}^{\prime}}
\mathrm{H}(y, f_\theta(x_l)))\]
\[\mathcal{L}_u=w \frac{1}{|\mathcal{U}^{\prime}|} \sum_{x_u, \hat{y} \in \mathcal{U}^{\prime}}
\mathrm{H}(\hat{y}, f_\theta(x_u)))\]
<p>In addition to these losses, the authors add a self-supervised loss. First, a new unlabeled batch
\(\hat{\mathcal{U}}^{\prime}\) of examples is created by rotating all of the examples with an angle \(r \sim\{0,90,180,270\}\). The rotated
examples are then used to compute a self-supervised loss, where the classification layer on top of the model predicts the correct applied
rotation, in addition to the cross-entropy loss over the rotated examples:</p>
\[\mathcal{L}_{SL} = w^{\prime}
\frac{1}{|\hat{\mathcal{U}}^{\prime}|} \sum_{x_u, \hat{y} \in \hat{\mathcal{U}}^{\prime}}
\mathrm{H}(\hat{y}, f_\theta(x_u))) + \lambda
\frac{1}{|\hat{\mathcal{U}}^{\prime}|} \sum_{x_u \in \hat{\mathcal{U}}^{\prime}}
\mathrm{H}(r, f_\theta(x_u)))\]
<h2 id="fixmatch">FixMatch</h2>
<p><a href="https://arxiv.org/abs/2001.07685">Kihyuk Sohn et al.</a> present FixMatch, a simple SSL algorithm that combines consistency regularization and pseudo-labeling.
In FixMatch (Figure 13), both the supervised and unsupervised losses are computed using a cross-entropy loss.
For labeled examples, the provided targets are used. For unlabeled examples \(x_u\), a weakly augmented version is first computed
using a weak augmentation function \(A_w\). As in self-training, the predicted label is then considered as a proxy label if
the highest class probability is greater than a threshold \(\tau\). With a proxy label for \(x_u\), \(K\) strongly augmented examples are generated
using a strong augmentation function \(A_s\), we then assign to these strongly augmented versions the proxy label obtained with the weakly
labeled version. With a batch of unlabeled examples of size \(\mathcal{D}_u\), the unsupervised loss can be written as follows:</p>
\[\mathcal{L}_u = w \frac{1}{K |\mathcal{D}_u|} \sum_{x_u \in \mathcal{D}_u} \sum_{i=1}^{K}
\mathbb{1}(\max (f_\theta(A_w(x_u))) \geq \tau)
\mathrm{H} (f_\theta(A_w(x_u)), f_\theta(A_s(x_u)))\]
<figure style="width: 75%" class="align-center">
<img src="https://yassouali.github.io//ml-blog/images/SSL/fixmatch.png" alt="" />
<figcaption>Fig. 13. FixMatch. The model prediction on a weakly augmented input is considered as target if the maximum output
class probability is above threshold, this target can then be used to train the model on a strongly augmented version of the same input using standard cross-entropy loss.
(Image source: <a href="https://arxiv.org/abs/2001.07685">Pham et al</a>)
</figcaption>
</figure>
<p><strong>Augmentations.</strong> Weak augmentations consist of a standard flip-and-shift augmentation strategy.
Specifically, the images are flipped horizontally with a probability of 50% on all datasets except SVHN, in addition to randomly translating
images by up to 12.5% vertically and horizontally. For the strong augmentations, RandAugment and CTAugment are used
where a given transformation (e.g.,, color inversion, translation, contrast adjustment, etc.) is randomly selected for each sample in a batch
of training examples, where the amplitude of the transformation is a hyperparameter that is optimized during training.</p>
<p>Other important factors in FixMatch are the usage of adam optimizer,
weight decay regularization and the learning rate schedule used, the authors propose to use a cosine learning rate decay with a decay of
\(\eta \cos (\frac{7 \pi t}{16 T})\),
where \(\eta\) is the initial learning rate, \(t\) is the current training step, and \(T\) is the total number of training steps.</p>
<h1 id="references">References</h1>
<p><sup>
[1] Chapelle et al. <a href="http://pages.cs.wisc.edu/~jerryzhu/pub/ssl_survey.pdf">Semi-supervised learning book</a>. IEEE Transactions on Neural Networks, 2009.<br />
[2] Xiaojin Jerry Zhu. <a href="http://www.acad.bg/ebook/ml/MITPress- SemiSupervised Learning.pdf">Semi-supervised learning literature survey</a>. Technical report, University of Wisconsin-Madison Department of Computer Sciences, 2005.<br />
[3] Rasmus et al. <a href="http://pages.cs.wisc.edu/~jerryzhu/pub/ssl_survey.pdf">Semi-supervised learning with ladder networks</a>. NIPS 2015.<br />
[4] Samuli Laine, Timo Aila. <a href="https://arxiv.org/abs/1610.02242">Temporal Ensembling for Semi-Supervised Learning</a>. ICLR 2017.<br />
[5] Harri Valpola et al. <a href="https://arxiv.org/abs/1411.7783">From neural PCA to deep unsupervised learning</a>. Advances in Independent Component Analysis and Learning Machines 2015.<br />
[6] Antti Tarvainen, Harri Valpola. <a href="https://arxiv.org/abs/1703.01780">Mean teachers are better role models:Weight-averaged consistency targets improve semi-supervised deep learning results</a>. NIPS 2017.<br />
[7] Takeru Miyato et al. <a href="https://arxiv.org/abs/1704.03976">Virtual adversarial training: a regularization method for supervised and semi-supervised learning.</a> Transactions on Pattern Analysis and Machine Intelligence 2018.<br />
[8] Ian Goodfellow et al. <a href="https://arxiv.org/abs/1412.6572">Explaining and harnessing adversarial examples.</a>. ICLR 2015.<br />
[9] Sungrae Park et al. <a href="https://arxiv.org/abs/1707.03631">Adversarial Dropout for Supervised and Semi-Supervised Learning.</a>. AAAI 2018.<br />
[10] Vikas Verma et al. <a href="https://arxiv.org/abs/1903.03825">Interpolation Consistency Training for Semi-Supervised Learning.</a>. IJCAI 2019.<br />
[11] Qizhe Xie et al. <a href="https://arxiv.org/abs/1904.12848">Unsupervised Data Augmentation for Consistency Training.</a>. arXiv 2019.<br />
[12] Zhanghan Ke et al. <a href="https://arxiv.org/abs/1909.01804">Dual Student: Breaking the Limits of the Teacher in Semi-supervised Learning.</a>. ICCV 2019.<br />
[13] Sebastian Ruder et al. <a href="https://arxiv.org/abs/1804.09530">Strong Baselines for Neural Semi-supervised Learning under Domain Shift.</a>. ACL 2018.<br />
[14] Hieu Pham et al. <a href="https://arxiv.org/abs/2003.10580">Meta Pseudo Labels.</a> Preprint 2020.<br />
[15] Jing Zhao et al. <a href="https://www.sciencedirect.com/science/article/abs/pii/S1566253516302032">Multi-view learning overview: Recent progress and new challenges.</a> Information Fusion, 2017.<br />
[16] Avrim Blum, Tom Michael. <a href="https://www.cs.cmu.edu/~avrim/Papers/cotrain.pdf">Combining labeled and unlabeled data with co-training.</a>, COLT 1992.
[17] Siyuan Qiao, Wei Shen, Zhishuai Zhang, Bo Wang, Alan Yuille. <a href="https://arxiv.org/abs/1803.05984">Deep Co-Training for Semi-Supervised Image Recognition.</a>, ECCV 2018.<br />
[18] Yan Zhou, Sally Goldman. <a href="http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.76.3152&rep=rep1&type=pdf">Democratic Co-Learning.</a>, ICTAI 2004.<br />
[19] Zhi-Hua Zhou, Ming Li. <a href="http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.487.2431&rep=rep1&type=pdf">Tri-Training: Exploiting Unlabled Data Using Three Classifiers.</a> IEEE Trans.Data Eng 2015.<br />
[20] Anders Søgaard. <a href="https://www.aclweb.org/anthology/P10-2038.pdf">Simple Semi-Supervised Training of Part-Of-Speech Taggers.</a> NIPS 2019.<br />
[21] Yves Grandvalet et al. <a href="https://papers.nips.cc/paper/2740-semi-supervised-learning-by-entropy-minimization.pdf">Semi-supervised learning by entropy minimization.</a> NIPS 2005.<br />
[22] David Berthelot et al. <a href="https://arxiv.org/abs/1905.02249">MixMatch: A Holistic Approach to Semi-Supervised Learning.</a> NIPS 2019.<br />
[23] David Berthelot et al. <a href="https://arxiv.org/abs/1911.09785">ReMixMatch: Semi-Supervised Learning with Distribution Matching and Augmentation Anchoring.</a> ICLR 2020.<br />
[24] Kihyuk Sohn et al. <a href="https://arxiv.org/abs/2001.07685">FixMatch: Simplifying Semi-Supervised Learning with Consistency and Confidence.</a> Preprint 2020.</sup></p>Yassineouali.yasine@gmail.comGiven the large amounts of training data required to train deep nets, but collecting big datasets is not cost nor time effective. As a result there is a growing need to develop data efficient methods. Semi-supervised learning (SSL) is possible solutions to such hurdles. In this blog post we present some of the new advance in SSL in the age of Deep Learning.