Fully Convolutional Self-Similarity (FCSS) Descriptor

Seungryong Kim1

Dongbo Min2

Bumsub Ham3

Stephen Lin4

Kwanghoon Sohn3

Korea University1, Ewha Womans University2, Yonsei University3, MSRA4

[CVPR'17 paper]

[CVPR'17 supp]

[TPAMI'20 paper]

[Code]

Visualization of our FCSS results: (a) source image, (b) target image, (c) warped source image using dense correspondences, (d), (e) enlarged windows for source and target images, (f), (g) local self- similarities computed by our FCSS descriptor between source and tar- get images. Even though there are significant differences in appearance among different instances within the same object category in (a) and (b), their local self-similarities computed by our FCSS descriptor are preserved as shown in (f) and (g), providing robustness to intra-class appearance and shape variations.

We present a descriptor, called fully convolutional self-similarity (FCSS), for dense semantic correspondence. Unlike traditional dense correspondence approaches for estimating depth or optical flow, semantic correspondence estimation poses additional challenges due to intra-class appearance and shape variations among different instances within the same object or scene category. To robustly match points across semantically similar images, we formulate FCSS using local self-similarity (LSS), which is inherently insensitive to intra-class appearance variations. LSS is incorporated through a proposed convolutional self-similarity (CSS) layer, where the sampling patterns and the self-similarity measure are jointly learned in an end-to-end and multi-scale manner. Furthermore, to address shape variations among different object instances, we propose a convolutional affine transformer (CAT) layer that estimates explicit affine transformation fields at each pixel to transform the sampling patterns and corresponding receptive fields. As training data for semantic correspondence is rather limited, we propose to leverage object candidate priors provided in most existing datasets and also correspondence consistency between object pairs to enable weakly-supervised learning. Experiments demonstrate that FCSS significantly outperforms conventional handcrafted descriptors and CNN-based descriptors on various benchmarks.

---