Visualization of the SSC and DSC descriptors. Our architecture consists of a hierarchical self-correlational layer, circular spatial pyramid pooling layer, non-linear gating layer, and normalization layer.
We present the deep self-correlation (DSC) descriptor for establishing dense correspondences between images taken under different imaging modalities, such as different spectral ranges or lighting conditions. In this descriptor, local self-similar structure is modeled in a hierarchical manner that yields more precise localization ability and greater robustness to non-rigid image deformations than state-of-the-art descriptors. Specifically, DSC first computes multiple self-correlation surfaces over a local support window for randomly sampled patches, and then builds hierarchical self-correlation surfaces through average pooling. The feature responses on the self-correlation surfaces are then encoded through spatial pyramid pooling in a circular configuration. To better handle geometric variations such as scale and rotation, we further propose the geometry-invariant DSC (GI-DSC) that leverages a multi-scale self-correlation surface and a canonical orientation estimation technique. In contrast to descriptors based on deep convolutional neural networks (CNNs), DSC and GI-DSC are training-free, i.e., handcrafted descriptors, are robust to cross-modal imaging, and cannot be overfitted to the appearance variations of specific modalities. The state-of-the-art performance of DSC and GI-DSC on challenging cases of cross-modal image pairs with photometric and geometric variations is demonstrated through extensive experiments.
