FCSS Descriptor

Abstract

    FCSS Descriptor
  • We present a descriptor, called fully convolutional self-similarity (FCSS), for dense semantic correspondence. Unlike

    traditional dense correspondence for estimating depth or optical flow, semantic correspondence estimation poses additional challenges

    due to intra-class appearance and shape variations among different instances within the same object or scene category. To robustly

    match points across semantically similar images, we formulate FCSS using local self-similarity (LSS), which is inherently insensitive to

    intra-class appearance variations. LSS is incorporated through a proposed convolutional self-similarity (CSS) layer, where the sampling

    patterns and the self-similarity measure are jointly learned in an end-to-end and multi-scale manner. Furthermore, to address shape

    variations among object instances, we propose a convolutional affine transformer (CAT) layer that estimates explicit affine

    transformation fields at each pixel to transform the sampling patterns and corresponding receptive fields. As training data for semantic

    correspondence is rather limited, we propose to leverage object candidate priors provided in most existing datasets and also

    correspondence consistency between object pairs to enable weakly-supervised learning. Experiments demonstrate that FCSS

    significantly outperforms conventional handcrafted descriptors and CNN-based descriptors on various benchmarks.

Publication

  • FCSS: Fully Convolutional Self-Similarity for Dense Semantic Correspondence

    Seungryong Kim, Dongbo Min, Bumsub Ham, Sangryul Jeon, Stephen Lin, and Kwanghoon Sohn

    IEEE Conference on Computer Vision and Pattern Recognition (CVPR) 2017

    Paper: [pdf]

    Supplementary: [pdf]

    Poster: [pdf]

  • FCSS: Fully Convolutional Self-Similarity for Dense Semantic Correspondence

    Seungryong Kim, Dongbo Min, Bumsub Ham, Stephen Lin, and Kwanghoon Sohn

    IEEE Transactions on Pattern Analysis and Machine Intelligence (Under Review)

    Paper: [pdf]

[Top]

Download

  • Code: MatconvNet (v1.0) [zip]

[Top]