Context-Aware Probabilities (CAP) for Semantic Image Segmentation

Semantic image segmentation based on Convolutional Neural Networks (CNNs) is one of the main approaches in computer vision area. The methods based on deep convolutional neural networks, typically use a pre-trained CNN trained on the large image classification datasets as a backend to extract features (image descriptors) from the images. Whereas, the special size of output features from CNN backends are smaller than the input images, by stacking multiple deconvolutional layers on the last layer of backend network, the dimension of the output will be the same as the input image. Segmentation using local image descriptors without involving relationships between these local descriptors yield weak and uneven segmentation results. Inspired by these observations, in this research we propose Context-Aware Probabilities (CAP) unit. CAP unit generates probabilities for classes using local-image descriptors. This unit can be used in any semantic image segmentation architectures. We used CAP unit in Fully Convolutional Network (FCN) and DeepLab-v3-plus architectures and propose the new FCN-CAP and DeepLab-v3-plus-CAP architectures. Training the proposed architectures on PASCAL VOC2012 dataset shows 1.9% and 0.4% accuracy improvement compared to the corresponding basic architectures, respectively.