Human segmentation by combining visible light and thermal cameras

For decades, the problem of segmenting human in video sequence has been a central issue in computer vision. Despite
its popularity, it remains to be a challenging problem because visual appearances are subjected to occlusion, illumination
change, hightlight, shadow and color confusion. Thermal infrared sensors are particularly useful for detecting human body as
due to the different temperature characteristics from that of the environment. We provide a system for robust segmentation of human in video sequences by fusing the visible-light and thermal imaginary.

The system first performs a simple calibration procedure to rectify the two camera views without knowing the cameras’ intrinsic characteristics. Then a blob-to-blob homography is learned on-the-fly by estimating the disparity of each blob so that a pixel level registration can be achieved. The multi-modality information is then combined under a two-tier tracking algorithm and a unified background model to attain precise segmentation. Preliminary experimental results shows significant improvements over existing schemes under various difficult scenarios. The prototype of our system can been seen in [1]

Quantitative measurement of the segmentation has been conducted. Twelve frames are randomly chosen and hand segmented by interactive graph cut algorithm [2]. We then compare the accuracy of segmentation by background segmentation alone and with fusion algorithm. In figure below, the three graphs in the first row are respectively image segmentation, infrared segmentation and fused segmentation. The leftmost graph in the second row is the ground truth segmentation. Then last two graphs are the image segmentation and fused segmentation results overlapped with ground truth. The pink part is the correct segment, red color denotes the false negative and the blue is the false positive. The average accuracy for the 12 frames can be seen in table below. We can see that through proper morphological operation, the fused segmentation algorithm promotes a very low false negative rate (1.6%) which is crucial for privacy protection and keep the false positive rate is at the same level as the image segmentation at the same time.

[2] C Rother, V Kolmogorov, A Blake, "GrabCut: interactive foreground extraction using iterated graph cuts", ACM Transactions on Graphics (TOG), 2004.