Automatic Real-Time Video Matting Using Time-of-Flight Camera and

Multichannel Poisson Equations

Liang Wang, Minglun Gong, Chenxi Zhang, Ruigang Yang, Cha Zhang, and Yee-Hong Yang

Project Abstract

We present an automatic real-time video matting system. The proposed system consists of two novel components. In order to automatically generate trimaps for live videos, we advocate a Time-of-Flight (TOF) camera-based approach to video bi-layer segmentation. Our algorithm combines color and depth cues in a probabilistic fusion framework. The scene depth information returned by the TOF camera is less sensitive to environment changes, which makes our method robust to illumination variation, dynamic background and camera motion. For the second step, we perform alpha matting based on the segmentation result. Our matting algorithm is based on a set of novel Poisson equations that are derived for handling multichannel color vectors, as well as the depth information captured. Real-time processing speed is achieved through optimizing the algorithm for parallel processing on graphics hardware. We demonstrate the effectiveness of our matting system on an extensive set of experimental results.

Binary Segmentation Datasets - color, depth images + ground truth binary segmentation

We grant permission to use and publish data sets used in this project. If you use our datasets, we request that you cite the paper [1] or [3]. [download] (167MB Zip file)

Supplemental Videos [download] (10MB Zip file)

References

[1] L. Wang, M. Gong, C. Zhang, R. Yang, C. Zhang, and Y-H. Yang. Automatic Real-Time Video Matting Using Time-of-Flight Camera and MultiChannel Poisson Equations. International Journal of Computer Vision (IJCV), accepted May 2011.

[2] M. Gong, L. Wang, R. Yang, and Y-H. Yang. Real-Time Video Matting Using MultiChannel Poisson Equations. In Proc. Graphics Interface (GI), 2010

[3] L. Wang, C. Zhang, R. Yang, and C. Zhang. TofCut: Towards Robust Real-Time Foreground Extraction Using a Time-of-Flight Camera. In Proc. The Fifth International Symposium on 3D Data Processing, Visualization and Transmission (3DPVT), 2010