Probabilistic Graphical Models

EE 639 Advanced Topics in Signal Processing and Communication

Fall 2013 


[Home] [Lectures] [Final Project Challenge]


For the final project, we will focus on the practical problem of multi-modal gesture recognition from audio and RGB-D video data. This challenge was conducted during the 15th ACM International Conference on Multimodal Interaction 2013 (ICMI) in Sydney, Australia. The website that describes this competition can be found here.

The goal of the project is that of a multiple-instance, user-independent learning of gestures, which means learning to recognize gestures from several instances for each category performed by different users, drawn from a gesture vocabulary. A gesture vocabulary is a set of unique gestures, generally related to a particular task. In this challenge, we will focus on the recognition of a vocabulary of 20 Italian cultural/ anthropological signs. Each data sample contains a single person performing a sequence with 8 to 15 gestures and their related oral expression captured using a Microsoft Kinect. RGB, depth and audio information, together with additional information such as the skeleton and user mask.

Figure 1: RGB, Depth, Mask, Skeleton Data


Due to the complexity of the project, the class is divided into two teams, “EE” and “CS”, with the following membership:





Po, Hasan

Dr. Samson Cheung


Anthony, Sourov, Zhenghao

Dr. Ramakanth Kavuluru

The role of the adviser is to provide a sounding board for your ideas. Each team must complete all work on their own and any external software package used must be acknowledged.


The evaluation will be conducted at and you will need to create an account. As the competition has already been closed, all the training, validation, and test data are available here. However, training your algorithms on the test data is STRICTLY FORBIDDENED! Due to the time constraint, you are highly recommended to base your approach on a hybrid of methods that have already been attempted. The winning methods and other useful information are described in this paper.



What to turn in

11/15 (Friday)

Two-page proposal that defines the problem and assumptions, as well as outline the approach, timeline, and division of labor.

11/25 (Monday)

One-page progress report that state the progresses and any difficulties encounted. Your team must meet with your adviser at least once to discuss the progress.

12/13 (Friday)

In-class demonstration and presentation (20 minutes for each team)

12/20 (Friday)

Final Report (6-page double-column format)