Michael S. Ryoo: Research description

Human Activity Recognition

The goal of this project is to construct a general methodology that is applicable for the recognition of human activities. We especially focus on the semantic-level analysis, toward recognition of high-level activities with complex temporal, spatial, and logical structure. A methodology to automatically recognize human activities are essential for various applications including high-level surveillance systems, real-time monitoring of elderly people or babies, and human-computer interactions.

	Human-human interactions We have developed a new description-based approach for the recognition of high-level interactions between two persons. A human activity is represented by decomposing it into multiple sub-events and by specifying their necessary relationships(temporal, spatial, and logical), and is recognized by matching the representation with input videos. Sub-events of one activity may be composed of multiple sub-events of itself, capturing the hierarchical structure of human activities. As a result, continued and recursive activities such as 'fighting', 'greeting', 'assault', and 'pursuit' of two persons are recognized. Related Publications M. S. Ryoo and J. K. Aggarwal, "Recognition of Composite Human Activities through Context-Free Grammar based Representation", Proceedings of IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR), Vol. 2, pp. 1709-1719, New York, NY, 2006. pdf M. S. Ryoo and J. K. Aggarwal, "Semantic Understanding of Continued and Recursive Human Activities", Proceedings of 18th International Conference on Pattern Recognition (ICPR), Vol. 1, pp. 379~382, Hong Kong, 2006. pdf
	Human-object interactions The framework is extended for the recognition of interactions between humans and multiple objects. Human activities involving objects, such as "a person stealing another's suitcase", are recognized by considering objects and their motion. Ability to probabilistically compensate for the failure of its components (object recognition for example) is given to the system for more reliable recognition. Related Publications M. S. Ryoo and J. K. Aggarwal, "Hierarchical Recognition of Human Activities Interacting with Objects", Proceedings of 2nd International Workshop on Semantic Learning Applications in Multimedia (SLAM), in conjunction with CVPR, Minneapolis, MN, June 2007. pdf
group stealing      group arresting click to play the video     click to play the video	Group activities We designed and implemented a methodology for recognition of high-level group activities. Recognition of groups and their activities makes detection of high-level events possible, which are semantically meaningful when overall actions of multiple persons are considered jointly but not when they are considered individually (e.g. a group of thieves stealing an object from shop owners by distracting them). The proposed algorithm detects groups and recognizes their activities simultaneously by searching for individual members who satisfy temporal, spatial, and logical constraints of the group activity. Related Publications M. S. Ryoo and J. K. Aggarwal, "Recognition of High-level Group Activities Based on Activities of Individual Members," Proceedings of IEEE Workshop on Motion and Video Computing (WMVC), Copper Mountain, CO, January 2008. pdf

Human-computer interaction

The goal of the project is to design a novel human-computer interaction system. We particularly focused on constructing 'intelligent workspaces', which encompass a particular group of human-computer interaction (HCI) systems motivated to help users with physical tasks.

Intelligent workspace We constructed an intelligent environment which visually observes tasks of users to help the users complete their tasks. The system is designed to analyze the status of ongoing tasks and to generate appropriate feedback guiding the user. Related Publications M. S. Ryoo and J. K. Aggarwal, "Robust Human-Computer Interaction System Guiding a User by Providing Feedback", Proceedings of International Joint Conference on Artificial Intelligence (IJCAI), Hyderabad, India, 2007. pdf

Applications

I have collaborated with other lab members for development of real-time applications of abnormal activity detections. By applying the activity recognition methodology I have proposed, we were able to recognize events of "baggage abandonment" and "illegal parking".

Abandoned baggage detection We have worked on the problem of the 'abandoned baggage detection', which is aimed at the detection of suspicious baggage in subway stations or airports. We have constructed a low-level component for identification of a segmented baggage, for obtaining appearance models of candidate owners of the bag, and for tracking detected bags and candidate owners. Applying the paradigm of my activity recognition [cvpr06], we were able to successfully detect unattended baggage in crowded scenes. Related Publications M. Bhargava, C.-C. Chen, M. S. Ryoo, and J. K. Aggarwal, "Detection of Abandoned Objects in Crowded Environments", Proceedings of IEEE International Conference on Advanced Video and Signal based Surveillance (AVSS), London, UK, September 2007. pdf

Illegally parked car detection Similarly, we have developed a real-time method to detect illegally parked vehicles. We have projected a vehicle into a one-dimensional line along the road so that the system can efficiently detect a car parked for a sufficient period at time on a roadway (i.e. a no-parking zone). Related Publications J. T. Lee, M. S. Ryoo, M. Riley, and J. K. Aggarwal, "Real-time Detection of Illegally Parked Vehicles using 1-D Transformation", Proceedings of IEEE International Conference on Advanced Video and Signal based Surveillance (AVSS), London, UK, September 2007. pdf