prof.irfanessa.com » 2004

Talk at USC’s IRIS (2004): “Temporal Reasoning from Video to Temporal Synthesis of Video”

Irfan Essa — Sun, 31 Oct 2004 01:09:39 +0000

Irfan Essa (2004), “Temporal Reasoning from Video to Temporal Synthesis of Video” Talk at USC’s IRIS-Vision Seminars (Fall 2004).

Temporal Reasoning from Video to Temporal Synthesis of Video

Abstract

In this talk, I will present some ongoing work on extracting spatio-temporal cues from video for both synthesis of novel video sequences, and recognition of complex activities. I will start off with some of our earlier work on Video Textures, where repeating information is extracted to generate extended sequences of videos. I will then describe some of our extensions to this approach that allow for controlled generation of animations of video sprites. We have developed various learning and optimization techniques that allow for video-based animations of photo-realistic characters. Then I will describe our new approach for image and video synthesis that builds on optimal patch-based copying of samples. I will show how our method allows for iterative refinement and extends to synthesis of both images and video from very limited samples. In the next part of my talk, I will describe how a similar analysis of video can be used to recognize what a person is doing in a scene. Such an analysis of video, aimed at recognition, requires more contextual information about the environment. I will show how we leverage contextual information shared between actions and objects to recognize what is happening in complex environments. I will also show that by adding some form of grammar (we use Stochastic Context Free Grammar) we can recognize very complex, multi-tasked activities.

If time permits, I will describe (very briefly) the Aware Home project at Georgia Tech, which is one primary area of ongoing and future research for me and my group. Further information on my work with videos is available from my webpage at http://www.cc.gatech.edu/~irfan

ESORICS Paper (2004): “Parameterized Authentication”

Irfan Essa — Fri, 01 Oct 2004 00:43:49 +0000

Computer Security – ESORICS 2004

Paper: ACM NPAR (2003) “Image and video based painterly animation”

Irfan Essa — Mon, 07 Jun 2004 15:42:04 +0000

James Hays and Irfan Essa (2004) “Image and video based painterly animation” In Proceedings of the 3rd international symposium on Non-photorealistic animation and rendering (NPAR 2004), Annecy, France, June 7-9, 2004, pages, 113 – 120, ISBN:1-58113-887-3, 2004 (DOI|PDF|Project Web Site).

Hays and Essa (2004), “Image and video based painterly animation,” in Proceedings of ACM Conference on Non-photorealistic animation and rendering (NPAR), New York, NY, USA, 2004, pp. 113-120. [BIBTEX]

@inproceedings{2004-Hays-IVBPA,
  Address = {New York, NY, USA},
  Author = {J. Hays and I. Essa},
  Booktitle = {Proceedings of ACM Conference on Non-photorealistic animation and rendering (NPAR)},
  Date-Modified = {2011-12-08 21:27:48 +0000},
  Pages = {113--120},
  Publisher = {ACM Press},
  Title = {Image and video based painterly animation},
  Year = {2004}}

ABSTRACT

We present techniques for transforming images and videos into painterly animations depicting different artistic styles. Our techniques rely on image and video analysis to compute appearance and motion properties. We also determine and apply motion information from different (user-specified) sources to static and moving images. These properties that encode spatio-temporal variations are then used to render (or paint) effects of selected styles to generate images and videos with a painted look. Painterly animations are generated using a mesh of brush stroke objects with dynamic spatio-temporal properties. Styles govern the behavior of these brush strokes as well as their rendering to a virtual canvas. We present methods for modifying the properties of these brush strokes according to the input images, videos, or motions. Brush stroke color, length, orientation, opacity, and motion are determined and the brush strokes are regenerated to fill the canvas as the video changes. All brush stroke properties are temporally constrained to guarantee temporally coherent non-photorealistic animations.

Paper: IEEE CVPR (2004) “Asymmetrically boosted HMM for speech reading”

Irfan Essa — Wed, 02 Jun 2004 22:46:44 +0000

Pei Yin Essa, I. Rehg, J.M. (2004) “Asymmetrically boosted HMM for speech reading,”, In Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2004 (CVPR 2004). Publication Date: 27 June-2 July 2004, Volume: 2, On page(s): II-755 – II-761 Vol.2 ISSN: 1063-6919, ISBN: 0-7695-2158-, INSPEC Accession Number:8161546, Digital Object Identifier: 10.1109/CVPR.2004.1315240

Abstract

Speech reading, also known as lip reading, is aimed at extracting visual cues of lip and facial movements to aid in recognition of speech. The main hurdle for speech reading is that visual measurements of lip and facial motion lack information-rich features like the Mel frequency cepstral coefficients (MFCC), widely used in acoustic speech recognition. These MFCC are used with hidden Markov models (HMM) in most speech recognition systems at present. Speech reading could greatly benefit from automatic selection and formation of informative features from measurements in the visual domain. These new features can then be used with HMM to capture the dynamics of lip movement and eventual recognition of lip shapes. Towards this end, we use AdaBoost methods for automatic visual feature formation. Specifically, we design an asymmetric variant of AdaBoost M2 algorithm to deal with the ill-posed multi-class sample distribution inherent in our problem. Our experiments show that the boosted HMM approach outperforms conventional AdaBoost and HMM classifiers. Our primary contributions are in the design of (a) boosted HMM and (b) asymmetric multi-class boosting.

Paper: IEEE CVPR (2004) “Propagation networks for recognition of partially ordered sequential action”

Irfan Essa — Wed, 02 Jun 2004 22:44:31 +0000

Yifan Shi, Yan Huang, Minnen, D., Bobick, A., Essa, I. (2004), “Propagation networks for recognition of partially ordered sequential action” In Proceedings of the 2004 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2004 (CVPR 2004). Volume: 2, page(s): II-862 – II-869 Vol.2, ISSN: 1063-6919, ISBN: 0-7695-2158-4, INSPEC Accession Number:8161557, Digital Object Identifier: 10.1109/CVPR.2004.1315255, 27 June-2 July 2004 (IEEEXplore)

Abstract

We present propagation networks (P-nets), a novel approach for representing and recognizing sequential activities that include parallel streams of action. We represent each activity using partially ordered intervals. Each interval is restricted by both temporal and logical constraints, including information about its duration and its temporal relationship with other intervals. P-nets associate one node with each temporal interval. Each node is triggered according to a probability density function that depends on the state of its parent nodes. Each node also has an associated observation function that characterizes supporting perceptual evidence. To facilitate real-time analysis, we introduce a particle filter framework to explore the conditional state space. We modify the original condensation algorithm to more efficiently sample a discrete state space (D-condensation). Experiments in the domain of blood glucose monitor calibration demonstrate both the representational power of P-nets and the effectiveness of the D-condensation algorithm.

Showcase: DVFX 2004 Video Productions

Irfan Essa — Thu, 22 Apr 2004 17:00:03 +0000

DVFX 2004 Video Productions

“Once Upon A Time”

The Digital Video Special Effects class of Spring 2004 (CS 4480/8803dfx) will present their final productions on April 22, 2004 in CoCB 102 at 12n. Students will show and discuss the short video segments (less than 1 minutes each) that they have produced to showcase the technical special effects they have generated during the course of the semester. This class combines understanding of the technical issues underlying the generation of special effects with the expression of artistic creativity by producing a short video. Students are required to do all aspects of the production from story-boarding to shot-planning to camera-work to writing code for special effects generation, to sound and & music editing and video editing. They also produce a “making-of” to explain their production. Most aspects of their production and all the important steps that led to it are listed below.

Thesis: Gabriel Brostow’s PhD (2004): “Novel Skeletal Representation for Articulated Creatures”

Irfan Essa — Fri, 09 Apr 2004 16:17:01 +0000

Gabriel Brostow (2004), “Novel Skeletal Representation for Articulated Creatures” PhD Thesis, Georgia Institute of Technology, College of Computing. (Advisor: Irfan Essa) [PDF] [URI]

Abstract

This research examines an approach for capturing 3D surface and structural data of moving articulated creatures. Given the task of non-invasively and automatically capturing such data, a methodologyand the associated experiments are presented, that apply to multiview videos of the subjects motion. Our thesis states: A functional structure and the timevarying surface of an articulated creature subject are contained in a sequence of its 3D data. A functional structure is one example of the possible arrangements of internal mechanisms (kinematic joints, springs, etc.) that is capable of performing the motions observed in the input data. Volumetric structures are frequently used as shape descriptors for 3D data. The capture of such data is being facilitated by developments in multi-view video and range scanning, extending to subjects that are alive and moving. In this research, we examine vision-based modeling and the related representation of moving articulated creatures using Spines. We define a Spine as a branching axial structure representing the shape and topology of a 3D objects limbs, and capturing the limbs correspondence and motion over time. The Spine concept builds on skeletal representations often used to describe the internal structure of an articulated object and the significant protrusions. Our representation of a Spine provides for enhancements over a 3D skeleton. These enhancements form temporally consistent limb hierarchies that contain correspondence information about real motion data. We present a practical implementation that approximates a Spines joint probability function to reconstruct Spines for synthetic and real subjects that move. In general, our approach combines the objectives of generalized cylinders, 3D scanning, and markerless motion capture to generate baseline models from real puppets, animals, and human subjects.