prof.irfanessa.com

Georgia Tech Alumna Named Rhodes Scholar | College of Computing.

Georgia Tech alumna Joy Buolamwini has been named a Rhodes scholar. She will attend the University of Oxford, where she plans to pursue degrees in African studies and global governance and diplomacy.

A 2012 computer science graduate from Memphis, Tenn., Buolamwini plans to use her skills to lower barriers in communication with the goal of increasing commerce, keeping governments accountable and improving the quality of life for millions of people.

“My mission is to show compassion through computation,” Buolamwini said. “The heart of computing is humanity, and as a Rhodes scholar, I will have an unprecedented opportunity to gain a deeper understanding of developing nations and global governance while connecting with world leaders who are committed to fighting the world’s fight – making sure each individual can reach her human potential.”

Irfan Essa
Georgia Tech
School of Interactive Computing
GVU and RIM @ GT Centers

October 19, 2012, 3:30 PM, NSH 1305

Abstract

In this talk, I will discuss a variety of approaches my group is working on for video analysis and enhancement. In particular, I will describe our approach for a video stabilizer, currently implemented and running on YouTube, and its extensions.

This method generates stabilized videos by employing L1-optimal camera paths to remove undesirable motions [1]. We compute camera paths that are optimally partitioned into constant, linear and parabolic segments mimicking the camera motions employed by professional cinematographers. We propose a linear programming framework to minimize the first, second, and third derivatives of the resulting camera path. Our method allows for video stabilization beyond the conventional filtering that only suppresses high frequency jitter. An additional challenge in videos shot from mobile phones are rolling shutter distortions. Modern CMOS cameras capture the frame one scan-line at a time, which results in non-rigid image distortions such as shear and wobble. I will demonstrate a solution based on a novel mixture model of homographies parametrized by scan-line blocks to correct these rolling shutter distortions [2]. Our method does not rely on a-priori knowledge of the readout time nor requires prior camera calibration. A thorough evaluation based on a user study and direct comparisons to other approaches, demonstrates a general preference for our algorithm.

I will conclude the talk by showcasing a live demo of the stabilizer. This work is in collaboration with Matthias Grundmann and Vivek Kwatra at Google, and appears in following two papers.

Time permitting, I will discuss some other projects we are working on, including video segmentation and retargetting.

[1] Matthias Grundmann, Vivek Kwatra, Irfan Essa, CVPR 2011, www.cc.gatech.edu/cpl/projects/videostabilization

[2] Matthias Grundmann, Vivek Kwatra, Daniel Castro Irfan Essa, ICCP 2012, Best paper, www.cc.gatech.edu/cpl/projects/rollingshutter

Host: Takeo Kanade

via Robotics Institute: Talks and Seminars.

Weakly Supervised Learning of Object Segmentations from Web-Scale Videos

• G. Hartmann, M. Grundmann, J. Hoffman, D. Tsai, V. Kwatra, O. Madani, S. Vijayanarasimhan, I. Essa, J. Rehg, and R. Sukthankar (2012), “Weakly Supervised Learning of Object Segmentations from Web-Scale Videos,” in Proceedings of ECCV 2012 Workshop on Web-scale Vision and Social Media, 2012. [PDF] [BIBTEX]

  @inproceedings{2012-Hartmann-WSLOSFWV,
    Author = {Glenn Hartmann and Matthias Grundmann and Judy Hoffman and David Tsai and Vivek Kwatra and Omid Madani and Sudheendra Vijayanarasimhan and Irfan Essa and James Rehg and Rahul Sukthankar},
    Booktitle = {Proceedings of ECCV 2012 Workshop on Web-scale Vision and Social Media},
    Date-Added = {2012-10-23 15:03:18 +0000},
    Date-Modified = {2012-10-23 15:07:04 +0000},
    Pdf = {http://www.cs.cmu.edu/~rahuls/pub/eccv2012wk-cp-rahuls.pdf},
    Title = {Weakly Supervised Learning of Object Segmentations from Web-Scale Videos},
    Year = {2012}}

Abstract

We propose to learn pixel-level segmentations of objects from weakly labeled (tagged) internet videos. Specically, given a large collection of raw YouTube content, along with potentially noisy tags, our goal is to automatically generate spatiotemporal masks for each object, such as dog”, without employing any pre-trained object detectors. We formulate this problem as learning weakly supervised classiers for a set of independent spatio-temporal segments. The object seeds obtained using segment-level classiers are further rened using graphcuts to generate high-precision object masks. Our results, obtained by training on a dataset of 20,000 YouTube videos weakly tagged into 15 classes, demonstrate automatic extraction of pixel-level object masks. Evaluated against a ground-truthed subset of 50,000 frames with pixel-level annotations, we conrm that our proposed methods can learn good object masks just by watching YouTube.

Presented at: ECCV 2012 Workshop on Web-scale Vision and Social Media, 2012, October 7-12, 2012, in Florence, ITALY.

Awarded the BEST PAPER AWARD!

Linguistic Transfer of Human Assembly Tasks to Robots

• N. Dantam, I. Essa, and M. Stilman (2012), “Linguistic Transfer of Human Assembly Tasks to Robots,” in Proceedings of Intelligent Robots and Systems (IROS), 2012. [PDF] [BIBTEX]

  @inproceedings{2012-Dantam-LTHATR,
    Author = {N. Dantam and I. Essa and M. Stilman},
    Booktitle = {Proceedings of Intelligent Robots and Systems (IROS)},
    Date-Added = {2012-10-23 15:07:46 +0000},
    Date-Modified = {2012-10-23 15:08:57 +0000},
    Pdf = {http://www.cc.gatech.edu/~ndantam3/papers/dantam2012assembly.pdf},
    Title = {Linguistic Transfer of Human Assembly Tasks to Robots},
    Year = {2012}}

Abstract

We demonstrate the automatic transfer of an assembly task from human to robot. This work extends efforts showing the utility of linguistic models in verifiable robot control policies by now performing real visual analysis of human demonstrations to automatically extract a policy for the task. This method tokenizes each human demonstration into a sequence of object connection symbols, then transforms the set of sequences from all demonstrations into an automaton, which represents the task-language for assembling a desired object. Finally, we combine this assembly automaton with a kinematic model of a robot arm to reproduce the demonstrated task.

Presented at: IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS 2012), October 7-12, 2012 Vilamoura, Algarve, Portugal.

Video Enhancement and Analysis: From Content Analysis to Video Stabilization for YouTube

Irfan Essa
Georgia Institute of Technology

October 14, 2012 15:00 – 16:00, Huxely Room. South Kensington Campus, Imperial College, London

Abstract

The talk will describe a variety of efforts undertaken on analysis of  video to enhancement and synthesis of video. An overview of the past work on representing and analyzing videos as a stochastic process and use of this in a form of Video Textures will be provided.  Majority of the talk will then focus on the recent effort which resulted in a widely-used video stabilizer currently implemented on YouTube and its extensions. This method generates stabilized videos by employing L1-optimal camera paths to remove undesirable motions. We compute camera paths that are optimally partitioned into constant, linear and parabolic segments mimicking the camera motions employed by professional cinematographers. To this end, we propose a linear programming framework to minimize the first, second, and third derivatives of the resulting camera path. Our method allows for video stabilization beyond the conventional filtering that only suppresses high frequency jitter. An additional challenge in videos shot from mobile phones are rolling shutter distortions.  We demonstrate a solution based on a novel mixture model of homographies parametrized by scanline blocks to correct these rolling shutter distortions. Our method does not rely on a-priori knowledge of the readout time nor requires prior camera calibration.  This work is in collaboration with Matthias Grundmann and Vivek Kwatra at Google.

Via Distinguished Seminar Series in Computer Science Irfan Essa – GA Tech.

Wednesday September 19, 2012, 7:00pm in the Hill Auditorium, High Museum, Altanta.

In this sixth installment of Lumière’s Fall Lecture Series, Shannon Perich, curator of the photographic history collection at the National Museum of American History, Smithsonian Institution, and Irfan Essa of the Georgia Institute of Technology will each speak to the future of art in a rapidly expanding digital culture. Their commentary will be followed by a panel discussion with audience participation. The panel will address the threats and opportunities created by a growing range of capabilities to create, distribute, and interact with art. Additional information is available at www.lumieregallery.net.This lecture is a collaborative event with the Atlanta Celebrates Photography 2012 Festival.

via Lumière’s Fall Lecture and Panel Discussion.

SLIDES now available here

At ACM sponsored, 14th International Conference on Ubiquitous Computing (Ubicomp 2012), Pittsburgh, PA, September 5 – 7, 2012.

Here are the highlights of my group’s participation in Ubicomp 2012.

• E. Thomaz, V. Bettadapura, G. Reyes, M. Sandesh, G. Schindler, T. Ploetz, G. D. Abowd, and I. Essa (2012), “Recognizing Water-Based Activities in the Home Through Infrastructure-Mediated Sensing,” in Proceedings of ACM International Conference on Ubiquitous Computing (UBICOMP), 2012. [PDF] [WEBSITE] (Oral Presentation at 2pm on Wednesday September 5, 2012).
• J. Wang, G. Schindler, and I. Essa (2012), “Orientation Aware Scene Understanding for Mobile Camera,” in Proceedings of ACM International Conference on Ubiquitous Computing (UBICOMP), 2012. [PDF[WEBSITE] (Oral Presentation at 2pm on Thursday September 6, 2012).

In addition, my colleague, Gregory Abowd has a position paper on “What next, Ubicomp? Celebrating an intellectual disappearing act” on Wednesday 11:15am session and my other colleague/collaborator Thomas Ploetz has a paper on “Automatic Assessment of Problem Behavior in Individuals with Developmental Disabilities” with his co-authors Nils Hammerla, Agata Rozga, Andrea Reavis, Nathan Call, Gregory Abowd on Friday September 6, in the 9:15am session.

Video Stabilization and Rolling Shutter Removal on YouTube

Abstract

In this talk, I will over a variety of approaches my group is working on for video analysis and enhancement. In particular, I will describe our approach for a video stabilizer (currently implemented on YouTube) and its extensions. This work is in collaboration with Matthias Grundmann and Vivek Kwatra at Google. This method generates stabilized videos by employing L1-optimal camera paths to remove undesirable motions [1]. We compute camera paths that are optimally partitioned into constant, linear and parabolic segments mimicking the camera motions employed by professional cinematographers. To this end, we propose a linear programming framework to minimize the first, second, and third derivatives of the resulting camera path. Our method allows for video stabilization beyond the conventional filtering that only suppresses high frequency jitter. An additional challenge in videos shot from mobile phones are rolling shutter distortions. Modern CMOS cameras capture the frame one scanline at a time, which results in non-rigid image distortions such as shear and wobble. I will demonstrate a solution based on a novel mixture model of homographies parametrized by scanline blocks to correct these rolling shutter distortions [2]. Our method does not rely on a-priori knowledge of the readout time nor requires prior camera calibration. A thorough evaluation based on a user study demonstrates a general preference for our algorithm.

I will conclude the talk by showcasing a live demo of the stabilizer and time permitting, I will discuss some other projects we are working on.

[1] Matthias Grundmann, Vivek Kwatra, Irfan Essa, CVPR 2011, www.cc.gatech.edu/cpl/projects/videostabilization

[2] Matthias Grundmann, Vivek Kwatra, Daniel Castro Irfan Essa, ICCP 2012, Best paper, www.cc.gatech.edu/cpl/projects/rollingshutter

Check out the Computational Photography | Coursera Course I am working on now.

This course is aimed at teaching you the basics of how computation has impacted the entire workflow of photography, from how images are captured, manipulated and collaborated on and shared. At the core of it photography means, drawing with light and how light can be captured to form images/videos. In this class you will learn about how the optics, and the sensor within a camera are generalized, as well as the lighting and other aspects of the environment are generalized to capture novel images. We will also cover post and preprocessing techniques to manipulate and improve images. Finally, we will consider the power of the web and the Internet for both analyzing and sharing images, as well as the impact of mobile smartphone cameras. This class builds on concepts from well known disciplines like computer vision, computer graphics, and image processing. Look forward to your participation in this class.

via Computational Photography | Coursera

At IEEE CVPR 2012 is in Providence RI, from Jun 16 – 21, 2012.

Busy week ahead meeting good friends and colleagues. Here are some highlights of what my group is involved with.

Paper in Main Conference

• K. Kim, D. Lee, and I. Essa (2012), “Detecting Regions of Interest in Dynamic Scenes with Camera Motions,” in Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2012. [PDF] [WEBSITE] [VIDEO] [Poster on Tuesday 6/19/2012]

Demo in Main Conference

• M. Grundmann, V. Kwatra, D. Castro, and I. Essa (2012), “Calibration-Free Rolling Shutter Removal,” in [WEBSITE] [VIDEO] (Paper in ICCP 2012) [Demo on Monday and Tuesday (6/18-19) at the Google Booth]

Invited Talk in Workshop

• Irfan Essa, Invited Talk in CVPR Workshop 2012: Large-Scale Video Search and Mining on Thursday 6/21/2012