MENU: Home Bio Affiliations Research Teaching Publications Videos Collaborators/Students Contact FAQ ©2007-14 RSS

At ICVSS (International Computer Vision Summer School) 2013, in Calabria, ITALY (July 2013)

July 11th, 2013 Irfan Essa Posted in Computational Photography, Computational Photography and Video, Daniel Castro, Matthias Grundmann, Presentations, S. Hussain Raza, Vivek Kwatra No Comments »

Teaching at the ICVSS 2013, in Calabria, Italy, July 2013 (Programme)

Computational Video: Post-processing Methods for Stabilization, Retargeting and Segmentation

Irfan Essa
(This work in collaboration with
Matthias Grundmann, Daniel Castro, Vivek Kwatra, Mei Han, S. Hussian Raza).

Abstract

We address a variety of challenges for analysis and enhancement of Computational Video. We present novel post-processing methods to bridge the difference between professional and casually shot videos mostly seen on online sites. Our research presents solutions to three well-defined problems: (1) Video stabilization and rolling shutter removal in casually-shot, uncalibrated videos; (2) Content-aware video retargeting; and (3) spatio-temporal video segmentation to enable efficient video annotation. We showcase several real-world applications building on these techniques.

We start by proposing a novel algorithm for video stabilization that generates stabilized videos by employing L1-optimal camera paths to remove undesirable motions. We compute camera paths that are optimally partitioned into con- stant, linear and parabolic segments mimicking the camera motions employed by professional cinematographers. To achieve this, we propose a linear program- ming framework to minimize the first, second, and third derivatives of the result- ing camera path. Our method allows for video stabilization beyond conventional filtering, that only suppresses high frequency jitter. An additional challenge in videos shot from mobile phones are rolling shutter distortions. Modern CMOS cameras capture the frame one scanline at a time, which results in non-rigid image distortions such as shear and wobble. We propose a solution based on a novel mixture model of homographies parametrized by scanline blocks to correct these rolling shutter distortions. Our method does not rely on a-priori knowl- edge of the readout time nor requires prior camera calibration. Our novel video stabilization and calibration free rolling shutter removal have been deployed on YouTube where they have successfully stabilized millions of videos. We also discuss several extensions to the stabilization algorithm and present technical details behind the widely used YouTube Video Stabilizer.

We address the challenge of changing the aspect ratio of videos, by proposing algorithms that retarget videos to fit the form factor of a given device without stretching or letter-boxing. Our approaches use all of the screens pixels, while striving to deliver as much video-content of the original as possible. First, we introduce a new algorithm that uses discontinuous seam-carving in both space and time for resizing videos. Our algorithm relies on a novel appearance-based temporal coherence formulation that allows for frame-by-frame processing and results in temporally discontinuous seams, as opposed to geometrically smooth and continuous seams. Second, we present a technique, that builds on the above mentioned video stabilization approach. We effectively automate classical pan and scan techniques by smoothly guiding a virtual crop window via saliency constraints.

Finally, we introduce an efficient and scalable technique for spatio-temporal segmentation of long video sequences using a hierarchical graph-based algorithm. We begin by over-segmenting a volumetric video graph into space-time regions grouped by appearance. We then construct a region graph over the ob- tained segmentation and iteratively repeat this process over multiple levels to create a tree of spatio-temporal segmentations. This hierarchical approach gen- erates high quality segmentations, and allows subsequent applications to choose from varying levels of granularity. We demonstrate the use of spatio-temporal segmentation as users interact with the video, enabling efficient annotation of objects within the video.

Part of this talks will will expose attendees to use the Video Stabilizer on YouTube and the video segmentation system at videosegmentation.com. Please find appropriate videos to test the systems.

Part of the work described above was done at Google, where Matthias Grundmann, Vivek Kwatra and Mei Han are, and Professor Essa is working as a Consultant. Part of the work were efforts of research by Matthias Grundmann, Daniel Castro and S. Hussain Raza, as part of their research efforts as students at GA Tech.

AddThis Social Bookmark Button

Summer in Barcelona. Teaching classes and such.

May 13th, 2013 Irfan Essa Posted in CnJ, Computational Photography, Daniel Castro, Study Abroad No Comments »

I am here in Barcelona for my 4th summer, participating in the Georgia Tech, College of Computing’s International Study Abroad Programs in Barcelona, ESP. Teaching two classes and spending time with 64 student participants, 4 teaching assistants, and 6 faculty from GA Tech.  Spending time with Faculty at Facultat d’Informàtica de Barcelona – UPC, our hosts here in Barcelona.

  • CS 4464: Computational Journalism: This class is aimed at understanding the computational and technological advancements in the area of journalism. Primary focus is on the study of technologies for developing new tools for (a) sense-making from diverse news information sources, (b) the impact of more and cheaper networked sensors (c) collaborative human models for information aggregation and sense-making, (d) mashups and the use of programming in journalism, (e) the impact of mobile computing and data gathering, (f) computational approaches to information quality, (g) data mining for personalization and aggregation, and (h) citizen journalism. 
  • CS 4475: Computational Photography: This class explores perceptual and technical aspects of pictures, and more precisely the capture and depiction of reality on a 2D medium. The scientific, perceptual, and artistic principles behind image-making will be emphasized. Topics include the relationship between pictorial techniques and the human visual system; intrinsic limitations of 2D representations and their possible compensations; and technical issues involving depiction. Technical aspects of image capture and rendering, and exploration of how such a medium can be used to its maximum potential, will be examined. Material from recent coursera offering of Computational Photography will be leveraged this class.
AddThis Social Bookmark Button

Computational Photography MOOC on Coursera, comes to a close.

May 7th, 2013 Irfan Essa Posted in Computational Photography, Computational Photography and Video, Coursera, Denis Lantsman No Comments »

The Computational Photography MOOC offering in Coursera came to a close with the following final announcement (abridged here) on May 7, 2013.

Computational photographers:

Thanks for joining us for an engaging 5 weeks of collaboratively learning the wonderful aspects of computational photography. We bid you all farewell now and hope to see some of you in a future reincarnation of this class, building on the feedback provided by many of you. Keep a lookout for the repeat of the same class, and for another class continuing to more advanced topics.

Final graded scores, and the the certificate of completion will be made available this week. All assignment solutions are available, as requested. We will also keep the class site open for a while.

Do remember that we still welcome your feedback, so use the forums. If you haven’t done so already, please do spend a few minutes to fill out the survey for the last week of class, which is part of a survey we are conducting to understand and evaluate online classes offerings like this one.

Again, thanks for participating, and good luck with your future endeavors. And remember to take good pictures, and to have fun computing with photographs.

via Announcements | Computational Photography.

AddThis Social Bookmark Button

Coursera Course on Computational Photography, NOW LIVE and RUNNING!

March 23rd, 2013 Irfan Essa Posted in Computational Photography, Coursera, Denis Lantsman 4 Comments »

We are live!

Welcome to the course website!

As you take a look around, there are a few things we want to bring to your attention:

1. Do note that while this is an introductory class, we do require that you have working knowledge of college level mathematics, which includes concepts like Linear Algebra and Calculus. In addition, the programming assignments will require access to a computer with Python and OpenCV. Instructions for installing this software are available here, which is also linked in the navigation bar of the site. The responsibility to get your computer system working with this software is entirely yours. The diverse nature of computer platforms and softwares do offer challenges in setting up such systems and we encourage folks to use the forums to help each other with challenges faced.Please get started with the software installation as soon as possible!

2. These online courses are still rather mysterious, we want to learn as much as we can about them What works? What doesnt?. Accordingly, we invite you to participate in occasional surveys that are part of the research study. The surveys will tell us about you, and how the course is working for you. The information you provide will help us make the course better and improve our ability to provide more intellectually engaging content. Participation in the surveys is completely optional and will not affect your grade in the course in any way. For your reference, here is a link to a PDF document Consent Form that describes the surveys and study in more detail.If you would like to participate, begin by filling out the background survey here. Also, keep an eye on the syllabus page and the weekly announcements for end-of-week surveys.

3. The syllabus page will be updated every week to reflect all of the course content as it becomes available. Please consult that page if you are unsure about what to do next. The page also contains useful information about class logistics, policies, and frequently asked questions.

4. Note that the lecture videos page contains links to the subtitles, slides, and video file downloads for each of the lecture videos.Thank you for your attention, and I hope you will have fun in the following few weeks!

-Denis

via Announcements | Computational Photography.

Computational Photography via Coursera

AddThis Social Bookmark Button

Computational Photography Classes at Georgia Tech | CS 4475 Spring 2013

January 7th, 2013 Irfan Essa Posted in Teaching No Comments »

I am teaching my usual Computational Photography Class at Georgia Tech | CS 4475 for Spring 2013. This site should have all the external materials for the class.  All internal GA Tech materials will be available via the T-square site.  I am also hoping to use the Piazza site.

AddThis Social Bookmark Button

Computation + Journalism Symposium 2013 on Jan 31 – Feb 1, at GA Tech.

January 2nd, 2013 Irfan Essa Posted in Brad Stenger, CnJ, Computational Journalism, Events, Interesting, Nick Diakopoulos No Comments »

Join us for the 2nd Computation + Journalism Symposium 2013 in Atlanta, GA on Jan 31 – Feb 1, 2013

What role does computation have in the practice of journalism today and in the near future? As computer-driven forces like automation and aggregation increasingly alter the role of journalists and journalism in society, how can computation become a force of deliberate, positive social impact in journalism and civic life? Five years after the first Computation and Journalism symposium at Georgia Tech, this event brings together leaders in both journalism and computation to discuss and debate current trends and future opportunities.

Join us for the second Symposium on Computation + Journalism to be held at the Georgia Institute of Technology in Atlanta on Jan 31, – Feb 1, 2012. Visit this site for additional details.

AddThis Social Bookmark Button

Online (and Free) Course on Computational Photography via Coursera

August 17th, 2012 Irfan Essa Posted in DVFX, Teaching No Comments »

Check out the Computational Photography | Coursera Course I am working on now.

This course is aimed at teaching you the basics of how computation has impacted the entire workflow of photography, from how images are captured, manipulated and collaborated on and shared. At the core of it photography means, drawing with light and how light can be captured to form images/videos. In this class you will learn about how the optics, and the sensor within a camera are generalized, as well as the lighting and other aspects of the environment are generalized to capture novel images. We will also cover post and preprocessing techniques to manipulate and improve images. Finally, we will consider the power of the web and the Internet for both analyzing and sharing images, as well as the impact of mobile smartphone cameras. This class builds on concepts from well known disciplines like computer vision, computer graphics, and image processing. Look forward to your participation in this class.

via Computational Photography | Coursera

AddThis Social Bookmark Button

Teaching: Spring 2012

January 11th, 2012 Irfan Essa Posted in CnJ, Computational Journalism, Computational Photography and Video, DVFX No Comments »

In Spring 2012, I am teaching 2 classes.

Advanced Computational Photography (CS 8803 PHO) [with Grant Schindler]

This is an advanced topics class in Computational Photography, building on my intro class and explores technical aspects of pictures, and more precisely the capture and depiction of reality on a 2D medium. The scientific, perceptual, and artistic principles behind image-making will be emphasized. Topics include the relationship between pictorial techniques and the human visual system; intrinsic limitations of 2D representations and their possible compensations; and technical issues involving depiction. Technical aspects of image capture and rendering, and exploration of how such a medium can be used to its maximum potential, will be examined. Students are strongly encouraged (not required) to bring their digital cameras and a laptop to facilitate experiments. The class will explore recent and state of the art paper in Computational Photography from leading conferences and journals in the area and students will do projects in a variety of topics.

Computation + Journalism (CS 4464 / CS 6465)

This class is aimed at understanding the computational and technological advancements in the area of journalism. Primary focus is on the study of technologies for developing new tools for (a) sense-making from diverse news information sources, (b) the impact of more and cheaper networked sensors (c) collaborative human models for information aggregation and sense-making, (d) mashups and the use of programming in journalism, (e) the impact of mobile computing and data gathering, (f) computational approaches to information quality, (g) data mining for personalization and aggregation, and (h) citizen journalism. Complete schedule and other information will be on the t-square site available to only students taking the class.

AddThis Social Bookmark Button

Teaching for Spring Semester 2011

January 6th, 2011 Irfan Essa Posted in Computational Photography and Video, Teaching No Comments »

In Spring 2011, I am teaching 2 classes.

Computational Photography (CS 4475 HP/ 8803 PHO)

This class explores perceptual and technical aspects of pictures, and more precisely the capture and depiction of reality on a 2D medium. The scientific, perceptual, and artistic principles behind image-making will be emphasized. Topics include the relationship between pictorial techniques and the human visual system; intrinsic limitations of 2D representations and their possible compensations; and technical issues involving depiction. Technical aspects of image capture and rendering, and exploration of how such a medium can be used to its maximum potential, will be examined. Students are strongly encouraged (not required) to bring their digital cameras and a laptop to facilitate experiments.

Computing, Society and Professionalism (CS 4001)

Although Computing, Society and Professionalism is a required course for CS majors, it is not a typical computer science course. Rather than dealing with the technical content of computing, it addresses the effects of computing on individuals, organizations, and society, and on what yourresponsibilities are as a computing professional in light of those impacts. The topic is a very broad one and one that you will have to deal with almost every day of your professional life. The issues are sometimes as intellectually deep as some of the greatest philosophical writings in history – and sometimes as shallow as a report on the evening TV news. This course can do little more than introduce you to the topics, but, if successful, will change the way you view the technology with which you work.

AddThis Social Bookmark Button

Final Projects 2010 « Computation + Journalism Class at Georgia Tech

April 1st, 2010 Irfan Essa Posted in CnJ, Computational Journalism, Teaching No Comments »

Final Projects 2010 « Computation + Journalism Class at Georgia Tech.

Check out the list of final projects for this term’s (Spring 2010) class on Computational Journalism.  Final reports expected in last week of April.  Stay tuned.

AddThis Social Bookmark Button