Auto-Directed Video Stabilization with Robust L1 Optimal Camera Paths
Posted by Matthias Grundmann, Vivek Kwatra, and Irfan Essa,

Earlier this year, we announced the launch of new features on the YouTube Video Editor, including stabilization for shaky videos, with the ability to preview them in real-time. The core technology behind this feature is detailed in this paper, which will be presented at the IEEE International Conference on Computer Vision and Pattern Recognition (CVPR 2011).

Casually shot videos captured by handheld or mobile cameras suffer from significant amount of shake. Existing in-camera stabilization methods dampen high-frequency jitter but do not suppress low-frequency movements and bounces, such as those observed in videos captured by a walking person. On the other hand, most professionally shot videos usually consist of carefully designed camera configurations, using specialized equipment such as tripods or camera dollies, and employ ease-in and ease-out for transitions. Our goal was to devise a completely automatic method for converting casual shaky footage into more pleasant and professional looking videos.

http://www.youtube.com/watch?v=0MiY-PNy-GU

Our technique mimics the cinematographic principles outlined above by automatically determining the best camera path using a robust optimization technique. The original, shaky camera path is divided into a set of segments, each approximated by either a constant, linear or parabolic motion. Our optimization finds the best of all possible partitions using a computationally efficient and stable algorithm.

To achieve real-time performance on the web, we distribute the computation across multiple machines in the cloud. This enables us to provide users with a real-time preview and interactive control of the stabilized result. Above we provide a video demonstration of how to use this feature on the YouTube Editor. We will also demo this live at Google’s exhibition booth in CVPR 2011.

For more details see the Project Site. See the youtube video of the system on youtube. See the paper in PDF, and a technical video of the work.

Full paper is

• Matthias Grundmann, Vivek Kwatra, and Irfan Essa (2011), “Auto-Directed Video Stabilization with Robust L1 Optimal Camera Paths,” In Proceedings of IEEE International Conference on Computer Vision and Pattern Recognition (CVPR 2011), Colorado Springs, CO, USA. [PDF][Video][Blog][Demo][Project Site]

ABSTRACT

Using off-the-shelf Global Positioning System (GPS) units, we reconstruct buildings in 3D by exploiting the reduction in signal to noise ratio (SNR) that occurs when the buildings obstruct the line-of-sight between the moving units and the orbiting satellites. We measure the size and height of skyscrapers as well as automatically constructing a density map representing the location of multiple buildings in an urban landscape.  If deployed on a large scale, via a cellular service provider’s GPS-enabled mobile phones or GPS-tracked delivery vehicles, the system could provide an inexpensive means of continuously creating and updating 3D maps of urban environments.

Investigator(s): Umakishore Ramachandran, (Principal Investigator), Irfan Essa (Co-Principal Investigator)

Dates: September 1, 2008 – August 31, 2009 (Estimated)

Abstract

From the western world to the third world, the use of handheld devices (cellphones, PDAs) has proliferated. The world of users is becoming both wireless and mobile. Web 2.0 has ushered in an age wherein the web is viewed as a provider of services and not just a repository of documents and/or information. Despite this advance, the web remains just that, a single web with an inherent assumption that a powerful computing and communication infrastructure supports it. Couldn’t mobile wireless devices in close proximity form a web of their own? This is the vision behind this project, the Web on Demand (WoD). WoD aims at bridging the gap between social networks and ad hoc networking. In other words, it aims to rethink the system software stack all the way from application to networking that would allow the creation and management of social networks without any assumption of infrastructure support. The core of the research is to develop software technologies for mobile devices that would allow the dynamic creation of thematic ad hoc overlay networks empowering (a) mobile people with similar interests (e.g., weather forecast), (b) friends and family (e.g., in a theme park), and (c) participants in mission critical applications (e.g., search and rescue), stay connected. WoD complements the World Wide Web (WWW) and leverages it when it is available, such as exploiting the ambient computing infrastructure to enhance user experience, and managing the dynamic creation of User Generated Content (UGC) by mobile users. The vision behind this project is to democratize access to services that are currently offered through WWW. In this sense, the results from this research can have far-reaching technological and societal consequences. Most importantly, the research will help breed a new class of computer scientists who are connected with societal causes in addition to advancing technology.