Abstract

Egocentric videos are a valuable source of information as a daily log of our lives. However, large fraction of ego-centric video content is typically irrelevant and boring to re-watch. It is an agonizing task, for example, to manually search for the moment when your daughter first met Mickey Mouse from hours-long egocentric videos taken at Disney-land. Although many summarization methods have been successfully proposed to create concise representations of videos, in practice, the value of the subshots to users may change according to their immediate preference/mood; thus summaries with fixed criteria may not fully satisfy users’ various search intents. To address this, we propose a sto-ryline representation that expresses an egocentric video as a set of jointly inferred, through MRF inference, story el-ements comprising of actors, locations, supporting objects and events, depicted on a timeline. We construct such a sto-ryline with very limited annotation data (a list of map loca-tions and weak knowledge of what events may be possible at each location), by bootstrapping the process with data ob-tained through focused Web image and video searches. Our representation promotes story-based search with queries in the form of AND-OR graphs, which span any subset of story elements and their spatio-temporal composition. We show effectiveness of our approach on a set of unconstrained YouTube egocentric videos of visits to Disneyland. 1.