Alignment of time series is an important problem to solve in many scientific disciplines. In particular, temporal alignment of two or more subjects performing similar activities is a challenging problem due to the large temporal scale difference between human actions as well as the inter/intra subject variability. In this paper we present canonical time warping (CTW), an extension of canonical correlation analysis (CCA) for spatio-temporal alignment of human motion between two subjects. CTW extends previous work on CCA in two ways: (i) it combines CCA with dynamic time warping (DTW), and (ii) it extends CCA by allowing local spatial deformations. We show CTW’s effectiveness in three experiments: alignment of synthetic data, alignment of motion capture data of two subjects performing similar actions, and alignment of similar facial expressions made by two people. Our results demonstrate that CTW provides both visually and qualitatively better alignment than state-of-the-art techniques based on DTW. 1

Subject This thesis in part of a larger project, called SignCom, which aims at improving the quality of real-time interaction between human and virtual characters, by using natural communication modalities such as gesture, facial expressions and gaze. Communication takes place in both directions: the user performs actions that are recognized by the system. The virtual agent provides answers after interpretation of the undertaken actions. The interaction is guided by the gradual creation of a virtual geometric space between the real and virtual characters. The construction of a 3D virtual scene is based on spatial mechanisms used in sign language. Thus, the spatial representation of gestures will be used to build interactive scenes, in which discourse entities such as characters and objects of the narrative scene are referenced and located in the 3D environment around the character. This thesis focuses on the generation and synthesis processes for the animation of a virtual character. It takes into account both high level representations (structural and semantic features) and low level representations of gestures (movement trajectories). A formal generation language will express the arrangement of atomic elements composing gestures. This language includes spatial and temporal dependency relations between parameters that describe signs. Gesture phrases will thus be written thanks to this language, and interpreted by a control model of multimodal

Temporal alignment of human motion performing similar activities has been a topic of recent interest due to its many applications in animation, tele-rehabilitation or activity recognition. This paper presents generalized time warping (GTW), an extension of dynamic time warping (DTW) for temporally aligning multi-modal sequences from multiple subjects performing similar activities. GTW solves three major drawbacks of existing approaches based on DTW: (1) GTW provides a feature weighting layer to adapt different modalities (e.g., video and motion capture data), (2) GTW extends DTW by allowing a more flexible time warping as combination of monotonic functions, (3) unlike DTW that typically has a quadratic cost, GTW has linear complexity in terms of the length of the sequence. Experimental results demonstrate that GTW can efficiently solve the multi-modal temporal alignment problem, and outperforms state-of-the-art methods for temporal alignment of signals with the same modality. 1.

Alignment of time series is an important problem to solve in many scientific disciplines (e.g. bioinformatics, computer vision). In particular, temporal alignment of two or more subjects performing similar activities is a challenging problem due to the large temporal scale difference between human actions as well as the inter/intra subject variability. In this paper we present canonical time warping (CTW), an extension of canonical correlation analysis (CCA) for spatio-temporal alignment of the behavior between two subjects. CTW extends previous work on CCA in two ways: (i) it combines CCA with dynamic time warping for temporal alignment; and (ii) it extends CCA to allow local spatial deformations. We show CTW’s effectiveness in three experiments: alignment of synthetic data, alignment of motion capture data of two subjects performing similar actions, and alignment of two people with similar facial expressions. Our results demonstrate that CTW provides both visually and qualitatively better alignment than state-of-the-art techniques based on dynamic time warping.