Abstract — Robust perception is a vital capability for robotic manipulation in unstructured scenes. In this context, full pose estimation of relevant objects in a scene is a critical step towards the introduction of robots into household environments. In this paper, we present an approach for building metric 3D models of objects using local descriptors from several images. Each model is optimized to fit a set of calibrated training images, thus obtaining the best possible alignment between the 3D model and the real object. Given a new test image, we match the local descriptors to our stored models online, using a novel combination of the RANSAC and Mean Shift algorithms to register multiple instances of each object. A robust initialization step allows for arbitrary rotation, translation and scaling of objects in the test images. The resulting system provides markerless 6-DOF pose estimation for complex objects in cluttered scenes. We provide experimental results demonstrating orientation and translation accuracy, as well a physical implementation of the pose output being used by an autonomous robot to perform grasping in highly cluttered scenes. I.

Abstract—We present a distributed vision-based architecture for smart robotics that is composed of multiple control loops, each with a specialized level of competence. Our architecture is subsumptive and hierarchical, in the sense that each control loop can add to the competence level of the loops below, and in the sense that the loops can present a coarse-to-fine gradation with respect to vision sensing. At the coarsest level, the processing of sensory information enables a robot to become aware of the approximate location of an object in its field of view. On the other hand, at the finest end, the processing of stereo information enables a robot to determine more precisely the position and orientation of an object in the coordinate frame of the robot. The processing in each module of the control loops is completely independent and it can be performed at its own rate. A control Arbitrator ranks the results of each loop according to certain confidence indices, which are derived solely from the sensory information. This architecture has clear advantages regarding overall performance of the system, which is not affected by the “slowest link, ” and regarding fault tolerance, since faults in one module does not affect the other modules. At this time we are able to demonstrate the utility of the architecture for stereoscopic visual servoing. The architecture has also been applied to mobile robot navigation and can easily be extended to tasks such as “assembly-on-the-fly.” Index Terms—Assembly-on-the-fly, automation, computer vision, distributed architectures, robotics, vision-based architecture,

Abstract. One main challenge in Augmented Reality (AR) applications is to keep track of video objects with their movement, orientation, size, and position accurately. This poses a challenging task to recover nonrigid shape and global pose in real-time AR applications. This paper proposes a novel two-stage scheme for online non-rigid shape recovery toward AR applications using Active Appearance Models (AAMs). First, we construct 3D shape models from AAMs offline, which do not involve processing of the 3D scan data. Based on the computed 3D shape models, we propose an efficient online algorithm to estimate both 3D pose and non-rigid shape parameters via local bundle adjustment for building up point correspondences. Our approach, without manual intervention, can recover the 3D non-rigid shape effectively from either real-time video sequences or single image. The recovered 3D pose parameters can be used for AR registrations. Furthermore, the facial feature can be tracked simultaneously, which is critical for many face related applications. We evaluate our algorithms on several video sequences. Promising experimental results demonstrate our proposed scheme is effective and significant for real-time AR applications.

We describe the architecture, algorithms, and experiments with HERB, an autonomous mobile manipulator that performs useful manipulation tasks in the home. We present new algorithms for searching for objects, learning to navigate in cluttered dynamic indoor scenes, recognizing and registering objects accurately in high clutter using vision, manipulating doors and other constrained objects using caging grasps, grasp planning and execution in clutter, and manipulation on pose and torque constraint manifolds. We also

We propose a learning algorithm for estimating the 3-D orientation of objects. Orientation learning is a difficult problem because the space of orientations is non-Euclidean, and in some cases (such as quaternions) the representation is ambiguous, in that multiple representations exist for the same physical orientation. Learning is further complicated by the fact that most man-made objects exhibit symmetry, so that there are multiple “correct ” orientations. In this paper, we propose a new representation for orientations—and a class of learning and inference algorithms using this representation—that allows us to learn orientations for symmetric or asymmetric objects as a function of a single image. We extensively evaluate our algorithm for learning orientations of objects from six categories. 1

In this paper we present a new method for tracking rigid objects using a modified version of the Active Appearance Model. Unlike most of the other appearance-based methods in the literature, such as [3,5,6,9,11], our method allows for both partial and self occlusion of the objects. We use ground-truth to demonstrate the accuracy of our tracking algorithm. We show that our method can be applied to track moving objects over wide variations in position and orientation of the object – one meter in translation and 140 degrees in rotation – with an accuracy of a few millimeters. 1.

Abstract — This paper presents a robust model-based visual tracking algorithm that can give accurate 3D pose of a rigid object. Our tracking algorithm uses an incremental pose update scheme in a prediction-verification framework. Extended Kalman filter is used to update the pose of a target incrementally to minimize the error between the expected map of the target model and the corresponding gradient edge in the image space. The main contributions of this paper include: 1) A novel approach to how we use the two extremities of straight-lines as features. By taking into account the measurement uncertainties associated with the locations of the extracted extremities of the straight-line, our approach can compare correctly two straight-lines of different lengths. 2) Our use of a test of mean criterion for initiating backtracking and our use of a variable threshold on the output of this criterion that makes nil-matching more effective. We have tested our tracking algorithm with image sequences containing highly cluttered backgrounds. The system successfully tracks objects even when they are highly occluded. Index Terms — object tracking, 3D pose estimation, feature representation, extended Kalman filter. I.

Abstract—We propose a learning algorithm for estimating the3-Dorientationofobjects.Orientationlearningisadifficult problem because the space of orientations is non-Euclidean, and in some cases (such as quaternions) the representation is ambiguous,inthatmultiplerepresentationsexistforthesame physical orientation. Learning is further complicated by the fact that most man-made objects exhibit symmetry, so that there are multiple “correct ” orientations. In this paper, we propose a new representation for orientations—and a class of learning and inference algorithms using this representation— thatallowsustolearnorientationsforsymmetricorasymmetric objectsasafunctionofasingleimage.Weextensivelyevaluate our algorithm for learning orientations of objects from six categories. 1 I.