Research in robotic grasping has flourished in the last 25 years. A recent survey by Bicchi [1] covered over 140 papers, and many more than that have been published. Stemming from our desire to implement some of the work in grasp analysis for particular hand designs, we created an interactive grasping simulator that can import a wide variety of hand and object models and can evaluate the grasps formed by these hands. This system, dubbed “GraspIt!,” has since expanded in scope to the point where we feel it could serve as a useful tool for other researchers in the field. To that end, we are making the system publicly available (GraspIt! is available for download for a variety of platforms from

In this article we present the integration of 3-D shape knowledge into a variational model for level set based image segmentation and tracking. Given a 3-D surface model of an object that is visible in the image of one or multiple cameras calibrated to the same world coordinate system, the object contour extracted by the segmentation method is applied to estimate the 3-D pose parameters of the object. Vice-versa, the surface model projected to the image plane helps in a top-down manner to improve the extraction of the contour. While common alternative segmentation approaches, which integrate 2-D shape knowledge, face the problem that an object can look very differently from various viewpoints, a 3-D free form model ensures that for each view the model can fit the data in the image very well. Moreover, one additionally solves the higher level problem of determining the object pose in 3-D space. Due to the variational formulation, the approach clearly states all model assumptions in a single energy functional that is locally minimized by our method. Its performance is demonstrated by experiments with a monocular and a stereo camera system. 1 1

This paper describes a synergistic integration of a grasping simulator and a real-time visual tracking system, that work in concert to 1) find an object’s pose, 2) plan grasps and movement trajectories, and 3) visually monitor task execution. Starting with a CAD model of an object to be grasped, the system can find the object’s pose through vision which then synchronizes the state of the robot workcell with an online, model-based grasp planning and visualization system we have developed called GraspIt. GraspIt can then plan a stable grasp for the object, and direct the robotic hand system to perform the grasp. It can also generate trajectories for the movement of the grasped object, which are used by the visual control system to monitor the task and compare the actual grasp and trajectory with the planned ones. We present experimental results using typical grasping tasks. 1

In registration-based motion tracking precise pose between a reference template and the current images is determined by warping image patches into the template coordinates and matching pixel-wise intensities. Efficient such algorithms are based on relating spatial and temporal derivatives using numerical optimization algorithms. We extend this approach from planar patches into a formulation where the 3D geometry of a scene is both estimated from uncalibrated video and used in the tracking of the same video sequence. Our tracking algorithm is different than traditional SSD tracking as it trackers a 3D pose global to all patches and not individual 2D image warps. Experimentally we compare convergence and accuracy of our uncalibrated 3D tracking to previous approaches. Notably, the 3D algorithm can successfully track over significantly larger pose changes than ones using only 2D planar regions. It also allows for the detection of occlusions and removal/insertion of tracking regions as appropriate in response.

We present a tracking method where full camera position and orientation is tracked from intensity differences in a video sequence. The camera pose is calculated based on plane equations, and hence does not depend on point correspondences. The plane based formulation also allows additional constraints to be naturally added, e.g. perpendicularity between walls, floor and ceiling surfaces, co-planarity of wall surfaces etc. A particular feature of our method is that the full 3D pose change is directly computed from temporal image differences without making a commitment to a particular intermediate (e.g. 2D feature) representation. We experimentally compared our method with regular 2D SSD tracking and found it more robust and stable. This is due to 3D consistency being enforced even in the low level registration of image regions. This yields better results than first computing (and hence committing to) 2D image features and then from these compute 3D pose.

We present an algorithm which can track the 3D pose of a hand held camera in real-time using predefined models of objects in the scene. The technique utilises and extends recently developed techniques for 3D tracking with a particle filter. The novelty is in the use of edge information for 3D tracking which has not been achieved before within a realtime Bayesian sampling framework. We develop a robust tracker by carefully designing the particle filter observation model: grouping line segments from a known model into 3D junctions and performing fast inlier/outlier counts on projected junction branches. Results demonstrate the ability to track full 3D pose in dense clutter whilst using a minimal number of junctions. 1

Abstract — This paper presents a robust model-based visual tracking algorithm that can give accurate 3D pose of a rigid object. Our tracking algorithm uses an incremental pose update scheme in a prediction-verification framework. Extended Kalman filter is used to update the pose of a target incrementally to minimize the error between the expected map of the target model and the corresponding gradient edge in the image space. The main contributions of this paper include: 1) A novel approach to how we use the two extremities of straight-lines as features. By taking into account the measurement uncertainties associated with the locations of the extracted extremities of the straight-line, our approach can compare correctly two straight-lines of different lengths. 2) Our use of a test of mean criterion for initiating backtracking and our use of a variable threshold on the output of this criterion that makes nil-matching more effective. We have tested our tracking algorithm with image sequences containing highly cluttered backgrounds. The system successfully tracks objects even when they are highly occluded. Index Terms — object tracking, 3D pose estimation, feature representation, extended Kalman filter. I.

In this paper, a short survey of vision-based robot control (generally called visual servoing) is presented. Visual servoing concerns several field of research including vision systems, robotics and automatic control. Visual servoing can be useful for a wide range of applications and it can be used to control many different dynamic systems (manipulator arms, mobile robots, aircraft, etc.). Visual servoing systems are generally classified depending on the number of cameras, on the position of the camera with respect to the robot, on the design of the error function to minimize in order to reposition the robot. In this paper, we describe the main visual servoing approaches proposed in the literature. For simplicity, the examples in the survey focuses on manipulator arms with a single camera mounted on the end-effector. Examples are taken from work made at the University of Cambridge for the European Long Term Research Project VIGOR (Visually guided robots using uncalibrated cameras).

Reliability of a model-based edge tracker critically depends on its ability to establish correct correspondences between points on the model edges and edge pixels in an image. This is a non-trivial problem especially in the presence of large inter-frame motions and in cluttered environments. We propose an online learning approach to solving this problem. An edge pixel is represented by a descriptor composed of a small segment of intensity patterns. From training examples the algorithm utilizes the randomized forest model to learn a posteriori distribution of correspondence given the descriptor. In a new frame, the edge pixels are classified using maximum a posteriori (MAP) estimation. The proposed method is very powerful and it enables us to apply the proposed tracker to many previously impossible scenarios with unprecedented robustness. 1.

Abstract. In this work, we present an approach to jointly segment a rigid object in a two-dimensional (2D) image and estimate its three-dimensional (3D) pose, using the knowledge of a 3D model. We naturally couple the two processes together into a shape optimization problem and minimize a unique energy functional through a variational approach. Our methodology differs from the standard monocular 3D pose estimation algorithms since it does not rely on local image features. Instead, we use global image statistics to drive the pose estimation process. This confers a satisfying level of robustness to noise and initialization for our algorithm and bypasses the need to establish correspondences between image and object features. Moreover, our methodology possesses the typical qualities of region-based active contour techniques with shape priors, such as robustness to occlusions or missing information, without the need to evolve an infinite dimensional curve. Another novelty of the proposed contribution is to use a unique 3D model surface of the object, instead of learning a large collection of 2D shapes to accommodate the diverse aspects that a 3D object can take when imaged by a camera. Experimental results on both synthetic and real images are provided, which highlight the robust performance of the technique in challenging tracking and segmentation applications. Key words. region-based segmentation and tracking, three-dimensional pose estimation, three-dimensional