Hidden Markov Models (HMMs) are used for automatic segmentation and recognition of user motions. A new algorithm for real-time HMM recognition was developed. The segmentation results are used to provide appropriate assistance in a combined curve following and object avoidance task. This assistance takes the form of a virtual fixture, whose compliance can be altered online. Recognition and assistance experiments were performed using force and position data recorded from a cooperative manipulation system, where a robot and a human operator hold an instrument simultaneously. Recognition accuracy exceeds 90%, even when the users training the HMMs differ from those executing the task. For a task consisting of both path following and avoidance motions, an HMM-based virtual fixture switches the compliance from low to high when the user is trying to move away from the path. The HMM method improves operator performance in comparison with a constant virtual fixture and no virtual fixture.

A forbidden-region virtual fixture (FRVF) is a computer-generated constraint that displays position or force limitations to a robot manipulator or operator, in order to prevent motion into forbidden regions of the workspace. We compare nine FRVFs on each of four common telemanipulator control architectures: position forward, position exchange, position forward/force feedback, and position exchange/force feedback. A one-degree-of-freedom telemanipulation system was used in an experiment designed to simulate users working near a known forbidden region. The metrics of tracking, safety, and submittance were used to analyze the performance of the system with six different users. The results indicate that different FRVF architectures perform best for each of the three metrics. No single FRVF scheme is the best over aH metrics, so selection of an FRVF architecture should be an application-dependent weighting of the three metrics. Across all control architectures, the results indicate that a very strong FRVF at the slave device in combination with no FRVF at the master device leads to poor telepresence.

Abstract—We propose a passive bilateral teleoperation control law for a pair of-degree-of-freedom (DOF) nonlinear robotic systems. The control law ensures energetic passivity of the closed-loop teleoperator with power scaling, coordinates motions of the master and slave robots, and installs useful task-specific dynamics for inertia scaling, motion guidance, and obstacle avoidance. Consequently, the closed-loop teleoperator behaves like a common passive mechanical tool. A key innovation is the passive decomposition, which decomposes the P-DOF nonlinear teleoperator dynamics into two robot-like systems without violating passivity: an-DOF shape system representing the master–slave position coordination aspect, and an-DOF locked system representing the dynamics of the coordinated teleoperator. The master–slave position coordination is then achieved by regulating the shape system, while programmable apparent inertia of the coordinated teleoperator is achieved by scaling the inertia of the locked system. To achieve this perfect coordination and inertia scaling, the proposed control law measures and compensates for environment and human forcing. Passive velocity field control and artificial potential field control are used to implement guidance and obstacle avoidance for the coordinated teleoperator. The designed control is also implemented in an intrinsically passive negative semidefinite structure to ensure energetic passivity of the closed-loop teleoperator, even in the presence of parametric model uncertainties and inaccurate force sensing. Experiments are performed to validate the properties of the proposed control framework. Index Terms—Avoidance, decomposition, guidance, inertia scaling, nonlinear teleoperator, passive control implementation, passivity, power scaling. I.

In this paper, we present experiments to determine the effect of different virtual fixture compliance levels in a human-machine cooperative manipulation system. Subjects used the JHU Steady Hand Robot with vision-based virtual fixtures to perform three common tasks: curve following, off-path targeting, and object avoidance. The virtual fixtures provided different levels of guidance to the operator, ranging from no guidance to complete guidance. User performance was evaluated based on the error and time for task execution. We developed an algorithm to determine the appropriate compliance level based on the nature of the task. Task parameters considered were time vs. accuracy and user constraint rs. freedom.

We present a method for implementing "steadyhand control" on teleoperators where the master device is of the impedance type. Typical steady-hand systems are admittance controlled cooperative robots that can implement very high damping. Such systems are ideal for implementing guidance virtual fixtures, which are constraints in software that assist a user in moving a tool along preferred paths. Our steady-hand teleoperation method implements a type of admittance control law on an impedance-type master, but requires no force sensor. Combined with guidance virtual fixtures, the system results in a slave device that is precisely constrained to preferred paths. Experimental results demonstrate the desirable behavior of the system. This research is applicable to impedance-type telemanipulation systems, particularly those used in robot-assisted minimally invasive surgery.

Abstract — This paper presents an approach to implement virtual fixtures for surgical robot assistants. Our approach uses a weighted, multi-objective (both linear and nonlinear) constrained optimization framework to formalize a library of virtual fixtures for task primitives. By our formulation, we provide a library of virtual fixtures on task primitives and a way to assemble multiple virtual fixture objects. We implement the constrained optimization problem with both linear and nonlinear constraints, and discuss the trade-offs between them. Moreover, we introduce the notion of “soft ” virtual fixture mechanism for robotic surgical assistance. The “soft ” virtual fixtures enable a surgical tool to have some resistance inside safety regions and no resistance in preferred regions. I.

This dissertation addresses three related topics in the application of virtual fix-tures to bilateral telemanipulation systems. Bilateral telemanipulation is the direct human control of a remote robot, with force and/or tactile feedback, and virtual fixtures are guidance modes, implemented in software, that assist the user in ac-complishing a telemanipulated task. The first topic addressed in this dissertation is the design of functional and stable forbidden-region virtual fixtures, which prevent robot motion into forbidden-regions of the workspace. Metrics are defined to evalu-ate the effectiveness of forbidden-region virtual fixtures, and a human-factors exper-iment uses these metrics to quantify how users interact with various combinations of forbidden-region virtual fixtures and telemanipulation control system. A method to predict system stability that incorporates an explicit model of the telemanipulator and bounding models of human users is created and experimentally verified. Next, a new condition is presented for the passivity of a virtual wall with sampling, sensor quantization, and friction effects, for an impedance-type robot. This condition is

A forbidden-region virtual fixture (FRVF) is a constraint, implemented in software, that keeps the slave manipulator of a master/slave telemanipulation system from entering into a forbidden region of the workspace. In this paper, we consider the problem of unstable vibrations of the slave against the FRVF for a general class of telemanipulator control architectures. The master and slave equilibrium positions resulting from a constant human input force are found, and the system is evaluated around this equilibrium. We consider two methods of analyzing the stability of the system around this equilibrium point. The first method uses tools developed for analysis of two-port networks. The second method converts the system to its discrete state-space form, and then uses the position of the eigenvalues to analyze system stability. We find that the discrete state-space method agrees with simulations, and can be easily used to design and analyze the stability and transient behavior of the telemanipulator. 1

Abstract — In this work, we have extended the concept of constrained motion control of robots to surgical tasks that require multiple robots. We present virtual fixtures to guide the motion of multiple robots such that spatial and temporal relationship is maintained between them. At the same time, our algorithm keeps the surgeon in the control loop. Moreover, we show that these virtual fixtures allow bimanual tasks to be completed using input for a single robot. That is, the user requires only one hand to cooperatively control multiple robots. This reduces the cognitive load on the surgeon and makes multiple-robot setup for surgery more relevant. We demonstrate this architecture by using an example of manipulating a surgical knot to position it at a target point. Significant improvement is observed in the accuracy when bimanual virtual fixture assistance is provided. Moreover, the accuracy when using a

Abstract — This paper describes a spatial motion constraints generation approach for human-machine collaborative surgical assistant system from registered CT models. We extend constrained optimization formulation incorporating task goals, anatomy-based constraints, “no fly zones”, etc. We use a fast potential collision constraint detection method based on 3D surface model and covariance tree data structure. These boundary constraints, along with task behaviors and joint limits, serve as constraint conditions for constrained robot control. We are able to follow a complex path inside a human skull phantom represented by a surface model composed of 99,000 vertices and 182,000 triangles in real time. Our approach enables realtime task-based control of surgical robot in a precise interactive minimally invasive surgery task. We illustrate our approach based on two example tasks which are analogous to the procedures in endoscopic sinus surgery and analyze the users performance on both teleoperation and cooperative control for one of the example tasks. The experimental results show that a robotic assistant employing our approach on spatial motion constraints can assist user in skilled manipulation tasks, while maintaining desired properties. Our approach is equally applicable to teleoperative and cooperative controlled robots. Index Terms — virtual fixtures, anatomy-based constraint, optimization robot control, surgical robot assistant I.