Under review
1 University of Michigan · 2 Massachusetts Institute of Technology · 3 Yonsei University
† Corresponding authors
Dexterous grasping requires deciding where to grasp, reaching the target, and maintaining stable contact. We connect these stages through a compact three-point interface that separates global geometric reasoning from local contact control. Given object geometry and optional language commands, our framework samples contact triples from a precomputed grasp-affordance heatmap. A model-based reactive controller tracks the object, avoids collisions, and guides the hand toward the selected contacts. In the final centimeters, a Reinforcement Learning (RL) policy uses proprioceptive feedback to refine and stabilize the grasp despite reaching and perception errors. It observes only finger joint states and its recent actions, with no target points, visual observations, or object geometry, so a single policy is shared across objects and grasp configurations. In simulation, we compare grasp-and-lift success against squeeze and end-to-end baselines, characterize reaching convergence, and demonstrate grasp steering; hardware demonstrations on two training objects and one unseen object illustrate the full pipeline. Our modular framework uses geometry to guide the reach and local feedback to secure the grasp.
An offline grasp-affordance heatmap over the object surface is sampled for three-point contact triples, and re-selected in real time as the object moves.
On the mustard bottle, each candidate is scored in six steps: sample the surface, place the wrap contacts, check wrap alignment, keep the fingers a fingertip apart, weight by COM torque, then score every vertex into the grasp-affordance field.
The same six-step construction applied to a Rubik's cube.
Grasp-affordance heatmaps for four hero objects (sugar box, mustard bottle, power drill, toy airplane), then the full YCB object set. Each heatmap is computed once per object and reused at any pose.
The cached heatmap rides a moving object while the controller re-picks three-point contacts as the pose changes, at a measured 8.5 ms median per re-selection.
A user-specified anchor, from a point or a language command, morphs the grasp-affordance heatmap toward a new region of the object, letting the same interface target different parts of the same object.
An anchor shifts which contacts get picked: the default heatmaps morph toward the requested region on the mustard bottle (top), wine glass (stem), pitcher (body), and wire cutters (handle).
As the anchor slides across the object, the heatmap re-weights and the selected contacts follow, at a measured 15.1 ms median per update.
Same object and pose, only the anchor changes: the full reactive stack executes top- vs. bottom-anchored grasps in simulation, then re-routes mid-reach when the anchor moves during execution.
The three-point interface steered to grasp a foam brick near its top.
The same foam brick steered and grasped at its midpoint.
The same foam brick steered and grasped near its base.
A model-based controller tracks the object and reaches the selected contacts, handing off to the learned policy once it is close enough or a fine-motion time budget expires.
In the clips below, the object is driven along a smooth path in simulation while the arm is already reaching. The examples vary the object, the path's shape and length, whether the object rotates, and when in the reach it starts moving. The controller tracks the live object pose on every tick. The orange band on the chart marks when the object is moving, and the trail on the table shows the path it has covered so far. The closing policy takes over once the object stops and the fingertips settle.
The apple slides 20 cm sideways over 3.4 s, starting just 0.3 s into the reach. Once the hand arrives it trails the moving apple at a steady ~28 mm, then closes at 22 mm as the apple comes to rest.
The can holds still until 1.3 s, when the fingertips are already within 40 mm, then slides 18 cm diagonally. The hand stays within 38 mm of its targets throughout and hands off at 16 mm.
The cube goes 12 cm one way, then reverses for 18 cm the other way: 30 cm of travel in 3.5 s at up to 14 cm/s. The hand turns around with it and hands off at 21 mm.
A 26 cm arc with a 90° turn along the way, so the selected contacts swing around the can as it moves. The reach keeps re-aiming at them and settles to 22 mm after the can stops.
No translation at all: the box turns 120° in place over 3 s. The fingertip targets orbit with its faces, the hand follows them around, and it hands off at 17 mm.
Two separate 10 cm shoves at 0.6 s and 2.4 s, with a pause between them. The second shove comes after the hand has closed in and pushes the error back out to 66 mm; the hand re-converges and hands off at 29 mm.
The reach to a potted meat can is fast at first, then crawls; it converges inside the 20 mm tolerance at 1.6 s of the 3 s fine-adjustment budget and hands off at 17 mm. Over 2,752 randomized reaches, more time barely helps: 78.3% converge at 3 s vs. 80.0% at 7 s, so the closing policy must absorb the residual error.
Six deploy-controller runs on six objects, approaching from 29° to 61° below horizontal: one take per cell, with the full reach from the home pose, then the close and the lift, followed by the approach-direction distribution over all 444 successful benchmark episodes.
In the final centimeters, a proprioceptive reinforcement-learning policy refines contact and stabilizes the lift, compared here against a naive squeeze baseline from the same reach.
The squeeze baseline guides the fingertips to the contacts, then grips at 3 N. A tomato soup can twists out of the grasp and falls, followed by 16 representative failures across the eight evaluated objects, played until the object lands.
From the same start, the squeeze baseline drops a tomato soup can while the learned policy, which sees only finger joint states and its recent actions, holds it; then four paired objects (pitcher base, apple, foam brick, potted meat can). Across the eight evaluated objects, the learned policy reaches 91.9% nominal grasp-and-lift success vs. 56.1% for naive squeezing.
Real-time hardware footage of the full pipeline (reach, contact refinement, and lift) on two training objects and one unseen object.
Training object, top-down grasp: full reach, contact refinement, and lift on a tomato soup can.
Training object, top-down grasp: full reach, contact refinement, and lift on a potted meat can.
Unseen object, side grasp: full reach, contact refinement, and lift on a Pringles can the policy never saw in training.
@misc{nguyen2026steerable,
title = {Steerable and Reactive Grasping Through Modular Design with a Three-Point Interface},
author = {Andrew Nguyen and Yonghyeon Lee and Sangbae Kim},
year = {2026},
eprint = {2609.24896},
archivePrefix = {arXiv},
primaryClass = {cs.RO},
url = {https://arxiv.org/abs/2609.24896}
}