Under review
1 University of Michigan · 2 Massachusetts Institute of Technology · 3 Yonsei University
† Corresponding authors
Dexterous grasping requires deciding where to grasp, reaching the target, and maintaining stable contact. We connect these stages through a compact three-point interface that separates global geometric reasoning from local contact control. Given object geometry and optional language commands, our framework samples contact triples from a precomputed grasp-affordance heatmap. A model-based reactive controller tracks the object, avoids collisions, and guides the hand toward the selected contacts. In the final centimeters, a Reinforcement Learning (RL) policy uses proprioceptive feedback to refine and stabilize the grasp despite reaching and perception errors. It observes only finger joint states and its recent actions, with no target points, visual observations, or object geometry, so a single policy is shared across objects and grasp configurations. In simulation, we compare grasp-and-lift success against squeeze and end-to-end baselines, characterize reaching convergence, and demonstrate grasp steering; hardware demonstrations on two training objects and one unseen object illustrate the full pipeline. Our modular framework uses geometry to guide the reach and local feedback to secure the grasp.
An offline grasp-affordance heatmap over the object surface is sampled for three-point contact triples, and re-selected in real time as the object moves.
On the mustard bottle, each candidate is scored in six steps: sample the surface, place the wrap contacts, check wrap alignment, keep the fingers a fingertip apart, weight by COM torque, then score every vertex into the grasp-affordance field.
The same six-step construction applied to a Rubik's cube.
Grasp-affordance heatmaps for four hero objects (sugar box, mustard bottle, power drill, toy airplane), then the full YCB object set. Each heatmap is computed once per object and reused at any pose.
The cached heatmap rides a moving object while the controller re-picks three-point contacts as the pose changes, at a measured 8.5 ms median per re-selection.
A user-specified anchor, from a point or a language command, morphs the grasp-affordance heatmap toward a new region of the object, letting the same interface target different parts of the same object.
An anchor shifts which contacts get picked: the default heatmaps morph toward the requested region on the mustard bottle (top), wine glass (stem), pitcher (body), and wire cutters (handle).
As the anchor slides across the object, the heatmap re-weights and the selected contacts follow, at a measured 15.1 ms median per update.
Same object and pose, only the anchor changes: the full reactive stack executes top- vs. bottom-anchored grasps in simulation, then re-routes mid-reach when the anchor moves during execution.
The three-point interface steered to grasp a foam brick near its top.
The same foam brick steered and grasped at its midpoint.
The same foam brick steered and grasped near its base.
A model-based controller tracks the object and reaches the selected contacts, handing off to the learned policy once it is close enough or a fine-motion time budget expires.
The reach to a potted meat can is fast at first, then crawls; it converges inside the 20 mm tolerance at 1.6 s of the 3 s fine-adjustment budget and hands off at 17 mm. Over 2,752 randomized reaches, more time barely helps: 78.3% converge at 3 s vs. 80.0% at 7 s, so the closing policy must absorb the residual error.
Six deploy-controller runs on six objects, approaching from 29° to 61° below horizontal: one take per cell, with the full reach from the home pose, then the close and the lift, followed by the approach-direction distribution over all 444 successful benchmark episodes.
In the final centimeters, a proprioceptive reinforcement-learning policy refines contact and stabilizes the lift, compared here against a naive squeeze baseline from the same reach.
The squeeze baseline guides the fingertips to the contacts, then grips at 3 N. A tomato soup can twists out of the grasp and falls, followed by 16 representative failures across the eight evaluated objects, played until the object lands.
From the same start, the squeeze baseline drops a tomato soup can while the learned policy, which sees only finger joint states and its recent actions, holds it; then four paired objects (pitcher base, apple, foam brick, potted meat can). Across the eight evaluated objects, the learned policy reaches 91.9% nominal grasp-and-lift success vs. 56.1% for naive squeezing.
Real-time hardware footage of the full pipeline (reach, contact refinement, and lift) on two training objects and one unseen object.
Training object, top-down grasp: full reach, contact refinement, and lift on a tomato soup can.
Training object, top-down grasp: full reach, contact refinement, and lift on a potted meat can.
Unseen object, side grasp: full reach, contact refinement, and lift on a Pringles can the policy never saw in training.
@misc{nguyen2026steerable,
title = {Steerable and Reactive Grasping Through Modular Design with a Three-Point Interface},
author = {Andrew Nguyen and Yonghyeon Lee and Sangbae Kim},
year = {2026}
}