摘要
Dexterous manipulation models under the vision-language-action (VLA) paradigm can generate stable action trajectories, yet fine grasping often fails due to pose misalignment of the dexterous hand and force-control deviations. To address the dual challenges of blind spots in unimodal visual perception and unstable contact force control, we propose a guided tactile multimodal framework (GTMF) that decodes visuo-tactile joint features into tactile thresholds and, with real-time feedback from fingertip tactile arrays, performs precise grasp refinement. Built on VLA’s cross-scene generalization, GTMF first uses a guided mask-attention network to calibrate the end-effector pose, then performs cross-modal alignment and decoding to produce scene-adaptive tactile thresholds, which drive real-time servo control for accurate closed-loop grasping. For data acquisition, we design a wearable visuo-tactile isomorphic module to efficiently collect human tactile signals together with end-effector vision and pose. Experiments verify the effectiveness of the approach, achieving grasp accuracy comparable to or exceeding mainstream baselines even when relying solely on tactile input. To the best of our knowledge, this is the first method that decodes tactile signals purely from RGB images and uses them for real-time grasp-pose adjustment, offering a new route to robust dexterous grasping within the VLA framework. © 2026 Science Press. All rights reserved.
| 投稿的翻译标题 | Adaptive dexterous manipulation for humanoid robots via a guided tactile multimodal foundation model |
|---|---|
| 源语言 | 中文 |
| 页(从-至) | 1526-1543 |
| 期刊 | Scientia Sinica Technologica |
| 卷 | 56 |
| 期 | 6 |
| DOI | |
| 出版状态 | 已出版 - 1 1月 2026 |
关键词
- Alignment
- Closed loop control systems
- Data acquisition
- End effectors
- Mice (computer peripherals)
- Tactile sensors
- Vision
- Visual languages
- Visual servoing
- Dexterous hands
- Dexterous manipulation
- Foundation models
- Guided mask attention
- Humanoid robot
- Multi-modal
- Multimodal frameworks
- Real- time
- Tactile sensing
- Visuo-tactile joint representation
- Force control
- dexterous manipulation
- guided mask attention
- tactile sensing
- visuo-tactile joint representation
收录
- EI
- Scopus
学术指纹
探究 '基于引导式触觉多模态大模型的人形机器人自适应灵巧操作策略' 的科研主题。它们共同构成独一无二的学术指纹。引用此
- APA
- Author
- BIBTEX
- Harvard
- Standard
- RIS
- Vancouver