跳到主要导航 跳到搜索 跳到主要内容

基于引导式触觉多模态大模型的人形机器人自适应灵巧操作策略

  • School of Artificial Intelligence

科研成果: 期刊稿件 › 论文 › 同行评审

摘要

Dexterous manipulation models under the vision-language-action (VLA) paradigm can generate stable action trajectories, yet fine grasping often fails due to pose misalignment of the dexterous hand and force-control deviations. To address the dual challenges of blind spots in unimodal visual perception and unstable contact force control, we propose a guided tactile multimodal framework (GTMF) that decodes visuo-tactile joint features into tactile thresholds and, with real-time feedback from fingertip tactile arrays, performs precise grasp refinement. Built on VLA’s cross-scene generalization, GTMF first uses a guided mask-attention network to calibrate the end-effector pose, then performs cross-modal alignment and decoding to produce scene-adaptive tactile thresholds, which drive real-time servo control for accurate closed-loop grasping. For data acquisition, we design a wearable visuo-tactile isomorphic module to efficiently collect human tactile signals together with end-effector vision and pose. Experiments verify the effectiveness of the approach, achieving grasp accuracy comparable to or exceeding mainstream baselines even when relying solely on tactile input. To the best of our knowledge, this is the first method that decodes tactile signals purely from RGB images and uses them for real-time grasp-pose adjustment, offering a new route to robust dexterous grasping within the VLA framework. © 2026 Science Press. All rights reserved.
投稿的翻译标题Adaptive dexterous manipulation for humanoid robots via a guided tactile multimodal foundation model
源语言中文
页(从-至)1526-1543
期刊Scientia Sinica Technologica
卷56
期6
DOI
出版状态已出版 - 1 1月 2026

关键词

  • Alignment
  • Closed loop control systems
  • Data acquisition
  • End effectors
  • Mice (computer peripherals)
  • Tactile sensors
  • Vision
  • Visual languages
  • Visual servoing
  • Dexterous hands
  • Dexterous manipulation
  • Foundation models
  • Guided mask attention
  • Humanoid robot
  • Multi-modal
  • Multimodal frameworks
  • Real- time
  • Tactile sensing
  • Visuo-tactile joint representation
  • Force control
  • dexterous manipulation
  • guided mask attention
  • tactile sensing
  • visuo-tactile joint representation

收录

  • EI
  • Scopus

学术指纹

探究 '基于引导式触觉多模态大模型的人形机器人自适应灵巧操作策略' 的科研主题。它们共同构成独一无二的学术指纹。

引用此