首页 | 本学科首页   官方微博 | 高级检索  
     

支持多模态交互的桌面增强显示系统
引用本文:王子珩,姜忠鼎.支持多模态交互的桌面增强显示系统[J].计算机工程,2022,48(7):177-188.
作者姓名:王子珩  姜忠鼎
作者单位:1. 复旦大学 软件学院, 上海 200438;2. 复旦大学 上海市数据科学重点实验室, 上海 200438
基金项目:教育部第二批“新工程”研究与实践项目(E-XTYR20200621)。
摘    要:传统的显示设备在受限的物理空间内难以向用户呈现大量画面和复杂内容,而AR头戴式显示设备通过将三维的可视化内容悬浮显示在用户眼前,在不占用额外物理空间的条件下可增强真实世界的画面显示,呈现形式更为丰富的内容。设计AR虚拟空间与真实电脑画面虚实融合的桌面增强显示系统。通过基于二维码识别的空间定位技术将真实电脑画面映射至虚拟空间内,实现交互空间的统一,同时构建窗口布局计算模型使得系统可以根据用户自定义参数自动生成窗口并设置其布局。在此基础上,利用蓝牙通信、网络传输、操作系统底层映射和结合视线检测的语音识别等技术支持手势、键鼠和语音的多模态交互方式,设计鼠标移动策略以扩展鼠标在三维空间下的多种操作模式。实验结果表明,与隔空手势交互、鼠标交互等传统交互方式相比,该系统在处理常见电脑任务时平均耗时节省10%~30%,具有较高的交互效率,且在跨窗口连续移动和瞬间跳转时能够正确显示鼠标位置。

关 键 词:头戴式显示设备  增强显示  虚实融合  多模态交互  人机交互  
收稿时间:2021-05-20
修稿时间:2021-07-14

Enhanced Desktop Display System Supporting Multi-Modal Interaction
WANG Ziheng,JIANG Zhongding.Enhanced Desktop Display System Supporting Multi-Modal Interaction[J].Computer Engineering,2022,48(7):177-188.
Authors:WANG Ziheng  JIANG Zhongding
Affiliation:1. Software School, Fudan University, Shanghai 200438, China;2. Shanghai Key Laboratory of Data Science, Fudan University, Shanghai 200438, China
Abstract:Traditional display devices are limited in their ability to present a large amount of complex content to users with in a given physical space.In contrast, Augmented Reality(AR) Head Mounted Display(HMD) devices can enhance users' perception of the real world by overlaying a three-dimensional visualization on their field of view to display richer content in several forms without occupying additional physical space.This study presents the design of an enhanced desktop display system that integrates virtual content in AR space with real screen.Through spatial positioning technology based on QR code recognition, real computer screens are mapped to aunified virtual interactive space.By establishing a window layout calculation model, the system can automatically generate windows and set their layouts according to user-defined parameters.The system supports multi-modal gesture interaction modes including keyboard and mouse input as well as voice control through technologies such as Bluetooth communication, network transmission, an underlying mapping of the operating system, and voice recognition based on gesture detection.The mouse movement strategy of the system is designed to expand the various operation modes of standard mouse devices into a three-dimensional space.The results of experiments show that users are able to complete common tasks with the proposed system in about 10%~30% less time on average compared with traditional input methods such as air gesture-based and mouse interaction.Moreover, the proposed approach exhibited a high interaction efficiency, and is able to display the cursor position correctly when moving across windows continuously and jumping instantaneously.
Keywords:Head Mounted Display(HMD) device  enhanced display  integration of virtuality and reality  multi-modal interaction  human-computer interaction  
点击此处可从《计算机工程》浏览原始摘要信息
点击此处可从《计算机工程》下载全文
设为首页 | 免责声明 | 关于勤云 | 加入收藏

Copyright©北京勤云科技发展有限公司  京ICP备09084417号