基于音视信息融合的桌面机械臂技能获取及控制系统

针对化学实验机械臂控制编程门槛高且技能获取准确率低等发展制约因素，设计一种基于音视信息融合算法的桌面型实验机械臂的控制系统，实验员通过边做边说的示教方式教给机械臂运动技能，进而代替实验员完成一些繁琐、具有危险性的实验工作。系统分为技能获取以及运动控制两部分。其技能获取部分使用改进的双流卷积网络实现动作检测；使用语音AI和正则表达式实现语音提取；再通过音视动作信息融合算法将动作检测和语音部分的识别信息相融合得出高重合度的运动技能，技能获取准确度可达81%以上。运动控制部分使用电机控制和抓取位姿识别，可实现更精细的控制和抓取。系统可用于具有特定流程化学实验的示教控制工作，在代替实验员来完成化学实验工作的同时大大降低了编程门槛，提高了效率。

Research on Acquiring Teaching Skills of Desktop Manipulator Based on Audio-visual Information Fusion

Aiming at developmental constraints such as the high threshold of control programming for chemical experiment manipulators and the low accuracy of skill acquisition, a control system for desktop experimental manipulators based on audio-visual information fusion algorithm is designed. Teach the mechanical arm movement skills, and then replace the experimenter to complete some tedious and dangerous experimental work. The system is divided into two parts: skill acquisition and movement control. Its skill acquisition part uses an improved dual-stream convolutional network to achieve motion detection; uses voice AI and regular expressions to achieve voice extraction; and then uses audio-visual motion information fusion algorithms to integrate motion detection and voice recognition information to obtain a high degree of coincidence The accuracy of skill acquisition can reach more than 81%. The motion control part uses motor control and grasping pose recognition, which can achieve more precise control and grasping. The system can be used for the teaching and control work of chemical experiments with specific processes. It can replace the experimenter to complete the chemical experiment work while greatly reducing the programming threshold and improving the efficiency.