Grasping an augmented object to analyse manipulative force control

Augmented reality allows changes to be made to the visual perception of object size even while the tangible components remain completely unaltered. It was, therefore, utilized in a study whose results are being reported here to provide the proper environment required to thoroughly observe the exact effect that visual change to object size had on programming fingertip forces when objects were lifted with a precision grip. Twenty-one participants performed repeated lifts of an identical grip apparatus to a height of 20 mm, maintained each lift for 8 seconds, and then replaced the grip apparatus on the table. While all other factors of the grip apparatus remained unchanged, visual appearance was altered graphically in a 3-D augmented environment. The grip apparatus measured grip and load forces independently. Grip and load forces demonstrated significant rates of increase as well as peak forces as the size of graphical images increased; an aspect that occurred in spite of the fact that extraneous haptic information remained constant throughout the trials. By indicating a human tendency to rely - even unconsciously - on visual input to program the forces in the initial lifting phase, this finding provides further confirmation of previous research findings obtained in the physical environment; including the possibility of extraneous haptic effects (Gordon et al. 1991a, Mon-Williams and Murray 2000, Kawai et al. 2000). The present results also suggest that existing knowledge concerning human manipulation tasks in the physical world may be applied to an augmented environment where the physical objects are enhanced by computer generated visual components.     

Shared space: An augmented reality approach for computer supported collaborative work

Virtual Reality (VR) appears a natural medium for three-dimensional computer supported collaborative work (CSCW). However the current trend in CSCW is to adapt the computer interface to work with the user's traditional tools, rather than separating the user from the real world as does immersive VR. One solution is through Augmented Reality, the overlaying of virtual objects on the real world. In this paper we describe the Shared Space concept—the application of Augmented Reality for three-dimensional CSCW. This combines the advantages of Virtual Reality with current CSCW approaches. We describe a collaborative experiment based on this concept and present preliminary results which show that this approach may be better for some applications.     

The use of force feedback and auditory cues for performance of an assembly task in an immersive virtual environment

Using an immersive virtual environment, this study investigated whether the inclusion of force feedback or auditory cues improved manipulation performance and subjective reports of usability for an assembly task. Twenty-four volunteers (12 males and 12 females) were required to assemble and then disassemble five interconnecting virtual parts with either auditory, force, or no feedback cues provided. Performance for the assembly task was measured using completion time and number of collisions between parts, while the users preferences across conditions were evaluated using subjective reports of usability. The results indicated that the addition of force feedback slowed completion time and led to more collisions between parts for males. In contrast, females exhibited no change in the mean completion time for the assembly task but did show an increase in collision counts. Despite these negative performance findings when adding force feedback, users did report perceived increases in realism, helpfulness and utility towards the assembly task when force feedback was provided. Unlike force feedback, the results showed that auditory feedback, indicating that parts had collided during the assembly task, had no negative performance effects on the objective measures while still increasing perceived realism and overall user satisfaction. When auditory cues and force feedback were presented together, performance times, number of collisions, and usability were not improved compared to conditions containing just auditory cues or force feedback alone. Based on these results, and given the task and display devices used in the present study, the less costly option of excluding auditory and force feedback cues would produce the best performance when measured by the number of collisions and completion time. However, if increased ratings of usability for an assembly task are desired while maintaining objective performance levels and reduced cost, then the inclusion of auditory feedback cues is best.     

Advanced teleoperation and control system for industrial robots based on augmented virtuality and haptic feedback

There are some industrial tasks that are still mainly performed manually by human workers due to their complexity, which is the case of surface treatment operations (such as sanding, deburring, finishing, grinding, polishing, etc.) used to repair defects. This work develops an advanced teleoperation and control system for industrial robots in order to assist the human operator to perform the mentioned tasks. On the one hand, the controlled robotic system provides strength and accuracy, holding the tool, keeping the right tool orientation and guaranteeing a smooth approach to the workpiece. On the other hand, the advanced teleoperation provides security and comfort to the user when performing the task. In particular, the proposed teleoperation uses augmented virtuality (i.e., a virtual world that includes non-modeled real-world data) and haptic feedback to provide the user an immersive virtual experience when remotely teleoperating the tool of the robot system to treat arbitrary regions of the workpiece surface. The method is illustrated with a car body surface treatment operation, although it can be easily extended to other surface treatment applications or even to other industrial tasks where the human operator may benefit from robotic assistance. The effectiveness of the proposed approach is shown with several experiments using a 6R robotic arm. Moreover, a comparison of the performance obtained manually by an expert and that obtained with the proposed method has also been conducted in order to show the suitability of the proposed approach.     

Grasping impact force control of a flexible robotic gripper using a piezoelectric actuator

This paper presents robust force tracking control of a flexible beam during a grasping operation using a piezoceramic actuator. Equations describing the motion of the gripper in conditions of contact and noncontact are derived based on the cantilever beam. In this study, contact force is regulated, in addition to the impact force generated at the instant of contact, based on variable structure model reference adaptive control theory using only force measurements. For the derivation of the control law, it is assumed that parameters of the beam and the stiffness of the object are unknown. Computer simulations show the effectiveness the controller. This work was presented, in part, at the Fourth International Symposium on Artificial Life and Robotics, Oita, Japan, January 19–22, 1999     

Inter-digit co-ordination and object-digit interaction when holding an object with five digits

The current study investigated inter-digit co-ordination and object-digit interaction during sustained object holding tasks by using five, six-component force/torque sensors. The sum of the individual finger normal forces and the thumb normal force showed a parallel variation with a mean median correlation coefficient of 0.941. The normal force traces demonstrated the lowest coefficient of variation (about 9% as averaged across digits) as compared with other force/torque traces. The sum for the variances of the normal forces of the index, middle, ring, and little fingers was about 50% of the variance of the summed normal force of the four fingers. Of the five digits, the thumb, index, middle, ring and little fingers accounted for 50.0, 15.4, 14.6, 11.7 and 7.3% of the total normal force; and 39.4, 9.9, 19.3, 14.0 and 17.5% of the total vertical shear force (i.e. the load), respectively. The ratios of the normal force to the resultant shear force were 2.6, 4.5, 1.8, 2.2 and 1.3 for the thumb, index, middle, ring and little finger, respectively. The centre of pressure migration area of a single digit at the object-digit surface during object holding ranged from 0.30 to 1.21 mm². The current study reveals a number of detailed object-digit mechanics and multiple digits co-ordination principle. The results of this study may help to improve ergonomic designs that involve the usage of multiple digits.     

Experimental research into human cognitive processing in an augmented reality environment for embedded training systems

Research into human factors issues associated with the use of augmented reality (AR) technology is very limited. Consequently, there is a need for formal human factors design guidelines to underpin the integration of AR into systems. The Defence Evaluation and Research Agency (DERA) Centre for Human Sciences (CHS) is evaluating the potential of AR for providing real-time training feedback in future advanced embedded training systems for the military. In order to understand the important human factors issues of augmented reality, DERA funded the Advanced VR Research Centre (AVRRC) at Loughborough University to investigate the cognitive ergonomics of this technology. An important aspect of this research is concerned with identifying any human information processing issues that may arise when information is presented via AR and overlaid upon one or more primary display surfaces such as a visual display unit (VDU). Two main issues are addressed in this research. First, the impact of AR on human information processing and second, subjective workload experienced when displaying information via the AR medium. The experiments reported in this paper assess issues of reaccommodation and reaction times to alarms on different display formats. They demonstrate also that AR performs as well as standard display formats.     

Haptic force control based on impedance/admittance control aided by visual feedback

This paper demonstrates a haptic device for interaction with a virtual environment. The force control is added by visual feedback that makes the system more responsive and accurate. There are two popular control methods widely used in haptic controller design. First, is impedance control when user motion input is measured, and then, the reaction force is fed back to the operator. The alternative method is admittance control, when forces exerted by user are measured and motion is fed back to the user. Both, impedance and admittance control are also basic ways for interacting with a virtual environment. In this paper, several experiments were performed to evaluate the suitability of force-impedance control for haptic interface development. The difference between conventional application of impedance control in robot motion control and its application in haptic interface development is investigated. Open loop impedance control methodology is implemented for static case and a general-purpose robot under open loop impedance control was developed as a haptic device, while a closed loop model based impedance control was used for haptic controller design in both static and dynamic case. The factors that could affect to the performance of a haptic interface are also investigated experimentally using parametric studies. Experimental results for 1 DOF rotational motion and 2 DOF planar translational motion systems are presented. The results show that the impedance control aided by visual feedback broaden the applicability of the haptic device and makes the system more responsive and accurate.

Augmented reality as an interface to adaptive hypermedia systems

Adaptive Hypermedia has sought to tackle the problems of dealing with complex, heavily structured information and the presentation of views of that structure to users. Increasingly, adaptive content is achieved through different forms of context. Using two case-study applications, we will reflect on how Augmented Reality may present solutions to a number of Adaptive Hypermedia presentation problems. Each case study describes a different physical interaction metaphor for exposing the complex adaptation of hypermedia content in an intuitive way. The preliminary findings of our early evaluations are discussed. Finally, conclusions are drawn as to how Augmented Reality applications could use the modelling techniques of the Adaptive Hypermedia community to deal more easily with complex information.     

The application of augmented reality technologies for the improvement of occupational safety in an industrial environment

In many branches of industry, occupational safety experts identified two main causes of worker injuries related to the usage of modern electro-mechanical machines and systems: inadequate training and insufficient work experience, and monotonicity of the tasks often performed repeatedly. In this paper, we present a system based on augmented reality (AR) technologies that can be useful in reducing these factors of risk at work and decreasing the error rate and preventing injuries. The system that is implemented on mobile devices is intended to project augmented reality instructions directly at the work place. A worker is led by the AR-system step by step through various work and safety procedures that should be performed. Each procedure consists of steps specified by a series of instructions accessed through an interactive check list. To ensure the safeness, if a confirmation is missing because of a skipped, incompletely, or wrongly performed step of a procedure, the AR-system blocks further implementation of the procedure and returns the worker to the previous step until the correct actions are carried out. At the same time, interactive work with the checklist breaks the monotonicity of the job. The system is personalized according to skills of a worker by taking into account his professional training and work experience. Depending on that it is determined the amount of data to be displayed to a worker helping even less skilled workers to perform a task.As a case study, the proposed approach is implemented as an instructional and occupational safety system for work at a universal lathe, which is an element of many technological processes of Thermal Power Plant Ugljevik in the Republika Srpska, Bosnia and Herzegovina, where this AR-system was experimentally implemented and verified.     

Toward AI-enabled augmented reality to enhance the safety of highway work zones: Feasibility,requirements, and challenges

Highway work zones are considered among the most hazardous working environments. In 2018 alone, 124 workers lost their lives to fatal accidents. The lack of predictive safety systems that notify workers of upcoming dangers in advance is a major reason to blame in the highway maintenance and operation community. This article presents an integrative design framework for bringing recent advances in Augmented Reality (AR) and Artificial Intelligence (AI) to enhance the safety of highway workers through real-time multimodal notifications on-spot. To this end, this article conceptualizes and co-designs three major pillars: (1) AR user interface design for multimodal notification, (2) real-time AI at the edge for vehicle detection/classification from distance, and (3) real-time wireless communication in work zone setting to enable latency-aware operation between AI and AR components. Our early results demonstrate that we can achieve 24.83 FPS end-to-end execution latency on the Xavier AGX Jetson board with 48.7% mAP on BDD100K dataset, and a real-time communication covering 120 meters with an average latency of 5.1 ms at the farthest distance. Our mixed-method user research also reveals an acceptable level of excitement and engagement from the body of highway workers toward both the proposed technology and the designed user interface. Overall, this article provides a proof-of-concept toward AI-enabled AR safety systems in highway work zones.     

Using visual and auditory cues to locate out-of-view objects in head-mounted augmented reality

When looking for an object in a complex visual scene, Augmented Reality (AR) can assist search with visual cues persistently pointing in the target’s direction. The effectiveness of these visual cues can be reduced if they are placed at a different visual depth plane to the target they are indicating. To overcome this visual-depth problem, we test the effectiveness of adding simultaneous spatialized auditory cues that are fixed at the target’s location. In an experiment we manipulated which cue(s) were available (visual-only vs. visual + auditory), and which disparity plane relative to the target the visual cue was displayed on. Results show that participants were slower at finding targets when the visual cue was placed on a different disparity plane to the target. However, this slowdown in search performance could be substantially reduced with auditory cueing. These results demonstrate the importance of AR cross-modal cueing under conditions of visual uncertainty and show that designers should consider augmenting visual cues with auditory ones.     

Design of an augmented automatic choosing control with constrained input by Hamiltonian and absolute antiwindup measures

In this paper, we present a novel approach for a nonlinear feedback control which is called an augmented automatic choosing control (AACC) of the antiwindup type for nonlinear systems with constrained inputs. Constant terms which arise from section-wise linearization of a given nonlinear system are treated as coefficients of a stable zero dynamics. Parameters included in the control are suboptimally selected by the genetic algorithm so as to minimize a performance made of the Hamiltonian and the absolute antiwindup measure. This approach is applied to a field excitation control problem of a power system to demonstrate the superiority of the AACC. Simulation results show that the new controller can improve performance remarkably well. This work was presented in part at the 12th International Symposium on Artificial Life and Robotics, Oita, Japan, January 25–27, 2007     

Constrained model predictive force control of an electrohydraulic actuator

An analytical MPC controller was designed for force control of a single-rod electrohydraulic actuator. The controller based on a difference equation uses short control horizon. The constraints on both input and output variables are taken into consideration by the controller. The mechanism of output constraints satisfaction uses output prediction and makes possible to constrain the output values many sampling instants ahead. Thus, it extends capabilities of the analytical MPC controllers to the field reserved so far for much more computationally expensive numerical MPC algorithms. Results of real life experiments illustrate efficiency of the proposed controller. The results also show that the MPC controller has better tracking performance than conventional P and PI controllers. The MPC controller with the constraint handling mechanisms, though relatively simple, offers very good performance. As the design process is detailed, it is possible to relatively easy adapt the proposed approach to other control plants.     

A simplified approach to force control for electro-hydraulic systems

In this paper, a Lyapunov-based control algorithm is developed for force tracking control of an electro-hydraulic actuator. The developed controller relies on an accurate model of the system. To compensate for the parametric uncertainties, a Lyapunov-based parameter adaptation is applied. The adaptation uses a variable structure approach to account for asymmetries present in the system. The coupled control law and the adaptation scheme are applied to an experimental valve-controlled cylinder. Friction modeling and compensation are also discussed. The experimental results show that the nonlinear control algorithm, together with the adaptation scheme, gives a good performance for the specified tracking task. The original adaptive control law is then simplified in several stages with an examination of the output tracking at each stage of simplification. It is shown that the original algorithm can be significantly simplified without too significant a loss of performance. The simplest algorithm corresponds to an adaptive velocity feedback term coupled with a simple force error feedback.     

Self-tuning of quantitative feedback theory for force control of an electro-hydraulic test machine

The aim of this paper is to design and fabricate an electro-hydraulic test machine (EHTM) for conducting performance and stability test of force and/or position control in the bench system and to propose a robust adaptive controller—the self-tuning quantitative feedback theory (STQFT)—in order to improve the force control performance of electro-hydraulic actuators (EHAs). By using the gradient descent method to adjust the QFT parameters, the novel STQFT controller is able to adapt to a wide range of working conditions. Experiments were carried out to evaluate the effectiveness of the proposed control method applied to the EHTM.     

Position/force hybrid control system for high precision aligning of small gripper to ring object

A position/force hybrid control system based on impedance control scheme is designed to align a small gripper to a special ring object. The vision information provided by microscope vision system is used as the feedback to indicate the position relationship between the gripper and the ring object. Multiple image features of the gripper and the ring object are extracted to estimate the relative positions between them. The end-effector of the gripper is tracked using the extracted features to keep the gripper moving in the field of view. The force information from the force sensor serves as the feedback to ensure that the contact force between the gripper and the ring object is limited in a small safe range. Experimental results verify the effectiveness of the proposed control strategy.     

Human-machine interaction force control: using a model-referenced adaptive impedance device to control an index finger exoskeleton

Exoskeleton robots and their control methods have been extensively developed to aid post-stroke rehabilitation. Most of the existing methods using linear controllers are designed for position control and are not suitable for human-machine interaction （HMI） force control, as the interaction system between the human body and exoskeleton is uncertain and nonlinear. We present an approach for HMI force control via model reference adaptive impedance control （MRAIC） to solve this problem in case of index finger exoskeleton control. First, a dynamic HMI model, which is based on a position control inner loop, is for- mulated. Second, the theoretical MRAC framework is implemented in the control system. Then, the adaptive controllers are designed according to the Lyapunov stability theory. To verify the performance of the proposed method, we compare it with a proportional-integral-derivative （PID） method in the time domain with real experiments and in the frequency domain with simu- lations. The results illustrate the effectiveness and robustness of the proposed method in solving the nonlinear HMI force control problem in hand exoskeleton.