Using intermediate objects to improve the efficiency of visual search

When using a mobile camera to search for a target object, it is often important to maximize the efficiency of the search. We consider a method for increasing efficiency by searching only those subregions that are especially likely to contain the object. These subregions are identified via spatial relationships. Searches that use this method repeatedly find an “intermediate” object that commonly participates in a spatial relationship with the target object, and then look for the target in the restricted region specified by this relationship. Intuitively, such searches, calledindirect searches, seem likely to provide efficiency increases when the intermediate objects can be recognized at low resolutions and hence can be found with little extra overhead, and when they significantly restrict the area that must be searched for the target. But what is the magnitude of this increase, and upon what other factors does efficiency depend? Although the idea of exploiting spatial relationships has been used in vision systems before, few have quantitatively examined these questions. We present a mathematical model of search efficiency that identifies the factors affecting efficiency and can be used to predict their effects. The model predicts that, in typical situations, indirect search provides up to an 8-fold increase in efficiency. Besides being useful as an analysis tool, the model is also suitable for use in an online system for selecting intermediate objects.     

Using visual and auditory cues to locate out-of-view objects in head-mounted augmented reality

When looking for an object in a complex visual scene, Augmented Reality (AR) can assist search with visual cues persistently pointing in the target’s direction. The effectiveness of these visual cues can be reduced if they are placed at a different visual depth plane to the target they are indicating. To overcome this visual-depth problem, we test the effectiveness of adding simultaneous spatialized auditory cues that are fixed at the target’s location. In an experiment we manipulated which cue(s) were available (visual-only vs. visual + auditory), and which disparity plane relative to the target the visual cue was displayed on. Results show that participants were slower at finding targets when the visual cue was placed on a different disparity plane to the target. However, this slowdown in search performance could be substantially reduced with auditory cueing. These results demonstrate the importance of AR cross-modal cueing under conditions of visual uncertainty and show that designers should consider augmenting visual cues with auditory ones.     

Selection of key visual cues in real and virtual environments for assembly tasks

Previous research identified that learning assembly tasks in Virtual Environments (VEs) is more difficult than in Real Environments (REs). This work's objective is to identify the key visual areas for both environments when performing an assembly task for ten consecutive cycles, when following visual instructions and when having visual distractors. Using an eye-tracker, we identified the key visual areas required for an assembly task in both environments. Results indicate that practice allowed participants to reduce their assembly time in both environments. They also indicate that two areas, Assembly Area and Blocks, concentrated a higher proportion of eye-fixations; 59.98% for REs and 81.48% for VEs, with a statistically significant observation difference between environments (t-test value = −14.23, with p-value <0.00001 and Cohen's d = 6.36). We conclude that participants considered the same key visual areas for both environments and that VE interaction has a significant role in observation behavior.     

Localization of virtual objects in the near visual field

We examined errors in the localization of nearby virtual objects presented via see-through helmet-mounted displays as a function of viewing conditions and scene content in four experiments using a total of 38 participants. Monocular, biocular, and stereoscopic presentation of the virtual objects, accommodation (required focus), participants' age, and the position of physical surfaces were examined. Nearby physical surfaces were found to introduce localization errors that differ depending on the other experimental factors. These errors apparently arise from the occlusion of the physical background by the optically superimposed virtual objects, but they are modified by participants' accommodative competence and specific viewing conditions. The apparent physical size and transparency of the virtual objects and physical surfaces, respectively, are influenced by their relative position when superimposed. The design implications of the findings are discussed in a concluding section. Head-mounted displays of virtual objects are currently being evaluated as aids for mechanical assembly and equipment maintenance. Other applications include telesurgery, surgical planning, telerobotics, and visualization aids for robotic programming.     

Precise and efficient pose estimation of stacked objects for mobile manipulation in industrial robotics challenges

Object manipulation tasks such as picking up, carrying and placing should be executed based on the information of objects which are provided by the perception system. A precise and efficient pose estimation system has been developed to address the requirements and to achieve the objectives for autonomous packaging, specifically picking up of stacked non-rigid objects. For fine pose estimation, a drawing pin shaped kernel and pinhole filtering methods are used on the roughly estimated pose of objects. The system has been applied in a realistic industrial environment as a challenging scenario for the Challenge 2 – Shop Floor Logistics and Manipulation on a mobile manipulator in the context of the European Robotics Challenges (EuRoC) project.     

利用问题求解理论来研究交互式复杂信息的可视分析行为

面对大数据的挑战,力图将人的推理能力和计算系统的数据处理能力相结合的交 互式可视分析研究变得愈发重要。然而目前仍缺乏有效的认知理论来指导面向复杂信息的可视 分析系统的设计,诸如意义构建等现有的理论框架通常着眼于分析行为的外在特征,未能对此 类行为的内在认知机理进行深入研究。因此提出将问题求解作为一种理论框架来解释交互可视 分析行为的基本认知活动,并建议从非良构问题的角度来描述可视分析过程中用户所面临的主 要挑战,还从问题表征及问题求解策略等角度分析了可视分析系统对分析行为的影响。本研究 在理论上,将认知心理学领域的问题求解理论引入到交互可视分析行为的研究中,该方法对设 计面向复杂信息分析的其他类型交互系统也有启示作用;在实践层面上,从问题求解的支持角 度探索了可视分析系统的设计和评估问题。     

The affect of contact force sensations on user performance in virtual assembly tasks

The development of a realistic virtual assembly environment is challenging because of the complexity of the physical processes and the limitation of available VR technology. Many research activities in this domain primarily focused on particular aspects of the assembly task such as the feasibility of assembly operations in terms of interference between the manipulated parts. The virtual assembly environment reported in this research is focused on mechanical part assembly. The approach presented addresses the problem of part-to-part contacts during the mating phase of assembly tasks. The system described calculates contact force sensations by making their intensity dependent on the depth of penetration. However the penetration is not visible to the user who sees a separate model, which does not intersect the mating part model. The two 3D models of the part, the off-screen rendered model and the on-screen rendered model are connected by a spring-dumper arrangement. The force calculated is felt by the operator through the haptic interface when parts come in contact during the mating phase of the assembly task. An evaluation study investigating the effect of contact force sensation on user performance during part-to-part interface was conducted. The results showed statistically significant effect of contact force sensation on user performance in terms of task completion time. The subjective evaluation based on feedback from users confirmed that contact force sensation is a useful cue for the operator to find the relative positions of components in the final assembly state.     

A novel virtual node approach for interactive visual analytics of big datasets in parallel coordinates

Big data is a collection of large and complex ​datasets that commonly appear in multidimensional and multivariate data formats. It has been recognized as a big challenge in modern computing/information sciences to gain (or find out) due to its massive volume and complexity (e.g. its multivariate format). Accordingly, there is an urgent need to find new and effective techniques to deal with such huge ​datasets. Parallel coordinates is a well-established geometrical system for visualizing multidimensional data that has been extensively studied for decades. There is also a variety of associated interaction techniques currently used with this geometrical system. However, none of these existing techniques can achieve the functions that are covered by the Select layer of Yi’s Seven-Layer Interaction Model. This is because it is theoretically impossible to find a select of data items via a mouse-click (or mouse-rollover) operation over a particular visual poly-line (a visual object) with no geometric region. In this paper, we present a novel technique that uses a set of virtual nodes to practically achieve the Select interaction which has hitherto proven to be such a challenging sphere in parallel coordinates visualization.     

Interaction techniques in desktop virtual environment: the study of visual feedback and precise manipulation method

Gesture-based systems allow users to interact with a virtual reality application in a natural way. Visual feedback for the gesture-based interaction technique has an impact on the performance and the hand instability making the manipulation of the object less precise. This paper investigated two new interaction techniques in a virtual environment. It describes the influence of natural and non-natural virtual feedback in the selection process using the GITDVR-G interaction technique, which consists of a grasping visual feedback. The GITDVR-G was evaluated in a virtual knee surgery training system. The results showed that it was effective in terms of the task completion time, and that the participants preferred the natural grasping visual feedback. Besides that, the precise manipulation in a newly-designed interaction technique (Precise GITDVR-G) was evaluated. The Precise GITDVR-G includes a normal manipulation mode and a precise manipulation mode that can be triggered by hand gestures. During the precise manipulation mode, an inset view will appear and move with the selected object to provide a better view to users, while the movements of the virtual hand are scaled down to improve the precision. Four different configurations of the precise manipulation technique were evaluated, and the results showed that the unimanual control method with an inset view performed better in terms of the task performance time and the subjective feedback. The finding suggested that the realistic virtual grasping visual feedback can be applied in a virtual hand interaction technique, and that the inset view feature is helpful in the precise manipulation.     

桌面虚拟技术提升高职院校运维服务质量

随着各种虚拟化产品的不断出现,虚拟化技术已经成为计算机技术发展的趋势.本文就桌面虚拟化技术的结构、功能效果方面做了说明,高职院校可以通过引入桌面虚拟化方案来提升数据安全、简化管理和节能减排,及时满足灵活的教学模式,保证了教学的顺利开展.     

Applications of virtual reality in maintenance during the industrial product lifecycle: A systematic review

As a state-of-the-art computer technology, virtual reality (VR) is considered to play an important role in helping manufacturing companies stay competitive in the international market. However, despite the achievements made in the field of VR, it is still an emerging technology that lacks deeper exploration and development in industrial application scenarios, especially in the coming fourth industrial revolution (Industry 4.0). This paper aims to systematically investigate the applications of VR in industrial maintenance to discover evidence of its values, limitations, and future directions so that VR can be guided to better serve manufacturing enterprises in remaining competitive in the coming Industry 4.0. A systematic literature review (SLR) methodology is adopted to review primary studies on this topic, by which 86 studies are ultimately included. The results show that VR has proved its value in benefiting maintenance issues through the product lifecycle. However, VR is still not an indispensable element for the lifecycle management of products regarding maintenance-related issues. Several key findings are concluded based on the analysis of the 86 studies. This review is valuable for researchers who are interested in the application of VR technology in maintainability design, maintenance training or maintenance task assistance.     

The use of force feedback and auditory cues for performance of an assembly task in an immersive virtual environment

Using an immersive virtual environment, this study investigated whether the inclusion of force feedback or auditory cues improved manipulation performance and subjective reports of usability for an assembly task. Twenty-four volunteers (12 males and 12 females) were required to assemble and then disassemble five interconnecting virtual parts with either auditory, force, or no feedback cues provided. Performance for the assembly task was measured using completion time and number of collisions between parts, while the users preferences across conditions were evaluated using subjective reports of usability. The results indicated that the addition of force feedback slowed completion time and led to more collisions between parts for males. In contrast, females exhibited no change in the mean completion time for the assembly task but did show an increase in collision counts. Despite these negative performance findings when adding force feedback, users did report perceived increases in realism, helpfulness and utility towards the assembly task when force feedback was provided. Unlike force feedback, the results showed that auditory feedback, indicating that parts had collided during the assembly task, had no negative performance effects on the objective measures while still increasing perceived realism and overall user satisfaction. When auditory cues and force feedback were presented together, performance times, number of collisions, and usability were not improved compared to conditions containing just auditory cues or force feedback alone. Based on these results, and given the task and display devices used in the present study, the less costly option of excluding auditory and force feedback cues would produce the best performance when measured by the number of collisions and completion time. However, if increased ratings of usability for an assembly task are desired while maintaining objective performance levels and reduced cost, then the inclusion of auditory feedback cues is best.     

The influence of the availability of visual cues on the accurate perception of spatial dimensions in architectural virtual environments

Several authors have observed that spatial dimensions tend to be underestimated in virtual environments. In this study, we hypothesize that the availability of visual cues in virtual environments has an influence on the accuracy of perception. An experiment was conducted to compare spatial perception in real and virtual environments that were modeled differently and visualized using a head-mounted display. Results suggest that the greater the availability of visual cues, the greater the level of accuracy in the estimates, especially for egocentric dimensions (p < 0.001). In the end, this study contributes to a better understanding of how architectural virtual environments should be modeled for use in professional or commercial applications where accurate and reliable simulations are required.     

Psychophysiological evaluation of seafarers to improve training in maritime virtual simulator

Over the years, safety in maritime industries has been reinforced by many state-of-the-art technologies. However, the accident rate hasn’t dropped significantly with the advanced technology onboard. The main cause of this phenomenon is human errors which drive researchers to study human factors in the maritime domain. One of the key factors that contribute to human performance is their mental states such as cognitive workload and stress. In this paper, we propose and implement an Electroencephalogram (EEG)-based psychophysiological evaluation system to be used in maritime virtual simulators for monitoring, training and assessing the seafarers. The system includes an EEG processing part, visualization part, and an evaluation part. By using the processing part of the system, different brain states including cognitive workload and stress can be identified from the raw EEG data recorded during maritime exercises in the simulator. By using the visualization part, the identified brain states, raw EEG signals, and videos recorded during the maritime exercises can be synchronized and displayed together. By using the evaluation part of the system, an indicative recommendation on “pass”, “retrain”, or “fail” of the seafarers’ performance can be obtained based on the EEG-based cognitive workload and stress recognition. Detailed analysis of the demanding events in the maritime tasks is provided by the system for each seafarer that could be used to improve their training. A case study is presented using the proposed system. EEG data from 4 pilots were recorded when they were performing maritime tasks in the simulator. The data are processed and evaluated. The results show that one pilot gets a “pass” recommendation, one pilot gets a “retrain” recommendation, and the other two get “fail” results regarding their performance in the simulator.     

Detection and interactive isolation of faults in steam turbines to support maintenance decisions

The maintenance of steam turbines is expensive, particularly if dismantling is required. A concept for the provision of support for the maintenance engineer in determining steam turbine status in relation to the recommended maintenance interval is presented here. The concept embodies an artificial neural network which is conditioned to recognise patterns known to be related to faults. The faults simulated are not known to be recognized on-line and the concept is in an early stage of development. An example of a Bayesian network structure containing expert knowledge is proposed to be used, in a dialogue with the operator, to isolate the root causes of a number of fault types. The aim is to be well informed about the statue of the turbine in order to take earlier and better informed maintenance actions. The detection procedure has been validated in a simulation environment.     

The application of parallel projections to three-dimensional object location in industrial assembly

Flexible industrial assembly under visual control requires fast and simple picture processing. Parallel projection optics can help to achieve this because it provides constant size images, high immunity to ambient illumination and easy combination of the results from multiple cameras to determine the position of objects in 3-D space. This paper describes the optical system itself, a simple method of calibration, and the application of the system to a simple problem in 3-D assembly.     

The application of computer vision to the removal of tie-wires in a live-line maintenance robotics manipulation

This paper presents a computer vision application for the detection of a tie-wire in a live-line maintenance operation for the replacement of insulators. The tie-wire is wrapped around the electrical conductor and must be unwrapped by a special tool fixed to the end effector of a manipulator. The computer vision algorithm uses pyramidal image reduction, edge image analysis, and inverse perspective projection to count the number of tie-wire turns around the conductor and compute the three-dimensional coordinates of each turn (or the distance between adjacent turns).