A new AR authoring tool using depth maps for industrial procedures

Several augmented reality systems have been proposed for different target fields such as medical, cultural heritage and military. However, most of the current AR authoring tools are actually programming interfaces that are exclusively suitable for programmers. In this paper, we propose an AR authoring tool which provides advanced visual effect, such as occlusion or media contents. This tool allows non-programming users to develop low-cost AR applications, specially oriented to on-site assembly and maintenance/repair tasks. A new 3D edition interface is proposed, using photos and Kinect depth information to improve 3D scenes composition. In order to validate our AR authoring tool, two evaluations have been performed, to test the authoring process and the task execution using AR. The evaluation results show that overlaying 3D instructions on the actual work pieces reduces the error rate for an assembly task by more than a 75%, particularly diminishing cumulative errors common in sequential procedures. Also, the results show how the new edition interface proposed, improves the 3D authoring process making possible create more accurate AR scenarios and 70% faster.     

An augmented reality approach for supporting panel alignment in car body assembly

One of the major problems in automotive assembly consists in achieving alignment within the specified tolerances between car panels that make up the exterior bodywork. To reduce errors and time requested to perform the assigned assembly tasks, workers should be guided during these panel fitting operations. Augmented Reality (AR) could be particularly suitable in this regard, as it represents one of the most promising tools to support personnel, with constantly growing applications in production processes. Following this trend, the present work aims to present an AR prototype system for supporting the operator during panel fitting operations of car body assembly, by providing instructions to correct alignment errors in terms of gap and flushness. A real case study concerning the fine alignment of car body panels with respect to the front light projector is also presented.     

Exploring augmented reality for worker assistance versus training

This paper aims at advancing the fundamental understanding of the affordances of Augmented Reality (AR) as a workplace-based learning and training technology in supporting manual or semi-automated manufacturing tasks that involve both complex manipulation and reasoning. Between-subject laboratory experiments involving 20 participants are conducted on a real-life electro-mechanical assembly task to investigate the impacts of various modes of information delivery through AR compared to traditional training methods on task efficiency, number of errors, learning, independence, and cognitive load. The AR application is developed in Unity and deployed on HoloLens 2 headsets. Interviews with experts from industry and academia are also conducted to create new insights into the affordances of AR as a training versus assistive tool for manufacturing workers, as well as the need for intelligent mechanisms that enable adaptive and personalized interactions between workers and AR. The findings indicate that despite comparable performance between the AR and control groups in terms of task completion time, learning curve, and independence from instructions, AR dramatically decreases the number of errors compared to traditional instruction, which is sustained after the AR support is removed. Several insights drawn from the experiments and expert interviews are discussed to inform the design of future AR technologies for both training and assisting incumbent and future manufacturing workers on complex manipulation and reasoning tasks.     

Identifying the task variables that influence perceived object assembly complexity

There is a general lack of understanding as to what issues affect assembly task performance when using diagrammatic instructions because few of the task variables contributing to assembly complexity have been identified. Using a task analysis of a range of self-assembly products, seven task variables hypothesized to predict assembly complexity were identified and studied in the instruction comprehension phase of assembly. Experiment 1 took nine real world assembly instructions and described each in terms of the seven task variables. Seventy-two participants gave a subjective rating of assembly difficulty for each assembly, showing a clear relationship between the task variables and perceived assembly difficulty. As real world assemblies provide little control a second experiment used an orthogonal design to systematically vary the values of each of the assembly task variables in 16 abstract assemblies. Forty-two participants compared the 16 assembly instructions to a final assembly. There was a clear relationship between the task variables and the time taken to view the instructions. Further, it was found that it is possible to predict the complexity of assembly tasks based upon the levels of the task variables identified. The task variables identified are a significant step towards identifying the factors that influence assembly complexity, together with providing progress towards a tool for predicting assembly complexity.     

Identifying the task variables that influence perceived object assembly complexity

There is a general lack of understanding as to what issues affect assembly task performance when using diagrammatic instructions because few of the task variables contributing to assembly complexity have been identified. Using a task analysis of a range of self-assembly products, seven task variables hypothesized to predict assembly complexity were identified and studied in the instruction comprehension phase of assembly. Experiment 1 took nine real world assembly instructions and described each in terms of the seven task variables. Seventy-two participants gave a subjective rating of assembly difficulty for each assembly, showing a clear relationship between the task variables and perceived assembly difficulty. As real world assemblies provide little control a second experiment used an orthogonal design to systematically vary the values of each of the assembly task variables in 16 abstract assemblies. Forty-two participants compared the 16 assembly instructions to a final assembly. There was a clear relationship between the task variables and the time taken to view the instructions. Further, it was found that it is possible to predict the complexity of assembly tasks based upon the levels of the task variables identified. The task variables identified are a significant step towards identifying the factors that influence assembly complexity, together with providing progress towards a tool for predicting assembly complexity.     

Cross-organizational collaboration supported by Augmented Reality

This paper presents a study where Augmented Reality (AR) technology has been used as a tool for supporting collaboration between the rescue services, the police and military personnel in a crisis management scenario. There are few studies on how AR systems should be designed to improve cooperation between actors from different organizations while at the same time supporting individual needs. In the present study, an AR system was utilized for supporting joint planning tasks by providing organization specific views of a shared map. The study involved a simulated emergency event conducted in close to real settings with representatives from the organizations for which the system is developed. As a baseline, a series of trials without the AR system was carried out. Results show that the users were positive toward the AR system and would like to use it in real work. They also experience some performance benefits of using the AR system compared to their traditional tools. Finally, the problem of designing for collaborative work as well as the benefits of using an iterative design processes is discussed.     

Impact of monitor-based augmented reality for on-site industrial manual operations

This work evaluates the impact of augmented reality (AR) technology to support operators during manual industrial tasks. The work focuses specifically on monitor-based augmented reality as a solution to provide instructions to the operator. Instructions are superimposed directly onto a video representation of the physical workspace and are displayed on a standard monitor. In contrast to previous AR solutions proposed to support manual tasks, the current work is more industrially acceptable because it meets most of the industrial requirements and it is cost effective for a deployment in an industrial environment. The developed prototype is described and evaluated by means of a user study to compare the monitor-based augmented reality solution to provide instructions and the traditional method. The test shows that AR can be a valid substitute to support the operators during manual tasks because it allows them to be more time efficient and it reduces their mental workload compared to traditional instructional manuals.     

An experimental study on augmented reality assisted manual assembly with occluded components

Augmented Reality (AR) technology has increasingly been applied to facilitate manufacturing tasks such as training, maintenance, and safety management, in which manual assembly frequently occurs. Previously, studies confirmed the values of AR-based assembly systems, but few explored support of spatially restricted assembly where components are visually occluded from operator. In this regard, this research aims to develop new AR functions that assist manual assembly in such situations. The focus is on validating the effectiveness of various assistive information in AR, including assembly interface, operator’s hand movement, and operator held components. Subjective and objective measures are used to evaluate assembly experiments of different degrees of difficulty. Analyses of the experimental data reveal whether or not and how effectively each information offers guidance in occluded condition. In particular, a time reduction in more difficult assembly is realized by showing operator hand movement in AR. The hand model initially offers an operator a visual clue for quickly and roughly locating the assembly interface in a large unseen area, prior to precisely localizing in a smaller region guided by tactile sensing. However, the effectiveness of incorporating the held component is not evident, as positional deviation between real and virtual objects may reduce human’s hand-eye coordination. These findings not only provide preliminary design guidelines of AR assembly functions for occluded components but also demonstrates a novel yet practical application of AR technology in smart manufacturing.     

Worst‐case execution time analysis for a Java processor

In this paper, we propose a solution for a worst‐case execution time (WCET) analyzable Java system: a combination of a time‐predictable Java processor and a tool that performs WCET analysis at Java bytecode level. We present a Java processor, called JOP, designed for time‐predictable execution of real‐time tasks. The execution time of bytecodes, the instructions of the Java virtual machine, is known to cycle accuracy for JOP. Therefore, JOP simplifies the low‐level WCET analysis. A method cache, which fills whole Java methods into the cache, simplifies cache analysis. The WCET analysis tool is based on integer linear programming. The tool performs the low‐level analysis at the bytecode level and integrates the method cache analysis. An integrated data‐flow analysis performs receiver‐type analysis for dynamic method dispatches and loop‐bound analysis. Furthermore, a model checking approach to WCET analysis is presented where the method cache can be exactly simulated. The combination of the time‐predictable Java processor and the WCET analysis tool is evaluated with standard WCET benchmarks and three real‐time applications. The WCET friendly architecture of JOP and the integrated method cache analysis yield tight WCET bounds. Comparing the exact, but expensive, model checking‐based analysis of the method cache with the static approach demonstrates that the static approximation of the method cache is sufficiently tight for practical purposes. Copyright © 2010 John Wiley & Sons, Ltd.     

Impact of information display on worker performance for wood frame wall assembly using AR HMD under different task conditions

Augmented Reality (AR) head-mounted displays (HMDs) provide users with an immersive virtual experience in the real world. The portability of this technology affords various information display options for construction workers that are not possible otherwise. However, the impact of these different information presentation options on human performance should be carefully evaluated before such technology is deployed in the jobsite. In this paper, we describe a research effort examining how different information displays presented via AR HMD influence task performance when assembling three sized wooden wall frame assembly tasks. We asked 18 construction engineering students with framing experience to finish three wood frame assembly tasks (large, medium, and small) using one of the three information displays (AR 3D conformal, AR 2D tag-along, and paper blueprints). The task performance was measured by time of completion and framing errors, which were analyzed and compared among each factor.     

Knowledge intensive Petri net framework for concurrent intelligent design of automatic assembly systems

The integration of design and planning of flexible assembly system has been recognized as a tool for achieving efficient assembly in a production environment that demands assembly with a high degree of flexibility. This paper proposes a concurrent intelligent approach and framework for the design of robotic flexible assembly systems. The principle of the proposed approach is based on the knowledge Petri net formalisms, incorporating Petri nets with more general problem-solving strategies in AI using knowledge-based system techniques. The complex assembly systems are modeled and analyzed by adopting a formal representation of the system dynamic behaviors through knowledge Petri net modeling from the specifications and the analysis of those models. A template is first defined for a knowledge Petri net model, and then the models for assembly system individuals are established in the form of instances of the template. The design of assembly systems is implemented through a knowledge Petri net-based function–behavior–structure model. The research results show that the proposed knowledge Petri net approach is applicable for design, simulation, analysis and evaluation, and even layout optimization of the flexible assembly system in an integrated intelligent environment. The integration of assembly design and planning process can help reduce the development time of assembly systems.     

Augmented Reality (AR) for Assembly Processes Design and Experimental Evaluation

Augmented reality (AR) for assembly processes is a new kind of computer support for a traditional industrial domain. This new application of AR technology is called ARsembly. The intention of this article is to describe a typical scenario for assembly and service personnel and how they might be supported by AR. For this purpose, tasks with different degrees of difficulty were selected from an authentic assembly process. In addition, 2 other kinds of assembly support media (a paper manual and a tutorial by an expert) were examined in order to compare them with ARsembly. The results showed that the assembly times varied according to the different support conditions. AR support proved to be more suitable for difficult tasks than the paper manual, whereas for easier tasks the use of a paper manual did not differ significantly from AR support. Tasks done under the guidance of an expert were completed most rapidly. Some of the information obtained in this investigation also indicated important considerations for improving future ARsembly applications.     

Augmented Reality-Based Manual Assembly Support With Visual Features for Different Degrees of Difficulty

This research investigates different visual features for augmented reality (AR)–based assembly instructions. Since the beginning of AR research, one of its most popular application areas has been manual assembly assistance. A typical AR assembly application indicates the necessary manual assembly operations by generating visual representations of parts that are spatially registered with, and superimposed on, a video representation of the physical product to be assembled. Research in this area indicates the advantages of this type of assembly instruction presentation. This research investigates different types of visual features for different assembly operations. The hypothesis is that in order to gain an advantage from AR, the visual features used to explain a particular assembly operation must correspond to its relative difficulty level. The final goal is to associate different types of visual features to different levels of task complexity. A user study has been conducted in order to compare different visual features at different operation complexity levels. The results support the hypothesis.     

An indoor localization dataset and data collection framework with high precision position annotation

We introduce a novel technique and an associated high resolution dataset that aim to precisely evaluate wireless signal based indoor positioning algorithms. The technique implements an augmented reality (AR) based positioning system that is used to annotate the wireless signal parameter data samples with high precision position data. We track the position of a practical and low cost navigable setup of cameras and a Bluetooth Low Energy (BLE) beacon in an area decorated with AR markers. We maximize the performance of the AR-based localization by using a redundant number of markers. Video streams captured by the cameras are subjected to a series of marker recognition, subset selection and filtering operations to yield highly precise pose estimations. Our results show that we can reduce the positional error of the AR localization system to a rate under 0.05 meters. The position data are then used to annotate the BLE data that are captured simultaneously by the sensors stationed in the environment, hence, constructing a wireless signal dataset with the ground truth, which allows a wireless signal based localization system to be evaluated accurately.     

Human-object integrated assembly intention recognition for context-aware human-robot collaborative assembly

Human-robot collaborative (HRC) assembly combines the advantages of robot's operation consistency with human's cognitive ability and adaptivity, which provides an efficient and flexible way for complex assembly tasks. In the process of HRC assembly, the robot needs to understand the operator's intention accurately to assist the collaborative assembly tasks. At present, operator intention recognition considering context information such as assembly objects in a complex environment remains challenging. In this paper, we propose a human-object integrated approach for context-aware assembly intention recognition in the HRC, which integrates the recognition of assembly actions and assembly parts to improve the accuracy of the operator's intention recognition. Specifically, considering the real-time requirements of HRC assembly, spatial-temporal graph convolutional networks (ST-GCN) model based on skeleton features is utilized to recognize the assembly action to reduce unnecessary redundant information. Considering the disorder and occlusion of assembly parts, an improved YOLOX model is proposed to improve the focusing capability of network structure on the assembly parts that are difficult to recognize. Afterwards, taking decelerator assembly tasks as an example, a rule-based reasoning method that contains the recognition information of assembly actions and assembly parts is designed to recognize the current assembly intention. Finally, the feasibility and effectiveness of the proposed approach for recognizing human intentions are verified. The integration of assembly action recognition and assembly part recognition can facilitate the accurate operator's intention recognition in the complex and flexible HRC assembly environment.     

基于Transformer重建的时序数据异常检测与关系提取

现有时序异常检测方法存在计算效率低和可解释性差的问题。考虑到Transformer模型在自然语言处理任务中表现出并行效率高且能够跨距离提取关系的优势，提出基于Transformer的掩膜时序建模方法。建立时序数据的并行无方向模型，并使用掩膜机制重建当前时间步，从而实现整段序列的重建。在存储系统数据集和NASA航天器数据集上的实验结果表明，与基于长短期记忆网络模型的检测方法相比，该方法可节约80.7%的计算时间，Range-based指标的F1得分达到0.582，并且其通过可视化关系矩阵可准确反映人为指令与异常的关系。     

Selection of key visual cues in real and virtual environments for assembly tasks

Previous research identified that learning assembly tasks in Virtual Environments (VEs) is more difficult than in Real Environments (REs). This work's objective is to identify the key visual areas for both environments when performing an assembly task for ten consecutive cycles, when following visual instructions and when having visual distractors. Using an eye-tracker, we identified the key visual areas required for an assembly task in both environments. Results indicate that practice allowed participants to reduce their assembly time in both environments. They also indicate that two areas, Assembly Area and Blocks, concentrated a higher proportion of eye-fixations; 59.98% for REs and 81.48% for VEs, with a statistically significant observation difference between environments (t-test value = −14.23, with p-value <0.00001 and Cohen's d = 6.36). We conclude that participants considered the same key visual areas for both environments and that VE interaction has a significant role in observation behavior.     

Tolerance dataset: mating process of plug-in cable connectors for wire harness assembly tasks

In this research, we studied the mating tolerance of various plug-in cable connectors and provide a mating tolerance dataset of 70 different connectors. This dataset will be highly advantageous to industries for wire harness assembly tasks using robots. Understanding the mating tolerance is crucial for automating the mating process because it is closely related to the control specifications of a robotic manipulator. Our system uses a 2-finger Robotiq adaptive gripper attached to a 6 degree-of-freedom industrial robot (ABB Robotics) to test the mating process of wire harness assembly tasks. In addition, we use 70 types of wire harness connectors with different numbers of pins widths, lengths, and thicknesses, and various shapes, to test the mating tolerance. The results indicate that the connector mating tolerance of our dataset is more generous than the repeatability of conventional industrial manipulators, and further demonstrate the suitability of the position control methods to wire harness assembly tasks.

     

The role of user-centered AR instruction in improving novice spatial cognition in a high-precision procedural task

AR instruction is a kind of virtual information presented on a human–computer interface. It allows users to view the geometric state, spatial relationship, operation method, and other information involved in the physical task, to form the spatial cognition of the current interaction process. At present, AR instructions cannot support high-precision procedural tasks. The reason is that the existing research work is to use visual elements to express the spatial relationship of physical tasks, without considering transforming the long-term accumulated potential experience of advanced users into a series of effective visual features and interaction modes, to promote the new users to quickly conceive the task intention. In this paper, a user-centered AR instruction(UcAI) is defined and tested for the first time in a procedural task. The control experiment and behavior analysis of 30 participants designing two tasks with different operation precision show that UcAI is more beneficial to improve the user's spatial cognitive ability than conventional AR instruction. Especially in the high-precision operation task, UcAI plays an important role. Our research results have a certain guiding significance for advanced AR instruction design, which extends AR technology to physical tasks with high cognitive complexity.     

Recurrent type-1 fuzzy functions approach for time series forecasting

Forecasting the future values of a time series is a common research topic and is studied using probabilistic and non-probabilistic methods. For probabilistic methods, the autoregressive integrated moving average and exponential smoothing methods are commonly used, whereas for non-probabilistic methods, artificial neural networks and fuzzy inference systems (FIS) are commonly used. There are numerous FIS methods. While most of these methods are rule-based, there are a few methods that do not require rules, such as the type-1 fuzzy function (T1FF) approach. While it is possible to encounter a method such as an autoregressive (AR) model integrated with a T1FF, no method that includes T1FF and the moving average (MA) model in one algorithm has yet been proposed. The aim of this study is to improve forecasting by taking the disturbance terms into account. The input dataset is organized using the following variables. First, the lagged values of the time series are used for the AR model. Second, a fuzzy c-means clustering algorithm is used to cluster the inputs. Third, for the MA, the residuals of fuzzy functions are used. Hence, AR, MA, and the degree of memberships of the objects are included in the input dataset. Because the objective function is not derivative, particle swarm optimization is preferable for solving it. The results on several datasets show that the proposed method outperforms most of the methods in literature.