Assessing the SALSA architecture for developing agent-based ambient computing applications

We have proposed the use of autonomous agents for coping with some of the challenges of creating ambient computing systems. The motivation of this research is that ambient computing environments are characterized by the distribution, reactivity, collaboration and adaptation of their artifacts, which are also characteristics attributed to software agents. To assist developers in creating the software entities of an ambient computing environment, the Simple Agent Library for Smart Ambients (SALSA) was created. The SALSA middleware and architecture enables the creation of autonomous agents reactive to the context of the ambient computing environment. SALSA agents can represent users, resources, or wrap complex system functionality of the environment. The aim of this paper is to provide evidence that SALSA facilitates the implementation of ambient computing services through autonomous agents. Unfortunately, the literature on Ubicomp development frameworks has, aside from a few exceptions, not reported experimental evaluation of their usability. The evaluations of Ubicomp development platforms have not addressed this issue since their evaluations have been mostly focused on performance and to prove feasibility. We present the results of an empirical evaluation conducted to assess the use of SALSA. This study included in-lab programming experiments and design exercises to evaluate the programming facilities provided by SALSA agents. Even though for some of the evaluation participants the use of autonomous agents as an abstraction for the development of ubiquitous computing systems was not innate, the evaluation results demonstrate that the execution model of SALSA and its facilities to implement Ubicomp systems are comprehensible.     

Augmented reality visualization: A review of civil infrastructure system applications

In Civil Infrastructure System (CIS) applications, the requirement of blending synthetic and physical objects distinguishes Augmented Reality (AR) from other visualization technologies in three aspects: (1) it reinforces the connections between people and objects, and promotes engineers’ appreciation about their working context; (2) it allows engineers to perform field tasks with the awareness of both the physical and synthetic environment; and (3) it offsets the significant cost of 3D Model Engineering by including the real world background. This paper reviews critical problems in AR and investigates technical approaches to address the fundamental challenges that prevent the technology from being usefully deployed in CIS applications, such as the alignment of virtual objects with the real environment continuously across time and space; blending of virtual entities with their real background faithfully to create a sustained illusion of co-existence; and the integration of these methods to a scalable and extensible computing AR framework that is openly accessible to the teaching and research community. The research findings have been evaluated in several challenging CIS applications where the potential of having a significant economic and social impact is high. Examples of validation test beds implemented include an AR visual excavator-utility collision avoidance system that enables workers to “see” buried utilities hidden under the ground surface, thus helping prevent accidental utility strikes; an AR post-disaster reconnaissance framework that enables building inspectors to rapidly evaluate and quantify structural damage sustained by buildings in seismic events such as earthquakes or blasts; and a tabletop collaborative AR visualization framework that allows multiple users to observe and interact with visual simulations of engineering processes.     

AIR-MODELLING: A tool for gesture-based solid modelling in context during early design stages in AR environments

Augmented reality (AR) technologies are just being used as interface in CAD tools allowing the user to perceive 3D models over a real environment. The influence of the use of AR in the conceptualization of products whose configuration, shape and dimensions depend mainly on the context remains unexplored. We aimed to prove that modelling in AR environments allows to use the context in real-time as an information input for making the iterative design process more efficient. In order to prove that, we developed a tool called AIR-MODELLING in which the designer is able to create virtual conceptual products by hand gestures meanwhile he/she is interacting directly with the real scenario. We conducted a test for comparing designers’ performance using AIR-MODELLING and a traditional CAD system. We obtained an average reduction of 44% on the modeling time in 76% of the cases. We found that modelling in AR environments using the hands as interface allows the designer to quickly and efficiently conceptualize potential solutions using the spatial restrictions of the context as an information input in real-time. Additionally, modelling in a natural scale, directly over the real scene, prevents the designer from drawing his/her attention on dimensional details and allows him/her to focus on the product itself and its relation with the environment.     

Information at a glance [augmented reality user interfaces]

What if we could visualize and interact with information directly in the context of our surroundings? Our research group is exploring how augmented reality (AR) could someday make this possible. AR integrates a complementary virtual world with the physical world-for example, by using head-tracked see-through head-worn displays to overlay graphics on what we see. Instead of looking back and forth between the real world and a PDA, we look directly at the real world and the virtual information overlaid on it. At the heart of this approach is context-aware computing, computing systems that are sensitive to the context in which they operate, ranging from human relationships to physical location. For example, information might be tied to specific locations within a global, Earth-centered, coordinate system. How can we design effective mobile AR user interfaces? We've been trying to answer this question in part by developing experimental AR research prototypes. In AR, as in work on information visualization using desktop technologies, the amount of information available can far exceed what a system can legibly display at a given time, necessitating information filtering. Julier et al. (2000) have developed information filtering techniques for AR that depend on the user's goals, object importance, and proximity. We assume that a system can accomplish information filtering of this sort and that our system is displaying everything it should.     

Augmented reality navigation systems

The augmented reality (AR) research community has been developing a manifold of ideas and concepts to improve the depiction of virtual objects in a real scene. In contrast, current AR applications require the use of unwieldy equipment which discourages their use. In order to essentially ease the perception of digital information and to naturally interact with the pervasive computing landscape, the required AR equipment has to be seamlessly integrated into the user’s natural environment. Considering this basic principle, this paper proposes the car as an AR apparatus and presents an innovative visualization paradigm for navigation systems that is anticipated to enhance user interaction.     

Interactive 3D content insertion in images for multimedia applications

This article addresses the problem of creating interactive mixed reality applications where virtual objects interact in images of real world scenarios. This is relevant to create games and architectural or space planning applications that interact with visual elements in the images such as walls, floors and empty spaces. These scenarios are intended to be captured by the users with regular cameras or using previously taken photographs. Introducing virtual objects in photographs presents several challenges, such as pose estimation and the creation of a visually correct interaction between virtual objects and the boundaries of the scene. The two main research questions addressed in this article include, the study of the feasibility of creating interactive augmented reality (AR) applications where virtual objects interact in a real world scenario using the image detected high-level features and, also, verifying if untrained users are capable and motivated enough to perform AR initialization steps. The proposed system detects the scene automatically from an image with additional features obtained using basic annotations from the user. This operation is significantly simple to accommodate the needs of non-expert users. The system analyzes one or more photos captured by the user and detects high-level features such as vanishing points, floor and scene orientation. Using these features it will be possible to create mixed and augmented reality applications where the user interactively introduces virtual objects that blend with the picture in real time and respond to the physical environment. To validate the solution several system tests are described and compared using available external image datasets.     

Proactive Fuzzy Control and Adaptation Methods for Smart Homes

Proactive, context-aware computing isn't new. In 2000, David Tennenhouse called for a change in the boundary between the physical and virtual worlds. He identified proactive computing as an alternative to interactive computing and defined how future systems should become more involved with the real world. He also considered context-aware control systems with online adaptation especially promising. Today, ambient-intelligence researchers show increasing interest in both proactive applications and context-aware applications. Using different context-recognition methods, researchers can easily gather application-specific information from the environment and enable context-triggered actions. According to Hee Eon Byun and Keith Cheverst, a context-aware home can serve its inhabitants more flexibly and adaptively than an ordinary home. They also claim that proactive systems can be built using machine-learning algorithms with context recognition. In addition to context recognition, adaptivity is essential in intelligent environments. In terms of computing systems, the environment can adjust itself using adaptation mechanisms to comply with user preferences; it can become unobtrusive and better support user activities. A home adapting to its inhabitants' living style is much more convenient than a user adapting to the home's behavior. In this article, we show how to develop a proactive, adaptive, fuzzy home-control system, present the algorithm we used for adaptation, and evaluate the test results we obtained.     

Let’s keep in touch online: a Facebook aware virtual human interface

A virtual human is an effective interface for interacting with users and plays an important role in carrying out certain tasks. As social networking sites are getting more and more popular, we propose a Facebook aware virtual human. The social networking sites are used to empower virtual humans for interpersonal conversational interaction in this paper. We combine Internet world, physical world and 3D virtual world together to create a new interface for users to interact with an autonomous virtual human which can behave like a real modern human. In order to take advantages of social networking sites, virtual human gathers information of a user from its profile, its likes, dislikes and gauge mood from most recent status update. In two user studies, we investigated whether and how this new interface can enhance human–virtual human interaction. Some positive results concluded from these studies will be guidelines on research and development of future virtual human interfaces.     

Comprehensible Visualization for Augmented Reality

This article presents interactive visualizations to support the comprehension of spatial relationships between virtual and real world objects for augmented reality (AR) applications. To enhance the clarity of such relationships we discuss visualization techniques and their suitability for AR. We apply them on different AR applications with different goals, e.g. in X-Ray vision or in applications which draw a user's attention to an object of interest. We demonstrate how Focus and Context (F+C) visualizations are used to affect the user's perception of hidden or nearby objects by presenting contextual information in the area of augmentation. We discuss the organization and the possible sources of data for visualizations in augmented reality and present cascaded and multi level F+C visualizations to address complex, cluttered scenes that are inevitable in real environments. This article also shows filters and tools to interactively control the amount of augmentation. It compares the impact of real world context preserving to a pure virtual and uniform enhancement of these structures for augmentations of real world imagery. Finally this paper discusses the stylization of sparse object representations for AR to improve x-ray vision.     

虚拟教学环境中基于草图的自适应用户界面研究

将草图技术、自适应技术与传统的虚拟现实技术相结合应用于虚拟教学过程中,通过增强用户和系统之间个性化需求的交互处理来提高系统的智能性和友好性,进而有效提高虚拟教学环境的应用效果.旨在虚拟环境中构建基于草图的自适应用户界面并应用于虚拟教学来满足特定教学需求,着重结合实例分析了基于自适应草图用户界面的虚拟教学环境中的草图上下文处理机制.在上述研究基础上,设计开发了一个虚拟教学原型系统,实验证明该系统在用户体验上有明显的改善.     

A Virtual Reality Interface for Space Planning Tasks

Manipulating and assembling elements in a 3D space is a task which interests a huge number of potential applications whether they deal with real or abstract objects. Direct manipulation techniques in traditional interactive systems use 2D devices and do not allow an easy manipulation of 3D objects. To facilitate user interaction, we have studied direct manipulation techniques in a virtual reality environment. A VR interface is naturally object-oriented and allows the definition of real-world metaphors. Operators can thus work in the virtual world in a similar way to the real world: they perceive the position of objects through the depth cue of stereo view, and can grab and push them in any direction by means of avirtual handuntil they reach their destination. They can put an object on top of another and line it up with other objects. We model the virtual world as ajob-orientedworld which is governed by a few simple rules which facilitate object positioning. In this paper, we describe the design and implementation strategies to obtain a real-time performance on a low-level workstation.     

Situated Computing: A Paradigm for the Mobile User-Interaction with Multimedia Sources

Today, users interact with computers in an explicit manner and the system's response is independent from their situations. Hence, it is difficult to integrate computers with working life as embedded tools, which can facilitate users to accomplish real world objectives easily. Situated computing is a new paradigm for mobile computer users based on their physical context and activities carried out as a part of their working business. It provides the mechanism to have a mobile computer as a utility to satisfy the user's real world requirements as well as an infrastructure for the situated interaction using applications. In this paper, we are presenting a metaphor called situation metaphor to model interaction between the user and mobile computers in order to achieve expectations of situated computing. A three-layered schema is followed in developing situation metaphor. We discuss extensively the theoretical foundation and framework for the situation metaphor, and followed by applications developed based on the framework.     

Real world teleconferencing

We've been exploring how augmented reality (AR) technology can create fundamentally new forms of remote collaboration for mobile devices. AR involves the overlay of virtual graphics and audio on reality. Typically, the user views the world through a handheld or head-mounted display (HMD) that's either see-through or overlays graphics on video of the surrounding environment. Unlike other computer interfaces that draw users away from the real world and onto the screen, AR interfaces enhance the real world experience. For example, with this technology doctors could see virtual ultrasound information superimposed on a patient's body.     

A usability study of multimodal input in an augmented reality environment

In this paper, we describe a user study evaluating the usability of an augmented reality (AR) multimodal interface (MMI). We have developed an AR MMI that combines free-hand gesture and speech input in a natural way using a multimodal fusion architecture. We describe the system architecture and present a study exploring the usability of the AR MMI compared with speech-only and 3D-hand-gesture-only interaction conditions. The interface was used in an AR application for selecting 3D virtual objects and changing their shape and color. For each interface condition, we measured task completion time, the number of user and system errors, and user satisfactions. We found that the MMI was more usable than the gesture-only interface conditions, and users felt that the MMI was more satisfying to use than the speech-only interface conditions; however, it was neither more effective nor more efficient than the speech-only interface. We discuss the implications of this research for designing AR MMI and outline directions for future work. The findings could also be used to help develop MMIs for a wider range of AR applications, for example, in AR navigation tasks, mobile AR interfaces, or AR game applications.     

Deep learning-based smart task assistance in wearable augmented reality

Wearable augmented reality (AR) smart glasses have been utilized in various applications such as training, maintenance, and collaboration. However, most previous research on wearable AR technology did not effectively supported situation-aware task assistance because of AR marker-based static visualization and registration. In this study, a smart and user-centric task assistance method is proposed, which combines deep learning-based object detection and instance segmentation with wearable AR technology to provide more effective visual guidance with less cognitive load. In particular, instance segmentation using the Mask R-CNN and markerless AR are combined to overlay the 3D spatial mapping of an actual object onto its surrounding real environment. In addition, 3D spatial information with instance segmentation is used to provide 3D task guidance and navigation, which helps the user to more easily identify and understand physical objects while moving around in the physical environment. Furthermore, 2.5D or 3D replicas support the 3D annotation and collaboration between different workers without predefined 3D models. Therefore, the user can perform more realistic manufacturing tasks in dynamic environments. To verify the usability and usefulness of the proposed method, we performed quantitative and qualitative analyses by conducting two user studies: 1) matching a virtual object to a real object in a real environment, and 2) performing a realistic task, that is, the maintenance and inspection of a 3D printer. We also implemented several viable applications supporting task assistance using the proposed deep learning-based task assistance in wearable AR.     

Improving quality-of-service in fog computing through efficient resource allocation

In today's world, large group migration of applications to the fog computing is registered in the information technology world. The main issue in fog computing is providing enhanced quality of service (QoS). QoS management consists of various method used for allocating fog-user applications in the virtual environment and selecting suitable method for allocating virtual resources to physical resource. The resources allocation in effective manner in the fog environment is also a major problem in fog computing; it occurs when the infrastructure is build using light-weight computing devices. In this article, the allocation of task and placement of virtual machine problems is explained in the single fog computing environment. The experiment is done and the result shows that the proposed framework improves QoS in fog environment.     

A systematic framework for on‐line calibration of a head‐mounted projection display for augmented‐reality systems

Abstract— Augmented reality (AR) is a technology in which computer‐generated virtual images are dynamically superimposed upon a real‐world scene to enhance a user's perceptions of the physical environment. A successful AR system requires that the overlaid digital information be aligned with the user's real‐world senses — a process known as registration. An accurate registration process requires the knowledge of both the intrinsic and extrinsic parameters of the viewing device and these parameters form the viewing and projection transformations for creating the simulations of virtual images. In our previous work, an easy off‐line calibration method in which an image‐based automatic matching method was used to establish the world‐to‐image correspondences was presented, and it is able to achieve subpixel accuracy. However, this off‐line method yields accurate registration only when a user's eye placements relative to the display device coincides with locations established during the offline calibration process. A likely deviation of eye placements, for instance, due to helmet slippage or user‐dependent factors such as interpupillary distance, will lead to misregistration. In this paper, a systematic on‐line calibration framework to refine the off‐line calibration results and to account for user‐dependent factors is presented. Specifically, based on an equivalent viewing projection model, a six‐parameter on‐line calibration method to refine the user‐dependent parameters in the viewing transformations is presented. Calibration procedures and results as well as evaluation experiments are described in detail. The evaluation experiments demonstrate the improvement of the registration accuracy.     

Real-time optical markerless tracking for augmented reality applications

Augmented reality (AR) technology consists in adding computer-generated information (2D/3D) to a real video sequence in such a manner that the real and virtual objects appear coexisting in the same world. To get a realistic illusion, the real and virtual objects must be properly aligned with respect to each other, which requires a robust real-time tracking strategy—one of the bottlenecks of AR applications. In this paper, we describe the limitations and advantages of different optical tracking technologies, and we present our customized implementation of both recursive tracking and tracking by detection approaches. The second approach requires the implementation of a classifier and we propose the use of a Random Forest classifier. We evaluated both approaches in the context of an AR application for design review. Some conclusions regarding the performance of each approach are given.