Augmented reality agents for user interface adaptation

Most augmented reality (AR) applications are primarily concerned with letting a user browse a 3D virtual world registered with the real world. More advanced AR interfaces let the user interact with the mixed environment, but the virtual part is typically rather finite and deterministic. In contrast, autonomous behavior is often desirable in ubiquitous computing (Ubicomp), which requires the computers embedded into the environment to adapt to context and situation without explicit user intervention. We present an AR framework that is enhanced by typical Ubicomp features by dynamically and proactively exploiting previously unknown applications and hardware devices, and adapting the appearance of the user interface to persistently stored and accumulated user preferences. Our framework explores proactive computing, multi‐user interface adaptation, and user interface migration. We employ mobile and autonomous agents embodied by real and virtual objects as an interface and interaction metaphor, where agent bodies are able to opportunistically migrate between multiple AR applications and computing platforms to best match the needs of the current application context. We present two pilot applications to illustrate design concepts. Copyright © 2007 John Wiley & Sons, Ltd.     

A component framework for reusing a proprietary computer-aided engineering environment

Nowadays, it is becoming more frequent for engineers to deal with problems and situations that require specific software and the commercially available applications may not result adequate. Because of this, the reuse of software components is becoming a normal practice for improving the productivity of the software programmers, and the quality of the products that they develop. The employment of reusable components presents a number of advantages, such as the reduction in time necessary to develop new software, or the simplification of many computational tasks.There are some proprietary software environments in the engineering domain that are practically de facto standards, since they offer a broad functionality, robustness and constant updating. Therefore, those environments could be ideal candidates to be reusable components when developing new software. An example of this is Matlab, which can be encapsulated, in order to use it as a true reusable component. This will provide the user with efficient tools for designing his/her own applications.Moreover, there is an increasing importance for any software to offer distributed services. To perform this task, it is fundamental to have at one’s disposal reusable components that support Internet-based distributed applications and services development.In the present work, a software component framework that effectively encapsulates Matlab is described. This software allows developers the reuse of Matlab, for both local and distributed applications. To address those issues the .NET technology was employed. The component framework developed can be integrated in the toolbars of software development environments supporting the .NET framework. This facilitates the construction of applications that can reuse the components, since the software developers can use and test them and change their properties in design-time.     

An artificial pancreas system in android phones: A dual app architecture

An Artificial Pancreas (AP) system centered around an Android Smartphone is proposed in this paper. This unique architecture involves two separate but interacting modular Android applications (Apps) that are designed for unique functionalities. Even though an artificial pancreas system is a safety-critical system that demands a complex architecture and careful coding process, owing to their modular nature, these apps could be designed and developed through rapid prototyping processes. The first app (App-A), which runs in the front-end, serves as the user interface and acts as a connection hub for the hardware devices, was developed on the Android studio platform. Necessary communication protocols to enable communication with the hardware devices are incorporated into this app. The second app (App-B), which runs in the back-end, is developed using Simulink’s Android support package. It contains a safety-critical model predictive control algorithm with appropriate constraints to compute the required insulin rate, augmented with a Kalman Filter for estimating the states. There is an additional safety logic as well to prevent insulin over-dosage, thereby augmenting to the cause of a safety-critical control algorithm. The contribution of this work is a modular software framework and prototyping methodology that can be used for rapid development, testing and deployment of Android based AP-systems.     

A user-centric network communication broker for multimedia collaborative computing

The development of collaborative multimedia applications today follows a vertical development approach, where each application is built on top of low-level network abstractions such as the socket interface. This stovepipe development process is a major inhibitor that drives up the cost of development and slows down the innovation pace of new generations of communication applications. In this paper, we propose a network communication broker (NCB) that provides a unified higher-level abstraction for the class of multimedia collaborative applications. We demonstrate how NCB encapsulates the complexity of network-level communication control and media delivery, and expedites the development of applications with various communication logics. We investigate the minimum necessary requirements for the NCB abstraction. We identify that the concept of user-level sessions involving multiple parties and multiple media, is critical to designing a reusable NCB to facilitate next-generation multimedia communications. Furthermore, the internal design of NCB decouples the user-level sessions from network-level sessions, so that the NCB framework can accommodate heterogeneous networks, and applications can be easily ported to new network environments. In addition, we demonstrate how the extensible and self-managing design of NCB supports dynamic adaptation in response to changes in network conditions and user requirements.     

Usability Principles for Augmented Reality Applications in a Smartphone Environment

Through the rapid spread of smartphones, users have access to many types of applications similar to those on desktop computer systems. Smartphone applications using augmented reality (AR) technology make use of users' location information. As AR applications will require new evaluation methods, improved usability and user convenience should be developed. The purpose of the current study is to develop usability principles for the development and evaluation of smartphone applications using AR technology. We develop usability principles for smartphone AR applications by analyzing existing research about heuristic evaluation methods, design principles for AR systems, guidelines for handheld mobile device interfaces, and usability principles for the tangible user interface. We conducted a heuristic evaluation for three popularly used smartphone AR applications to identify usability problems. We suggested new design guidelines to solve the identified problems. Then, we developed an improved AR application prototype of an Android-based smartphone, which later was conducted a usability testing to validate the effects of usability principles.     

Webizing mobile augmented reality content

This paper presents a content structure for building mobile augmented reality (AR) applications in HTML5 to achieve a clean separation of the mobile AR content and the application logic for scaling as on the Web. We propose that the content structure contains the physical world as well as virtual assets for mobile AR applications as document object model (DOM) elements and that their behaviour and user interactions are controlled through DOM events by representing objects and places with a uniform resource identifier. Our content structure enables mobile AR applications to be seamlessly developed as normal HTML documents under the current Web eco-system.     

The programmable network prototyping system

The programmable network prototyping system (PNPS) uses a collection of reusable hardware modules that implement generic communications functions such as transmission, reception, signal propagation, and pattern matching. These modules are interconnected and configured to emulate a variety of communication networks whose behavior can be monitored under different load conditions. The user specifies a network as a set of interacting components using available software tools. These tools are accessible within a prototyping environment that includes a control system for configuring the hardware modules and interconnecting them according to the component specifications. Previously designed components are stored in a library and can be used to specify new networks. Although PNPS is designed to provide a prototyping environment for communication networks, some of the basic ideas can be useful in other contexts     

Client-server computing on Shrimp

Technological advances in network and processor speeds do not lead to equally large improvements in the performance of client-server systems. For instance, hardware performance improvements do not translate into faster user applications. This is primarily because software overhead dominates communication. The Shrimp project at Princeton University seeks solutions to this problem. Shrimp (Scalable High-Performance Really Inexpensive Multiprocessor) supports protected user-level communication between processes by mapping memory pages between virtual address spaces. This virtual memory-mapped network interface has several advantages, including flexible user-level communication and very low overhead for initiating data transfers. Here, we examine two remote procedure call (RPC) protocols and one socket implementation for Shrimp that deliver almost undiminished hardware performance to user applications     

Scalable communication architectures for massively parallel hardware multi-processors

Modern complex embedded applications in multiple application fields impose stringent and continuously increasing functional and parametric demands. To adequately serve these applications, massively parallel multi-processor systems on a single chip (MPSoCs) are required. This paper is devoted to the design of scalable communication architectures of massively parallel hardware multi-processors for highly-demanding applications. We demonstrated that in the massively parallel hardware multi-processors the communication network influence on both the throughput and circuit area dominates the processors influence, while the traditionally used flat communication architectures do not scale well with the increase of parallelism. Therefore, we propose to design highly optimized application-specific partitioned hierarchical organizations of the communication architectures through exploiting the regularity and hierarchy of the actual information flows of a given application. We developed related communication architecture synthesis strategies and incorporated them into our quality-driven model-based multi-processor design methodology and related automated architecture exploration framework. Using this framework we performed a large series of architecture synthesis experiments. Some of the results of the experiments are presented in this paper. They demonstrate many features of the synthesized communication architectures and show that our method and related framework are able to efficiently synthesize well scalable communication architectures even for the high-end massively parallel multi-processors that have to satisfy extremely stringent computation demands.     

The NEXUS open system for integrating robotic software

In this paper a framework for constructing flexible, robust and efficient software applications for robots is described. The basic concepts needed to integrate complex, multidisciplinary robot software architectures are identified, and the methods to achieve them are taken from different areas of research (programming languages, network communication systems, real-time systems, etc.). The result is an open software system called NEXUS which includes the basic characteristics needed for the integration of very different software modules, minimizing the effort of integration and maximizing the reusability, efficiency and robustness of the resulting software applications. This software has proven to be a basis for more sophisticated tools that help in reducing the cost of modifications to and the complexity of multidisciplinary projects, allowing highly structured and reusable designs to be implemented. Although it has been currently implemented for mobile robots, it is a sufficiently generic framework suitable for use in other control systems.     

Implementing a generic component‐based framework for telecontrol applications

The rapid growth of telecontrol systems is one of the major trends in today's network‐oriented community. The implementation of generic frameworks, consisting of reusable components that can form the basis for the development of such systems, is a necessity. There is a plethora of associated applications that can be developed in a distributed environment, such as audio/video teleconferencing, groupware and collaborative computing environments, remote controlled services, etc. In this paper we design and implement a generic framework of components that can be used for the realization of telecontrol applications. This category of applications focuses primarily on the issues of managing distributed units on remote end‐systems. Such applications contain remote units and administrators that are connected and exchange data and control messages. We analyse the outlined architecture of our framework and the most important system operations. We also describe the communication protocol used in message exchanges between the constituent components. Finally, we illustrate the usefulness of our framework by presenting two applications that were created by extending the basic software infrastructure. Copyright © 2007 John Wiley & Sons, Ltd.     

Toward AI-enabled augmented reality to enhance the safety of highway work zones: Feasibility,requirements, and challenges

Highway work zones are considered among the most hazardous working environments. In 2018 alone, 124 workers lost their lives to fatal accidents. The lack of predictive safety systems that notify workers of upcoming dangers in advance is a major reason to blame in the highway maintenance and operation community. This article presents an integrative design framework for bringing recent advances in Augmented Reality (AR) and Artificial Intelligence (AI) to enhance the safety of highway workers through real-time multimodal notifications on-spot. To this end, this article conceptualizes and co-designs three major pillars: (1) AR user interface design for multimodal notification, (2) real-time AI at the edge for vehicle detection/classification from distance, and (3) real-time wireless communication in work zone setting to enable latency-aware operation between AI and AR components. Our early results demonstrate that we can achieve 24.83 FPS end-to-end execution latency on the Xavier AGX Jetson board with 48.7% mAP on BDD100K dataset, and a real-time communication covering 120 meters with an average latency of 5.1 ms at the farthest distance. Our mixed-method user research also reveals an acceptable level of excitement and engagement from the body of highway workers toward both the proposed technology and the designed user interface. Overall, this article provides a proof-of-concept toward AI-enabled AR safety systems in highway work zones.     

A rapid prototyping software infrastructure for user interfaces in ubiquitous augmented reality

Recent user interface concepts, such as multimedia, multimodal, wearable, ubiquitous, tangible, or augmented-reality-based (AR) interfaces, each cover different approaches that are all needed to support complex human–computer interaction. Increasingly, an overarching approach towards building what we call ubiquitous augmented reality (UAR) user interfaces that include all of the just mentioned concepts will be required. To this end, we present a user interface architecture that can form a sound basis for combining several of these concepts into complex systems. We explain in this paper the fundamentals of DWARFs user interface framework (DWARF standing for distributed wearable augmented reality framework) and an implementation of this architecture. Finally, we present several examples that show how the framework can form the basis of prototypical applications.     

MagicMeeting: A Collaborative Tangible Augmented Reality System

We describe an augmented reality (AR) system that allows multiple participants to interact with 2D and 3D data using tangible user interfaces. The system features face-to-face communication, collaborative viewing and manipulation of 3D models, and seamless access to 2D desktop applications within the shared 3D space. All virtual content, including 3D models and 2D desktop windows, is attached to tracked physical objects in order to leverage the efficiencies of natural two-handed manipulation. The presence of 2D desktop space within 3D facilitates data exchange between the two realms, enables control of 3D information by 2D applications, and generally increases productivity by providing access to familiar tools. We present a general concept for a collaborative tangible AR system, including a comprehensive set of interaction techniques, a distributed hardware setup, and a component-based software architecture that can be flexibly configured using XML. We show the validity of our concept with an implementation of an application scenario from the automotive industry.     

Preference-Based Human Factors Knowledge Repository for Designing User Interfaces

This article proposes and describes a representational framework and a supporting tool environment for embedding and propagating human factors expertise into high level user interface design and development platforms. The proposed framework allows user interface designers to elicit, accommodate, and articulate user interface guidelines and results of experimental studies into reusable, evolutionary, and "living" design cases. The building blocks of the representational framework are a set of primitive constructs for consolidating the semantics of human factors knowledge into a design representation that characterizes the physical level of interaction. This is achieved through the development of a logical framework based on preference constraints and an initial set of preference and indifference expressions. The preference constraints provide a reasoning engine and a proof strategy for compiling a preference ordering of competing design alternatives and subsequently aggregating them into indifferent classes of design options per interaction element. The article also reports the implications of the proposed technique for user interface designers and the underlying requirements of user interface development platforms. Finally, the representational sufficiency of the proposed approach is discussed in the context of recent case studies aiming to consolidate human factors knowledge into a reusable repository supporting the ergonomic design of user interfaces in two example application domains, namely nonvisual hypermedia accessible to blind people and interpersonal communication aids for speech-motor-impaired and language-cognitive-impaired users.     

A system integration framework for coupled multiphysics simulations

Multiphysics simulations are playing an increasingly important role in computational science and engineering for applications ranging from aircraft design to medical treatments. These simulations require integration of techniques and tools from multiple disciplines, and in turn demand new advanced technologies to integrate independently developed physics solvers effectively. In this paper, we describe some numerical, geometrical, and system software components required by such integration, with a concrete case study of detailed, three-dimensional, parallel rocket simulations involving system-level interactions among fluid, solid, and combustion, as well as subsystem-level interactions. We package these components into a software framework that provides common-refinement based methods for transferring data between potentially non-matching meshes, novel and robust face-offsetting methods for tracking Lagrangian surface meshes, as well as integrated support for parallel mesh optimization, remeshing, algebraic manipulations, performance monitoring, and high-level data management and I/O. From these general, reusable framework components we construct domain-specific building blocks to facilitate integration of parallel, multiphysics simulations from high-level specifications that are easy to read and can also be visualized graphically. These reusable building blocks are integrated with independently developed physics codes to perform various multiphysics simulations.     

The effect of tracking technique on the quality of user experience for augmented reality mobile navigation

Augmented reality (AR) technology is changing the way we interact with the world by making the journey seamless and interactive. This is done by layering digital enhancements over an existing reality or real life scenario. However, employing AR technologies in wayfinding and navigation does not automatically bring positive experiences. We argue that tracking techniques are important in mobile AR navigation and fundamentally affect the quality of user experience. In this paper, we propose two different tourist AR applications based on different tracking techniques. In addition to the analysis of the applications, we conducted a user evaluation to study the effect of different AR tracking techniques on the performance of users and the quality of user experience.