Designing for engagement in mixed reality experiences that combine projection mapping and camera-based interaction

Abstract

Projection mapping is a group of techniques for projecting imagery onto physical three-dimensional objects in order to augment the object or space with digital content. Most projection mapping experiences and events are non-interactive, partly because of the many inherent design problems in designing for interaction is such situations. This article explores one approach to this tricky issue by analysing a group of four interactive mixed reality experiences that combine projection mapping with camera-based interaction techniques. The approaches used are described, and their deployment in a variety of situations is analysed using engagement and play theories. The works suggest that some projection mapping techniques can be used in the service of creating engaging interactive exhibits and installations, but that the effectiveness of the approach relies on it placing the experience and interactions clearly in our own physical world, rather than behind a screen or in an artificial image space.     

Ink: designing for performative literary interactions

In this article, we present the interactive literary installation Ink, an experimental installation displayed at the Roskilde library, and the Roskilde Festival, which is designed to prompt public reflection on the nature and role of digital literature. By manipulating three books embedded with sensors, and watching text visualized on a 55” screen, members of the public select from a range of predefined sentences, previously created by a Danish author, and stored in a database. Squeezing the books alters each poetic line by degrees. The final poems print onto a library receipt for their producers and upload to a blog for public viewing. We present (1) design challenges to an installation meant to persuade people to engage affectively with the ergodic nature of digital literature and (2) an in-depth analysis of the empirical findings from the installation studies at the Roskilde Festival. In particular, we highlight the broad, rich range of performative interactions facilitated by the interactive setup. This analysis examines the performative writing and reading achieved by the public through interaction with Ink and also with the resultant poems. From this exploration, we discuss general tendencies when designing affectively engaging literary interactions at the edge of art and design.     

Mining data,making art

Abstract

This survey essay presents a range of contemporary media art projects exploring ecological themes, deriving their content through the use of scientific information and data. Artists incorporate a variety of technological approaches including online environments, interactive installation and real-time data visualisations to procure environmental data, making visible the often-kept hidden aspects of natural or human-built systems, resource consumption and ecosystem disruptions. The projects engage the public while informing about such things as a building's energy consumption, air quality in their vicinity, mapping of water usage on a digital landscape, and the calculating CO₂ in air travel, to name a few.     

3-D projection installations—three design strategies for a new medium

Abstract

Three-dimensional projection installations are particular kinds of augmented spaces in which a digital 3-D model is projected onto a physical three-dimensional object, thereby fusing the digital content and the physical object. Based on interaction design research and media studies, this article contributes to the understanding of the distinctive characteristics of such a new medium, and identifies three strategies for designing 3-D projection installations: establishing space; interplay between the digital and the physical; and transformation of materiality. The principal empirical case, From Fingerplan to Loop City, is a 3-D projection installation presenting the history and future of city planning for the Copenhagen area in Denmark. The installation was presented as part of the 12th Architecture Biennale in Venice in 2010.     

Analysis of large format camera photographs of the Po Delta,Italy, for topographic and thematic mapping

Abstract

Photographs produced by two generations of photogrammetric cameras, the first generation metric camera and the second generation large format camera, are compared with regard to their interpretability for topographic and thematic mapping. The latter part of this paper deals with interpretation problems which depend mainly on the characteristics of the black and white film. It can be shown that digitization and digital image enhancement techniques improve the interpretability to a certain extent. The conclusions, when looking forward to a second flightof the metric camera, are that the installation of forward motion compensation in combination with high resolution films will be a step forward in the development of a European photogrammetric camera.     

Aesthetic geometry evolution in a generic interactive evolutionary design framework

An interactive evolutionary design framework is presented which provides a testbed for the development and exploration of a range of visual aesthetic design spaces. A commercial 3D digital content development software package is used to leverage the strength of existing 3D algorithm implementations and familiar interfaces and tools. The domain of non-representational 3D forms sculpted with a series of non-linear deformers is discussed as the primary example. Additional simple problem domains are shown which are being developed to serve as examples for non-programmer artists and designers who are familiar with digital content authoring in Maya. These examples will enable such individuals to rapidly construct their own interactive evolution systems. Matthew Lewis, Ph.D.: He is a researcher at The Advanced Computing Center for the Arts and Design (ACCAD) at The Ohio State University, where he received his M.S. (1993) and Ph.D. (2001) degrees in Computer Science. His research interests include evolutionary design, generative art, and data remapping for artistic performance and installation. Keith Ruston: He is a student at The Ohio State University where he has earned his B.S. in Computer Science & Engineering (2003) and is currently pursuing his M.S. His major field of study is graphics and animation. His career goals include working in the animated film industry and/or computer game development.     

User centred design for a digital welding machine

The design of the user interface of a digital welding machine based on a user centred design process is described in this paper. Due to the iterative design process and the involvement of end users, interaction techniques have been designed which are optimised for the mental work model of the target user groups.     

Play in the smart city context: exploring interactional,bodily, social and spatial aspects of situated media interfaces

ABSTRACT

Urban media interfaces can assume many forms, offering rich possibilities for interactions and social encounters. We focus on a particular type of urban interfaces, namely urban media installations, which are situated, fixed (i.e. non-mobile) and outdoor interactive interfaces enhanced by digital technologies. Drawing on research on play in the context of the smart city, we aim to clarify how the digital and the physical worlds of urban media installations can coalesce into enjoyable and socially thriving playscapes. Two case studies of urban media installations are presented: the Appearing Rooms, a seasonal art installation in London, and the Mirror Pool, a permanent, large-scale urban installation in Bradford, UK. Each of these interfaces incorporates a design paradigm that differs from the other in terms of context, duration of implementation, and scale. Following a longitudinal approach based on non-participant observations and time-lapse photography, we analyse emergent interactions and focus in particular on playful encounters at different levels and scales: from the micro scale of the bodily engagement to the macro scale of the spatial and social configurations. Our case studies highlight that the urban spatial layout is a key element in defining the emerging interactions and encounters around the urban situated interfaces.     

Animistic design: how to reimagine digital interaction between the human and the nonhuman

ABSTRACT

This article puts forward the notion of animistic design as an uncertainty-driven strategy to reimagine human–machine interaction as a milieu of human and nonhuman. Animistic design is suggested as capable of fostering affects, sensibilities and thoughts that capitalize on the uncertain, the unpredictable and the nonlinear, and their capacity to trigger creative pathways. Informed by post-human philosophies, theories of mediation and materiality, as well as by affect, agency and aesthesia, animistic design eschews the anthropomorphic and the cute playfulness often associated with animism. Instead, it proposes a practical–theoretical framework to articulate the nexus of digital innovation, interaction design practices, technical materialities and affective responses already emerging in the digital cohabitation of the human and the nonhuman. Using a ‘research through making’ approach, the article describes in detail a series of animistic design experiments and prototyping methods that explore ways of rethinking interaction as an open-ended and creative enterprise. Animistic design offers an investigative strategy that exploits degrees of collaboratively curated uncertainty and unpredictability to imagine forms of digital interaction, and to engender creative human–nonhuman relationships within a given digital milieu.     

Participatory heritage innovation: designing dialogic sites of engagement

Abstract

Innovations in digital cultural communication for museums challenge us to develop appropriate methods for participation in curatorial processes and to rethink the role of audiences inside exhibitions. The article explores the potentials of scaffolding sites of dialogue and creative engagement through the design process and final exhibition. It draws upon experiences from an interactive exhibition project, Digital Natives, in which we combined principles from Participatory Design with issues of contemporary digital culture to explore possibilities for creating heritage innovation. We suggest three critical stages of the dialogic design process in which engagement between stakeholders, researchers, and audiences can be central to shaping and transforming future conceptions of digital cultural heritage, through process and final exhibition. In this way, we argue that a participatory design anthropological approach to digital culture can expand opportunities for heritage innovation through technological means of engagement in museums.     

Flying Cities: building a 3D world from vocal input

Abstract

Flying Cities is an artistic installation which generates imaginary cities from the speech of its visitors. Thanks to an original interactive process analysing people's vocal input to create 3D graphics, a tangible correspondence between speech and visuals opens new possibilities of interaction. This cross-modal interaction not only supports our artistic messages, but also aims at providing anyone with a pleasant and stimulating feedback from her/his speech activity. As the feedback we have received when presenting Flying Cities was very positive, our objective is now to cross the bridge between art and a potential application. We believe that it could become a new medium for creativity and a way to visually perceive a vocal performance in the context of the rehabilitation of people with reduced mobility or language impairments.     

SAGE: Semantic Annotation of Georeferenced Environments

The emergence of portable 3D mapping systems are revolutionizing the way we generate digital 3D models of environments. These systems are human-centric and require the user to hold or carry the device while continuously walking and mapping an environment. In this paper, we adapt this unique coexistence of man and machines to propose SAGE (Semantic Annotation of Georeferenced Environments). SAGE consists of a portable 3D mobile mapping system and a smartphone that enables the user to assign semantic content to georeferenced 3D point clouds while scanning a scene. The proposed system contains several components including touchless speech acquisition, background noise adaptation, real time audio and vibrotactile feedback, automatic speech recognition, distributed clock synchronization, 3D annotation localization, user interaction, and interactive visualization. The most crucial advantage of SAGE technology is that it can be used to infer dynamic activities within an environment. Such activities are difficult to be identified with existing post-processing semantic annotation techniques. The capability of SAGE leads to many promising applications such as intelligent scene classification, place recognition and navigational aid tasks. We conduct several experiments to demonstrate the effectiveness of the proposed system.     

HCI and business practices in a collaborative method for augmented reality systems

ContextEvery interactive system is composed of a functional core and a user interface. However, the software engineering (SE) and human–computer interaction (HCI) communities do not share the same methods, models or tools. This usually induces a large work overhead when specialists from the two domains try to connect their applicative studies, especially when developing augmented reality systems that feature complex interaction cores.ObjectiveWe present in this paper the essential activities and concepts of a development method integrating the SE and HCI development practices, from the specifications down to the design, as well as their application on a case study.MethodThe efficiency of the method was tested in a qualitative study involving four pairs of SE and HCI experts in the design of an application for which an augmented reality interaction would provide better user performance than a classic interactive system. The effectivity of the method was evaluated in a qualitative study comparing the quality of three implementations of the same application fragment (based on the same analysis model), using software engineering metrics.ResultsThe first evaluation confirmed the ease of use of our method and the relevance of our tools for guiding the design process, but raised concerns on the handling of conflicting collaborative activities. The second evaluation gave indications that the structure of the analysis model facilitates the implementation of quality software (in terms of coupling, stability and complexity).ConclusionIt is concluded that our method enables design teams with different backgrounds in application development to collaborate for integrating augmented reality applications with information systems. Areas of improvement are also described.     

A Systematic Mapping Study of HCI Practice Research

ABSTRACT

Human–computer interaction (HCI) practice has emerged as a research domain in the HCI field and is growing. The need to transfer HCI practices to the industry began significantly with the works of Nielsen on usability engineering. To date, methods and techniques for designing, evaluating, and implementing interactive systems for human use have continued to emerge. It is, therefore, justified to conduct a systematic mapping study to determine the landscape of HCI practice research. A Systematic Mapping Study method was used to map 142 studies according to research type, topic, and contribution. These were then analyzed to determine an overview of HCI practice research. The objective was to analyze studies on HCI practice and present prominent issues that characterize the HCI practice research landscape. Second, to identify pressing challenges regarding HCI practices in software/systems development companies. The results show that HCI practice research has steadily increased since 2012. The majority of the studies explored focused on evaluation research that largely contributed to the evaluation methods or processes. Most of the studies were on design tools and techniques, design methods and contexts, design work and organizational culture, and collaboration and team communication. Interviews, case studies, and survey methods have been prominently used as research methods. HCI techniques are mostly used during the initial phase of development and during evaluation. HCI practice challenges in companies are mostly process-related and on performance of usability and user experience activities. The major challenge seems to be to find a way to collect and incorporate user feedback in a timely manner, especially in agile processes. There are areas identified in this study as needing more research.     

引入力触觉的数字文物多模交互方法

目的 针对当前文物资源由传统的实体文物向虚拟展示和数字文物进行扩展的趋势,如何提供一种多模态的信息呈现方式就显得尤为重要。通过将力触觉技术引入3维文物展示领域,提出一种基于多模感知的3维文物交互式呈现的算法框架。在对文物的基本特征进行视、听、触觉多通道分析的基础上,依据用户与文物模型的接触状态对多通道信息进行计算和整合。方法 在力触觉计算渲染方面,基于嵌入深度构建弹簧系统模拟轮廓形状的接触过程,引入动摩擦和静摩擦因数来反映表面摩擦力这一材质特征,通过法线贴图来实现文物表面纹理的触觉处理;针对交互的环境由2维平面拓展至立体空间,结合力触觉设备将操作时的行为和状态映射为虚拟环境中的操作代理,借助操作代理构建"旋转"和"选择-移动-释放"两种基本的操作范式来实现用户意图;最后,物理引擎的引入将物体的基本运动规律集成至虚拟场景,提升场景交互的真实感.结果 使用Phantom Omni手控器搭建面向馆藏文物的多模感知实验系统,抽取志愿者对实验系统进行测评。实验结果表明：运用本文方法,用户可从视觉、听觉、触觉多个通道对数字文物的整体和细节信息进行感知,且交互的整体过程简单、自然、有效。结论 本文提出的基于多模感知的数字文物交互式呈现方法,可有效实现对各类数字遗产特别是3维文物的多模重现,在保证较高实时性的同时拥有良好的可用性和情感体验效果。     

Review

Abstract

The article analyses the creativity in a particular category of interactive products—personal portfolio websites. Most of the designers embraced the advantages of a digital portfolio which include the ability to make an unlimited number of copies at a very low price, availability through email and presence on the global job market. In trying to impress possible employers, designers often got more concerned with following advices offered by web design agencies on the web today rather than focusing on the quality of user experience. The question is can the rules of good web portfolio design be defined and/or can they be perceived as a process of creating new choices out of a prevailing set of options? The goal of this article is to provide an overview of work in this area and analyses it to speculate on what makes web portfolio creative as a narrative interactive product.     

Digital narrative conventions in heritage trail mobile apps

ABSTRACT

Galleries, libraries, archives and museums (GLAMs) are increasingly using digital technologies for storytelling and creating mobile applications (apps) for cultural heritage content, but how apps are used in practice to communicate information to users has not been widely studied. A team of people from a heritage organisation, a university, and mobile app development group plan to create a bespoke heritage trail app for Ireland, but to date design conventions/recommendations for this genre are lacking. This article applies a systematic approach to digital narrative content analysis to better understand how apps are being used specifically for heritage trails with the aim of identifying what the common features are, which modalities and narrative techniques are used. The selected corpus included 55 apps downloaded from the Google Play Store. The results of this content analysis—based on the App Walkthrough Method (Light, B., Burgess, J., & Duguay, S. (2018). The walkthrough method: An approach to the study of apps. New Media & Society, 20(3), 881–900)—show that there is a gap between academic research themes/trends and how digital narrative is actually being communicated in the current market, and it aims to inform the future development of heritage trail apps by including a list of design and content features common to this genre.     

Interactive specification and verification of behavioral adaptation contracts

ContextAdaptation is a crucial issue when building new applications by reusing existing software services which were not initially designed to interoperate with each other. Adaptation contracts describe composition constraints and adaptation requirements among these services. The writing of this specification by a designer is a difficult and error-prone task, especially when interaction protocols are considered in service interfaces.ObjectiveIn this article, we propose a tool-based, interactive approach to support the contract design process.MethodOur approach includes: (i) a graphical notation to define port bindings, and an interface compatibility measure to compare protocols and suggest some port connections to the designer, (ii) compositional and hierarchical techniques to facilitate the specification of adaptation contracts by building them incrementally, (iii) validation and verification techniques to check that the contract will make the involved services work correctly and as expected by the designer.ResultsOur results show a reduction both in the amount of effort that the designer has to put into building the contract, as well as in the number of errors present in the final result (noticeably higher in the case of manual specification).ConclusionWe conclude that it is important to provide integrated tool support for the specification and verification of adaptation contracts, since their incorrect specification induces erroneous executions of the system. To the best of our knowledge, such tool support has not been provided by any other approach so far, and hence we consider the techniques described in this paper as an important contribution to the area of behavioral software adaptation.     

快速虚顶点光子映射的焦散绘制

目的 模拟绘制焦散效果是真实感图形绘制的重要组成部分。利用可编程GPU硬件,基于图像的光子映射方法绘制速度快,但由于使用了近似采样和计算,会产生失真。为了克服这一现象,针对理想的镜面反射、折射体,提出一种快速绘制焦散效果的方法（VOBPBT）。方法 该方法首次定义了光子路径映射图的概念,并提出通过基于虚顶点光子映射的光子束跟踪来准确构建光子路径映射图的方法;此外,方法也创新性地提出利用光子路径映射图来查找焦散三角形,创建焦散映射图的方法。结果 实验结果表明,本文绘制结果真实,可以处理多次递归反射、折射,能够绘制连贯的高频焦散效果,同时可以达到交互的绘制性能。结论 本文VOBPBT方法在可交互计算机仿真、计算机游戏、虚拟漫游等应用领域具有一定的实用价值。     

A Case-Based Framework for Interactive Capture and Reuse of Design Knowledge

Aerospace design is a complex task requiring access to large amounts of specialized information. Consequently, intelligent systems that support and amplify the abilities of human designers by capturing and presenting relevant information can profoundly affect the speed and reliability of design generation. This article describes research on supporting aerospace design by integrating a case-based design support framework with interactive tools for capturing expert design knowledge through concept mapping. In the integrated system, interactive concept mapping tools provide crucial functions for generating and examining design cases and navigating their hierarchical structure, while CBR techniques facilitate retrieval and aid interactive adaptation of designs. Our goal is both to provide a useful design aid and to develop general interactive techniques to facilitate case acquisition and adaptation. Experiments illuminate the performance of the system's context-sensitive retrieval during interactive case adaptation and the conditions under which it provides the most benefit.