Visual Impairments and Mobile Touchscreen Interaction: State-of-the-Art,Causes of Visual Impairment,and Design Guidelines

ABSTRACT

This article identifies, catalogues, and discusses factors that are responsible for causing visual impairment of either a pathological or situational nature for touch and gesture input on smart mobile devices. Because the vast majority of interactions with touchscreen devices are highly visual in nature, any factor that prevents a clear, direct view of the mobile device’s screen can have potential negative implications on the effectiveness and efficiency of the interaction. This work presents the first overview of such factors, which are grouped in a catalogue of users, devices, and environments. The elements of the catalogue (e.g., psychological factors that relate to the user, or the social acceptability of mobile device use in public that relates to the social environment) are discussed in the context of current eye pathology classification from medicine and the recent literature in human–computer interaction on mobile touch and gesture input for people with visual impairments, for which a state-of-the-art survey is conducted. The goal of this work is to help systematize research on visual impairments and mobile touchscreen interaction by providing a catalogue-based view of the main causes of visual impairments affecting touch and gesture input on smart mobile devices.     

Collaboration among mobile agents for efficient energy allocation in mobile grid

The use of mobile devices in grid environments may have two interaction aspects: devices are considered as users of grid resources or as grid resources providers. Due to the limitation constraints on energy and processing capacity of mobile devices, their integration into the Grid is difficult. In this paper, we investigate the cooperation among mobile devices to balance the energy consumption and computation workloads. Mobile devices can have different roles such as buyer devices and seller devices. In the mobile grid, the energies of mobile devices are uneven, energy-poor devices can exploit other devices with spare energy. Our model consists of two actors: A buyer device agent represents the benefits of mobile buyer device that intends to purchase energy from other devices. A seller device agent represents the profits of mobile seller device that is willing to sell spare energy to other devices. The objective of optimal energy allocation in mobile grid is to maximize the utility of the system without exceeding the energy capacity, expense budget and the deadline. A collaboration algorithm among mobile agents for efficient energy allocation is proposed. In the simulation, the performance evaluation of collaboration algorithm among mobile agents is conducted.     

Refining visualization reference model for context information

Context-awareness can be used to decrease the need for interaction with a mobile device. This is increasingly important since the functionality of mobile devices and personal digital assistants gets more and more complex while the input and output capabilities remain restricted. An important aspect of context-awareness is to present the current context to the user. We propose a model for visualizing contextual information on the mobile terminal screen. The model is a refinement of a well-known visualization reference model; it takes into account the specific characteristics of mobile use and context information. We present the design of the model in detail, and discuss its applicability for a variety of contexts and tasks by providing a full-fledged use case.     

Supporting casual interactions between board games on public tabletop displays and mobile devices

As more interactive surfaces enter public life, casual interactions from passersby are bound to increase. Most of these users can be expected to carry a mobile phone or PDA, which nowadays offers significant computing capabilities of its own. This offers new possibilities for interaction between these users’ private displays and large public ones. In this paper, we present a system that supports such casual interactions. We first explore a method to track mobile phones that are placed on a horizontal interactive surface by examining the shadows which are cast on the surface. This approach detects the presence of a mobile device, as opposed to any other opaque object, through the signal strength emitted by the built-in Bluetooth transceiver without requiring any modifications to the devices’ software or hardware. We then go on to investigate interaction between a Sudoku game running in parallel on the public display and on mobile devices carried by passing users. Mobile users can join a running game by placing their devices on a designated area. The only requirement is that the device is in discoverable Bluetooth mode. After a specific device has been recognized, a client software is sent to the device which then enables the user to interact with the running game. Finally, we explore the results of a study which we conducted to determine the effectiveness and intrusiveness of interactions between users on the tabletop and users with mobile devices.     

Steerable projection: exploring alignment in interactive mobile displays

Emerging smartphones and other handheld devices are now being fitted with a set of new embedded technologies such as pico-projection. They are usually designed with the pico-projector embedded in the top of the device. Despite the potential of personal mobile projection to support new forms of interactivity such as augmented reality techniques, these devices have not yet made significant impact on the ways in which mobile data is experienced. We suggest that this ‘traditional’ configuration of fixed pico-projectors within the device is unsuited to many projection tasks because it couples the orientation of the device to the management of the projection space, preventing users from easily and simultaneously using the mobile device and looking at the projection. We present a study which demonstrates this problem and the requirement for steerable projection behaviour and some initial users’ preferences for different projection coupling angles according to context. Our study highlights the importance of flexible interactive projections which can support interaction techniques on the device and on the projection space according to task. This inspires a number of interaction techniques that create different personal and shared interactive display alignments to suit a range of different mobile projection situations.     

TouchTone: Interactive Local Image Adjustment Using Point‐and‐Swipe

Recent proliferation of camera phones, photo sharing and social network services has significantly changed how we process our photos. Instead of going through the traditional download‐edit‐share cycle using desktop editors, an increasing number of photos are taken with camera phones and published through cellular networks. The immediacy of the sharing process means that on‐device image editing, if needed, should be quick and intuitive. However, due to the limited computational resources and vastly different user interaction model on small screens, most traditional local selection methods can not be directly adapted to mobile devices. To address this issue, we present TouchTone, a new method for edge‐aware image adjustment using simple finger gestures. Our method enables users to select regions within the image and adjust their corresponding photographic attributes simultaneously through a simple point‐and‐swipe interaction. To enable fast interaction, we develop a memory‐ and computation‐efficient algorithm which samples a collection of 1D paths from the image, computes the adjustment solution along these paths, and interpolates the solutions to entire image through bilateral filtering. Our system is intuitive to use, and can support several local editing tasks, such as brightness, contrast, and color balance adjustments, within a minute on a mobile device.     

Studying the effect of perceived hedonic mobile device quality on user experience evaluations of mobile applications

When people interact with digital artefacts they perceive their pragmatic and hedonic qualities. In the case of interacting with mobile devices and applications, users seek utility as they try to satisfy certain needs, but at the same time they have certain feelings and emotions when, for example, they feel attached to their personal phone and/or trust its brand. Due to this strong relation between users and mobile devices a significant problem occurs when researchers want to evaluate the user experience of a mobile application in laboratory settings: the selection of an appropriate mobile device. Towards this end, this paper aims to unveil the effect of perceived hedonic quality of a mobile device on the user experience evaluation results of an application. Our results show that the perceived hedonic quality of a mobile device significantly affected the perceived pragmatic quality of the application, but not the hedonic one.     

Interactive visualization of medical volume models in mobile devices

Interactive visualization of volume models in standard mobile devices is a challenging present problem with increasing interest from new application fields like telemedicine. The complexity of present volume models in medical applications is continuously increasing, therefore increasing the gap between the available models and the rendering capabilities in low-end mobile clients. New and efficient rendering algorithms and interaction paradigms are required for these small platforms. In this paper, we propose a transfer function-aware compression and interaction scheme, for client-server architectures with visualization on standard mobile devices. The scheme is block-based, supporting adaptive ray-casting in the client. Our two-level ray-casting allows focusing on small details on targeted regions while keeping bounded memory requirements in the GPU of the client. Our approach includes a transfer function-aware compression scheme based on a local wavelet transformation, together with a bricking scheme that supports interactive inspection and levels of detail in the mobile device client. We also use a quantization technique that takes into account a perceptive metrics of the visual error. Our results show that we can have full interaction with high compression rates and with transmitted model sizes that can be of the order of a single photographic image.     

Secure mobile device structure for trust IoT

In the IoT environment, all devices are connected to each other, and mobile device is considered as key device. But hacking into mobile devices is increasing rapidly with the increase in mobile device users. As the market share of Android OS increases, hacking of mobile devices has focused on Android devices. Although there are many security solutions for mobile devices, they are fragmentary for mobile threats; that is, they are solutions for only several threats rather than comprehensive solutions. There is hence a limit to protecting user’s and company’s data stored or used on mobile devices from various types of hacking. To address this, we propose a mobile device protection technology based on domain isolation. Virtualization technology has emerged to increase CPU utilization in server-class PCs and to run various OSs in one system. As these virtualization technologies become lightweight, they are beginning to be applied to embedded devices. In this paper, we applied this lightweight embedded virtualization technology to mobile devices to divide mobile devices into two areas. Therefore, users can have hidden area from hacker attack in addition to Android OS area which can use same as existing mobile device. There is a hardware-based mobile security solution using an secure element, but this has to be reflected in the manufacturing process of the mobile device. However, since the domain separation technology using the virtualization, proposed in this paper, is a software solution, it has an advantage that it can be applied to a device that is already in use. In addition, to protect the hidden area, application authentication/authorization and user authentication technology were applied. And we use white-box cryptography to get root of trust of the key which is used for secure storage and data encryption/decryption. We believe this is a fundamental solution for protecting the mobile device users from hacking. We implemented and tested various mobile applications operating on a mobile device that incorporates our proposed structure based on domain isolation. There is some performance degradation caused by the domain separation, but it is negligible. According to https://www.wired.com/insights/2012/11/mobile-supercomputers/, the chips for mobile phones have evolved and mobile phones will soon become supercomputers. In this case, the addition of virtualization to the mobile device will have less impact on the computing power of the mobile device, and data protection stored in mobile devices and secure execution environment of security programs will become more important issues. Therefore, our TeeMo structure is a necessary technology to protect mobile device users.     

Composable continuous-space programs for robotic swarms

Programmability is an increasingly important barrier to the deployment of multi-robot systems, as no prior approach allows routine composition and reuse of general aggregate behaviors. The Proto spatial computing language, however, already provides this sort of aggregate behavior programming for non-mobile systems using an abstraction of the network as a continuous-space-filling device. We extend this abstraction to mobile systems and show that Proto can be applied to multi-robot systems with an actuator that turns a vector field into device motion. Proto programs operate on fields of values over an abstract device called the amorphous medium and can be joined together using functional composition. These programs are then automatically transformed for execution by individual devices, producing an approximation of the specified continuous-space behavior. We are thus able to build up a library of simple swarm behaviors, and to compose them together into highly succinct programs that predictably produce the desired complex swarm behaviors, as demonstrated in simulation and on a group of 40 iRobot SwarmBots.     

C2OF2N: a low power cooperative code offloading method for femtolet-based fog network

Power and delay aware cloud service provisioning to mobile devices has become a promising domain today. This paper proposes and implements a cooperative offloading approach for indoor mobile cloud network. In the proposed work mobile devices register under femtolet which is a home base station with computation and data storage facilities. The resources of the mobile devices are collaborated in such a way that different mobile devices can execute different types of computations based on cooperative federation. The proposed offloading scheme is referred as cooperative code offloading in femtolet-based fog network. If none of the mobile device can execute the requested computation, then femtolet executes the computation. Use of femtolet provides the mobile devices voice call service as well as cloud service access. Femtolet is used as the fog device in our approach. The proposed model is simulated using Qualnet version 7. The simulation results demonstrate that the proposed scheme minimizes the energy by 15% and average delay up to 12% approximately than the existing scheme. Hence, the proposed model is referred as a low power offloading approach.     

基于增强现实的多层次协同可视化系统

传统交互设备的信息展示维度存在一定局限性,增强现实技术能够扩展交互空间和信息维度,但在信息的全局交互和决策中则略显不足.该文将传统移动交互设备iPad与增强现实设备HoloLens进行结合,在iPad上展示整体的特征信息的同时,在HoloLens中进行细节的探索.基于这一设计思想创设了一种全新的交互环境,实现多层次多人...     

保留细节特征的轮廓线远程绘制算法

目前,智能掌上电脑、个人数字助理和智能手机等移动终端因为其便携性和可移动性,越来越多地被应用于分布式虚拟环境中,但是因为移动终端自身的局限性,在移动终端上的3维模型实时绘制问题仍未得到很好的解决。结合多分辨率网格模型和基于轮廓线的远程绘制的优点,提出了一种保留细节特征的轮廓线远程绘制算法。该算法预处理阶段,在原始精细网格模型上提取细节特征线,并进行多分辨率网格模型的预处理;运行时阶段,根据视点、帧率等交互信息使用选取策略对细节特征线进行选取,然后构建合适的多分辨率网格模型,并在该模型上提取出外围轮廓线,最后发送这两种轮廓线到移动终端进行绘制。实验结果表明,该算法在基于轮廓线的远程绘制中较好地保留了模型的细节特征,并保证了在移动终端的实时绘制。     

MicroMAIS: executing and orchestrating Web services on constrained mobile devices

Mobile devices with their more and more powerful resources allow the development of mobile information systems in which services are not only provided by traditional systems but also autonomously executed and controlled in the mobile devices themselves. Services distributed on autonomous mobile devices allow both the development of cooperative applications without a back‐end infrastructure and the development of applications blending distributed and centralized services. In this paper, we propose MicroMAIS: an integrated platform for supporting the execution of Web service‐based applications natively on a mobile device. The MicroMAIS platform is composed of mAS and μ‐BPEL. The former allows the execution of a single Web service, whereas the latter permits the orchestration of several Web services according to the WS‐BPEL standard. Copyright © 2011 John Wiley & Sons, Ltd.     

Multimedia layout adaptation through grammatical specifications

Online multimedia presentations, such as news, need to be constantly updated. Increasing demands are also being made for accessing online multimedia documents from mobile devices such as PDAs. There is an urgent need for a sound but practical formalism that supports automatic adaptation to the change of media content, display environments, and user intention. This paper presents a visual language approach to the layout adaptation of multimedia objects. The underlying theory of our approach is a context-sensitive graph grammar formalism enriched with facilities for spatial representation and specification. The paper focuses on the issues and techniques for size adaptation and style adaptation in response to the change of device requirements and user interactions.Published online: 2 February 2005 Correspondence to: Kang Zhang     

A point-based rendering approach for real-time interaction on mobile devices

Mobile device is an important interactive platform. Due to the limitation of computation, memory, display area and energy, how to realize the efficient and real-time interaction of 3D models based on mobile devices is an important research topic. Considering features of mobile devices, this paper adopts remote rendering mode and point models, and then, proposes a transmission and rendering approach that could interact in real time. First, improved simplification algorithm based on MLS and display resolution of mobile devices is proposed. Then, a hierarchy selection of point models and a QoS transmission control strategy are given based on interest area of operator, interest degree of object in the virtual environment and rendering error. They can save the energy consumption. Finally, the rendering and interaction of point models are completed on mobile devices. The experiments show that our method is efficient. Supported by the National Natural Science Foundation of China (Grant No. 60873159), the Program for New Century Excellent Talents in University (Grant No. NCET-07-0039), the National High-Tech Research & Development Progrom of China (Grant No. 2006AA01Z333)     

A color-based face tracking algorithm for enhancing interaction with mobile devices

A color-based face tracking algorithm is proposed to be used as a human-computer interaction tool on mobile devices. The solution provides a natural means of interaction enabling a motion parallax effect in applications. The algorithm considers the characteristics of mobile use-constrained computational resources and varying environmental conditions. The solution is based on color comparisons and works on images gathered from the front camera of a device. In addition to color comparisons, the coherency of the facial pixels is considered in the algorithm. Several applications are also demonstrated in this work, which use the face position to determine the viewpoint in a virtual scene, or for browsing large images. The accuracy of the system is tested under different environmental conditions such as lighting and background, and the performance of the system is measured in different types of mobile devices. According to these measurements the system allows for accurate (7% RMS error) face tracking in real time (20–100 fps).     

Mobile Web Browsing with Aural Flows: An Exploratory Study

Existing web applications force users focus their visual attention on mobile devices while browsing content and services while on the go. To support eyes-free, mobile experiences, designers can minimize interaction with a device by leveraging the auditory channel. Whereas acoustic interfaces have proven to be effective in reducing visual attention, a perplexing challenge is designing aural information architectures for the web. To address this problem, techniques to remodel existing information architectures as linear, aural flows were introduced and evaluated. Mobile web browsing with aural flows is exemplified in ANFORA News, a semiaural mobile site designed to browse large collections of news stories. An exploratory study involving frequent news readers (n?=?20) investigated the usability and navigation experience with ANFORA News in a mobile setting. Initial evidence suggests that aural flows are a promising paradigm to support eyes-free mobile navigation while on the go, but users still require assistance and additional learning to fully master the aural mechanics of the flows while on the go. Future work will improve on the mechanisms to customize content and control the aural navigation.     

Multidevice mobile sessions: A first look

The increasing number of users with multiple mobile devices underscores the importance of understanding how users interact, often simultaneously, with these multiple devices. However, most device based monitoring studies have focused only on a single device type. In contrast, we study the multidevice usage of a US-based panel through device based monitoring on panelist’s smartphone and tablet devices. We present a broad range of results from characterizing individual multidevice sessions to estimating device usage substitution. For example, we find that for panelists, 50% of all device interaction time can be considered multidevice usage.     

Mobile information access in the real world: A story of three wireless devices

The importance of the user perspective to the wireless information access experience cannot be understated: simply put, users will not indulge in devices that are perceived to be difficult to use and in technologies that do not offer quality infotainment – combined information and entertainment – content. In this paper, we investigate the impact that mobile devices have on the user wireless infotainment access experience in practice. To this end, we have undertaken an empirical study placed in a ‘real-world’ setting, in which participants undertook typical infotainment access tasks on three different wireless-enabled mobile devices: a laptop, a personal digital assistant and a head mounted display device. Results show that, with the exception of participants’ level of self-consciousness when using such devices in public environments, the user wireless information access experience is generally unaffected by device type. Location was shown, though, to be a significant factor when users engage in tasks such as listening to online music or navigation. Whilst the interaction between device and environment was found to influence entertainment-related tasks in our experiments, the informational ones were not affected. However, the interaction effects between device and user type was found to affect both types of tasks. Lastly, a user’s particular computing experience was shown to influence the perceived ease of wireless information access only in the case of online searching, irrespective of whether this is done for primarily informational purposes or entertainment ones.