Evaluation of motion-based interaction for mobile devices: A case study on image browsing

This article evaluates the usability of motion sensing-based interaction on a mobile platform using image browsing as a representative task. Three types of interfaces, a physical button interface, a motion-sensing interface using a high-precision commercial 3D motion tracker, and a motion-sensing interface using an in-house low-cost 3D motion tracker, are compared in terms of task performance and subjective preference. Participants were provided with prolonged training over 20 days, in order to compensate for the participants’ unfamiliarity with the motion-sensing interfaces. Experimental results showed that the participants’ task performance and subjective preference for the two motion-sensing interfaces were initially low, but they rapidly improved with training and soon approached the level of the button interface. Furthermore, a recall test, which was conducted 4 weeks later, demonstrated that the usability gains were well retained in spite of the long time gap between uses. Overall, these findings highlight the potential of motion-based interaction as an intuitive interface for mobile devices.     

Enhanced auditory feedback for Korean touch screen keyboards

With the increasing popularity of touch screen mobile devices, improving the usability and the user experience while inputting text on these devices is becoming increasingly important. Most conventional touch screen keyboards on mobile devices rely heavily on visual feedback, while auditory feedback seldom includes any useful information about what is being inputted by the user. The auditory feedback usually simply replicates the sounds produced by a physical keyboard. This paper describes the development of an enhanced auditory feedback mechanism for a Korean touch screen keyboard called the enhanced auditory feedback (EAF) mechanism. EAF has subtle phonetic auditory feedback generated using the acoustic phonetic features of human speech. While typing with EAF, users can acquire non-invasive auditory clues about the keys pressed. In this work, we compare conventional auditory feedback for a touch screen keyboard used in touch screen mobile devices with that of EAF and explore the possibility of using enhanced auditory feedback for touch screen keyboards.     

The performance of hand postures in front- and back-of-device interaction for mobile computing

Three studies of different mobile-device hand postures are presented. The first study measures the performance of postures in Fitts’ law tasks using one and two hands, thumbs and index fingers, horizontal and vertical movements, and front- and back-of-device interaction. Results indicate that the index finger performs well on both the front and the back of the device, and that thumb performance on the front of the device is generally worse. Fitts’ law models are created and serve as a basis for comparisons. The second study examines the orientation of shapes on the front and back of a mobile device. It shows that participants’ expectations of visual feedback for finger movements on the back of a device reverse the direction of their finger movements to favor a “transparent device” orientation. The third study examines letter-like gestures made on the front and back of a device. It confirms the performance of the index finger on the front of the device, while showing limitations in the ability for the index finger on the back to perform complex gestures. Taken together, these results provide an empirical foundation upon which new mobile interaction designs can be based. A set of design implications and recommendations are given based directly on the findings presented.     

基于XML的移动设备人机交互引擎

基于传统的移动设备领域人机交互界面的开发的现状以及特点设计了一种移动应用的开发模型。为了提高移动设备应用开发的效率和人机交互界面的质量,结合这种模型,提出了一种引擎设计方案。这种引擎(称为XmlSupervisor)采用面向服务的体系结构（SOA）的思想,通过将界面、逻辑代码和资源相分离的方案进行设计,可以支持利用可扩展标记语言(XML)的良好的扩展性、自描述性、跨平台性、树状存储结构等特点来开发的人机交互界面,为移动设备的人机交互界面的设计和开发提供了参考。     

Neck kinematics and muscle activity during mobile device operations

Neck pain is a significant health problem due to its high incident rates and economic costs. Increased use of touch screen mobile devices is becoming pervasive in the modern society and may further influence this already prevalent health problem. However, our current understanding of the cervical spine biomechanics during the operation of touch screen mobile devices is very limited. This study evaluated the neck extensor muscle activities and the kinematics of the cervical spine during the operation of a touch screen tablet and a smart phone. Three-variables, DEVICE, LOCATION and TASK were treated as the independent variables. NASA TLX revealed that “Gaming” was the least difficult task and “Typing” was the most difficult task. Participants of this study maintained significantly deeper neck flexion when operating a smart phone (44.7°), with the mobile devices set on a table (46.4°), and while performing a “Typing” task (45.6°). Lower levels of neck muscle activities were observed while performing a “Reading” task and holding mobile devices with hand. Lower levels of neck muscle activities were also observed when using a smart phone vs. a tablet, however such change was statistically insignificant.Relevance to industryThe current study demonstrated that users maintain deep neck flexion when using touchscreen mobile devices. In the recent years, there is an increasing popularity of mobile smart devices in various occupational environments. The findings of this study may be useful in implementing human-centered task designs to reduce neck injury risks among mobile device users.     

Exploiting the untapped potential of mobile distributed computing via approximation

Mobile computing is one of the largest untapped reservoirs in today’s pervasive computing world as it has the potential to enable a variety of in-situ, real-time applications. Yet, this computing paradigm suffers when the available resources–such as energy in the network, CPU cycles, memory, I/O data rate–are limited. In this article, the new paradigm of approximate computing is proposed to harness such potential and to enable real-time computation-intensive mobile applications in resource-limited and uncertain environments. A reduction in time and energy consumed by an application is obtained via approximate computing by decreasing the amount of computation needed; such improvement, however, comes with the potential loss in accuracy. Hence, a Mobile Distributed Computing framework, is introduced to determine offline the ‘approximable’ tasks in an application and a light-weight online algorithm is devised to select the approximate version of the tasks in an application during run time. The effectiveness of the proposed approach is validated through extensive simulation and testbed experiments by comparing approximate versus exact-computation performance.     

Enhanced user performance in an image gallery application with a mobile autostereoscopic touch display

In this study, we explored how stereoscopic depth affects performance and user experience in a mobile device with an autostereoscopic touch display. Participants conducted a visual search task with an image gallery application on three layouts with different depth ranges. The task completion times were recorded, and the participants were asked to rate their experiences. The results revealed that the image search times were facilitated by a mild depth effect and that too great a depth slowed search times and decreased user-experience ratings.     

Predicting time-sharing in mobile interaction

The era of modern personal and ubiquitous computers is beset with the problem of fragmentation of the user’s time between multiple tasks. Several adaptations have been envisioned that would support the performance of the user in the dynamically changing contexts in which interactions with mobile devices take place. This paper assesses the feasibility of sensor-based prediction of time-sharing, operationalized in terms of the number of glances, the duration of the longest glance, and the total and average durations of the glances to the interaction task. The data used for constructing and validating the predictive models was acquired from a field study (N = 28), in which subjects performing mobile browsing tasks were observed for approximately 1 h in a variety of environments and situations. The predictive accuracy achieved in binary classification tasks was about 70% (about 20% above default), and the most informative sensors were related to the environment and interactions with the mobile device. Implications to the feasibility of different kinds of adaptations are discussed.     

Musculoskeletal symptoms among mobile hand-held device users and their relationship to device use: A preliminary study in a Canadian university population

The study aims were, in a population of university students, staff, and faculty (n = 140), to: 1) determine the distribution of seven measures of mobile device use; 2) determine the distribution of musculoskeletal symptoms of the upper extremity, upper back and neck; and 3) assess the relationship between device use and symptoms. 137 of 140 participants (98%) reported using a mobile device. Most participants (84%) reported pain in at least one body part. Right hand pain was most common at the base of the thumb. Significant associations found included time spent internet browsing and pain in the base of the right thumb (odds ratio 2.21, 95% confidence interval 1.02–4.78), and total time spent using a mobile device and pain in the right shoulder (2.55, 1.25–5.21) and neck (2.72, 1.24–5.96). Although this research is preliminary, the observed associations, together with the rising use of these devices, raise concern for heavy users.     

Discovering determinants of users perception of mobile device functionality fit

In recent years, there has been an explosive growth in the use of mobile devices. The ubiquitous and multifunctional nature of these devices with internet connectivity and personalization features make them a unique context to investigate what factors shape mobile users perception of their mobile device functionality fit with their needs. In order to answer this question, we proposed a research model in which we introduced multifunctional use and perceived device-functionality fit as two new constructs. The results of our study show that a significant portion of individuals’ perceived device-functionality fit can be explained by their perceived enjoyment, perceived ease of use, perceived usefulness, and symbolic value of the device. In terms of the theoretical contribution, our research suggests revamping the concept of device-functionality fit when it comes to mobile devices by accounting for both hedonic and utilitarian aspects of mobile devices. In terms of practical implications, our study highlights the importance of the social image that mobile devices create in the society for their users as well as the importance of look-and-feel aspects of mobile devices in shaping users perception of fit between functionalities of their mobile devices and their needs.     

Human–battery interaction on mobile phones

Mobile phone users have to deal with limited battery lifetime through a reciprocal process we call human–battery interaction. We conducted three user studies in order to understand human–battery interaction and discover the problems in existing designs that prevent users from effectively dealing with the limited battery lifetime. The studies include a large-scale international survey, two long-term field trials including quantitative battery logging and qualitative inquiries, and structured interviews with twenty additional mobile phone users. We evaluated various aspects of human–battery interaction, including charging behavior, battery indicators, user interfaces for power-saving settings, user knowledge, and user reaction. We find that mobile phone users can be categorized into two types regarding human–battery interaction and often have inadequate knowledge regarding phone power characteristics. We provide qualitative and quantitative evidence that problems in state-of-the-art user interfaces have led to under-utilized power-saving settings, under-utilized battery energy, and dissatisfied users. Our findings provide insights into improving mobile phone design for users to effectively deal with the limited battery lifetime. Our work is the first to systematically address human–battery interaction on mobile phones and is complementary to the extensive research on energy-efficient design for a longer battery lifetime.     

Speech recognition for mobile devices

This article presents an overview of different approaches for providing automatic speech recognition (ASR) technology to mobile users. Three principal system architectures with respect to the employment of a wireless communication link are analyzed: Embedded Speech Recognition Systems, Network Speech Recognition (NSR) and Distributed Speech Recognition (DSR). An overview of the solutions having been standardized so far as well as a critical analysis of the latest developments in the field of speech recognition in mobile environments is given. Open issues, pros and cons of the different methodologies and techniques are highlighted. Special emphasis is placed on the constraints and limitations ASR applications are confronted with under different architectures.     

Accelerometer-based gesture control for a design environment

Accelerometer-based gesture control is studied as a supplementary or an alternative interaction modality. Gesture commands freely trainable by the user can be used for controlling external devices with handheld wireless sensor unit. Two user studies are presented. The first study concerns finding gestures for controlling a design environment (Smart Design Studio), TV, VCR, and lighting. The results indicate that different people usually prefer different gestures for the same task, and hence it should be possible to personalise them. The second user study concerns evaluating the usefulness of the gesture modality compared to other interaction modalities for controlling a design environment. The other modalities were speech, RFID-based physical tangible objects, laser-tracked pen, and PDA stylus. The results suggest that gestures are a natural modality for certain tasks, and can augment other modalities. Gesture commands were found to be natural, especially for commands with spatial association in design environment control.     

Using hand position for virtual object placement

Placement of an object in three-dimensional space is a six-degree-of-freedom (6D) operation requiring three variables to specify location and three more to specify orientation. This paper reports on an investigation using a six-degree-of-freedom digitizer to manipulate 3D scenes stored in a computer. Two experiments are presented, which evaluate speed and accuracy of placement under different conditions. Results show that placement times are faster with stereopsis, when z translations are disabled (when accurate translation is required) and when both rotations and translations are allowed simultaneously. The implications of these results are discussed.     

Multidevice mobile sessions: A first look

The increasing number of users with multiple mobile devices underscores the importance of understanding how users interact, often simultaneously, with these multiple devices. However, most device based monitoring studies have focused only on a single device type. In contrast, we study the multidevice usage of a US-based panel through device based monitoring on panelist’s smartphone and tablet devices. We present a broad range of results from characterizing individual multidevice sessions to estimating device usage substitution. For example, we find that for panelists, 50% of all device interaction time can be considered multidevice usage.     

移动终端声音增强菜单交互研究*

为了提高移动交互的可用性,进行了用非语音声音来增强移动终端菜单交互的研究,首先设计了一个移动终端的声音增强菜单,然后通过与相应的视觉菜单的对比实验来研究其是否能提高可用性。结果显示,声音增强移动菜单具有更好的可用性,声音增强移动菜单可以显著减少任务完成时间、降低工作负荷和脑力负荷。这说明有效利用声音增强移动菜单交互技术可以提高移动终端菜单交互的可用性。     

Stroke++: A new Chinese input method for touch screen mobile phones

Over the past few decades, users have been feeling clumsy inputting Chinese on mobile devices, partly because the layout of the keyboard/keypad is originally designed for inputting Latin alphabets. To improve this user experience, we propose Stroke++, a novel Chinese input method for touch screen mobile devices. More specifically, Stroke++ provides efficient keypad layout, a friendly user interface and a intelligent character/phrase candidate set generation algorithms. Stroke++ splits a Chinese character into multiple radicals. By leveraging hieroglyphic properties of Chinese characters, our method requires users to only input a subset of the radicals to identify the target character, making it much faster and easier to input Chinese on mobile phones. Our user study results show that Stroke++ outperforms most major Chinese input methods on mobile devices, including Stroke, Pinyin and Hand Writing Recognition (HWR), in terms of the input efficiency and usability. Moreover, we also demonstrate that Stroke++ offers a low entry barrier for Chinese-input novices.     

Natural language interface for information management on mobile devices

A natural language interface (NLI) enables the ease-of-use of information systems in performing sophisticated human - computer interaction. To address the challenges of mobile devices to user interaction in information management, we propose an NLI as a promising solution. In this paper, we review state-of-the-art NLI technologies and analyse user requirements for managing notable information on mobile devices. To minimize any technical difficulties arising from developing and improving the usability of NLI systems we develop general principles for NLI design, which fills in a gap in the literature. In order to satisfy user requirements for information management on mobile devices, we innovatively design NLI-enabled information management architecture. It is shown from two usage scenarios that the architecture could lead to reduced effort in user navigation and improved efficiency and effectiveness of managing information on mobile devices. We conclude the article with the implications of this study and suggestions for future direction.