Input Behavior When Using Two Fingers on a Multi-Touch Device

The purpose of this study is to analyze device input patterns using fingers in multi-touch mode and to derive Fitts’ law for multi-touch methods. Three specific input behaviors—dragging, rotating, and pinching—are investigated. Test participants performed two Fitts’ law experiments, a single-touch experiment using the thumb only and a multi-touch experiment using both the thumb and index finger. Within-subject factorial design was implemented with two levels of touch behavior (single-touch) and three levels of touch behavior (multi-touch), six levels of the index of difficulty (ID), four levels of direction, and two levels of repetition. The results indicate that a revised model of Fitts’ law is more appropriate for the multi-touch mode, achieving R² values above 0.895, and identifying the best nonlinear model among several regression models. The results of this study are used to determine the relationship between ID and movement time in terms of Fitts’ law and to investigate whether this relationship is sufficient to serve as a model to predict movement times. This study contributes to the prediction of movement times for products with multi-touch interfaces.     

Motion–Display Gain: A New Control–Display Mapping Reflecting Natural Human Pointing Gesture to Enhance Interaction with Large Displays at a Distance

The use of large displays is becoming increasingly prevalent, but development of the usability of three-dimensional (3D) interaction with large displays is still in the early stage. One way to improve the usability of 3D interaction is to develop appropriate control–display (CD) gain function. Nevertheless, unlike in desktop environments, the effects of the relationship between control space and display space in 3D interaction have not been investigated. Moreover, 3D interaction with large displays is natural and intuitive similar to how we work in the physical world. Therefore, a CD gain function that considers human behavior might improve the usability of interaction with large displays. The first experiment was conducted to identify the characteristics of user’s natural hand motion and the user perception of target in distal pointing. Thirty people participated and the characteristics of users’ natural hand movements and the 3D coordinates of their pointing positions were derived. These characteristics were considered in development of motion–display (MD) gain which is a new position-to-position CD mapping. Then, MD gain was experimentally verified by comparing it with Laser pointing, which is currently the best existing CD mapping technique; 30 people participated. MD gain was superior to the existing pointing technique in terms of both performance and subjective satisfaction. MD gain can also be personalized for further improvement. This is an initial attempt to reflect natural human pointing gesture in distal pointing technique, and the developed technique (MD gain) was experimentally proved to be superior to the existing techniques. This achievement is worthy because even a marginal improvement in the performance of pointing task, which is a fundamental and frequent task, can have a large effect on users’ productivity. These results can be used as a resource to understand the characteristics of user’s natural hand movement, and MD gain can be directly applied to situations in which distal pointing is needed, such as interacting with smart TVs or with wall displays. Furthermore, the concept that maps natural human behavior in motor space and an object in visual space can be applied to any interactive system.     

Effects of Laptop Touchpad Texturing on User Performance

This research assessed user performance with different laptop touchpad textures. In specific, the study measured discrete movement task time and accuracy. It was hypothesized that texturing would increase task times but improve accuracy by providing users with tactile references. A variable representing the frictional potential of pads was introduced into an established model of discrete movement performance (Fitts’ Law) in an attempt to accurately model user performance under experimental task conditions. Results revealed touchpad texturing to degrade task performance. However, accuracy in pointing tasks was not significantly affected. Results also revealed that the expanded form of Fitts’ Law, including a parameter for representing the frictional potential of pad texturing, was more predictive of actual movement times than the original form of the Law. Results from the study increase understanding of the effects of touchpad texture on human motor control behavior and provide some guidance for future pad design.     

Assisting Visually Impaired People to Acquire Targets on a Large Wall-Mounted Display

Large displays have become ubiquitous in our everyday lives, but these displays are designed for sighted people.This paper addresses the need for visually impaired people to access targets on large wall-mounted displays. We developed an assistive interface which exploits mid-air gesture input and haptic feedback, and examined its potential for pointing and steering tasks in human computer interaction（HCI）. In two experiments, blind and blindfolded users performed target acquisition tasks using mid-air gestures and two different kinds of feedback（i.e., haptic feedback and audio feedback）. Our results show that participants perform faster in Fitts＇ law pointing tasks using the haptic feedback interface rather than the audio feedback interface. Furthermore, a regression analysis between movement time（MT） and the index of difficulty（ID）demonstrates that the Fitts＇ law model and the steering law model are both effective for the evaluation of assistive interfaces for the blind. Our work and findings will serve as an initial step to assist visually impaired people to easily access required information on large public displays using haptic interfaces.     

Examining the costs of multiple trajectory pointing techniques

Pointing techniques that offer users multiple trajectories to a target have the potential to reduce pointing time by allowing a shorter than normal movement distance. However, such techniques potentially introduce additional elements into the pointing task involving identification of the alternative trajectories, assessment of their relative performance, and selection of the one to use. These additional tasks may reduce or negate the benefits of offering shorter paths. To better understand these issues, we developed a methodology for controlling the relative benefits of alternative target trajectories, and used it to evaluate three interfaces—a ‘pointer wrapping’ technique that allows the cursor to traverse from one screen edge to the opposing edge, and a system allowing users to choose between multiple cursors in two configurations (‘Ninja cursors’). We found that performance with these techniques was significantly worse than that predicted by Fitts' law for a single cursor, suggesting that the additional elements in their use are significant. Detailed analysis of behaviour during acquisitions showed that much of this cost is accrued in the mental preparation that precedes motor action, and in additional volatility in the pointing movements. We discuss how the method and findings may assist those developing enhanced pointing techniques.     

Behind Fitts’ law: kinematic patterns in goal-directed movements

Half a century ago, Paul Fitts first discovered that the time necessary to complete a pointing movement (MT) linearly increases with the amount of information (ID) necessary to specify the target width (W) relative to the distance (D). The so-called Fitts’ law states that , with ID being a logarithmic function of the D/W ratio. With the rising importance of pointing in human–computer interaction, Fitts’ law is nowadays an important tool for the quantitative evaluation of user interface design. We show that changes in ID give rise to systematic changes in the kinematics patterns that determine MT, and provide evidence that the observed patterns result from the interplay between basic oscillatory motion and visual control processes. We also emphasize the generality and abstract nature of Fitts’ robust model of human psychomotor behavior, and suggest that some adaptations in the design of the (computer-mediated) coupling of perception and production of movement might improve the efficiency of the interaction.     

Analysis of cursor movements with a mouse

With a mouse input device, 32 experienced computer users moved a cursor from a starting position to a target, both of which were displayed simultaneously on the computer’s screen. Cursor movements occurred under conditions of variations in the angle of approach to the target, the target size and shape, the distance to the target, and the nature of the task, drag-drop or point-click. In a fully crossed within-subjects design, all variables studied significantly affected movement time. Fitts’ law accounted for 44% or 97% of the variance in movement time, depending on the method of analysis. Fitts’ law was not equally effective under all combinations of the variables studied. An analysis of residuals showed that residuals were smaller for a point-click task in comparison to a drag-drop task, and residuals were lowest for the largest target displayed at the shortest distance from the starting position. The application of Fitts’ law to cursor movements with a mouse should be qualified by noting the conditions under which the movements were observed.     

Towards accurate cursorless pointing: the effects of ocular dominance and handedness

Pointing gestures are our natural way of referencing distant objects and thus widely used in HCI for controlling devices. Due to current pointing models’ inherent inaccuracies, most of the systems using pointing gestures so far rely on visual feedback showing users where they point at. However, in many environments, e.g., smart homes, it is rarely possible to display cursors since most devices do not contain a display. Therefore, we raise the question of how to facilitate accurate pointing-based interaction in a cursorless context. In this paper we present two user studies showing that previous cursorless techniques are rather inaccurate as they lack important considerations about users’ characteristics that would help in minimizing inaccuracy. We show that pointing accuracy could be significantly improved by acknowledging users’ handedness and ocular dominance. In a first user study (n=?33), we reveal the large effect of ocular dominance and handedness on human pointing behavior. Current ray-casting techniques neglect both ocular dominance and handedness as effects onto pointing behavior, precluding them from accurate cursorless selection. With a second user study (n=?25), we show that accounting for ocular dominance and handedness yields to significantly more accurate selections compared to two previously published ray-casting techniques. This speaks for the importance of considering users’ characteristics further to develop better selection techniques to foster more robust accurate selections.     

An Empirical Study on the Interaction Capability of Arm Stretching

Body-based motion gestures have been gaining popularity in designing interactive systems. However, theories and design guidelines on the use of body movement in design have not been fully evaluated. This article investigates human ability to perform discrete target selection tasks using stretching action through two controlled experiments. The experimental results indicate that: (1) the range of the discrete levels of depth that users can easily discriminate with arm stretching is up to 16 with full visual feedback, but is down to 4 without the feedback; (2) dwelling, a gesture with keeping a hand motionless and the cursor within a target area for a certain amount of time, may be the best gesture for confirmation command; (3) full visual feedback can improve the user performance; and (4) arm stretching action can be modeled using Fitts’ law. We also discuss the design potentials for Stretch Widgets based on these results.     

Hand-adaptive user interface: improved gestural interaction in virtual reality

Most interactive user interfaces (UIs) for virtual reality (VR) applications are based on the traditional eye-centred UI design principle, which primarily considers the user’s visual searching efficiency and comfort, but the hand operation performance and ergonomics are relatively less considered. As a result, the hand interaction in VR is often criticized as being less efficient and precise. In this paper, the user’s arm movement features, such as the choice of the hand being used and hand interaction position, are hypothesized to influence the interaction results derived from a VR study. To verify this, we conducted a free hand target selection experiment with 24 participants. The results showed that (a) the hand choice had a significant effect on the target selection results: for a left hand interaction, the targets located in spaces to the left were selected more efficiently and accurately than those in spaces to the right; however, in a right hand interaction, the result was reversed, and (b) the free hand interactions at lower positions were more efficient and accurate than those at higher positions. Based on the above findings, this paper proposes a hand-adaptive UI technique to improve free hand interaction performance in VR. A comprehensive comparison between the hand-adaptive UI and traditional eye-centred UI was also conducted. It was shown that the hand-adaptive UI resulted in a higher interaction efficiency and a lower physical exertion and perceived task difficulty than the traditional UI.

     

Pupil responses to continuous aiming movements

Pupillary response is associated with perceptual and cognitive loads in visual and cognitive tasks, but no quantitative link between pupil response and the task workload in visual−motor tasks has been confirmed. The objective of this study is to investigate how the changes of task requirement of a visual−motor task are reflected by the changes of pupil size. In the present study, a simple continuous aiming task is performed and the task requirement is manipulated and measured by Fitts’ Index of Difficulty (ID), calculated for different combinations of the target size and movement distance. Pupil response is recorded using a remote eye-tracker. The results show that event-triggered pupil dilations in continuous aiming movements respect Fitts’ Law, such that higher task difficulty evokes higher peak pupil dilation and longer peak duration. These findings suggest that pupil diameter can be employed as a physiological indicator to task workload evoked by the task requirement in visual−motor tasks.     

Effect of target shape on movement time in a Fitts task

Experiments on the effect of target shape in a Fitts movement task are reported. Two experiments investigated the movement time vs. index of difficulty relationship for targets that were square, circular, diamond and triangular in shape. The ratio of standard deviation of hits in the vertical and horizontal directions was constant for all target shapes and thus the movement time could be expressed in terms of the constraint in the direction of motion. Correlation of movement time in terms of an Index of Difficulty based on the standard deviation of hits in the direction of motion showed all data fitting on a single line. Thus the constraint effects of the various shapes occurred from the way in which the hits were distributed within each shape. Various models were tried to account for the effect of the target shape. It was found that the experimental data were best accounted for by a 'cookie-cutter' model in which the target shape was cut into a bivariate normal distribution of hits on the target to obtain a ‘shape factor’ that defined the difficulty of the task relative to the usual rectangular target used in Fitts task experiments.     

Navigation by pointing to GPS locations

This article deals with a method for interacting with a handheld navigation application, based on using the mobile device for pointing. When the user points the device in any direction, feedback will be provided based on if the user is aiming at the next point in the track or beside it. The presented study has been performed in order to get a better understanding of how the basic parameters in this type of interaction—like the angle for pointing and the size of the target—influence the navigation performance. We have applied a dual investigation by running a computer simulation varying additional parameters such as GPS accuracy and user behavior, and also running an in-context study with 15 participants in a realistic outdoor setting with real location-based GPS data. The study has resulted in general recommendations for angle intervals and the radius of the circles surrounding the track points.     

Prediction of pointing and dragging times in graphical user interfaces

An experiment is described which demonstrates that the point-drag sequence common on interactive systems can be modelled as two separate Fitts law tasks — a point-select task followed by a drag-select task. Strong prediction models were built; however, comparisons with previous models were not as close as the standard error coefficients implied. Caution is therefore warranted in follow-up applications of models built in research settings. Additionally, the previous claim that target height is the appropriate substitute for target width in calculating Fitts' index of difficulty in dragging tasks was not supported. The experiment described varied the dragging target's width and height independently. Models using the horizontal width of the drag target or the smaller of the target's width or height outperformed the target height model.     

A model of scrolling on touch-sensitive displays

Scrolling interaction is a common and frequent activity allowing users to browse content that is initially off-screen. With the increasing popularity of touch-sensitive devices, gesture-based scrolling interactions (e.g., finger panning and flicking) have become an important element in our daily interaction vocabulary. However, there are currently no comprehensive user performance models for scrolling tasks on touch displays. This paper presents an empirical study of user performance in scrolling tasks on touch displays. In addition to three geometrical movement parameters—scrolling distance, display window size, and target width, we also investigate two other factors that could affect the performance, i.e., scrolling modes—panning and flicking, and feedback techniques—with and without distance feedback. We derive a quantitative model based on four formal assumptions that abstract the real-world scrolling tasks, which are drawn from the analysis and observations of user scrolling actions. The results of a control experiment reveal that our model generalizes well for direct-touch scrolling tasks, accommodating different movement parameters, scrolling modes and feedback techniques. Also, the supporting blocks of the model, the four basic assumptions and three important mathematical components, are validated by the experimental data. In-depth comparisons with existing models of similar tasks indicate that our model performs the best under different measurement criteria. Our work provides a theoretical foundation for modeling sophisticated scrolling actions, as well as offers insights into designing scrolling techniques for next-generation touch input devices.     

“Beating” Fitts’ law: virtual enhancements for pointing facilitation

We survey recent research into new techniques for artificially facilitating pointing at targets in graphical user interfaces. While pointing in the physical world is governed by Fitts’ law and constrained by physical laws, pointing in the virtual world does not necessarily have to abide by the same constraints, opening the possibility for “beating” Fitts’ law with the aid of the computer by artificially reducing the target distance, increasing the target width, or both. The survey suggests that while the techniques developed to date are promising, particularly when applied to the selection of single isolated targets, many of them do not scale well to the common situation in graphical user interfaces where multiple targets are located in close proximity.     

Developing Multimodal Adaptation Algorithm for Mobility Impaired Users by Evaluating Their Hand Strength

Recent research on interactive electronic systems like computer, digital TV, smartphones can improve the quality of life of many disabled and elderly people by helping them to engage more fully to the world. Previously, a simulator was developed that reflects the effect of impairment on interaction with electronic devices and thus helps designers in developing accessible systems. In this article, the scope of the simulator has been extended to multiple pointing devices. The way that hand strength affects pointing performance of people with and without mobility impairment in graphical user interfaces was investigated for four different input modalities, and a set of linear equations to predict pointing time and average number of submovements for different devices was developed. These models were used to develop an adaptation algorithm to facilitate pointing in electronic interfaces by users with motor impairment using different pointing devices. The algorithm attracts a pointer when it is near a target and thus helps to reduce random movement during homing and clicking. The algorithm was optimized using the simulator and then tested on a real-life application with multiple distractors involving three different pointing devices. The algorithm significantly reduces pointing time for different input modalities.     

The Effect of Target Precuing on Pointing With Mouse and Touchpad

In point-and-click interfaces the location of targets is sometimes known to the user before visually identifying it, and sometimes not. This experiment investigates how pointing is affected by whether the target location is precued so that users know it in advance or nonprecued so that users learn it only at the onset of pointing trials. The study investigates this for young, adult, and elderly participants pointing with mouse and touchpad. Target precuing affects the trial completion time, the reaction time, the sheer movement time, and multiple movement kinematics. In addition, target precuing interacts with the use of either mouse or touchpad, with target distance, and with target size, but surprisingly little with participant age. Because the target location was always made known to participants no later than at the onset of the pointing trial, the effects of target precuing must be due to the different possibilities for mental and motor preparations.     

Developing Predictive Equations to Model the Visual Demand of In-Vehicle Touchscreen HMIs

Touchscreen human–machine interfaces (HMIs) are commonly employed as the primary control interface and touch-point of vehicles. However, there has been very little theoretical work to model the demand associated with such devices in the automotive domain. Instead, touchscreen HMIs intended for deployment within vehicles tend to undergo time-consuming and expensive empirical testing and user trials, typically requiring fully functioning prototypes, test rigs, and extensive experimental protocols. While such testing is invaluable and must remain within the normal design/development cycle, there are clear benefits, both fiscal and practical, to the theoretical modeling of human performance. We describe the development of a preliminary model of human performance that makes a priori predictions of the visual demand (total glance time, number of glances, and mean glance duration) elicited by in-vehicle touchscreen HMI designs, when used concurrently with driving. The model incorporates information theoretic components based on Hick–Hyman Law decision/search time and Fitts’ Law pointing time and considers anticipation afforded by structuring and repeated exposure to an interface. Encouraging validation results, obtained by applying the model to a real-world prototype touchscreen HMI, suggest that it may provide an effective design and evaluation tool, capable of making valuable predictions regarding the limits of visual demand/performance associated with in-vehicle HMIs, much earlier in the design cycle than traditional design evaluation techniques. Further validation work is required to explore the behavior associated with more complex tasks requiring multiple screen interactions, as well as other HMI design elements and interaction techniques. Results are discussed in the context of facilitating the design of in-vehicle touchscreen HMI to minimize visual demand.