Functionality-power-packaging considerations in context aware wearable systems

Wearable computing places tighter constraints on architecture design than traditional mobile computing. The architecture is described in terms of miniaturization, power-awareness, global low-power design and suitability for an application. In this article we present a new methodology based on three different system properties. Functionality, power and electronic Packaging metrics are proposed and evaluated to study different trade offs. We analyze the trade offs in different context recognition scenarios. The proof of concept case study is analyzed by studying (a) interaction with household appliances by a wrist worn device (acceleration, light sensors) (b) studying walking behavior with acceleration sensors, (c) computational task and (d) gesture recognition in a wood-workshop using the combination of accelerometer and microphone sensors. After analyzing the case study, we highlight the size aspect by electronic packaging for a given functionality and present the miniaturization trends for ‘autonomous sensor button’.     

MagicBoard: A contribution to an intelligent office environment

In this paper, we describe an augmented reality tool for collaborative work called the MagicBoard. The MagicBoard is based on an ordinary white board which has been enhanced by a video-projector and a steerable camera. A supervisor coordinates the cooperation of several modules including gesture recognition, finger tracking and white board scanning for digitalisation of the content. The gesture recognition module uses an approach based on local spatio-temporal appearance of activities. The tracking module is designed for use with cluttered backgrounds and variable lighting conditions. The white board scanner eliminates global luminosity differences by adaptive thresholding and the result can serve to digitise the content of the board. The supervisor is based on a rule-based architecture and is easily extendable. The selected modules fit together to a compact system, that largely increases the functionality of a white board and makes it a useful tool in the future office environments.     

Combining monoSLAM with object recognition for scene augmentation using a wearable camera

In wearable visual computing, maintaining a time-evolving representation of the 3D environment along with the pose of the camera provides the geometrical foundation on which person-centred processing can be built. In this paper, an established method for the recognition of feature clusters is used on live imagery to identify and locate planar objects around the wearer. Objects’ locations are incorporated as additional 3D measurements into a monocular simultaneous localization and mapping process, which routinely uses 2D image measurements to acquire and maintain a map of the surroundings, irrespective of whether objects are present or not. Augmenting the 3D maps with automatically recognized objects enables useful annotations of the surroundings to be presented to the wearer. After demonstrating the geometrical integrity of the method, experiments show its use in two augmented reality applications.     

Position-aware activity recognition with wearable devices

Reliable human activity recognition with wearable devices enables the development of human-centric pervasive applications. We aim to develop a robust wearable-based activity recognition system for real life situations where the device position is up to the user or where a user is unable to collect initial training data. Consequently, in this work we focus on the problem of recognizing the on-body position of the wearable device ensued by comprehensive experiments concerning subject-specific and cross-subjects activity recognition approaches that rely on acceleration data. We introduce a device localization method that predicts the on-body position with an F-measure of 89% and a cross-subjects activity recognition approach that considers common physical characteristics. In this context, we present a real world data set that has been collected from 15 participants for 8 common activities where they carried 7 wearable devices in different on-body positions. Our results show that the detection of the device position consistently improves the result of activity recognition for common activities. Regarding cross-subjects models, we identified the waist as the most suitable device location at which the acceleration patterns for the same activity across several people are most similar. In this context, our results provide evidence for the reliability of physical characteristics based cross-subjects models.     

Multi-sensor based real-time 6-DoF pose tracking for wearable augmented reality

Wearable augmented reality (WAR) combines a live view of a real scene with computer-generated graphic on resource-limited platforms. One of the crucial technologies for WAR is a real-time 6-DoF pose tracking, facilitating registration of virtual components within in a real scene. Generally, artificial markers are typically applied to provide pose tracking for WAR applications. However, these marker-based methods suffer from marker occlusions or large viewpoint changes. Thus, a multi-sensor based tracking approach is applied in this paper, and it can perform real-time 6-DoF pose tracking with real-time scale estimation for WAR on a consumer smartphone. By combining a wide-angle monocular camera and an inertial sensor, a more robust 6-DoF motion tracking is demonstrated with the mutual compensations of the heterogeneous sensors. Moreover, with the help of the depth sensor, the scale initialization of the monocular tracking is addressed, where the initial scale is propagated within the subsequent sensor-fusion process, alleviating the scale drift in traditional monocular tracking approaches. In addition, a sliding-window based Kalman filter framework is used to provide a low jitter pose tracking for WAR. Finally, experiments are carried out to demonstrate the feasibility and robustness of the proposed tracking method for WAR applications.     

Detecting work-related stress with a wearable device

Excessive stress will lower work efficiency, lead to negative emotions and even various illnesses. This paper aims at detecting work-related stress based on physiological signals measured by a wearable device. Different from common binary stress detection, this study detects three levels of stress, i.e., no stress, moderate stress and high perceived stress. The Montreal Imaging Stress Task (MIST) is used to simulate the different stress conditions, including both mental stress and psychosocial stress factors that are relevant at the workplace. A sensor-based wearable device is used to acquire the electrocardiogram (ECG) and respiration (RSP) signals from 39 participants. We extract stress-related features from ECG and RSP, and the Random Forest is used to select the optimal feature combination, which is later fed to the classifier. Four classifiers are investigated about their ability to predict the three stress levels. Finally, the combination of Random Forest and Support Vector Machine (SVM) achieve the best performance. With this method, the accuracy is improved from 78% to 84% in three states classification. And in binary stress detection, the accuracy is 94%.     

Architecture and applications of the FingerMouse: a smart stereo camera for wearable computing HCI

In this paper we present a visual input HCI system for wearable computers, the FingerMouse. It is a fully integrated stereo camera and vision processing system, with a specifically designed ASIC performing stereo block matching at 5 Mpixel/s (e.g. QVGA 320 × 240 at 30 fps) and a disparity range of 47, consuming 187 mW (78 mW in the ASIC). It is button-sized (43 mm × 18 mm) and can be worn on the body, capturing the user’s hand and processing in real-time its coordinates as well as a 1-bit image of the hand segmented from the background. Alternatively, the system serves as a smart depth camera, delivering foreground segmentation and tracking, depth maps and standard images, with a processing latency smaller than 1 ms. This paper describes the FingerMouse functionality and its applications, and how the specific architecture outperforms other systems in size, latency and power consumption.     

Exploring MARS: developing indoor and outdoor user interfaces to a mobile augmented reality system

We describe an experimental mobile augmented reality system (MARS) testbed that employs different user interfaces to allow outdoor and indoor users to access and manage information that is spatially registered with the real world. Outdoor users can experience spatialized multimedia presentations that are presented on a head-tracked, see-through, head-worn display used in conjunction with a hand-held pen-based computer. Indoor users can get an overview of the outdoor scene and communicate with outdoor users through a desktop user interface or a head- and hand-tracked immersive augmented reality user interface.     

基于RGB-D摄像头的实时手指跟踪与手势识别

近些年,基于视觉的手部跟踪与手势识别一直是人机交互和计算机视觉等领域的研究热点。传统方法主要是使用单目或多目RGB摄像头等设备获得手部位置、方向等信息,但RGB摄像头易受到复杂背景、光照变化、纹理的限制,导致其准确性、实时性和鲁棒性都较差。随着可获得场景深度信息的家用RGB-Depth(RGB-D)摄像头的发展和上市,可以利用深度信息较好地克服上述环境问题。首先定义了一个基于RGB-D摄像头的3D交互空间,根据深度信息将手部区域从复杂背景、多变的光照条件下进行分割;然后提出了一种基于深度摄像头的手指识别和跟踪方法,该方法基于手部轮廓对人手及手指进行识别和跟踪;最后通过对手指位置和轨迹的跟踪进行手势识别,从而实现人机交互。对提出的方法进行的实验验证了它的准确性、实时性和鲁棒性。     

Application level performance evaluation of wearable devices for stress classification with explainable AI

Stress has become one of the most prominent problems of modern societies and a key contributor to major health issues. Dealing with stress effectively requires detecting it in real-time, informing the user, and giving instructions on how to manage it. Over the past few years, wearable devices equipped with biosensors that can be utilized for stress detection have become increasingly popular. Since they come with various designs and technologies and acquire biosignals from different body locations, choosing a suitable device for a particular application has become a challenge for researchers and end-users. This study compares seven common wearable biosensors for stress detection applications. This was accomplished by collecting physiological sensor data during Baseline, Stress, Recovery, and Cycling sessions from 32 participants. Machine learning algorithms were used to classify four stress classes, and the results obtained from all wearables were compared. Following this, a state-of-the-art explainable artificial intelligence method was employed to clarify our models’ predictions and investigate the influence different features have on the models’ outputs. Despite the results showing that ECG wearables perform slightly better than the rest of the devices, adding a second biosignal (EDA) improved the results significantly, tipping the balance toward multisensor wearables. Finally, we concluded that although the output results of each model can be affected by various factors, in most cases, there is no significant difference in the accuracy of stress detection by different wearables. However, the decision to select a particular wearable for stress detection applications must be made carefully considering the trade-off between the users’ expectations and preferences and the pros and cons of each device.     

Human activity recognition via wearable devices using enhanced ternary weight convolutional neural network

Human activity recognition (HAR) has been known as an active area for more than a decade, and there are still crucial aspects that are intended as challenging problems. Providing detailed and appropriate information about the activities and behaviors of people is one of the most important fields in ubiquitous computing. There are numerous applications in this field, among which healthcare, security, and entertainment scenarios can be listed. Human activity recognition can be carried out with the assistance of smartphone sensors such as accelerometers and gyroscopes or images captured from webcams. Today, the application of deep neural networks in this domain has received much attention and has led to more accurate and effective results compared to traditional techniques. The deep neural network performs arithmetic operations on a various number of hidden layers. In this article, a new approach called HAR-CT is proposed to enhance the accuracy of human activity recognition in various classes by adopting a convolutional neural network (CNN). Subsequently, an optimization technique using the TWN model is also suggested to reduce the complexity of the deep neural network approach that decreases the energy consumption of mobile devices. To this end, the float precision weights of the convolutional neural network are quantized and converted into ternary weights, while the decline in the accuracy is very low compared to the initial deep neural network. The evaluation results of both networks demonstrate that the proposed methods outperform the recently published approaches in human activity recognition.     

A computer vision system for the detection and classification of vehicles at urban road intersections

The paper presents a real-time vision system to compute traffic parameters by analyzing monocular image sequences coming from pole-mounted video cameras at urban crossroads. The system uses a combination of segmentation and motion information to localize and track moving objects on the road plane, utilizing a robust background updating, and a feature-based tracking method. It is able to describe the path of each detected vehicle, to estimate its speed and to classify it into seven categories. The classification task relies on a model-based matching technique refined by a feature-based one for distinguishing between classes having similar models, like bicycles and motorcycles. The system is flexible with respect to the intersection geometry and the camera position. Experimental results demonstrate robust, real-time vehicle detection, tracking and classification over several hours of videos taken under different illumination conditions. The system is presently under trial in Trento, a 100,000-people town in northern Italy.     

Respiration-based emotion recognition with deep learning

Different physiological signals are of different origins and may describe different functions of the human body. This paper studied respiration (RSP) signals alone to figure out its ability in detecting psychological activity. A deep learning framework is proposed to extract and recognize emotional information of respiration. An arousal-valence theory helps recognize emotions by mapping emotions into a two-dimension space. The deep learning framework includes a sparse auto-encoder (SAE) to extract emotion-related features, and two logistic regression with one for arousal classification and the other for valence classification. For the development of this work an international database for emotion classification known as Dataset for Emotion Analysis using Physiological signals (DEAP) is adopted for model establishment. To further evaluate the proposed method on other people, after model establishment, we used the affection database established by Augsburg University in Germany. The accuracies for valence and arousal classification on DEAP are 73.06% and 80.78% respectively, and the mean accuracy on Augsburg dataset is 80.22%. This study demonstrates the potential to use respiration collected from wearable deices to recognize human emotions.     

Probabilistic people tracking with appearance models and occlusion classification: The AD-HOC system

AD-HOC (Appearance Driven Human tracking with Occlusion Classification) is a complete framework for multiple people tracking in video surveillance applications in presence of large occlusions. The appearance-based approach allows the estimation of the pixel-wise shape of each tracked person even during the occlusion. This peculiarity can be very useful for higher level processes, such as action recognition or event detection. A first step predicts the position of all the objects in the new frame while a MAP framework provides a solution for best placement. A second step associates each candidate foreground pixel to an object according to mutual object position and color similarity. A novel definition of non-visible regions accounts for the parts of the objects that are not detected in the current frame, classifying them as dynamic, scene or apparent occlusions. Results on surveillance videos are reported, using in-house produced videos and the PETS2006 test set.     

A machine-vision system to measure the parameters describing the performance of a Foucault pendulum

We describe a machine-vision system that makes real-time measurements of the kinematics of a Foucault pendulum. Images are taken from a downward-facing camera placed close to the pendulum suspension point. The bob is detected via background subtraction and located by fitting circles to the resulting contour segments. The bob trajectory is then modelled by fitting ellipses to recent positions. Parameters are improved through Kalman filtering. Experimental results are shown. Our implementation is a robust and accurate tool for visualization of the pendulum kinematics as well as troubleshooting and maintenance of the mechanical elements.     

尺度补偿的相关核滤波器跟踪

相关核滤波器跟踪是工程实际中非常实用的跟踪算法,它的算法简单,只需要在下一帧进行一个样本的密集采样就能对目标进行跟踪,但是对于有尺度变化的目标跟踪的适用性不足。采用了尺度补偿的相关核滤波器跟踪算法,对相关核滤波器跟踪进行了改进。首先使用了点跟踪补偿机制对相关核滤波器尺度和位移进行补偿;其次采用了压缩感知提取的特征建立模板对目标进行建模,在关键帧进行目标的重检测来防止尺度估计带来的跟踪误差。通过实验对提出的算法进行了标准视频库的测试,并在中心点误差和实际跟踪覆盖率两个指标上与原算法进行了对比分析。实验结果表明,提出的具有尺度补偿的跟踪算法提高了相关核滤波器跟踪在有尺度属性变化视频序列中的准确率和实用性。     

A computer vision system for Extravehicular Activity Helper/Retriever

The Extravehicular Activity Helper/Retriever (EVAHR) is a free-flying robot currently being developed by the Automation and Robotics Division at the NASA Johnson Space Center to support activities in the neighborhood of Space Station Freedom or planetary habitat. The EVAHR's primary responsibilities are rescue of crew, and retrieval of critical equipment. It will also perform extravehicular activities in cooperation with crew members. The stated responsibilities could never be fulfilled without a robust and versatile real time computer vision system. This paper presents a preliminary design of the EVAHR's vision system and its initial implementation. The preliminary design consists of a vision system planner, and many sub-modules for performing various vision functions. Top-down and bottom-up approaches have been taken for initial implementation of the preliminary design. While the top-down approach focuses on building a control mechanism and laying down a framework for the vision system planner, the bottom-up approach emphasizes the design and implementation of various computation skills such as search, tracking, and pose estimation. Experimental results of the initial implementation are included in the paper.     

基于云计算的视频取证监控系统*

在视频取证过程中,面对多摄像头非协作工作方式的视频取证的缺陷以及海量的视频数据和复杂的取证计算问题,提出了一种基于云计算的视频取证监控系统的解决方案。在该方案中,各摄像头采用协作工作方式,监控系统中的视频数据保存在云计算系统中,终端用户需要的视频监控服务由云计算平台来提供,取证过程中的目标识别和跟踪等复杂计算也由云计算平台提供。该系统可以充分利用云计算平台的虚拟存储和虚拟计算能力,解决取证现场的多摄像头的协作工作能力,提高视频取证的处理效率和取证的准确性以及提高各种终端用户的监控灵活性和方便性。     

Face detection and recognition in an unconstrained environment for mobile visual assistive system

We present a visual assistive system that features mobile face detection and recognition in an unconstrained environment from a mobile source using convolutional neural networks. The goal of the system is to effectively detect individuals that approach facing towards the person equipped with the system. We find that face detection and recognition becomes a very difficult task due to the movement of the user which causes camera shakes resulting in motion blur and noise in the input for the visual assistive system. Due to the shortage of related datasets, we create a dataset of videos captured from a mobile source that features motion blur and noise from camera shakes. This makes the application a very challenging aspect of face detection and recognition in unconstrained environments. The performance of the convolutional neural network is further compared with a cascade classifier. The results show promising performance in daylight and artificial lighting conditions while the challenges lie for moonlight conditions with the need for reduction of false positives in order to develop a robust system. We also provide a framework for implementation of the system with smartphones and wearable devices for video input and auditory notification from the system to guide the visually impaired.     

Hand tracking in a natural conversational environment by the interacting multiple model and probabilistic data association (IMM-PDA) algorithm

Traditional image based hand tracking algorithms use a single model Kalman filter to estimate and predict the hand state (position, velocity, and acceleration) and do not consider multiple measurements with noise and false alarms. However, these approaches may fail in the case of large maneuvers and/or a clutter measurement environment. In this paper, we apply the interacting multiple model (IMM) to catch hand maneuvers and the probabilistic data association (PDA) method to process noisy measurements and false alarms. A theoretical framework of image based hand tracking by the IMM-PDA algorithm is set up. Experiment results from several long video segments show that the IMM-PDA algorithm gives a superior performance compared to single model based Kalman filters.