Improving performance on object recognition for real-time on mobile devices

Augmented reality has been on the rise due to the proliferation of mobile devices. At the same time, object recognition has also come to the fore. In particular, many studies have focused on object recognition based on markerless matching. However, most of these studies have focused on desktop systems, which can have high performance in terms of CPU and memory, rather than investigating the use of mobile systems, which have been previously unable to provide high-performance object recognition based on markerless matching. In this paper, we propose a method that uses the OpenCV mobile library to improve real-time object recognition performance on mobile systems. First, we investigate the original object recognition algorithm to identify performance bottlenecks. Second, we optimize the algorithm by analyzing each module and applying appropriate code enhancements. Last, we change the operational structure of the algorithm to improve its performance, changing the execution frequency of the object recognition task from every frame to every four frames for real-time operation. During the three frames in which the original method is not executed, the object is instead recognized using the mobile devices accelerometer. We carry out experiments to reveal how much each aspect of our method improves the overall object recognition performance; overall, experimental performance improves by approximately 800 %, with a corresponding reduction of approximately 1 % in object recognition accuracy. Therefore, the proposed technique can be used to significantly improve the performance of object recognition based on markerless matching on mobile systems for real-time operation.     

A point-based rendering approach for real-time interaction on mobile devices

Mobile device is an important interactive platform. Due to the limitation of computation, memory, display area and energy, how to realize the efficient and real-time interaction of 3D models based on mobile devices is an important research topic. Considering features of mobile devices, this paper adopts remote rendering mode and point models, and then, proposes a transmission and rendering approach that could interact in real time. First, improved simplification algorithm based on MLS and display resolution of mobile devices is proposed. Then, a hierarchy selection of point models and a QoS transmission control strategy are given based on interest area of operator, interest degree of object in the virtual environment and rendering error. They can save the energy consumption. Finally, the rendering and interaction of point models are completed on mobile devices. The experiments show that our method is efficient. Supported by the National Natural Science Foundation of China (Grant No. 60873159), the Program for New Century Excellent Talents in University (Grant No. NCET-07-0039), the National High-Tech Research & Development Progrom of China (Grant No. 2006AA01Z333)     

Robust and real-time pose tracking for augmented reality on mobile devices

This paper addresses robust and ultrafast pose tracking on mobile devices, such as smartphones and small drones. Existing methods, relying on either vision analysis or inertial sensing, are either too computational heavy to achieve real-time performance on a mobile platform, or not sufficiently robust to address unique challenges in mobile scenarios, including rapid camera motions, long exposure time of mobile cameras, etc. This paper presents a novel hybrid tracking system which utilizes on-device inertial sensors to greatly accelerate the visual feature tracking process and improve its robustness. In particular, our system adaptively resizes each video frame based on inertial sensor data and applies a highly efficient binary feature matching method to track the object pose in each resized frame with little accuracy degradation. This tracking result is revised periodically by a model-based feature tracking method (Hare et al. 2012) to reduce accumulated errors. Furthermore, an inertial tracking method and a solution of fusing its results with the feature tracking results are employed to further improve the robustness and efficiency. We first evaluate our hybrid system using a dataset consisting of 16 video clips with synchronized inertial sensing data and then assess its performance in a mobile augmented reality application. Experimental results demonstrated our method’s superior performance to a state-of-the-art feature tracking method (Hare et al. 2012), a direct tracking method (Engel et al. 2014) and the Vuforia SDK (Ibañez and Figueras 2013), and can run at more than 40 Hz on a standard smartphone. We will release the source code with the pubilication of this paper.     

Speeding-up homography estimation in mobile devices

A critical problem faced by computer vision on mobile devices is reducing the computational cost of algorithms and avoiding visual stalls. In this paper, we introduce a procedure for reducing the number of samples required for fitting a homography to a set of noisy correspondences using a random sampling method. This is achieved by means of a geometric constraint that detects invalid minimal sets. In the experiments conducted, we show that this constraint not only reduces the number of random samples at a negligible computational cost but also balances the processor workload over time preventing visual stalls. In extreme situations of very large outlier proportion and noise level, it reduces in about one order of magnitude the number of required random samples.     

Depth-assisted rectification for real-time object detection and pose estimation

RGB-D sensors have become in recent years a product of easy access to general users. They provide both a color image and a depth image of the scene and, besides being used for object modeling, they can also offer important cues for object detection and tracking in real time. In this context, the work presented in this paper investigates the use of consumer RGB-D sensors for object detection and pose estimation from natural features. Two methods based on depth-assisted rectification are proposed, which transform features extracted from the color image to a canonical view using depth data in order to obtain a representation invariant to rotation, scale and perspective distortions. While one method is suitable for textured objects, either planar or non-planar, the other method focuses on texture-less planar objects. Qualitative and quantitative evaluations of the proposed methods are performed, showing that they can obtain better results than some existing methods for object detection and pose estimation, especially when dealing with oblique poses.     

Integrated approach of streaming 3d multimedia contents in real-time for mobile devices

It is popular to watch a 3D video through a 3D display nowadays. However, it is still difficult to enjoy the 3D multimedia contents with a mobile device even if a mobile device with a 3D display is currently introduced into the market. The main technological challenges for watching 3D contents via the mobile devices can be identified as the following: generating and streaming 3D contents. Generating 3D contents requires extra computational resources. Moreover, streaming 3D contents demands additional network bandwidth for receiving and transmitting the 3D data. To overcome these technological challenges, we propose ReMA, a novel 3D video streaming system in this paper. We devised a novel architecture for transmitter, receiver, and a distribution system to efficiently disseminate and generate 3D videos for the mobile devices. We implemented ReMA in a real test-bed and conducted a thorough empirical evaluation study to see the feasibility of streaming 3D contents for the mobile devices. Based on our empirical study, the resulting system presents a great promise in streaming 3D video in real-time to the mobile devices.     

Modelling on mobile devices

Software and Systems Modeling - Modelling is central to many disciplines in engineering and the natural and social sciences. A wide variety of modelling languages and tools have been proposed along...     

Efficient location aware intrusion detection to protect mobile devices

This paper addresses the problem of efficient intrusion detection for mobile devices via correlating the user’s location and time data. We developed two statistical profiling approaches for modeling the normal spatio–temporal behavior of the users: one based on an empirical cumulative probability measure and the other based on the Markov properties of trajectories. An anomaly is detected when the probability of a particular (location, time) evolution matching the normal behavior of a given user becomes lower than a certain threshold, determined by controlling the recall rate of the model of the normal user’s behavior. We used compression techniques to reduce processing overhead while maintaining high accuracy. Our evaluation based on the Reality Mining and Geolife data sets shows that the proposed system is capable of detecting a potential intrusion within 15 min and with 94 % accuracy.     

Intrusion detection for mobile devices using the knowledge-based, temporal abstraction method

In this paper, a new approach for detecting previously unencountered malware targeting mobile device is proposed. In the proposed approach, time-stamped security data is continuously monitored within the target mobile device (i.e., smartphones, PDAs) and then processed by the knowledge-based temporal abstraction (KBTA) methodology. Using KBTA, continuously measured data (e.g., the number of sent SMSs) and events (e.g., software installation) are integrated with a mobile device security domain knowledge-base (i.e., an ontology for abstracting meaningful patterns from raw, time-oriented security data), to create higher level, time-oriented concepts and patterns, also known as temporal abstractions. Automatically-generated temporal abstractions are then monitored to detect suspicious temporal patterns and to issue an alert. These patterns are compatible with a set of predefined classes of malware as defined by a security expert (or the owner) employing a set of time and value constraints. The goal is to identify malicious behavior that other defensive technologies (e.g., antivirus or firewall) failed to detect. Since the abstraction derivation process is complex, the KBTA method was adapted for mobile devices that are limited in resources (i.e., CPU, memory, battery). To evaluate the proposed modified KBTA method a lightweight host-based intrusion detection system (HIDS), combined with central management capabilities for Android-based mobile phones, was developed. Evaluation results demonstrated the effectiveness of the new approach in detecting malicious applications on mobile devices (detection rate above 94% in most scenarios) and the feasibility of running such a system on mobile devices (CPU consumption was 3% on average).     

Speech recognition for mobile devices

This article presents an overview of different approaches for providing automatic speech recognition (ASR) technology to mobile users. Three principal system architectures with respect to the employment of a wireless communication link are analyzed: Embedded Speech Recognition Systems, Network Speech Recognition (NSR) and Distributed Speech Recognition (DSR). An overview of the solutions having been standardized so far as well as a critical analysis of the latest developments in the field of speech recognition in mobile environments is given. Open issues, pros and cons of the different methodologies and techniques are highlighted. Special emphasis is placed on the constraints and limitations ASR applications are confronted with under different architectures.     

Visualizing information on mobile devices

Visualization can make a wide range of mobile applications more intuitive and productive. The mobility context and technical limitations such as small screen size make it impossible to simply port visualization applications from desktop computers to mobile devices, but researchers are starting to address these challenges. From a purely technical point of view, building more sophisticated mobile visualizations become easier due to new, possibly standard, software APIs such as OpenGLES and increasingly powerful devices. Although ongoing improvements would not eliminate most device limitations or alter the mobility context, they make it easier to create and experiment with alternative approaches.     

High-performance code optimizations for mobile devices

The Journal of Supercomputing - Mobile devices have seen their performance increased in latest years due to improvements on System on Chip technologies. These shared memory systems now integrate...     

Efficient discriminative multiresolution cascade for real-time human detection applications

Human detection is fundamental in many machine vision applications, like video surveillance, driving assistance, action recognition and scene understanding. However in most of these applications real-time performance is necessary and this is not achieved yet by current detection methods.This paper presents a new method for human detection based on a multiresolution cascade of Histograms of Oriented Gradients (HOG) that can highly reduce the computational cost of detection search without affecting accuracy. The method consists of a cascade of sliding window detectors. Each detector is a linear Support Vector Machine (SVM) composed of HOG features at different resolutions, from coarse at the first level to fine at the last one.In contrast to previous methods, our approach uses a non-uniform stride of the sliding window that is defined by the feature resolution and allows the detection to be incrementally refined as going from coarse-to-fine resolution. In this way, the speed-up of the cascade is not only due to the fewer number of features computed at the first levels of the cascade, but also to the reduced number of windows that need to be evaluated at the coarse resolution. Experimental results show that our method reaches a detection rate comparable with the state-of-the-art of detectors based on HOG features, while at the same time the detection search is up to 23 times faster.     

Super-resolution image reconstruction for mobile devices

With advanced mobile devices, the mobile applications of the high-definition display attract a lot of attentions nowadays. The existing image super-resolution methods are computationally inefficient for the high-definition display on the mobile devices. In this paper, we point out that the above critical issue deteriorates the display quality of the high-definition mobile devices. We propose an efficient and effective algorithm to reconstruct the high-resolution images for the mobile devices. Our algorithm outperforms previous approaches in not only smaller running time but also the higher quality of the super-resolution image reconstruction for the mobile devices.     

Interactive multi-frame reconstruction for mobile devices

The small size of handheld devices, their video capabilities and multiple cameras are under-exploited assets. Properly combined, the features can be used for creating novel applications that are ideal for pocket-sized devices, but may not be useful in laptop computers, such as interactively capturing and analyzing images on the fly. In this paper we consider building mosaic images of printed documents and natural scenes from low resolution video frames. High interactivity is provided by giving a real-time feedback on the video quality, while simultaneously guiding the user’s actions. In our contribution, we analyze and compare means to reach interactivity and performance with sensor signal processing and GPU assistance. The viability of the concept is demonstrated on a mobile phone. The achieved usability benefits suggest that combining interactive imaging and energy efficient high performance computing could enable new mobile applications and user interactions.     

Keystroke dynamics-based authentication for mobile devices

Recently, mobile devices are used in financial applications such as banking and stock trading. However, unlike desktops and notebook computers, a 4-digit personal identification number (PIN) is often adopted as the only security mechanism for mobile devices. Because of their limited length, PINs are vulnerable to shoulder surfing and systematic trial-and-error attacks. This paper reports the effectiveness of user authentication using keystroke dynamics-based authentication (KDA) on mobile devices. We found that a KDA system can be effective for mobile devices in terms of authentication accuracy. Use of artificial rhythms leads to even better authentication performance.     

基于Mobile Ambients的网络实时模拟方法

在通信协议中,很多性质都与时间相关.为了研究通信协议的时间性质,需要一种能够描述时间的形式化方法,在Mobile Ambients的基础上,用时间对其做扩展,提出一种新的形式系统--类型化实时Mobile Ambients演算.并采用实时Mo-bile Ambients描述了三次握手协议,结果表明了该方法的可行性.