A live video streaming system for intuitive human-system interaction

In this paper, we propose an intuitive live video streaming system based on virtual reality technologies among people who are far apart. This system is a kind of server-client system, and can provide remote users with virtual 3D audiovisual fields in real time via a very-high-speed network. The server captures audio and video data from its clients, compiles them into a 3D audiovisual scene at a virtual conference, and broadcasts it to the clients. At the present stage, our system captures 2 videos and creates one 3D video at a time. Our system can play 3D audiovisual contents on Windows XP systems as well as on CAVE systems. Currently, our system can play the 3D video contents at about 2.36 fps under a LAN environment. This work was presented in part at the 12th International Symposium on Artificial Life and Robotics, Oita, Japan, January 25–27, 2007     

A-BI+: A framework for Augmented Business Intelligence

Augmented reality allows users to superimpose digital information (typically, of operational type) upon real-world objects. The synergy of analytical frameworks and augmented reality opens the door to a new wave of situated analytics, in which users within a physical environment are provided with immersive analyses of local contextual data. In this paper, we propose an approach named A-BI⁺ (Augmented Business Intelligence) that, based on the sensed augmented context (provided by wearable and smart devices), proposes a set of relevant analytical queries to the user. This is done by relying on a mapping between the objects that can be recognized by the devices and the elements of the enterprise multidimensional cubes, and also by taking into account the queries preferred by users during previous interactions that occurred in similar contexts. A set of experimental tests evaluates the proposed approach in terms of efficiency, effectiveness, and user satisfaction.     

一种基于视沉的增强现实三维注册算法

为了提高增强现实系统三维注册的精度和效率，提出了一种基于计算机视觉的增强现实三维注射算法，并已将其用于采用透射式头盔显示器的增强现实系统中，该算法有如下特点：（1）构架简单，实用性强，一般情况下只需4个平面标志物就可实现三维注册；（2）工作范围大，甚至可以应用到室外的增强现实系统中；（30数值求解过程最线性过程，误差小，可以满足增强现实系统高精度三维注册的要求，另外，应用这种算法所需要作的图象处理计算量很小，其典型系统包括一个彩色CCD和几个不同颜色的标志点，由于很容易获得标志点的对应像面位置，而且不需要计算图象对，因此实时性好，是一般图形工作站和PC机上的增强现实系统进行实时注册的算法基础。     

Optically based direct manipulation for augmented reality

Augmented reality (AR) constitutes a very powerful three-dimensional user interface for many “hands-on” application scenarios. To fully exploit the AR paradigm, the computer must not only augment the real world, but also accept feedback from it. In this paper, we present an optical approach for collecting such feedback by analyzing video sequences to track users and the objects they work with. Our system can be set up in any room after quickly placing a few known optical targets in the scene. We present two demonstration scenarios to illustrate the overall concept and potential of our approach and then discuss the research issues involved.     

基于增强现实的人机物理交互仿真系统研究

在增强现实系统中,真实物体和虚拟物体除了要实现视觉上的融合之外,还要能够实现视觉交互和物理交互.仿真系统采用了基于单目视觉的ARToolKit增强现实开发平台,首先对CMOS摄像头采集的图像进行处理,利用模式识别的方法对标识物进行图像匹配,然后对从标识物坐标系到摄像机坐标系的变换矩阵进行计算,实现虚实注册,最后通过对标识物空间位置的改变实现了虚实物体问的单向物理交互.实验结果表明,该系统操作简单直观,具有良好的实时性,交互性及可扩展性.     

基于移动平台的增强现实研究

对移动增强现实技术的研究现状和进展进行了较为全面地总结。首先,梳理了移动增强现实技术的研究之路,归纳并分析了国内外的研究机构、研究特点以及相关资源。其次,详细地阐述了该领域所涉及的理论与技术,对当前研究热点领域进行了概括、分类和对比。最后,讨论了该领域研究中存在的重、难点问题,并对未来的发展进行了展望。     

Fiducial Marker Based on Projective Invariant for Augmented Reality

Fiducial marker based Augmented Reality has many applications.So far the inner pattern of the fiducial marker is always used to encode the markers.Thus a large portion of the fiducial marker image is used for encoding instead of providing corresponding feature points for pose accuracy.This paper presents a novel method which utilizes directly the projective invariant contained in the positional relation of the corresponding feature points to encode the marker.The proposed method does not require the region of pattern image for encoding any more and can provide more corresponding feature points so that higher pose accuracy can be achieved easily.Many related approaches such as cmnulative distribution function,reprojection verification and robust process are proposed to overcome the problem of sensibility of the projective invariant.Experimental results show that the proposed fiducial marker system is reliable and robust,and can provide higher pose accuracy than that achieved by existing fiducial marker systems.     

Augmented reality on large screen for interactive maintenance instructions

We present an empirical study that evaluates the effectiveness of technical maintenance assisted with interactive augmented reality instructions. Our approach consists in an augmented visualization on a large screen and a combination of multiple fixed and mobile cameras. We used commercially available solutions. In our test, 14 participants completed a set of 4 maintenance tasks based on manual inspections of a motorbike engine. Tool selection, removal of bolts, and part dis\assembly, are supported by visual labels, 3D virtual models and 3D animations. All participants executed similar operations in two modalities: paper manuals and augmented instructions. Statistical analyses proved that augmented instructions reduced significantly participants’ overall execution time and error rate.     

Augmented and Virtual Reality techniques for footwear

The use of 3D imaging techniques has been early adopted in the footwear industry. In particular, 3D imaging could be used to aid commerce and improve the quality and sales of shoes. Footwear customization is an added value aimed not only to improve product quality, but also consumer comfort. Moreover, customisation implies a new business model that avoids the competition of mass production coming from new manufacturers settled mainly in Asian countries.     

Modeling video sources for real-time scheduling

What is the impact of the autocorrelation of variable-bit-rate (VBR) sources on real-time scheduling algorithms? Our results show that the impact of long term, or interframe, autocorrelation is negligible, while the impact of short term, or intraframe, autocorrelation can be significant. Such results are essentially independent of the video coding scheme employed. To derive these results, video sequences are modeled as a collection of stationary subsequences called scenes. Within a scene, a statistical model is derived for both the sequence of frames and of slices. The model captures the distribution and the autocorrelation function of real-time video data. In previous work, the pseudoperiodicity of the slice level auto-correlation function made it difficult to develop a simple yet accurate model. We present a generalization of previous methods that can easily capture this pseudoperiodicity and is suited for modeling a greater variety of autocorrelation functions. By simply tuning a few parameters, the model reproduces the statistic behavior of sources with different types and levels of correlation on both the frame and the slice level.     

Glove Based User Interaction Techniques for Augmented Reality in an Outdoor Environment

This paper presents a set of pinch glove-based user interface tools for an outdoor wearable augmented reality computer system. The main form of user interaction is the use of hand and head gestures. We have developed a set of augmented reality information presentation techniques. To support direct manipulation, the following three selection techniques have been implemented: two-handed framing, line of sight and laser beam. A new glove-based text entry mechanism has been developed to support symbolic manipulation. A scenario for a military logistics task is described to illustrate the functionality of this form of interaction.     

Real-time software-based video coder for multimedia communication systems

A novel software-based video compression algorithm, the Popular Video Coder (PVC), is presented in this paper, and a video phone system, the Popular Phone, is also implemented based on the PVC. The PVC simplifies the traditional video coder by removing the transform and the motion estimation parts and modifies the quantizer and entropy coder. Two novel coding algorithms, the adaptive quantizer and the modified windowed Huffman-like coder, are used in the PVC to encode the video data with a quality picture at a high compression ratio. The video quality of the proposed coder is as good as that of the MPEG coder when the input is a low-resolution and slow-motion video, and the computational complexity of the PVC is much lower than that of the Motion Picture Expert Group (MPEG). Since no compression hardware is needed for the PVC to encode and decode video data, the cost and complexity of developing multimedia applications, such as video phone and multimedia e-mail systems, can be greatly reduced. Furthermore, some networking issues, such as error control and flow control, are discussed in connection with applying the PVC to implement the Popular Phone.     

增强现实中的虚实注册技术研究

增强现实是把计算机产生的虚拟物体或其他信息合成到用户看到的真实世界中的一种技术 .实现虚拟信息与真实景象的正确匹配或有机结合是每个增强现实实用系统都必须完成的一项重要任务 ,且它的具体实现需依赖于计算机硬件和软件系统的相互配合及协调 .为了有效地降低增强现实应用系统中的虚实注册误差 ,在对基于位置跟踪和相机定标技术的传统虚实注册技术进行分析的基础上 ,首先研究了仿射变换技术理论在增强现实中的应用 ,并提出了在全局仿射坐标系下 ,获取像素相对深度参数的一种策略 ;然后在对 AR中的虚实注册问题进行研究分析的基础上 ,进一步讨论了图象分析技术在虚实注册问题中的应用问题 ,并提出了一种基于数值背景表达的快速虚实注册策略 ,以提高虚实注册过程中 ,场景特征点提取的精度和速度 ;最后还分别用该仿射理论进行了动态和静态的虚实注册实验 ,实验结果表明 ,该方法能有效地降低增强现实应用系统的虚实注册误差 .     

INTERVALES: INTERactive Virtual and Augmented framework for industriaL Environment and Scenarios

One of Industry 4.0’s greatest challenges for companies is the digitization of their processes and the integration of new related technologies such as virtual reality (VR) and augmented reality (AR), which can be used for training purposes, design, or assistance during industrial operations. Moreover, recent results and industrial proofs of concept show that these technologies demonstrate critical advantages in the industry. Nevertheless, the authoring and editing process of virtual and augmented content remains time-consuming, especially in complex industrial scenarios. While the use of interactive virtual environments through virtual and augmented reality presents new possibilities for many domains, a wider adoption of VR/AR is possible only if the authoring process is simplified, allowing for more rapid development and configuration without the need for advanced IT skills. To meet this goal, this study presents a new framework: INTERVALES. First, framework architecture is proposed, along with its different modules; this study then shows that the framework can be updated by not only IT workers, but also other job experts. The UML data model is presented to format and simplify the authoring processes for both VR and AR. This model takes into account virtual and augmented environments, the possible interactions, and ease operations orchestration. Finally, this paper presents the implementation of an industrial use case composed of collaborative robotic (cobotic) and manual assembly workstations in VR and AR based on INTERVALES data.     

一个高分辨率视频的创新加密算法

当今,多媒体的应用正在飞速发展.视频通信的保密性是商业实践上的主要考虑,例如收费性视频或者多方视频会议.特定的加密算法在实时多媒体通信中被用来满足严格的时效要求.引入了一个创新的视频加密,名为Puzzle,来为软件中的视频数据加密.它具有实时所要求的速度并提供了足够的安全保护.该算法能够随时嵌入至已存在的多媒体系统中.     

An enhanced client-centric approach for efficient video broadcast

Periodic broadcast is a cost-effective solution for large-scale distribution of popular videos. Regardless of the number of video requests, this strategy guarantees a constant worst service latency to all clients, making it possible to serve a large community with a minimal amount of broadcast bandwidth. Although many efficient periodic broadcast techniques have been proposed, most of them impose rigid requirements on client receiving bandwidth. They either demand clients to have the same bandwidth as the video server, or limit them to receive no more than two video streams at any one time. In our previous work, we addressed this problem with a Client-Centric Approach (CCA). This scheme takes into consideration both server broadcast bandwidth and client receiving bandwidth and allows clients to use all their receiving capability for prefetching broadcast data. As a result, given a fixed broadcast bandwidth, a shorter broadcast period can be achieved with an improved client communication capability. In this paper, we present an enhanced version of CCA to further leverage client bandwidth for more efficient video broadcast. The new scheme reduces the broadcast latency up to 50% as compared to CCA. We prove the correctness of this new technique and provide an analytical evaluation to show its performance advantage as compared with some existing techniques.

Educational impact of an Augmented Reality (AR) application for teaching structural systems to non-engineering students

This study evaluates the effectiveness of Augmented Reality (AR) in improving undergraduate non-engineering students’ (n = 103) achievement of learning outcomes and their perception of AR as a learning tool. An AR application for teaching structural systems, Virtual and Augmented Reality for Structures (VARS), is developed and tested using a quasi-experiment and a questionnaire. The study shows that VARS AR significantly improves the posttest quiz score of the experimental group. In addition, a conceptual framework linking design considerations and achievement of learning outcomes is created. Based on the questionnaire survey, authentic context and user interface are important design considerations influencing students’ perceived achievement of learning outcomes. It was found that experience using AR and background knowledge did not influence the perceived achievement of learning outcomes and change in quiz score.     

Rule-based spatiotemporal query processing for video databases

In our earlier work, we proposed an architecture for a Web-based video database management system (VDBMS) providing an integrated support for spatiotemporal and semantic queries. In this paper, we focus on the task of spatiotemporal query processing and also propose an SQL-like video query language that has the capability to handle a broad range of spatiotemporal queries. The language is rule-based in that it allows users to express spatial conditions in terms of Prolog-type predicates. Spatiotemporal query processing is carried out in three main stages: query recognition, query decomposition, and query execution.Received: 11 October 2001, Accepted: 3 October 2003, Published online: 12 December 2003Edited by: A. Buchmann Correspondence to: Özgür UlusoyThis work is supported by the Scientific and Research Council of Turkey (TÜBITAK) under Project Code 199E025. This work was done while the first author was at Bilkent University.     

Parallel neural networks for multimodal video genre classification

Improvements in digital technology have made possible the production and distribution of huge quantities of digital multimedia data. Tools for high-level multimedia documentation are becoming indispensable to efficiently access and retrieve desired content from such data. In this context, automatic genre classification provides a simple and effective solution to describe multimedia contents in a structured and well understandable way. We propose in this article a methodology for classifying the genre of television programmes. Features are extracted from four informative sources, which include visual-perceptual information (colour, texture and motion), structural information (shot length, shot distribution, shot rhythm, shot clusters duration and saturation), cognitive information (face properties, such as number, positions and dimensions) and aural information (transcribed text, sound characteristics). These features are used for training a parallel neural network system able to distinguish between seven video genres: football, cartoons, music, weather forecast, newscast, talk show and commercials. Experiments conducted on more than 100 h of audiovisual material confirm the effectiveness of the proposed method, which reaches a classification accuracy rate of 95%.

An EyeTap video-based featureless projective motion estimation assisted by gyroscopic tracking for wearable computer mediated reality

In this paper we present a computationally economical method of recovering the projective motion of head mounted cameras or EyeTap devices, for use in wearable computer-mediated reality. The tracking system combines featureless vision and inertial methods in a closed loop system to achieve accurate robust head tracking using inexpensive sensors. The combination of inertial and vision techniques provides the high accuracy visual registration needed for fitting computer graphics onto real images and the robustness to large interframe camera motion due to fast head rotations. Operating on a 1.2 GHz Pentium III wearable computer with graphics accelerated hardware, the system is able to register live video images with less than 2 pixels of error (0.3 degrees) at 12 frames per second. Fast image registration is achieved by offloading computer vision computation onto the graphics hardware, which is readily available on many wearable computer systems. As an application of this tracking approach, we present a system which allows wearable computer users to share views of their current environments that have been stabilised to another viewer's head position.