Real-time single camera natural user interface engine development

Natural user interfaces (NUIs) provide human computer interaction (HCI) with natural and intuitive operation interfaces, such as using human gestures and voice. We have developed a real-time NUI engine architecture using a web camera as a means of implementing NUI applications. The system captures video via the web camera, implements real-time image processing using graphic processing unit (GPU) programming. This paper describes the architecture of the engine and the real-virtual environment interaction methods, such as foreground segmentation and hand gesture recognition. These methods are implemented using GPU programming in order to realize real-time image processing for HCI. To verify the efficacy of our proposed NUI engine, we utilized it in the development and implementation of several mixed reality games and touch-less operation applications, using the developed NUI engine and the DirectX SDK. Our results confirm that the methods implemented by the engine operate in real time and the interactive operations are intuitive.     

Using GOMS and NASA-TLX to Evaluate Human–Computer Interaction Process in Interactive Segmentation

HCI plays an important role in interactive medical image segmentation. The Goals, Operators, Methods, and Selection rules (GOMS) model and the National Aeronautics and Space Administration Task Load Index (NASA-TLX) questionnaire are different methods that are often used to evaluate the HCI process. In this article, we aim at improving the HCI process of interactive segmentation using both the GOMS model and the NASA-TLX questionnaire to: 1) identify the relations between these two methods and 2) propose HCI design suggestions based on the synthesis of the evaluation results using both methods. For this, we conducted an experiment where three physicians used two interactive segmentation approaches to segment different types of organs at risk for radiotherapy planning. Using the GOMS model, we identified 16 operators and 10 methods. Further analysis discovered strong relations between the use of GOMS operators and the results of the NASA-TLX questionnaire. Finally, HCI design issues were identified, and suggestions were proposed based on the evaluation results and the identified relations.     

面向公安应用的智能移动翻译软件的研究与实现*

应用语音中间件、嵌入式语音引擎、多语言信息服务系统架构和多语言模板翻译引擎等多项已处于实用化阶段的高新技术成果,研发面向首都公安应用的多语言智能移动翻译软件,为首都公安干警在社会治安、出入境管理、交通管理和边防检查等几个方面提供信息查询和语言翻译的手段,以推进首都公安的信息化建设,提高奥运安保工作水平,适应新时代公安工作需要.     

Model-driven development of interactive groupware systems: Integration into the software development process

Modeling interactive groupware systems is a complex and multi-disciplinary task. It is necessary to provide designers and engineers with a set of methods, notations and tools to specify the different aspects to consider when designing this type of systems. In this work we present a methodological framework based on the integration of several notations and processes for modeling some of these aspects, in particular: interaction, collaboration and functionality. The objective of this work is to provide a more complete support to the design of groupware systems, considering different viewpoints and modeling perspectives of the several stakeholders involved in the development of such applications.     

Embarking on Spoken-Language NL Interface Design

Natural language (NL) user interfaces are growing in popularity. Unfortunately, the complexity of NL interaction makes these interfaces difficult to design. For NL interfaces to become successful, universal tools are needed to help support the NL design process. What work practice should these tools explicitly support? Interviews with NL designers and our own experiments have identified a specific work practice that designers should consider as they begin to incorporate NL into user interface designs. The work practice study highlights the value of using Wizard of Oz prototyping in NL design. We describe a tool that we have built, called SUEDE, to explicitly support the first stage of NL design for spoken-language user interfaces. Our tools and tools like it will help make NL in human-computer interaction (HCI) more commonplace.     

An Architecture for Interactive Context-Aware Applications

The interactive behavior of context-aware applications depends on the physical and logical context in which the interaction occurs. The main difference between traditional HCI design and context-aware design is how we deal with context. In this article, we pick up on recent ubicomp community trends, drawing from sociology and focusing on interaction's communicative aspects. So, starting from the premise that interaction is communication, we propose a new interaction model for context-aware applications. We then derive an architectural framework that developers can use to implement our interaction model. The main benefit of our architecture is that, by modeling context in the user interface, developers can represent the application's inferences visually for users     

SpidersRUs: Creating specialized search engines in multiple languages

While small-scale search engines in specific domains and languages are increasingly used by Web users, most existing search engine development tools do not support the development of search engines in languages other than English, cannot be integrated with other applications, or rely on proprietary software. A tool that supports search engine creation in multiple languages is thus highly desired. To study the research issues involved, we review related literature and suggest the criteria for an ideal search tool. We present the design of a toolkit, called SpidersRUs, developed for multilingual search engine creation. The design and implementation of the tool, consisting of a Spider module, an Indexer module, an Index Structure, a Search module, and a Graphical User Interface module, are discussed in detail. A sample user session and a case study on using the tool to develop a medical search engine in Chinese are also presented. The technical issues involved and the lessons learned in the project are then discussed. This study demonstrates that the proposed architecture is feasible in developing search engines easily in different languages such as Chinese, Spanish, Japanese, and Arabic.     

Designing and implementing interactive and realistic augmented reality experiences

In this paper, we propose an approach for supporting the design and implementation of interactive and realistic Augmented Reality (AR). Despite the advances in AR technology, most software applications still fail to support AR experiences where virtual objects appear as merged into the real setting. To alleviate this situation, we propose to combine the use of model-based AR techniques with the advantages of current game engines to develop AR scenes in which the virtual objects collide, are occluded, project shadows and, in general, are integrated into the augmented environment more realistically. To evaluate the feasibility of the proposed approach, we extended an existing game platform named GREP to enhance it with AR capacities. The realism of the AR experiences produced with the software was assessed in an event in which more than 100 people played two AR games simultaneously.

     

Speech synthesis with face embeddings

Human beings are capable of imagining a person’s voice according to his or her appearance because different people have different voice characteristics. Although researchers have made great progress in single-view speech synthesis, there are few studies on multi-view speech synthesis, especially the speech synthesis using face images. On the basis of implicit relationship between the speaker’s face image and his or her voice, we propose a multi-view speech synthesis method called SSFE (Speech Synthesis with Face Embeddings). The proposed SSFE consists of three parts: a voice encoder, a face encoder and an improved multi-speaker text-to-speech (TTS) engine. On the one hand, the proposed voice encoder generates the voice embeddings from the speaker’s speech and the proposed face encoder extracts the voice features from the speaker’s face as f-voice embeddings. On the other hand, the multi-speaker TTS engine would synthesize the speech with voice embeddings and f-voice embeddings. We have conducted extensive experiments to evaluate the proposed SSFE on the synthesized speech quality and face-voice matching degree, in which the Mean Opinion Score of the SSFE is more than 3.7 and the matching degree is about 1.7. The experimental results prove that the proposed SSFE method outperforms state-of-the-art methods on the synthesized speech in terms of speech quality and face-voice matching degree.

     

On integration of interface design methods: Can debates be resolved?

There have been many debates on how to design the human–computer interface (HCI). Often, one can find that different views in a debate are simply because these views are attached to different aspects which embody the same thing. In other words, prior to giving an effective judgment of a debate, one needs to establish an understanding of the ‘total’ aspects of a thing the debate is about. Following this line of thinking, in this paper, we propose an understanding of the ‘total’ aspects of designing HCI, which is called the total interface design framework. We then judge several debates under this framework with the purpose of exemplifying the judgment process for any other debate related to designing HCI. At the end, the debates used for exemplifying our judgment process can be resolved. The effectiveness of the total interface design framework for integrating the different HCI approaches is also demonstrated.     

An evaluation of multimodal interaction techniques for 3D layout constraint solver in a desktop-based virtual environment

We propose a new approach to the 3D layout problems based on the integration of constraint programming and virtual reality interaction techniques. Our method uses an open-source constraint solver integrated in a popular 3D game engine. We designed multimodal interaction techniques for the system, based on gesture and voice input. We conducted a user study with an interactive task of laying out room furniture to compare and evaluate the mono- and multimodal interaction techniques. Results showed that voice command provided the best performance and was most preferred by participants, based on the analysis of both objective and subjective data. Results also revealed that there was no significant difference between the voice and multimodal input (voice and gesture). Our original approach opens the way to multidisciplinary theoretical work and promotes the development of high-level applications for the VR applications.     

A novel approach to improve the planning of adaptive and interactive sessions for the treatment of Major Depression

Human Computer Interaction (HCI) is a research field which aims to improve the relationship between users and interactive computer systems. A main objective of this research area is to make the user experience more pleasant and efficient, minimizing the barrier between the users׳ cognition of what they want to accomplish and the computer׳s understanding of the user׳s tasks, by means of user-friendly, useful and usable designs. A bad HCI design is one of the main reasons behind user rejection of computer-based applications, which in turn produces loss of productivity and economy in industrial environments.In the eHealth domain, user rejection of computer-based systems is a major barrier to exploiting the maximum benefit from those applications developed to support the treatment of diseases, and in the worst cases a poor design in these systems may cause deterioration in the clinical condition of the patient. Thus, a high level of personalisation of the system according to users׳ needs is extremely important, making it easy to use and contributing to the system׳s efficacy, which in turn facilitates the empowerment of the target users. Ideally, the content offered through the interactive sessions in these applications should be continuously assessed and adapted to the changing condition of the patient. A good HCI design and development can improve the acceptance of these applications and contribute to promoting better adherence levels to the treatment, preventing the patient from further relapses.In this work, we present a mechanism to provide personalised and adaptive daily interactive sessions focused on the treatment of patients with Major Depression. These sessions are able to automatically adapt the content and length of the sessions to obtain personalised and varied sessions in order to encourage the continuous and long-term use of the system. The tailored adaptation of session content is supported by decision-making processes based on: (i) clinical requirements; (ii) the patient’s historical data; and (iii) current responses from the patient. We have evaluated our system through two different methodologies: the first one performing a set of simulations producing different sessions from changing input conditions, in order to assess different levels of adaptability and variability of the session content offered by the system. The second evaluation process involved a set of patients who used the system for 14–28 days and answered a questionnaire to provide feedback about the perceived level of adaptability and variability produced by the system. The obtained results in both evaluations indicated good levels of adaptability and variability in the content of the sessions according to the input conditions.     

The Human–Artifact Model: An Activity Theoretical Approach to Artifact Ecologies

Although devices of all shapes and sizes currently dominate the technological landscape, human–computer interaction (HCI) as a field is not yet theoretically equipped to match this reality. In this article we develop the human–artifact model, which has its roots in activity theoretical HCI. By reinterpreting the activity theoretical foundation, we present a framework that helps addressing the analysis of individual interactive artifacts while embracing that they are part of a larger ecology of artifacts. We show how the human–artifact model helps structuring the understanding of an artifact's action-possibilities in relation to the artifact ecology surrounding it. Essential to the model is that it provides four interconnected levels of analysis and addresses the possibilities and problems at these four levels.

Artifacts and their use are constantly developing, and we address development in, and of, use. The framework needs to support such development through concepts and methods. This leads to a methodological approach that focuses on new artifacts to supplement and substitute existing artifacts. Through a design case, we develop the methodological approach and illustrate how the human–artifact model can be applied to analyze present artifacts and to design future ones. The model is used to structure such analysis and to reason about findings while providing leverage from activity theoretical insights on mediation, dialectics, and levels of activity.     

An Evaluation of the Potential of Task Analysis in the Evolution of Interactive Work Systems

Developed forms of task analysis allow designers to focus on both utility and usability issues in the development of interactive work systems. The models they generate represent aspects of the human, computer and domain elements of an interactive work system. Many interactive work systems are embedded in an organisational context. Pressure for changes are present in this context and provide impetus to stakeholders to change work tasks and the supporting tools. Interactive work systems also provide evolutionary pressures of their own, changing the very task they were designed to support. One approach to coping with change has been to evolve interactive work systems. Currently none of these techniques place focus on the performance of tasks as central, and consideration of usability is minimal. However, an evolutionary design approach forces an evolutionary experience upon users, and we cannot be sure whether this approach enhances the user’s experience or degrades their performance. Given the strength of task analysis it is likely that it will be applied within evolutionary contexts. Yet, little work has been undertaken to examine whether its role will, or could be different. We ask how we can move task analysis towards being used in a principled manner in the evolution of interactive work systems. This paper examines a number of features of the approach called task knowledge structures that may be useful in evolving interactive work systems. We look at tasks and their representativeness, roles, goals, objects (their attributes, relationships, typicality and centrality) and actions. We present a developing framework for examining other task analysis approaches for their utility in supporting interactive work systems evolution. Finally, we discuss future work within the area of applying task analysis in the evolution of interactive work systems.     

基于可重构阵列架构的强化学习计算引擎

现有神经网络处理器已广泛应用于计算机视觉、自然语言处理等领域。然而,现有片上加速方案对控制领域的强化学习算法支持较少,而基于神经网络的强化学习是智能系统决策技术的核心。该文采用可重构阵列体系结构,通过片上配置、动作与奖励存储的系统设计方案,可实现多种神经网 络算法的灵活部署,并支持强化学习使用模式。基于 65 nm CMOS 工艺的逻辑综合结果显示,处理器主频为 200 MHz 时,计算模块面积仅需 0.32 mm2,计算功率约 15.46 mW。     

HCI and business practices in a collaborative method for augmented reality systems

ContextEvery interactive system is composed of a functional core and a user interface. However, the software engineering (SE) and human–computer interaction (HCI) communities do not share the same methods, models or tools. This usually induces a large work overhead when specialists from the two domains try to connect their applicative studies, especially when developing augmented reality systems that feature complex interaction cores.ObjectiveWe present in this paper the essential activities and concepts of a development method integrating the SE and HCI development practices, from the specifications down to the design, as well as their application on a case study.MethodThe efficiency of the method was tested in a qualitative study involving four pairs of SE and HCI experts in the design of an application for which an augmented reality interaction would provide better user performance than a classic interactive system. The effectivity of the method was evaluated in a qualitative study comparing the quality of three implementations of the same application fragment (based on the same analysis model), using software engineering metrics.ResultsThe first evaluation confirmed the ease of use of our method and the relevance of our tools for guiding the design process, but raised concerns on the handling of conflicting collaborative activities. The second evaluation gave indications that the structure of the analysis model facilitates the implementation of quality software (in terms of coupling, stability and complexity).ConclusionIt is concluded that our method enables design teams with different backgrounds in application development to collaborate for integrating augmented reality applications with information systems. Areas of improvement are also described.     

基于学术社区的学术搜索引擎设计

学术社区和学术搜索引擎在科研活动中日趋重要。给出了一个基于学术社区的学术搜索引擎的设计方案,指出了它应具备的功能,提出了应着重解决的关键问题,并对部分问题提出了实现思路。给出了系统的架构设计,并讨论了文献资料的整合算法,将分散在不同位置、提供不同内容的学术信息组合为一个整体,有效地解决了文献提取问题。针对普通中文分词组件在对姓名进行分词时准确率较低的问题,设计了一个专门针对姓名进行分词的高效的算法。在开源框架Nutch和HBase的基础上,实现了一个学术搜索引擎,并在实验中验证了设计的有效性。