Analyzing, modelling, and specifying visual interaction

 This paper discusses the processes underlying human-computer visual interaction, thereby analysing the characteristics of visual interaction; a model and a theory of visual interaction, from which a formal specification of visual interactive systems that are trustable by their users can be derived, are also illustrated. Such a theory is called theory of visual sentences, since each message on the computer screen is described as a visual sentence, i.e., an element of a visual language that specifies the interaction. The concept of relational structure is introduced to take into account different kinds of relations which can exist among the characteristic patterns present in a visual sentence. A formal model of the dynamics of visual interaction is presented, which is specified as transformations of visual sentences, modelled through visual rewriting systems, and transformations of relational structures. Particular attention is given to usability issues so as to satisfy relevant features needed to allow non-ambiguity of interpretation, adequate communication, determinism and system viability. An example of visual interaction is given where an immunologist interacts with a simulation of the human immune system. This work was supported by the Italian Ministry of University and Research under the Research Project of National Interest “Specification, Design and Development of Visual Interactive Systems”.     

Automating visual language generation

A system to generate and interpret customized visual languages in given application areas is presented. The generation is highly automated. The user presents a set of sample visual sentences to the generator. The generator uses inference grammar techniques to produce a grammar that generalizes the initial set of sample sentences, and exploits general semantic information about the application area to determine the meaning of the visual sentences in the inferred language. The interpreter is modeled on an attribute grammar. A knowledge base, constructed during the generation of the system, is then consulted to construct the meaning of the visual sentence. The architecture of the system and its use in the application environment of visual text editing (inspired by the Heidelberg icon set) enhanced with file management features are reported     

Defining visual languages for interactive computing

A novel definition of visual languages allows a uniform approach to satisfying the needs of visual reasoning faced in visual human-computer interaction. The way the machine associates a computational meaning with an image, and conversely, the way it generates an image on the screen from a computation are formally described. A definition of a visual sentence and of a visual language as a set of visual sentences is discussed. A hierarchy of visual languages is derived in relation to the requirements for intelligible, manageable and trustable interaction between humans and computers     

A visual language compiler for information retrieval by visualreasoning

When a database increases in size, retrieving the data becomes a major problem. An approach based on data visualization and visual reasoning is described. The main idea is to transform the data objects and present sample data objects in a visual space. The user can use a visual language to incrementally formulate the information retrieval request in the visual space. A prototype system is described with the following features: (1) it is built on top of the SIL-ICON visual language compiler and therefore can be customized for different application domains; (2) it supports a fuzzy icon grammar to define reasonable visual sentences; (3) it incorporates a semantic model of the database for fuzzy visual query translation; and (4) it incorporates a VisualNet which stores the knowledge learned by the system in its interaction with the user so that the VisualReasoner can adapt its behavior     

基于深度学习的连续手语语句识别算法

目前,关于连续手语语句识别的研究相对较少,原因在于难以有效地分割出手语词。该文利用卷积神经网络提取手语词的手型特征,同时利用轨迹归一化算法提取手语词的轨迹特征,并在此基础上完成长短期记忆网络的构建,从而为手语语句识别准备好手语词分类器。对于一个待识别的手语语句,采用基于右手心轨迹信息的分割算法来检测过渡动作。由过渡动作可以将语句分割为多个片段,考虑到某些过渡动作可能是手语词内部的动作,所以将若干个片段拼接成一个复合段,并按照层次遍历的次序对所有复合段运用手语词分类器进行识别。最后,采用跨段搜索的动态规划算法寻找最大后验概率的词汇序列,从而完成手语语句的识别。实验结果表明,该算法可以对47个常用手语词组成的语句做出识别,且具有较高的准确性和实时性。     

Formalizing visual interaction with historical databases

Recent database applications are typically oriented towards a large set of non-expert users, and therefore, they need to be equipped with suitable interfaces facilitating the interaction with the system. Moreover, the incorporation of the time dimension in database systems is a desirable feature. Indeed, several temporal data models and the corresponding textual query languages have been proposed. However, there is a limited amount of research concerning the investigation of user-oriented languages for querying temporal databases. Our proposal addresses such a need. In particular, we propose a visual query environment, namely Temporal Visual Query Environment (TVQE) which provides an easier interaction of the user with temporal databases. The system adopts a diagrammatic representation of the database schema (including temporal classes and relationships) and a “graphical notebook” as interaction metaphor. In our approach, non-database experts are released from syntactical difficulties which are typical of textual languages, and they can easily express temporal queries by means of elementary graphical operations (e.g. click on a node label). Differently from many proposals in the field of visual query languages, the language underlying TVQE is provided with formal syntax and semantics. It is based on a minimal set of temporal graphical primitives (TGPs), which are defined on a Temporal Graph Model (TGM), with visual syntax and object-based semantics. In this paper we mainly concentrate on the formal aspects of TVQE, and provide some hints on the visual interaction mechanisms and implementation issues.     

Spatial and temporal aspects in visual interaction

The formal modeling of visual interaction processes, where images are composed by placing elementary pictures on a rectangular space, has to take into account both temporal and spatial aspects. Time is relevant in that some moves are enabled or disabled by previous ones, while for the spatial aspect one can impose constraints on the positions which can be occupied by certain pictures. Interaction can then be seen as a sort of visual game, in which players have to synchronise their moves in order to make the interaction progress. However, due to this dual nature, traditional models of concurrent computation are not sufficient to give reason of specific types of non-determinism encountered in visual interaction processes. We propose a characterisation of visual interaction in terms of games, and show that the algebraic structure derived from the association of temporal and spatial structures is canonically associated with a system of logics, in which the limitations on the possible inferences imposed by this type of non-determinism can be adequately expressed. Moreover, the logics can be naturally extended by the introduction of operators which simultaneously model both temporal and modal qualifications of formulae. In particular, it becomes possible to give a formal account of the “surprises” experienced by users in visual interaction processes or, equivalently, by players of visual games.     

Benefitting InfoVis with visual difficulties

Many well-cited theories for visualization design state that a visual representation should be optimized for quick and immediate interpretation by a user. Distracting elements like decorative "chartjunk" or extraneous information are avoided so as not to slow comprehension. Yet several recent studies in visualization research provide evidence that non-efficient visual elements may benefit comprehension and recall on the part of users. Similarly, findings from studies related to learning from visual displays in various subfields of psychology suggest that introducing cognitive difficulties to visualization interaction can improve a user's understanding of important information. In this paper, we synthesize empirical results from cross-disciplinary research on visual information representations, providing a counterpoint to efficiency-based design theory with guidelines that describe how visual difficulties can be introduced to benefit comprehension and recall. We identify conditions under which the application of visual difficulties is appropriate based on underlying factors in visualization interaction like active processing and engagement. We characterize effective graph design as a trade-off between efficiency and learning difficulties in order to provide Information Visualization (InfoVis) researchers and practitioners with a framework for organizing explorations of graphs for which comprehension and recall are crucial. We identify implications of this view for the design and evaluation of information visualizations.     

A visual environment for visual languages

The visual environment GENGED supports the visual definition of visual languages (VLs). Each VL is defined by an alphabet and a grammar. From a specific VL-definition, a VL-specification is generated which is the input of a graphical editor allowing for syntax-directed editing of diagrams over the specified VL. GENGED as well as each VL is based on the well-defined concepts of algebraic graph transformation and graphical constraint solving. The underlying formalism is hidden from the user, but it is essential for a formal presentation and manipulation of graphical structures. In this contribution, the GENGED concepts and environment are briefly proposed and illustrated by the definition of a simple kind of the well-known statechart language.     

Toward a formal specification of menu-based systems

As software systems continue to increase in sophistication and complexity, so do the interface requirements that support the corresponding user interaction. To select the proper blend of ingredients that constitutes an adequate user interface, it is essential that the system designer have a firm understanding of the interaction process, i.e., how the selected dialogue format interacts with the user and with the underlying task software. To promote such an understanding, this paper presents a model that characterizes one particular dialogue format: menu-based interaction. This model is actually a sequence of models, hierarchically structured, in which each successive model builds on its predecessor by introducing additional characterization elements. The first model describes the minimal set of elements inherent to any menu-based interface; successive models extend this minimal set by introducing task actions, incremental history sequences, and frame-associated memory. These principal model elements enable the characterization of fundamental, menu-based operations like computational and decision processes, user response reversal, and user directed movement. Moreover, because the principal model elements correspond directly to “real world” objects, an intuitive as well as formal understanding of menu-based interaction can be achieved. Effectively, the model elements and the hierarchical structure imposed by these elements provide an ideal environment for characterizing and classifying menu-based systems at various levels of sophistication.     

Towards Syntax-Aware Editors for Visual Languages

Editors for visual languages should provide a user-friendly environment supporting end users in the composition of visual sentences in an effective way. Syntax-aware editors are a class of editors that prompt users into writing syntactically correct programs by exploiting information on the visual language syntax. In particular, they do not constrain users to enter only correct syntactic states in a visual sentence. They merely inform the user when visual objects are syntactically correct. This means detecting both syntax and potential semantic errors as early as possible and providing feedback on such errors in a non-intrusive way during editing. As a consequence, error handling strategies are an essential part of such editing style of visual sentences.In this work, we develop a strategy for the construction of syntax-aware visual language editors by integrating incremental subsentence parsers into free-hand editors. The parser combines the LR-based techniques for parsing visual languages with the more general incremental Generalized LR parsing techniques developed for string languages. Such approach has been profitably exploited for introducing a noncorrecting error recovery strategy, and for prompting during the editing the continuation of what the user is drawing.     

Automated formal verification of visual modeling languages by model checking

Graph transformation has recently become more and more popular as a general, rule-based visual specification paradigm to formally capture (a) requirements or behavior of user models (on the model-level), and (b) the operational semantics of modeling languages (on the meta-level) as demonstrated by benchmark applications around the Unified Modeling Language (UML). The current paper focuses on the model checking-based automated formal verification of graph transformation systems used either on the model-level or meta-level. We present a general translation that inputs (i) a metamodel of an arbitrary visual modeling language, (ii) a set of graph transformation rules that defines a formal operational semantics for the language, and (iii) an arbitrary well-formed model instance of the language and generates a transitions system (TS) that serve as the underlying mathematical specification formalism of various model checker tools. The main theoretical benefit of our approach is an optimization technique that projects only the dynamic parts of the graph transformation system into the target transition system, which results in a drastical reduction in the state space. The main practical benefit is the use of existing back-end model checker tools, which directly provides formal verification facilities (without additional efforts required to implement an analysis tool) for many practical applications captured in a very high-level visual notation. The practical feasibility of the approach is demonstrated by modeling and analyzing the well-known verification benchmark of dining philosophers both on the model and meta-level.     

Formal Modeling and Reconfiguration of User Interfaces for Reduction of Errors in Failure Handling of Complex Systems

Controlling and observing complex systems is central to the study of human–machine interaction. In our understanding, there is much to be gained from integrating formal modeling and analysis, including the reconfiguration of user interfaces, with the development of user interfaces with high usability. To this end, we introduce a new approach to modeling and reconfiguration of user interfaces jointly with a newly developed set of tools for interactive and visual creation and automatic transformation of user interfaces' interaction logic to a formal language based on Petri nets. Reconfiguration will be embedded into a process for adapting user interfaces to the user's cognitive representation of the controlled system. This process involves practicing the use of a given user interface, adapting it to the user's needs through reconfiguration, and applying the resulting adaptations to the formally defined interaction logic. An evaluation study confirms that this process reduces errors in interaction.     

A visual news processing environment

CPP-TRS (Tonfoni 1989–94) is both a methodology and a language. Through the methodology (CPP — Communicative Positioning Program) those invisible aspects of communication are identified and represented by the meta-language (TRS — Text Representation Systems), which complements natural language. CPP-TRS is a new paradigm which applies to any communicative context.Icons in CPP-TRS are not intended to represent words or sentences in order to substitute them; they are rather intended to represent visually what natural language does not convey naturally. The improvement of communication is a challenge of the present and the future: CPP-TRS is designed explicitly for it.The CPP-TRS system thus is able to support the user with a set of visual tools, that are specifically suited to structuring text and communicating effectively.The same tools can support communication very effectively in situations where ambiguity can highly compromise the final result.The system may be used in a way that is either opaque or transparent to the reader of the text.Author of CPP-TRS¢ and CPP-TRS¢ Applications Designer.     

A graph-based framework for multiparadigmatic visual access todatabases

Describes an approach for multiparadigmatic visual access integration of different interaction paradigms. The user is provided with an adaptive interface augmented by a user model, supporting different visual representations of both data and queries. The visual representations are characterized on the basis of the chosen visual formalisms, namely forms, diagrams and icons. To access different databases, a unified data model called the “graph model” is used as a common underlying formalism to which databases, expressed in the most popular data models, can be mapped. Graph model databases are queried through the adaptive interface. The semantics of the query operations is formally defined in terms of graphical primitives. Such a formal approach permits us to define the concept of an “atomic query”, which is the minimal portion of a query that can be transferred from one interaction paradigm to another and processed by the system. Since certain interaction modalities and visual representations are more suitable for certain user classes, the system can suggest to the user the most appropriate interaction modality as well as the visual representation, according to the user model. Some results on user model construction are presented     

RECOGNITION CAN BE HARDER THAN PARSING

The work presented here attempts to bring out some fundamental concepts that underlie some known parsing algorithms, usually called chart or dynamic programming parsers, in the hope of guiding the design of similar algorithms for other formalisms that could be considered for describing the "surface" syntax of languages. The key idea is that chart parsing is essentially equivalent to a simple construction of the intersection of the language (represented by its grammar) with a regular set containing only the input sentence to be parsed (represented by a finite state machine). The resulting grammar for that intersection is precisely what is usually called a shared forest: it represents all parses of a syntactically ambiguous sentence. Since most techniques for processing ill-formed input can be modeled by considering a nonsingleton regular set of input sentences, we can expect to generalize these ill-formed input processing techniques to all parsers describable with our approach.     

Formalisation method for the text expressed knowledge

This paper describes a method of knowledge representation as a set of text expressed statements. The method is based on the identification of word-categories/phrases and their semantic relationships within the observed statement. Furthermore, the identification of semantic relationships between words/phrases using wh-questions that clarify the role of the word/phrase in the relationship is described. A conceptual model of the computer system based on the formalization method of text-expressed knowledge is proposed. The subsystem text formalization is described in detail, especially its parts: syntactic analysis of the sentence, sentence formalization, phrase structure grammar and lexicon. The phrase structure grammar is formed by induction and it is used to generate the language of the formalized notation of a sentence. The derivation of grammar is based on the simple phrase structure grammar which was used for the syntactical analysis of informal language notation. In its base, the suggested method translates sentences of the informal language into formal language sentences which are generated by the derivated phrase structure grammar. Current limitations of the method that also set the path of its further development are shown. Next concrete steps in the development of the method are also described.     

多语种自然语言生成系统中的预映射句子规划器

自然语言生成是研究如何用计算机来生成自然语言文本的研究领域 ,经典结构是宏观规划、微观规划和表层生成的流水线结构 .分析了多语种自然语言生成系统 ACNL G中的微观规划器 ,提出了“基于语言资源预映射的句子优化器”,它不仅对句子进行优化 ,而且负责将语种无关的文章内容映射到具体语种的表达方式中 .其核心思想就是将处理过程同具体语种的语言资源相分离 ,但同时又以语言资源为导向 ,既继承了现有方法的优点 ,同时也解决了多语种条件下各语种之间细微差别的处理问题 ,使句子优化能够针对目标语种的特点进行优化操作 ,从而使优化效果有实质的提高 ,进一步发展了多语种文本生成的理论     

A proof procedure for normal default theories

Recent research by Delgrande [6] and Geffner and Pearl [10] suggests two different semantic interpretations for normal defaults with one single representation as conditional sentences. However, they both need additional formal mechanisms for handling irrelevant information when their approaches are applied to formalising default reasoning. Delgrande in [5, 6] suggests two meta-strategies which he considers to be adequately strong to handle the orderings of defaults, and he claims they are equivalent. Furthermore, each of Delgrande's strategies is defined in terms of all sentences of the object language. In this paper, we shall prove that Delgrande's claim that his meta-strategies are equivalent is incorrect and that one of his meta-strategies can be reformulated within the framework of First Order Predicate Calculus (FOPC) and without having to consider every sentence of the object language. One advantage of such a reformalisation is its computational simplicity: to give an extension of a default theory there is only a need to consider those sentences which occur in the default theory under consideration rather than every sentence in the object language; furthermore, to provide a proof procedure for Delgrande's system as based on the meta-strategy we have formalised, one need only employ a FOPC proof procedure, rather than a conditional one.