Comprehension Grammars Generated from Machine Learning of Natural Languages

We are developing a theory of probabilistic language learning in the context of robotic instruction in elementary assembly actions. We describe the process of machine learning in terms of the various events that happen on a given trial, including the crucial association of words with internal representations of their meaning. Of central importance in learning is the generalization from utterances to grammatical forms. Our system derives a comprehension grammar for a superset of a natural language from pairs of verbal stimuli like Go to the screw! and corresponding internal representations of coerced actions. For the derivation of a grammar no knowledge of the language to be learned is assumed but only knowledge of an internal language.We present grammars for English, Chinese, and German generated from a finite sample of about 500 commands that are roughly equivalent across the three languages. All of the three grammars, which are context-free in form, accept an infinite set of commands in the given language.     

Gestures that people can understand and use

Recent advances in computing devices push researchers to envision new interaction modalities that go beyond traditional mouse and keyboard input. Typical examples are large displays for which researchers hope to create more “natural” means of interaction by using human gestures and body movements as input. In this article, we reflect about this goal of designing gestures that people can easily understand and use and how designers of gestural interaction can capitalize on the experience of 30 years of research on visual languages to achieve it. Concretely, we argue that gestures can be regarded as “visual expressions to convey meaning” and thus are a visual language. Based on what we have learned from visual language research in the past, we then explain why the design of a generic gesture set or language that spans many applications and devices is likely to fail. We also discuss why we recommend using gestural manipulations that enable users to directly manipulate on-screen objects instead of issuing commands with symbolic gestures whose meaning varies among different users, contexts, and cultures.     

Scaffolding game-based learning: Impact on learning achievements,perceived learning,and game experiences

One of the central challenges of integrating game-based learning in school settings is helping learners make the connections between the knowledge learned in the game and the knowledge learned at school, while maintaining a high level of engagement with game narrative and gameplay. The current study evaluated the effect of supplementing a business simulation game with an external conceptual scaffold, which introduces formal knowledge representations, on learners' ability to solve financial-mathematical word problems following the game, and on learners' perceptions regarding learning, flow, and enjoyment in the game. Participants (M_age = 10.10 years) were randomly assigned to three experimental conditions: a “study and play” condition that presented the scaffold first and then the game, a “play and study” condition, and a “play only” condition. Although no significant gains in problem-solving were found following the intervention, learners who studied with the external scaffold before the game performed significantly better in the post-game problem-solving assessment. Adding the external scaffold before the game reduced learners' perceived learning. However, the scaffold did not have a negative impact on reported flow and enjoyment. Flow was found to significantly predict perceived learning and enjoyment. Yet, perceived learning and enjoyment did not predict problem-solving and flow directly predicted problem solving only in the “play and study” condition. We suggest that presenting the scaffold may have “problematized” learners' understandings of the game by connecting them to disciplinary knowledge. Implications for the design of scaffolds for game-based learning are discussed.     

End-user strategy programming

Rule-based programming systems can be fragile because they force the user to account for all logical alternatives. If an unconsidered case does arise during execution, program behavior falls through the cracks into unspecified behavior. We investigate rule-based, end-user strategy programming by introducing our Interactive Football Playbook—a domain specific, end-user programming environment to allow American football coaches to create animated football scenarios by associating strategy information with virtual football players. We address the problem of rule explosion through “rule bending” to support a minimalist, scaffolding-driven programming environment. Additionally, we introduce visual language representations for logical and sequential “and” to mitigate end-user confusion with the semantic meaning of these “and” constructs.     

The opposite of control: a Deweyan perspective on intrinsic motivation in “After 3” technology programs

A central feature of the design of many “After 3” technology programs is the assumption that student learning and motivation requires that they have choice and control of their activity. Similarly, the dominant cognitive-rational perspective of motivation portrays effective learners as having control of themselves and their environment. In this article, we build on Dewey's (1934. Art as experience. New York: Perigree.) aesthetics and epistemology — as most fully developed in “Art as experience” — to suggest that to be deeply engaged in learning, to be truly moved, requires not only control, but also the “opposite of control”. In “Art as experience” Dewey proposed that aesthetic experience — compelling, transformative experience — requires doing (acting on the world), reflection (standing back from the world), and undergoing (being acted upon by the world). Furthermore, grasping the meaning of these experiences emerges through a qualitative sense in addition to intentional analysis and reflection. Thus, intrinsic motivation, or what we shall call transformative experience, finds a balance between control and its opposite. We elaborate our conception of the “opposite of control” and discuss how this idea helps us appreciate heretofore unilluminated qualities of intrinsic motivation in “After 3” technology programs.     

Generating pallet loading patterns with considerations of item stacking on end and side surfaces

In late 1979 a two phase heuristic algorithm employing dynamic programming was presented by Steudel for solving the two-dimensional cutting stock problem where all the small rectangles were of the same dimensions, but withour any restrictions that the cutting be performed in a purely “guillotine” fashion. The algorithm was applied to solving the common problem of loading rectangular items of size l by w on a rectangular pallet of size L and W so as to maximize the number of items per layer on the pallet deckboard. In this paper, a new three-phase heuristic is presented which extends the 1979 recursive procedure and evaluates the option of stacking items on their end and/or side surface within the best loading pattern of bottom-stacked items. The resulting pattern is then projected into the third dimension to generate the total “cubic” pallet load. Computation results show that end and/or side stacking (when applicable) can yield average improvements in the range of 5% in items per pallet load.     

The inner and outer approaches to the design of recursive neural architectures

Feedforward neural network architectures work well for numerical data of fixed size, such as images. For variable size, structured data, such as sequences, d dimensional grids, trees, and other graphs, recursive architectures must be used. We distinguish two general approaches for the design of recursive architectures in deep learning, the inner and the outer approach. The inner approach uses neural networks recursively inside the data graphs, essentially to “crawl” the edges of the graphs in order to compute the final output. It requires acyclic orientations of the underlying graphs. The outer approach uses neural networks recursively outside the data graphs and regardless of their orientation. These neural networks operate orthogonally to the data graph and progressively “fold” or aggregate the input structure to produce the final output. The distinction is illustrated using several examples from the fields of natural language processing, chemoinformatics, and bioinformatics, and applied to the problem of learning from variable-size sets.     

Abstract meaning representation for legal documents: an empirical research on a human-annotated dataset

Natural language processing techniques contribute more and more in analyzing legal documents recently, which supports the implementation of laws and rules using computers. Previous approaches in representing a legal sentence often based on logical patterns that illustrate the relations between concepts in the sentence, often consist of multiple words. Those representations cause the lack of semantic information at the word level. In our work, we aim to tackle such shortcomings by representing legal texts in the form of abstract meaning representation (AMR), a graph-based semantic representation that gains lots of polarity in NLP community recently. We present our study in AMR Parsing (producing AMR from natural language) and AMR-to-text Generation (producing natural language from AMR) specifically for legal domain. We also introduce JCivilCode, a human-annotated legal AMR dataset which was created and verified by a group of linguistic and legal experts. We conduct an empirical evaluation of various approaches in parsing and generating AMR on our own dataset and show the current challenges. Based on our observation, we propose our domain adaptation method applying in the training phase and decoding phase of a neural AMR-to-text generation model. Our method improves the quality of text generated from AMR graph compared to the baseline model. (This work is extended from our two previous papers: “An Empirical Evaluation of AMR Parsing for Legal Documents”, published in the Twelfth International Workshop on Juris-informatics (JURISIN) 2018; and “Legal Text Generation from Abstract Meaning Representation”, published in the 32nd International Conference on Legal Knowledge and Information Systems (JURIX) 2019.).

     

Conceptual and content-based annotation of (multimedia) documents

This paper focuses on the techniques used in an NKRL environment (NKRL = Narrative Knowledge Representation Language) to deal with a general problem affecting the so-called “semantic/conceptual annotations” techniques. These last, mainly ontology-based, aim at “annotating” multimedia documents by representing, in some way, the “inner meaning/deep content” of these documents. For documents of sufficient size, the content modeling operations are separately executed on ‘significant fragments’ of the documents, e.g., “sentences” for natural language texts or “segments” (minimal units for story advancement) in a video context. The general problem above concerns then the possibility of collecting all the partial conceptual representations into a global one. This integration operation must, moreover, be carried out in such a way that the meaning of the full document could go beyond the simple addition of the ‘meanings’ conveyed by the single fragments. In this context, NKRL makes use of second order knowledge representation structures, “completive construction” and “binding occurrences”, for collecting within the conceptual annotation of a whole “narrative” the basic building blocks corresponding to the representation of its composing elementary events. These solutions, of a quite general nature, are discussed in some depth in this paper. This last includes also a short “state of the art” in the annotation domain and some comparisons with the different methodologies proposed in the past for solving the above ‘integration’ problem.     

Learning from natural instructions

Machine learning is traditionally formalized and investigated as the study of learning concepts and decision functions from labeled examples, requiring a representation that encodes information about the domain of the decision function to be learned. We are interested in providing a way for a human teacher to interact with an automated learner using natural instructions, thus allowing the teacher to communicate the relevant domain expertise to the learner without necessarily knowing anything about the internal representations used in the learning process. In this paper we suggest to view the process of learning a decision function as a natural language lesson interpretation problem, as opposed to learning from labeled examples. This view of machine learning is motivated by human learning processes, in which the learner is given a lesson describing the target concept directly and a few instances exemplifying it. We introduce a learning algorithm for the lesson interpretation problem that receives feedback from its performance on the final task, while learning jointly (1) how to interpret the lesson and (2) how to use this interpretation to do well on the final task. traditional machine learning by focusing on supplying the learner only with information that can be provided by a task expert. We evaluate our approach by applying it to the rules of the solitaire card game. We show that our learning approach can eventually use natural language instructions to learn the target concept and play the game legally. Furthermore, we show that the learned semantic interpreter also generalizes to previously unseen instructions.     

Put: language-based interactive manipulation of objects

Our approach to scene generation capitalizes the expressive power of natural language by separating its aptness in specifying spatial relations from the difficulties of understanding text. We are implementing an object-placement system called Put that uses a combination of linguistic commands and direct manipulation. The system is language-based, meaning that its design and structure are guided by natural language. Our approach (inspired by research in cognitive linguistics) is to analyze the natural use of spatial relations, define a well-understood class of fundamental relationships, and gradually build a coherent and natural spatial-manipulation system. Just a few simple spatial relationships, such as in, on, and at, parameterized by a limited number of environmental variables can provide comfortable object manipulation. These natural commands can be used to quickly prototype a complex scene and constrain object placement. We believe that we have an extensible, predictable, and computationally feasible environment for object manipulation. We have focused first on spatial relationships because they are fundamental to many conceptual domains beyond object placement, including motion and time. These particular domains are very important to areas of computer graphics such as animation. Uses of spatial relationships in these areas can be quite complex. We briefly introduce the complexities of understanding spatial relations and summarize related work. Then we describe the core of the Put placement system, followed by its linguistic, procedural, and interactive interfaces. We conclude by discussing future enhancements to the system     

User interface improvements in computer-assisted instruction,the challenge

The purpose of this paper is threefold: (1) to identify user interface problems as they relate to computer-assisted instruction (CAI); (2) to review the learning theories and instructional theories related to CAI user interface; and (3) to present potential CAI user interface improvements for research and development based on learning and instruction theory. User interface is defined as “hardware, software (including menus, screen design, keyboard commands, and command language), or both that allows a user to interact with and perform operations on a system, program, or device” (McDaniel, 1994, IBM dictionary of computing (10th ed.). New York: McGraw-Hill, p. 724). Not all learning theories and instructional theories are discussed, but those most influential to CAI are included. Likewise, not all potential CAI user interface improvements are addressed but, rather, the focus is on CAI screen design user interface improvements. Proposed improvements are those which could be easily researched and incorporated into CAI design.     

Part and Attribute Discovery from Relative Annotations

Part and attribute based representations are widely used to support high-level search and retrieval applications. However, learning computer vision models for automatically extracting these from images requires significant effort in the form of part and attribute labels and annotations. We propose an annotation framework based on comparisons between pairs of instances within a set, which aims to reduce the overhead in manually specifying the set of part and attribute labels. Our comparisons are based on intuitive properties such as correspondences and differences, which are applicable to a wide range of categories. Moreover, they require few category specific instructions and lead to simple annotation interfaces compared to traditional approaches. On a number of visual categories we show that our framework can use noisy annotations collected via “crowdsourcing” to discover semantic parts useful for detection and parsing, as well as attributes suitable for fine-grained recognition.     

Enhancing multimodal learning through personalized gesture recognition

Gestural recognition systems are important tools for leveraging movement‐based interactions in multimodal learning environments but personalizing these interactions has proven difficult. We offer an adaptable model that uses multimodal analytics, enabling students to define their physical interactions with computer‐assisted learning environments. We argue that these interactions are foundational to developing stronger connections between students' physical actions and digital representations within a multimodal space. Our model uses real time learning analytics for gesture recognition, training a hierarchical hidden‐Markov model with a “one‐shot” construct, learning from user‐defined gestures, and accessing 3 different modes of data: skeleton positions, kinematics features, and internal model parameters. Through an empirical comparison with a “pretrained” model, we show that our model can achieve a higher recognition accuracy in repeatability and recall tasks. This suggests that our approach is a promising way to create productive experiences with gesture‐based educational simulations, promoting personalized interfaces, and analytics of multimodal learning scenarios.     

Hierarchical search support for hypertext on-line documentation

Effectively finding relevant passages in a full-text database of software documentation calls for a user interface that does more than mimic a printed book. A hypertext approach, with a network of links among passages, offers great flexibility but often at the cost of high cognitive overhead and a disorienting lack of contextual cues. A tree-based approach guides users along branching paths through a hierarchy of text nodes. The “natural”, sequential implementation of such hierarchical access, however, is psychologically inept in large databases because it is order-dependent, discriminates awkwardly among key terms, clarifies each node's context incompletely, and often involves much semantic redundancy. An alternative, mixed approach, recently implemented in the on-line documentation system at the National Energy Research Supercomputer Center (NERSC), overcomes three of these four problems. It displays only local tree structure in response to “zoomin” or “zoomout” commands issued to focus a search begun with typical hypertext moves. This combination approach enjoys the benefits of cued, spatially interpreted hierarchical search while avoiding most of its known pitfalls. Usage monitoring at NERSC shows the ready acceptance of both zoom commands by documentation readers.     

A preferential,pattern-seeking,Semantics for natural language inference

The paper describes the way in which a Preference Semantics system for natural language analysis and generation tackles a difficult class of anaphoric inference problems: those requiring either analytic (conceptual) knowledge of a complex sort, or requiring weak inductive knowledge of the course of events in the real world. The method employed converts all available knowledge to a canonical template form and endeavors to create chains of non-reductive inferences from the unknowns to the possible referents. Its method for this is consistent with the overall principle of “semantic preference” used to set up the original meaning representation.     

Relational Reinforcement Learning

Relational reinforcement learning is presented, a learning technique that combines reinforcement learning with relational learning or inductive logic programming. Due to the use of a more expressive representation language to represent states, actions and Q-functions, relational reinforcement learning can be potentially applied to a new range of learning tasks. One such task that we investigate is planning in the blocks world, where it is assumed that the effects of the actions are unknown to the agent and the agent has to learn a policy. Within this simple domain we show that relational reinforcement learning solves some existing problems with reinforcement learning. In particular, relational reinforcement learning allows us to employ structural representations, to abstract from specific goals pursued and to exploit the results of previous learning phases when addressing new (more complex) situations.