Modeling robotic operations controlled by natural language

There are multiple ways to control a robotic system. Most of them require the users to have prior knowledge about robots or get trained before using them. Natural language based control attracts increasing attention due to its versatility and less requirements for users. Since natural language instructions from users cannot be understood by the robots directly, the linguistic input has to be processed into a formal representation which captures the task specification and removes the ambiguity inherent in natural language. For most of existing natural language controlled robotic system, they assume the given language instructions are already in correct orders. However, it is very likely for untrained users to give commands in a mixed order based on their direct observation and intuitive thinking. Simply following the order of the commands can lead to failures of tasks. To provide a remedy for the problem, we propose a novel framework named dependency relation matrix (DRM) to model and organize the semantic information extracted from language input, in order to figure out an executable sequence of subtasks for later execution. In addition, the proposed approach projects abstract language input and detailed sensory information into the same space, and uses the difference between the goal specification and temporal status of the task under implementation to monitor the progress of task execution. In this paper, we describe the DRM framework in detail, and illustrate the utility of this approach with experiment results.     

RALPH static planner: CAD-based manipulator assembly task planning for CSG-modeled objects

This article describes a manipulator assembly task planner that processes the knowledge of the working environment and generates a sequence of general, manipulator independent commands. the planner takes a very high level command, such as “insert PEG into HOLE” without further specifications, reasons about the involved object features using the information from the CAD system, and generates a process plan for the manipulator to automatically perform the task. In this planner, the grasp planning and the path planning are developed and implemented for a static world.     

LIA: A Virtual Assistant that Can Be Taught New Commands by Speech

ABSTRACT

In the imminent future, people are likely to engage with smart devices by instructing them in natural language. A fundamental question to ask is how might intelligent agents interpret such instructions and learn new tasks. In this article we present the first speech-based virtual assistant that can be taught new commands by speech. A user study on our agent has shown that people can teach it new commands. We also show that people see great advantage in using an instructable agent, and determine what users believe are the most important use cases of such an agent.     

Event,State, And Process In Arrow Logic

Artificial agents, which are embedded in a virtual world, need to interpret a sequence of commands given to them adequately, considering the temporal structure for each command. In this paper, we start with the semantics of natural language and classify the temporal structures of various eventualities into such aspectual classes as action, process, and event. In order to formalize these temporal structures, we adopt Arrow Logic. This logic specifies the domain for the valuation of a sentence as an arrow. We can connect, or give order to, arrows by defining inter-arrow operations, and can give different views for sentences. Thereafter we formalize the rules of aspectual shifts in situated inference, in the style of a logic programming language. Thus, we not only describe the static representation of temporal features, but also show the dynamic process to deduce how each eventuality is viewed. The rules are applied to the information flow through the sequence of commands; therefore, we consider how the temporal structure of a command affects the succeeding commands.     

语义图驱动的面向复杂逻辑关系的自然语言问答

传统的基于知识库的问答难以处理具有复杂逻辑关系的自然语言问题,而此类问题在实际应用中广泛存在。为此,该文提出了语义图驱动的自然语言问答框架。框架核心是用主链、支链、环结构等图形化结构及其拼接,表达领域中的事件及事件之间的语义关系。进一步的,构造语义图的线性编码形式,利用路径生成模型将复杂自然语言问题翻译成语义图的线性序列。为验证框架有效性,该文面向公开的医疗领域数据,半自动地构建了3000个具有复杂逻辑关系的问题与答案。将问句进行实体识别、实体对齐,得到语义图线性序列,最后通过槽填充后在知识库中查询得到答案。其中,基于注意力机制的序列到序列模型达到了97.67%的准确率,启发式规则的槽填充达到94.88%的准确率,系统整体性能达到91.5%。     

A case-based reasoning approach for automating disassembly process planning

One of the first processes for preparing a product for reuse, remanufacture or recycle is disassembly. Disassembly is the process of systematic removal of desirable constituents from the original assembly so that there is no impairment to any useful component. As the number of components in a product increases, the time required for disassembly, as well as the complexity of planning for disassembly rises. Thus, it is important to have the capability to generate disassembly process plans quickly in order to prevent interruptions in processing especially when multiple products are involved. Case-based reasoning (CBR) approach can provide such a capability. CBR allows a process planner to rapidly retrieve, reuse, revise, and retain solutions to past disassembly problems. Once a planning problem has been solved and stored in the case memory, a planner can retrieve and reuse the product's disassembly process plan at any time. The planner can also adapt an original plan for a new product, which does not have an existing plan in case memory. Following adaptation and application, a successful plan is retained in the case memory for future use. This paper presents the procedures to initialize a case memory for different product platforms, and to operate a CBR system, which can be used to plan disassembly processes. The procedures are illustrated using examples.     

Comprehension Grammars Generated from Machine Learning of Natural Languages

We are developing a theory of probabilistic language learning in the context of robotic instruction in elementary assembly actions. We describe the process of machine learning in terms of the various events that happen on a given trial, including the crucial association of words with internal representations of their meaning. Of central importance in learning is the generalization from utterances to grammatical forms. Our system derives a comprehension grammar for a superset of a natural language from pairs of verbal stimuli like Go to the screw! and corresponding internal representations of coerced actions. For the derivation of a grammar no knowledge of the language to be learned is assumed but only knowledge of an internal language.We present grammars for English, Chinese, and German generated from a finite sample of about 500 commands that are roughly equivalent across the three languages. All of the three grammars, which are context-free in form, accept an infinite set of commands in the given language.     

Exploiting temporal coherence in nonlinear plan construction

Correct conventional nonlinear planners operate in accordance with Chapman's modal truth criterion (MTC). The MTC characterizes the conditions under which an assertion will be true at a point in a nonlinear plan. However, the MTC is not all one requires in order to build a realistic planning system: it merely sanctions the use of a number of plan modifications in order to achieve each assertion in a developing plan. The number of modifications that can be made is usually very large. To avoid breadth-first search a planner must have some idea of which plan modification to consider. We describe a domain-independent search heuristic called temporal coherence , which helps guide the search through the space of partial plans defined by the MTC. Temporal coherence works by suggesting certain orderings of goal achievement as more appealing than others, and thus by finding bindings for plan variables consistent with the planner's overall goals. Our experience with a real nonlinear planner has highlighted the need for such a heuristic. In this paper, we give an example planning problem and use it to illustrate how temporal coherence can speed the search for an acceptable plan. We also prove that if a solution exists in the partial plan search space defined by the MTC, then there exists a path to that solution which is sanctioned by temporal coherence.     

The Berkeley UNIX Consultant Project

UC (UNIX Consultant) is an intelligent, natural-language interface thatallows naive users to learn about the UNIX operating system. UC wasundertaken because the task was thought to be both a fertile domain forArtificial Intelligence research and a useful application of AI work inplanning, reasoning, natural language processing, and knowledgerepresentation. The current implementation of UC comprises the followingcomponents: A language analyzer, called ALANA, that produces arepresentation of the content contained in an utterance; aninference component called a concretion mechanism that furtherrefines this content; a goal analyzer, PAGAN, that hypothesizes theplans and goals under which the user is operating; an agent, calledUCEgo, that decides on UC's goals and proposes plans for them; adomain planner, called KIP, that computes a plan to address the user'srequest; an expression mechanism, UCExpress, that determines thecontent to be communicated to the user, and a language productionmechanism, UCGen, that expresses UC's response in English. UC alsocontains a component called KNOME that builds a model of the user'sknowledge state with respect to UNIX. Another mechanism, UCTeacher,allows a user to add knowledge of both English vocabulary and factsabout UNIX to UC's knowledge base. This is done by interacting with theuser in natural language. All these aspects of UC make use of knowledgerepresented in a knowledge representation system called KODIAK. KODIAKis a relation-oriented system that is intended to have widerepresentational range and a clear semantics, while maintaining acognitive appeal. All of UC's knowledge, ranging from its most generalconcepts to the content of a particular utterance, is represented inKODIAK.     

Planning in highly dynamic environments: an anytime approach for planning under time constraints

In this paper, we present a novel and domain-independent planner aimed at working in highly dynamic environments with time constraints. The planner follows the anytime principles: a first solution can be quickly computed and the quality of the final plan is improved as long as time is available. This way, the planner can provide either fast reactions or very good quality plans depending on the demands of the environment. As an on-line planner, it also offers important advantages: our planner allows the plan to start its execution before it is totally generated, unexpected events are efficiently tackled during execution, and sensing actions allow the acquisition of required information in partially observable domains. The planning algorithm is based on problem decomposition and relaxation techniques. The traditional relaxed planning graph has been adapted to this on-line framework by considering information about sensing actions and action costs. Results also show that our planner is competitive with other top-performing classical planners.     

Controlling a robot manipulator with fuzzy voice commands using a probabilistic neural network

Natural language commands are generated by intelligent human beings. As a result, they contain a lot of information. Therefore, if it is possible to learn from such commands and reuse that knowledge, it will be a very efficient process. In this paper, learning from such information rich voice commands for controlling a robot is studied. First, new concepts of fuzzy coach-player system and sub-coach are proposed for controlling robots with natural language commands. Then, the characteristics of the subjective human decision making process are discussed and a Probabilistic Neural Network (PNN) based learning method is proposed to learn from such commands and to reuse the acquired knowledge. Finally, the proposed concept is demonstrated and confirmed with experiments conducted using a PA-10 redundant manipulator.     

From event calculus to the scheduling problem. Semantics of action and temporal reasoning in aircraft maintenance

In this work we are interested in the logical and semantical aspects of reasoning about actions in a scheduling process. We present an adaptation of the event calculus of Kowalski and Sergot to the problem of determining the temporal structure of the operations that must be performed during the realization of some complex objectives. Our application domain is aircraft maintenance. We try to reason about the actions which are performed during an overhaul in order to help to schedule them. The original model reasons about changes, i.e. events which initiate or terminate propositions. The first step of this work was to improve the initial model by adding a temporal relation between events and propositions because in our field we also have to reason about events which only inform us about some propositions without affecting them. The second step of this work is to build a set of specific rules which temporally interpret the semantics of the usual specifications of the actions to be considered. This interpretation aims to associate each action with two events and some temporal relations which are usable by the general model. Temporal reasoning uses pertinent knowledge about the specific universe (here, the aircraft that we consider and the actions which may be performed on it). We outline a generative methodology to formalize this relevant knowledge efficiently. This cognitive approach brings more informational economy in temporal reasoning because only the relevant information is considered The temporal reasoning model and the methodology have been exemplified and tested on a complex part of an aircraft. In the future, adapted tools based on this approach will be developed, in order to solve several problems of aircraft maintenance scheduling.     

Toward a programming theory for rational agents

An important problem in agent verification is a lack of proper understanding of the relation between agent programs on the one hand and agent logics on the other. Understanding this relation would help to establish that an agent programming language is both conceptually well-founded and well-behaved, as well as yield a way to reason about agent programs by means of agent logics. As a step toward bridging this gap, we study several issues that need to be resolved in order to establish a precise mathematical relation between a modal agent logic and an agent programming language specified by means of an operational semantics. In this paper, we present an agent programming theory that provides both an agent programming language as well as a corresponding agent verification logic to verify agent programs. The theory is developed in stages to show, first, how a modal semantics can be grounded in a state-based semantics, and, second, how denotational semantics can be used to define the mathematical relation connecting the logic and agent programming language. Additionally, it is shown how to integrate declarative goals and add precompiled plans to the programming theory. In particular, we discuss the use of the concept of higher-order goals in our theory. Other issues such as a complete axiomatization and the complexity of decision procedures for the verification logic are not the focus of this paper and remain for future investigation. Part of this research was carried out while the first author was affiliated with the Nijmegen Institute for Cognition and Information, Radboud University Nijmegen.     

Planning procedural advice

A planner which generates advice about the procedures which should be carried out by a human agent in order to achieve a goal is described. The fact that the agent is a person, not a robot, makes it possible to develop plans cooperatively with the user in the course of a dialogue, but imposes special requirements on the planner. The planner should be capable of taking advantage of the user's knowledge and abilities; of providing partial plans; of planning even in the absence of complete knowledge about the user's current state; of re-planning when the execution does not succeed or the situation changes; and of providing explanations of its advice. The paper considers the implications of these requirements on the design of such an advisory planner, implemented as part of the ‘Advice System’, a knowledge-based system for advising members of the public about welfare benefits.     

An organization of knowledge for problem solving and language comprehension

Plan synthesis and language comprehension, or more generally, the act of discovering how one perception relates to others, are two sides of the same coin, because they both rely on a knowledge of cause and effect—algorithmic knowledge about how to do things and how things work. I will describe a new theory of representation for commonsense algorithmic world knowledge, then show how this knowledge can be organized into larger memory structures, as it has been in a LISP implementation of the theory. The large-scale organization of the memory is based on structures called bypassable causal selection networks. A system of such networks serves to embed thousands of small commonsense algorithm patterns into a larger fabric which is directly usable by both a plan synthesizer and a language comprehender. Because these bypassable networks can adapt to context, so will the plan synthesizer and a language comprehender. I will propose that the model is an approximation to the way humans organize and use algorithmic knowledge, and as such, that it suggests approaches not only to problem solving and language comprehension, but also to learning. I'll describe the commonsense algorithm representation, show how the system synthesizes plans using this knowledge, and trace through the process of language comprehension, illustrating how it threads its way through these algorithmic structures.     

The SINIX Consultant – Towards a Theoretical Treatment of Plan Recognition

We have realized the help system SINIX Consultant (SC) for SINIX¹ users. The system is capable of answering – in German – natural language questions concerning SINIX commands, objects, and concepts. But not only does this help system react to inquiries – additionally, the system is capable of activating itself. If the user employs a sequence of SINIX commands (a plan) in order to reach a specific goal, the help system proposes a sequence which reaches the same goal, but, with fewer commands. In this paper, a brief survey of the SINIX Consultant and the realized plan recognizer REPLIX is first given. Then, an initial attempt of a theoretical treatment of plan recognition is presented. This is done within the logical framework. We show how we can use an interval-based logic of time to describe actions, atomic plans, non-atomic plans, action execution, and simple plan recognition. We also show that the recognition of inserted sub-plans managed by REPLIX can be handled as well. Then, we present a problem which cannot be treated in the formalism. Thus, in this paper, we don't present a full developed theory but nevertheless, a step towards it. This revised version was published online in August 2006 with corrections to the Cover Date.     

ON TEMPORAL DEDUCTIVE DATABASES

This article introduces a temporal deductive database system featuring a logic programming language and an algebraic front-end. The language, called Temporal DATALOG, is an extension of DATALOG based on a linear-time temporal logic in which the flow of time is modeled by the set of natural numbers. Programs of Temporal DATALOG are considered as temporal deductive databases, specifying temporal relationships among data and providing base relations to the algebraic front-end. The minimum model of a given Temporal DATALOG program is regarded as the temporal database the program models intensionally. The algebraic front-end, called TRA, is a point-wise extension of the relational algebra upon the set of natural numbers. When needed during the evaluation of TRA expressions, slices of temporal relations over intervals can be retrieved from a given temporal deductive database by bottom-up evaluation strategies.
A modular extension of Temporal DATALOG is also proposed, through which temporal relations created during the evaluation of TRA expressions may be fed back to the deductive part for further manipulation. Modules therefore enable the algebra to have full access to the deductive capabilities of Temporal DATALOG and to extend it with nonstandard algebraic operators. This article also shows that the temporal operators of TRA can be simulated in Temporal DATALOG by program clauses.