Integrating vision, motion and language through mind

This paper presents an integrated comprehension and generation system for vision, motion and language. First, a computer model of mind proposed previously is reviewed as the basis of the integration. It consists of nine domains according to the contents of mental activities and five levels along the process of concept formation.Then, vision and motion are discussed based on the model. Real world objects are divided into four categories: substance, event, attribute and others. The first three are named as noun, verb and adjective, respectively. The comprehension process of the named objects and the generation process of the named motion are shown along the five levels. Furthermore, vision and motion are connected and controlled by planning.Next, language is considered to describe any data or processing in mind in our model. From this are derived three functions: surveying any mental object, organizing high-level thought, and communicating with others. The realization of these functions is discussed from the point of generation.Next, an integrated system for vision, motion, and language is implemented on workstations. The system, or an intellectual and emotional agent, simulates the protagonist or fox of an Aesop fable. Its mental and physical behavior is shown by graphic displays, a voice generator, and a music generator which expresses its emotional states.In conclusion, this system could be expected to provide a good framework for integrated comprehension and generation systems from the technological point of view.