Automated facility inspection using robotics and BIM: A knowledge-driven approach

Facility inspection is crucial for ensuring the performance of built assets. A traditional inspection, characterized by humans’ physical presence, is laborious, time-consuming, and becomes difficult to implement because of travel restrictions amid the pandemic. This laborious practice can potentially be automated by emerging smart technologies such as robotics and building information model (BIM). However, little has been known on how such automation can be achieved, concerning the knowledge-intensive nature of facility inspection. To fill the gap, this research aims to develop a knowledge-driven approach that can synergize knowledge of diverse sources (e.g., explicit knowledge from BIM, and tacit experience in human minds) to allow autonomous implementation of facility inspection by robotic agents. At the core the approach is an integrated scene-task-agent (iSTA) model that formalizes engineering priori in facility management and integrates the rich contextual information from BIM. Experiments demonstrated the applicability of the approach, which can endow robots with autonomy and knowledge to navigate the challenging built environments and deliver facility inspection outcomes. The iSTA model is publicized online, in hope of further extension by the research community and practical deployment to enable automated facility inspection using robotics and BIM.     

Autonomic self healing and recovery informed by environment knowledge

An important goal of autonomic computing is the development of computing systems that are capable of self healing with a minimum of human intervention. Typically, recovery from even a simple fault will require knowledge of the environment in which a computing system operates. To meet this need, we present an approach to self healing and recovery informed by environment knowledge that combines case based reasoning (CBR) and rule based reasoning. Specifically, CBR is used for fault diagnosis and rule based reasoning for fault remediation, recovery, and referral. We also show how automated information gathering from available sources in a computing system’s environment can increase problem solving efficiency and help to reduce the occurrence of service failures. Finally, we demonstrate the approach in an intelligent system for fault management in a local printer network.     

Integrity constraints in the information source tracking method

We study the issue of integrity constraints in the Information Source Tracking (IST) method. The IST method is an approach to the management of uncertain data in database systems. The main idea behind IST is that database information is supplied, or confirmed, by information sources. The accuracy of data is modeled by the reliability of the contributing information sources. In practice, information sources that are generally quite reliable may supply conflicting data at times. A database system based on IST must accept such data, and be capable of functioning correctly despite the inconsistency. We concentrate on cases where conflicting data is in violation of “key constraints”, a special form of functional dependencies resulting from the imposition of a key for each relation in the database. We present an approach, based on the adjustment of conflicting information source reliabilities, to handle data in violation of key constraints     

Comparing three methods for modeling the uncertainty in knowledge discovery from area-class soil maps

Knowledge discovery has been demonstrated as an effective approach to extracting knowledge from existing data sources for soil classification and mapping. Soils are spatial entities with fuzzy boundaries. Our study focuses on the uncertainty associated with class assignments when classifying such entities. We first present a framework of knowledge representation for categorizing spatial entities with fuzzy boundaries. Three knowledge discovery methods are discussed next for extracting knowledge from data sources. The methods were designed to maintain information for modeling the uncertainties associated with class assignments when using the extracted knowledge for classification. In a case study of knowledge discovery from an area-class soil map, all three methods were able to extract knowledge embedded in the map to classify soils at accuracies comparable to that of the original map. The methods were also able to capture membership gradations and helped to identify transitional zones and areas of potential problems on the source map when measures of uncertainties were mapped. Among the three methods compared, a fuzzy decision tree approach demonstrated the best performance in modeling the transitions between soil prototypes.     

Legacy information system replacement: Pursuing quality design of operational information systems

Legacy information systems (LIS) present unique design challenges by moving static processes across functional boundaries and into changing environments. Questions emerge for practice on how to capitalize on the diverse knowledge in the core team and which sources of external knowledge to tap. We develop a framework of practice and knowledge sources based on information elaboration theory (IET) that encourages highly interactive decision processes among diverse team members under conditions of complexity due to change. The results serve to advise practice and extend IET to consider the sourcing of knowledge and specific techniques for exploiting diverse knowledge sources.     

A hybrid integrated architecture for energy consumption prediction

Irresponsible and negligent use of natural resources in the last five decades has made it an important priority to adopt more intelligent ways of managing existing resources, especially the ones related to energy. The main objective of this paper is to explore the opportunities of integrating internal data already stored in Data Warehouses together with external Big Data to improve energy consumption predictions. This paper presents a study in which we propose an architecture that makes use of already stored energy data and external unstructured information to improve knowledge acquisition and allow managers to make better decisions. This external knowledge is represented by a torrent of information that, in many cases, is hidden across heterogeneous and unstructured data sources, which are recuperated by an Information Extraction system. Alternatively, it is present in social networks expressed as user opinions. Furthermore, our approach applies data mining techniques to exploit the already integrated data. Our approach has been applied to a real case study and shows promising results. The experiments carried out in this work are twofold: (i) using and comparing diverse Artificial Intelligence methods, and (ii) validating our approach with data sources integration.     

Agent-based framework for integrated facility engineering

An agent-based framework for the development of integrated facility engineering environments in support of collaborative design is introduced. This framework aims at integrating design software by allowing better software interoperability. Within their framework, design agents represent various existing design and planning systems that communicate their design information and knowledge partially and incrementally using the Agent Communication Language (ACL). ACL is a formal language proposed as a communication standard for disparate software. It is based on a logic-based language called Knowledge Interchange Format (KIF) and a message protocol called Knowledge Query Manipulation Language (KQML). Design agents are linked and their communication of design information is coordinated via system programs called facilitators in a federation architecture. The federation architecture specifies the way design agents and facilitators communicate in an integrated software environment. In concert with pursuing fundamental research concepts, we have been developing an integrated design software environment that spans different phases of the facility life cycle. This environment serves to demonstrate the primary aspects of this research methodology. In this paper, we first discuss the integration problem and review related research projects. We then present the major aspects of agent-based software engineering methodology and its application to integrated facility engineering. A highlight of the current integrated design environment development is given to illustrate the advantages of this approach. Finally, we summarize and discuss some of the important research issues in light of previous research.     

A type-2 fuzzy system-based approach for image data fusion to create building information models

Building Information Modelling (BIM) is a standard digital process that fuses buildings information from different sources into a 3D model during their lifecycle. For new construction sites using BIM, it is possible to monitor the cost, schedule, and changes throughout the lifecycle; however, existing buildings do not have a BIM model. Manually creating the BIM models for existing buildings is a high-cost task, both in time and money, hence there is a need for extracting information from available paper-based documentation and fuse it into a BIM model. The struggle of facility management and utility companies to fully adopt a BIM process (due to their high volumes of paper-based documentation of existing buildings) has led to the research on creating these 3D BIM models from 2D floor plan images.This paper presents a novel processing pipeline to extract 2D digital information from floorplans, fusing it into a 3D BIM model. The work focuses on fusing the available information to create the structure of the building in BIM format, which is considered the essential step before looking on working with other sources of data. In this process, we introduce a type-2 fuzzy logic based Explainable Artificial Intelligence (XAI) approach for the semantic segmentation step. The approach consists of using the outputs of type-2 fuzzy logic systems to classify a pixel as wall or background, by using information around and from the pixel of interest as the inputs to the system. After the semantic segmentation step, the output of the type-2 fuzzy logic goes through a noise removal process and finally a transformation from 2D to 3D by assigning the corresponding BIM tag to each identified element. The proposed type-2 fuzzy logic semantic segmentation approach produced comparable results (97.3% mean Intersection over Union (IoU) performance metric value) to the opaque box model approach based on Convolutional Neural Network (CNN) (99.3% mean IoU performance metric value). However, the type-2 fuzzy XAI system benefits from being an augmentable and interpretable model, which means that human users can understand the decision process and modify the model using their expert knowledge.     

Reduced Modeling of Unknown Trajectories

This paper deals with model order reduction of parametrical dynamical systems. We consider the specific setup where the distribution of the system’s trajectories is unknown but the following two sources of information are available: (i) some “rough” prior knowledge on the system’s realisations; (ii) a set of “incomplete” observations of the system’s trajectories. We propose a Bayesian methodological framework to build reduced-order models (ROMs) by exploiting these two sources of information. We emphasise that complementing the prior knowledge with the collected data provably enhances the knowledge of the distribution of the system’s trajectories. We then propose an implementation of the proposed methodology based on Monte-Carlo methods. In this context, we show that standard ROM learning techniques, such e.g., proper orthogonal decomposition or dynamic mode decomposition, can be revisited and recast within the probabilistic framework considered in this paper. We illustrate the performance of the proposed approach by numerical results obtained for a standard geophysical model.     

A general framework for time-aware decision support systems

In this paper we present a general framework for time-aware decision support systems. The framework uses the state-of-the-art tOWL language for the representation of temporal knowledge and enables temporal reasoning over the information that is represented in a knowledge base. Our approach uses state-of-the-art Semantic Web technology for handling temporal data. Through such an approach, the designer of a system can focus on the application intelligence rather than enforcing/checking data related restrictions manually. Also, there is an increased support for reuse of temporal reasoning tools across applications. We illustrate the applicability of our framework by building a market recommendations aggregation system. This system automatically collects market recommendations from online sources and, based on the past performance of the analysts that issued a recommendation, generates an aggregated recommendation in the form of a buy, hold, or sell advice. We illustrate the flexibility of our proposed system by implementing multiple methods for the aggregation of market recommendations.     

Knowledge capitalisation through case bases and knowledge engineering for road safety analysis

New generation knowledge-based systems should be fully integrated into their environment, by exploiting existing information sources, and should be flexible and easily extensible. This article describes the architecture of an organisational memory (OM) for road safety analysis. Starting from the design of a knowledge-based system, we show how we address knowledge capitalisation issues through the building of an OM. We present its main components and describe how knowledge engineering techniques can be exploited to build and enrich it. We then describe the major task that exploits the OM as decision support for site analysis. We also explain how domain knowledge can be exploited and capitalised using case-based reasoning and collaborative work.     

Visual Exploration of Financial Data with Incremental Domain Knowledge

Modelling the dynamics of a growing financial environment is a complex task that requires domain knowledge, expertise and access to heterogeneous information types. Such information can stem from several sources at different scales, complicating the task of forming a holistic impression of the financial landscape, especially in terms of the economical relationships between firms. Bringing this scattered information into a common context is, therefore, an essential step in the process of obtaining meaningful insights about the state of an economy. In this paper, we present Sabrina 2.0, a Visual Analytics (VA) approach for exploring financial data across different scales, from individual firms up to nation-wide aggregate data. Our solution is coupled with a pipeline for the generation of firm-to-firm financial transaction networks, fusing information about individual firms with sector-to-sector transaction data and domain knowledge on macroscopic aspects of the economy. Each network can be created to have multiple instances to compare different scenarios. We collaborated with experts from finance and economy during the development of our VA solution, and evaluated our approach with seven domain experts across industry and academia through a qualitative insight-based evaluation. The analysis shows how Sabrina 2.0 enables the generation of insights, and how the incorporation of transaction models assists users in their exploration of a national economy.     

An application of metadata-based image retrieval system for facility management

For facility management, photography is an efficient and accurate method of recording the physical state of infrastructure. However, without an effective organizational scheme, the difficulty of retrieving relevant photos from historical databases can become overly burdensome for highly complex or long-lived assets. To make strategic decisions, it is crucial to retrieve the right information from a plurality of sources in a timely manner. The main objective of this paper is to present a method for organizing and retrieving photos from massive facility management photo databases using photo-metadata: photographed location, camera perspective, and image semantic content information. Indoor localization experiments were performed using Bluetooth technology to infer the location information. Perspective is inferred from the device’s on-board inertial measurement unit (IMU). Image semantic content is inferred using a Convolutional Neural Network (CNN)-based deep learning algorithm. Fusing these three features, seven query options were provided for the user when retrieving images. Leveraging Building Information Modeling (BIM) as a process and Geographic Information Systems (GIS) as a framework, this paper also envisions a federated information management by connecting 2D and 3D facility assets with our real-world map which can be smoothly bridged with our image retrieval system. The realization of the integrated application with BIM and GIS is significantly beneficial for the facility management domain by advancing the understanding of projects in a broader view with a federated data platform. In this research, the framework is illustrated with 21 institutional buildings within the University of Texas at Austin’s main campus, and the authors conclude that the proposed metadata-based image retrieval system can ultimately enhance the better-informed decision-making process through rapid information retrieval.     

An Object Algebra Approach to Multidatabase Query Decomposition in Donají

This paper presents an approach to query decomposition in a multidatabase environment. The unique aspect of this approach is that it is based on performing transformations over an object algebra that can be used as the basis for a global query language. In the paper, we first present our multidatabase environment and semantic framework, where a global conceptual schema based on the Object Data Management Group standard encompasses the information from heterogeneous data sources that include relational databases as well as object-oriented databases and flat file sources. The meta-data about the global schema is enhanced with information about virtual classes as well as virtual relationships and inheritance hierarchies that exist between multiple sources. The AQUA object algebra is used as the formal foundation for manipulation of the query expression over the multidatabase. AQUA is enhanced with distribution operators for dealing with data distribution issues. During query decomposition we perform an extensive analysis of traversals for path expressions that involve virtual relationships and hierarchies for access to several heterogeneous sources. The distribution operators defined in algebraic terms enhance the global algebra expression with semantic information about the structure, distribution, and localization of the data sources relevant to the solution of the query. By using an object algebra as the basis for query processing, we are able to define algebraic transformations and exploit rewriting techniques during the decomposition phase. Our use of an object algebra also provides a formal and uniform representation for dealing with an object-oriented approach to multidatabase query processing. As part of our query processing discussion, we include an overview of a global object identification approach for relating semantically equivalent objects from diverse data sources, illustrating how knowledge about global object identity is used in the decomposition and assembly processes.     

Supporting geographically-aware web document foraging and sensemaking

This paper reports on the development and application of strategies and tools for geographic information seeking and knowledge building that leverages unstructured text resources found on the web. Geographic knowledge building from unstructured web sources starts with web document foraging during which the quantity, scope and diversity of web-based information create incredible cognitive burdens on an analyst’s or researcher’s ability to judge information relevancy. Determining information relevancy is ultimately a process of sensemaking. In this paper, we present our research on visually supporting web document foraging and sensemaking. In particular, we present the Sense-of-Place (SensePlace) analytic environment. The scientific goal of SensePlace is to visually and computationally support analyst sensemaking with text artifacts that have potential place, time, and thematic relevance to an analytical problem through identification and visual highlighting of named entities (people, places, times, and organizations) in documents, automated inference to determine document relevance using stored knowledge, and a visual interface with coupled geographic map, timeline, and concept graph displays that are used to contextualize the contexts of potentially relevant documents. We present the results of a case study analysis using SensePlace to uncover potential population migration, geopolitical, and other infectious disease dynamics drivers for measles and other epidemics in Niger. Our analysis allowed us to demonstrate how our approach can support analysis of complex situations along (a) multi-scale geographic dimensions (i.e., vaccine coverage areas), (b) temporal dimensions (i.e., seasonal population movement and migrations), and (c) diverse thematic dimensions (effects of political upheaval, food security, transient movement, etc.).     

Semantic information and knowledge integration through argumentative reasoning to support intelligent decision making

The availability of integrated, high quality information is a pre-requisite for a decision support system (DSS) to aid in the decision-making process. The introduction of semantic web ensures the seamless integration of information derived from diverse sources and transforms the DSS to an adoptable and flexible Semantic Web-DSS (Web-DSS). However, due to the monotonic nature of the layered development of semantic web, it lacks the capability to represent, reason and integrate incomplete and conflicting information. This, in turn, renders an enterprise incapable of knowledge integration; that is, integration of information about a subject that could potentially be incomplete, inconsistent and distributed among different Web-DSS within or across enterprises. In this article, we address the issues of incomplete and inconsistent semantic information and knowledge integration by using argumentation and argumentation schemes. We discuss the Argumentation-enabled Information Integration Web-DSS (Web@IDSS) along with its syntax and semantics for semantic information integration, and devise a methodology for sharing the results of Web@IDSS in Argument Interchange Format (AIF) format. We also discuss Argumentation-enabled Knowledge Integration Web-DSS (Web@KIDSS) for semantic knowledge integration. We provide formal syntax and semantics for the Web@KIDSS, propose a conceptual framework, and describe it in detail. We present the algorithms for knowledge integration and the prototype application for validation of results.     

Incremental maintenance of schema-restructuring views in SchemaSQL

The integration of data, especially from heterogeneous sources, is a hard and widely studied problem. One particularly challenging issue is the integration of sources that are semantically equivalent but schematically heterogeneous. While two such data sources may represent the same information, one may store the information inside tuples (data) while the other may store it in attribute or relation names (schema). The SchemaSQL query language is a recent solution to this problem powerful enough to restructure such sources into each other without the loss of information. We propose the first incremental view maintenance strategy for such schema-restructuring views. Our strategy, based on an algebraic representation of the view query, correctly transforms a data update or a schema change to a source into sequences of schema and data updates to be applied to the view. We also introduce an optimization of incremental maintenance using batching. We present a proof of correctness of the propagation approach. We also describe the implementation of our SchemaSQL Query Processor and View Maintainer. Last, our experimental results demonstrate that, in many cases, incremental SchemaSQL view maintenance is significantly faster than complete view recomputation.     

A distributed problem-solving approach to collaborative facility engineering

A framework for collaborative facility engineering is presented. The framework is based on a distributed problem-solving approach to collaborative facility engineering and employs an integration approach called Agent-Based Software Engineering as an implementation vehicle of this approach. The focal entity of this framework is a Multiagent Design Team (MDT) that comprises a collection of software agents (e.g. design software applications with a certain standard communication interface) and a design specialist, which together perform specific design tasks. Multiagent design teams are autonomous and form an organizational structure based on a federation architecture. Every multiagent design team surrenders its autonomy to a system program called facilitator, which coordinates the interaction among software agents in the federation architecture. Facilitators can be viewed as representatives of one or more teams that facilitate the exchange of design information and knowledge in support of the design tasks they perform. In the federation architecture, design specialists collaborate by exchanging design information with others via their software agents, and by identifying and resolving design conflicts by negotiation. In addition to a discussion of the framework's primary components, its realization in an integrated distributed environment for collaborative building engineering is described.     

Interactive focus + context medical data exploration and editing

Volumetric datasets are increasingly used in medical applications. In many of these applications, visualization and interaction is generally performed on cross‐sectional two‐dimensional (2D) views of three‐dimensional (3D) imaging modalities. Displaying 3D volumetric medical datasets on traditional 2D screens can present problems such as occlusion and information overload, especially when multiple data sources are present. Displaying desired information while showing the relationship to the rest of the dataset(s) can be challenging. In this paper, we present an interactive focus + context visualization approach that uses the volumetric Magic Lens interaction paradigm. We propose to use the Magic Lens as a volumetric brush to perform volume editing tasks, therefore combining data exploration with volumetric editing. Polygon‐assisted ray casting methods are used for real‐time rendering and editing frame rates, while providing compact storage of editing states for undo/redo operations. We discuss the application of our methods to radiation therapy, which is an important cancer treatment modality. We envision that this approach will improve the treatment planning process by improving the therapists' understanding of information from various sources and will help identify if the alignment of the patient in the treatment room coincides with the prepared treatment plan. Copyright © 2013 John Wiley & Sons, Ltd.     

Ontology-based information extraction and integration from heterogeneous data sources

In this paper we present the design, implementation and evaluation of SOBA, a system for ontology-based information extraction from heterogeneous data resources, including plain text, tables and image captions. SOBA is capable of processing structured information, text and image captions to extract information and integrate it into a coherent knowledge base. To establish coherence, SOBA interlinks the information extracted from different sources and detects duplicate information. The knowledge base produced by SOBA can then be used to query for information contained in the different sources in an integrated and seamless manner. Overall, this allows for advanced retrieval functionality by which questions can be answered precisely. A further distinguishing feature of the SOBA system is that it straightforwardly integrates deep and shallow natural language processing to increase robustness and accuracy. We discuss the implementation and application of the SOBA system within the SmartWeb multimodal dialog system. In addition, we present a thorough evaluation of the different components of the system. However, an end-to-end evaluation of the whole SmartWeb system is out of the scope of this paper and has been presented elsewhere by the SmartWeb consortium.