An innovative web-based collaborative platform for video annotation

Large scale labeled datasets are of key importance for the development of automatic video analysis tools as they, from one hand, allow multi-class classifiers training and, from the other hand, support the algorithms’ evaluation phase. This is widely recognized by the multimedia and computer vision communities, as witnessed by the growing number of available datasets; however, the research still lacks in annotation tools able to meet user needs, since a lot of human concentration is necessary to generate high quality ground truth data. Nevertheless, it is not feasible to collect large video ground truths, covering as much scenarios and object categories as possible, by exploiting only the effort of isolated research groups. In this paper we present a collaborative web-based platform for video ground truth annotation. It features an easy and intuitive user interface that allows plain video annotation and instant sharing/integration of the generated ground truths, in order to not only alleviate a large part of the effort and time needed, but also to increase the quality of the generated annotations. The tool has been on-line in the last four months and, at the current date, we have collected about 70,000 annotations. A comparative performance evaluation has also shown that our system outperforms existing state of the art methods in terms of annotation time, annotation quality and system’s usability.     

Data Comets: Designing a Visualization Tool for Analyzing Autonomous Aerial Vehicle Logs with Grounded Evaluation

Autonomous unmanned aerial vehicles are complex systems of hardware, software, and human input. Understanding this complexity is key to their development and operation. Information visualizations already exist for exploring flight logs but comprehensive analyses currently require several disparate and custom tools. This design study helps address the pain points faced by autonomous unmanned aerial vehicle developers and operators. We contribute: a spiral development process model for grounded evaluation visualization development focused on progressively broadening target user involvement and refining user goals; a demonstration of the model as part of developing a deployed and adopted visualization system; a data and task abstraction for developers and operators performing post-flight analysis of autonomous unmanned aerial vehicle logs; the design and implementation of Data Comets , an open-source and web-based interactive visualization tool for post-flight log analysis incorporating temporal, geospatial, and multivariate data; and the results of a summative evaluation of the visualization system and our abstractions based on in-the-wild usage. A free copy of this paper and source code are available at osf.io/h4p7g     

Product interface design: A participatory approach based on virtual reality

The usability of the user interface is a key aspect for the success of several industrial products. This assumption has led to the introduction of numerous design methodologies addressed to evaluate the user-friendliness of industrial products. Most of these methodologies follow the participatory design approach to involve the user in the design process. Virtual Reality is a valid tool to support Participatory Design, because it facilitates the collaboration among designers and users.The present study aims to evaluate the feasibility and the efficacy of an innovative Participatory Design approach where Virtual Reality plays a ‘double role’: a tool to evaluate the usability of the virtual product interface, and a communication channel that allows users to be directly involved in the design process as co-designers.In order to achieve these goals, we conducted three experiments: the purpose of the first experiment is to determine the influence of the virtual interface on the usability evaluation by comparing “user–real product” interaction and “user–virtual product” interaction. Subsequently, we tested the effectiveness of our approach with two experiments involving users (directly or through their participation in a focus group) in the redesign of a product user interface. The experiments were conducted with two typologies of consumer appliances: a microwave oven and a washing machine.     

Touching annotations: A visual metaphor for navigation of annotation in digital documents

Direct touch manipulation interactions with technology are now commonplace and significant interest is building around their use in the culture and heritage domain. Such interactions can give people the opportunity to explore materials and artefacts in ways that would otherwise be unavailable. These are often heavily annotated and can be linked to a large array of related digital content, thus enriching the experience for the user. Research has addressed issues of how to present digital documents and their related annotations but at present it is unclear what the optimal interaction approach to navigating these annotations in a touch display context might be.In this paper we investigate the role of two alternative approaches to support the navigation of annotations in digitised documents in the context of a touch interface. Through a control study we demonstrate that, whilst the navigation paradigm displays a significant interaction with the type of annotations task performed, there is no discernible advantage of using a natural visual metaphor for annotation in this context. This suggests that design of digital document annotation navigation tools should account for the context and navigation tasks being considered.     

Ontology-driven collaborative annotation in shared workspaces

In the last decade, collaboration and sharing on the Web have become mainstream. Digital, remote interaction happens on a daily basis, not only to share digital resources, but also to create, manage and discuss them, in every possible situation where collaboration is required: from work teams to groups of friends, from community committees to no-profit organizations. In this paper we address the task of collaborative management of digital resources within a team, with a special focus on the task of semantic annotation, where team members, possibly supported by automated reasoning, enrich resources with properties that help in organizing, retrieving and creating connections between contents of different types. We focus in particular on the problem of reaching an agreement on the annotation itself among the participants. The paper presents a qualitative user study aimed at observing users behavior when faced with this task. The results of the study are then analyzed in order to draw guidelines, which are then implemented in a tool for collaborative annotation. This study is carried out in the context of the Semantic Table Plus Plus (Sem T++) Project, a framework supporting collaboration over thematic workspaces, whose goal is to enhance cooperation through awareness, enhanced communication and easy sharing of digital content.     

Precedence-constrained path planning of messenger UAV for air-ground coordination

This paper addresses an unmanned aerial vehicle (UAV) path planning problem for a team of cooperating heterogeneous vehicles composed of one UAV and multiple unmanned ground vehicles (UGVs). The UGVs are used as mobile actuators and scattered in a large area. To achieve multi-UGV communication and collaboration, the UAV serves as a messenger to fly over all task points to collect the task information and then flies all UGVs to transmit the information about tasks and UGVs. The path planning of messenger UAV is formulated as a precedence-constrained dynamic Dubins traveling salesman problem with neighborhood (PDDTSPN). The goal of this problem is to find the shortest route enabling the UAV to fly over all task points and deliver information to all requested UGVs. When solving this path planning problem, a decoupling strategy is proposed to sequentially and rapidly determine the access sequence in which the UAV visits task points and UGVs as well as the access location of UAV in the communication neighborhood of each task point and each UGV. The effectiveness of the proposed approach is corroborated through computational experiments on randomly generated instances. The computational results on both small and large instances demonstrate that the proposed approach can generate high-quality solutions in a reasonable time as compared with two other heuristic algorithms.     

Situated cognitive engineering for crew support in space

Space crews are in need for excellent cognitive support to perform nominal and off-nominal actions. This paper presents a coherent cognitive engineering methodology for the design of such support, which may be used to establish adequate usability, context-specific support that is integrated into astronaut’s task performance and/or electronic partners who enhance human–machine team’s resilience. It comprises (a) usability guidelines, measures and methods, (b) a general process guide that integrates task procedure design into user interface design and a software framework to implement such support and (c) theories, methods and tools to analyse, model and test future human–machine collaborations in space. In empirical studies, the knowledge base and tools for crew support are continuously being extended, refined and maintained.     

Network-aware support for mobile distributed teams

An experiment evaluated network-aware support to increase understanding of the factors that are important for successful teamwork in mobile geographically dispersed teams of first responders. Participants performed a simulated search and rescue team task and were equipped with a digitized map and real-time situation updates on the location of other participants in a simulated disaster area. The connection to a server, however, was made deliberately error-prone, leading to occasional losses of network connections. Consequently, participants were not provided with real-time situation updates. To deal with this problem we equipped team members with a network-aware application that signaled network loss to them and adapted the graphical representation of the location of fellow team members accordingly to the quality of location information present. The experiment revealed that presenting complete and reliable geospatial information improves teamwork. Teams connected to a server over a fast and reliable link showed superior performance over teams with no network connection whatsoever to a server. The present study failed, however, to demonstrate the added value of network-aware support when teams had to collaborate in the presence of an unreliable communications infrastructure. Although participants indicated a slight preference for the network-aware application over a condition without support signaling network loss, no differences were observed in team process and outcome measures.     

A collaborative augmented reality annotation tool for the inspection of prefabricated buildings

The inspection of prefabricated buildings involves different stages and tasks such as the collection of measurements, the visual inspection of components and the written annotation of defects. Traditionally, inspectors have documented the process, the kind of defects and the proposed correction measures in paper format, hindering the collaboration with other experts (either simultaneously or asynchronously) and the collection of other types of annotations (e.g. images, 3D elements). In this paper, we present an AR tool designed to aid inspectors during this process. The tool has many benefits, as it allows simultaneously performing a collaborative inspection, taking multitype and geolocated annotations, their monitoring and edition, and performing in situ augmented visualizations. The quantitative and qualitative user evaluation carried out with our tool in a real environment (including usability and satisfaction evaluations) shows the relevance that such a technology might bring to the field and prove that our tool is usable and fulfils most of the inspectors’ expectations.

     

Managing design ideas with a shared drawing tool

The generation of design ideas in group discussion is a complex and dynamic process. The fluent expression of ideas and the ability to interact and build on representations created by others contributes significantly to the process. When group members are separated by physical distance, the fluency of this process is interrupted. Computerized shared drawing tools re-establish this fluency. Such tools need to aid not only the drawing process, but also the management of design ideas during the process. The paper lays the groundwork for the design of the idea management portion of a shared drawing tool. It presents a list of group idea management behaviours and identifies user requirements in support of these activities. A prototype shared drawing system is described which attempts to incorporate the user requirements into an interface design. The paper then presents the results of a usability study conducted on the prototype. The interface performed miserably, but the study provided support for the user requirements and gave detailed insight into how a shared drawing interface should be designed.     

Eras: Improving the quality control in the annotation process for Natural Language Processing tasks

The increasing amount of valuable, unstructured textual information poses a major challenge to extract value from those texts. We need to use NLP (Natural Language Processing) techniques, most of which rely on manually annotating a large corpus of text for its development and evaluation. Creating a large annotated corpus is laborious and requires suitable computational support. There are many annotation tools available, but their main weaknesses are the absence of data management features for quality control and the need for a commercial license. As the quality of the data used to train an NLP model directly affects the quality of the results, the quality control of the annotations is essential. In this paper, we introduce ERAS, a novel web-based text annotation tool developed to facilitate and manage the process of text annotation. ERAS includes not only the key features of current mainstream annotation systems but also other features necessary to improve the curation process, such as the inter-annotator agreement, self-agreement and annotation log visualization, for annotation quality control. ERAS also implements a series of features to improve the customization of the user’s annotation workflow, such as: random document selection, re-annotation stages, and warm-up annotations. We conducted two empirical studies to evaluate the tool’s support to text annotation, and the results suggest that the tool not only meets the basic needs of the annotation task but also has some important advantages over the other tools evaluated in the studies. ERAS is freely available at https://github.com/grosmanjs/eras.     

A Cognitive Modeling Approach to Decision Support Tool Design for Anesthesia Provider Crisis Management

Prior research has revealed existing operating room (OR) patient monitors to provide limited support for prompt and accurate decision making by anesthesia providers during crises. Decision support tools (DSTs) developed for this purpose typically alert the anesthesia provider to existence of a problem but do not recommend a treatment plan. There is a need for a human-centered approach to the design and development of a crisis management DST. A hierarchical task analysis was conducted to identify anesthesia provider procedures in detecting, diagnosing, and treating a critical incident and a cognitive task analysis to elicit goals, decisions, and information requirements. This information was coded in a computational cognitive model using GOMS (Goals, Operators, Methods, Selection rules) Language. An OR monitor interface was prototyped to present output from the cognitive model following ecological interface design principles. A preliminary assessment of the DST was performed with anesthesiology and usability experts. The anesthesiologists indicated they would use the tool in the perioperative environment and would recommend its use by junior anesthesia providers. Future research will focus on formal validation of the DST design approach and comparison of tool output to actual anesthesia provider decisions in real or simulated crises.     

Using a hand-drawn sketch to control a team of robots

In this paper, we describe a prototype interface that facilitates the control of a mobile robot team by a single operator, using a sketch interface on a Tablet PC. The user draws a sketch map of the scene and includes the robots in approximate starting positions. Both path and target position commands are supported as well as editing capabilities. Sensor feedback from the robots is included in the display such that the sketch interface acts as a two-way communication device between the user and the robots. The paper also includes results of a usability study, in which users were asked to perform a series of tasks.     

A method and advisor tool for multimedia user interface design

This paper describes a multimedia user interface design method and a design assistant tool which supports the method. The method covers specification of user requirements and information architecture, selection of appropriate media to represent the information content, design for directing attention to important information and interaction design to enhance user engagement. Guidelines for media selection and design for attractiveness, i.e. usability and user experience, are given. The method was evaluated in a case study design of a crowd control simulation training system, which demonstrated the method was usable and gave good solutions against an expert gold standard design. The tool provides advice on media selection and attention effects that match specification of the information content expressed as information types and communication goals. A usability evaluation was carried out to measure the usefulness and effectiveness of the tool in comparison to the method, and the results showed that the tool has a positive impact on multimedia design.     

A graphical tool for parasite genome annotation

A graphical tool to facilitate rapid primary annotation of genomic sequence has been developed. Within a single interface the user can import sequences or database entries, run feature prediction programs and similarity searches, filter results, add additional manually found features and notes, and finally export annotations for database submission. Integrated rule-based feature corroboration and a novel decision support heuristic using ORF orientation, length and base-composition further enhances the efficiency of the annotation process without compromising flexibility. The program has been explicitly tailored to use in protozoan parasite genome projects, but can constitute a useful tool for prokaryote annotation as well. It is successfully being used by our lab in the Trypanosoma cruzi genome project, and can be obtained from the authors upon request.     

Systematic evaluation of design choices for software development tools

Most design and evaluation of software tools is based on the intuition and experience of the designers. Software tool designers consider themselves typical users of the tools that they build and tend to subjectively evaluate their products rather than objectively evaluate them using established usability methods. This subjective approach is inadequate if the quality of software tools is to improve and the use of more systematic methods is advocated. This paper summarises a sequence of studies that show how user interface design choices for software development tools can be evaluated using established usability engineering techniques. The techniques used included guideline review, predictive modelling and experimental studies with users.     

External tools for collaborative medication scheduling

Medication adherence—taking medication as prescribed—is critical for successful self-care, especially among older adults. Adherence depends on developing and implementing plans for taking medications. Age-related cognitive declines that affect adherence may be mitigated using external tools that support patient-provider collaboration needed to develop these adherence plans. We tested the effectiveness of structured collaborative medication tools to support better medication planning and adherence practices. Evidence for benefits of collaborative tools has been mixed in terms of their usefulness for medication-scheduling tasks, perhaps because the tools have not been explicitly designed to support patient-provider collaboration. A total of 144 community-dwelling older adults participated in pairs and performed the role of a patient or provider in a simulated patient-provider medication-scheduling task. Each pair worked with a structured (MedTable^? and e-MedTable) or unstructured (Medcard) scheduling tool and completed four problems (2 simple and 2 complex). Performance was measured using the following: problem-solving (medication schedule) accuracy, problem-solving time, solution (schedule) optimality, tool usability, collaborative effectiveness, and subjective workload involved in creating the medication schedules. Participants using structured tools produced more accurate and optimal schedules. They also rated subjective workload as lower and thought that the structured tools were easier to use, reduced subjective workload associated with creating the schedules. There was also suggestive evidence that participants using the structured tools rated more highly the quality of their collaboration. Structured medication-scheduling tools have the potential to improve medication adherence among older adults because they support collaborative planning and reduce the cognitive load involved in creating these adherence plans.     

Collaborative video searching on a tabletop

Almost all system and application design for multimedia systems is based around a single user working in isolation to perform some task yet much of the work for which we use computers to help us, is based on working collaboratively with colleagues. Groupware systems do support user collaboration but typically this is supported through software and users still physically work independently. Tabletop systems, such as the DiamondTouch from MERL, are interface devices which support direct user collaboration on a tabletop. When a tabletop is used as the interface for a multimedia system, such as a video search system, then this kind of direct collaboration raises many questions for system design. In this paper we present a tabletop system for supporting a pair of users in a video search task and we evaluate the system not only in terms of search performance but also in terms of user–user interaction and how different user personalities within each pair of searchers impacts search performance and user interaction. Incorporating the user into the system evaluation as we have done here reveals several interesting results and has important ramifications for the design of a multimedia search system.     

IoT-Enabled Autonomous System Collaboration for Disaster-Area Management

Timely investigating post-disaster situations to locate survivors and secure hazardous sources is critical, but also very challenging and risky. Despite first responders putting their lives at risk in saving others, human-physical limits cause delays in response time, resulting in fatality and property damage. In this paper, we proposed and implemented a framework intended for creating collaboration between heterogeneous unmanned vehicles and first responders to make search and rescue operations safer and faster. The framework consists of unmanned aerial vehicles (UAVs), unmanned ground vehicles (UGVs), a cloud-based remote control station (RCS). A light-weight message queuing telemetry transport (MQTT) based communication is adopted for facilitating collaboration between autonomous systems. To effectively work under unfavorable disaster conditions, antenna tracker is developed as a tool to extend network coverage to distant areas, and mobile charging points for the UAVs are also implemented. The proposed framework’s performance is evaluated in terms of end-to-end delay and analyzed using architectural analysis and design language (AADL). Experimental measurements and simulation results show that the adopted communication protocol performs more efficiently than other conventional communication protocols, and the implemented UAV control mechanisms are functioning properly. Several scenarios are implemented to validate the overall effectiveness of the proposed framework and demonstrate possible use cases.