An object oriented user interface for analysis of biological data

In a previous paper we described a self-documented file and a collection of general purpose programs or tools that facilitates the management and analysis of biological data. The tools can be specified in a pipeline to accomplish a specific analysis task. However, we found that it was difficult for investigators to learn the UNIX command language for specifying pipelines, specify selection tasks through a command language, and visualize the data as they were transformed and rearranged. To alleviate these problems we developed an object-oriented user interface for the pipeline programs. The system consists of four major programs for visualization: Vedit, Vgraf, Vscan, and V spread. Vedit is a simple text editor, Vgraf is a flexible graphics program, Vscan facilitates scanning graphically through large files, and Vspread provides spreadsheet-like capabilities. To demonstrate how the visualization programs are used together to accomplish the needed analysis we describe two case studies and then discuss how well the system accomplished the goals of visualization, short learning curve, and user adaptability.     

Trans-Proteomic Pipeline,a standardized data processing pipeline for large-scale reproducible proteomics informatics

Democratization of genomics technologies has enabled the rapid determination of genotypes. More recently the democratization of comprehensive proteomics technologies is enabling the determination of the cellular phenotype and the molecular events that define its dynamic state. Core proteomic technologies include MS to define protein sequence, protein:protein interactions, and protein PTMs. Key enabling technologies for proteomics are bioinformatic pipelines to identify, quantitate, and summarize these events. The Trans-Proteomics Pipeline (TPP) is a robust open-source standardized data processing pipeline for large-scale reproducible quantitative MS proteomics. It supports all major operating systems and instrument vendors via open data formats. Here, we provide a review of the overall proteomics workflow supported by the TPP, its major tools, and how it can be used in its various modes from desktop to cloud computing. We describe new features for the TPP, including data visualization functionality. We conclude by describing some common perils that affect the analysis of MS/MS datasets, as well as some major upcoming features.     

基于本体的动态工作流系统

项目采办过程中通常涉及很多企业和组织,信息量大,时问长,工作流过程控制比较复杂。本工作流系统把工作流模型以知识的形式存储在本体知识库中,用户可以通过知识库服务器动态修改过程模型,实现它的动态运转。组织模型和资源模型也能够随着根据业务的变化及时做出调整。同时可以利用逻辑推理功能插件扩展对知识的推理功能,在任务转移提供辅助信息,实现工作流的智能运转。     

Exploring temporal behaviour of app users completing ecological momentary assessments using mental health scales and mood logs

ABSTRACT

Smartphone-based digital phenotyping can provide insight into mood, cognition and behaviour. In this study, data analytics was carried out with data generated from a maternal mental health app to address the following question: what is the temporal behaviour of users when completing ecological momentary assessments (EMAs) with EMAs in the form of mental health scales versus EMAs in the form of mood logs? The methodology involved using the Health Interaction Log Data Analytics (HILDA) pipeline to analyse 1461 app users. Clustering was used to characterise archetypical user engagement with the two forms of EMA. Users preferred mood log EMAs, with 6993 mood log completions compared to 2129 scale completions. Users are more willing to log moods at 9am and 12pm and complete mental health scales between 8pm and 10pm. The fewest number of mood logs and scale completions take place on Saturday followed by a Sunday. Whilst ‘happiness’ is the dominant mood during day times, ‘anxiety’ and ‘sadness’ peak during night times. The overall findings are that users prefer completing mood log EMAs and that the temporal behaviour of users engaging with EMAs in the form of mental health scales are distinctly different from how they engage with mood logs.     

Fine-grained visualization pipelines and lazy functional languages

The pipeline model in visualization has evolved from a conceptual model of data processing into a widely used architecture for implementing visualization systems. In the process, a number of capabilities have been introduced, including streaming of data in chunks, distributed pipelines, and demand-driven processing. Visualization systems have invariably built on stateful programming technologies, and these capabilities have had to be implemented explicitly within the lower layers of a complex hierarchy of services. The good news for developers is that applications built on top of this hierarchy can access these capabilities without concern for how they are implemented. The bad news is that by freezing capabilities into low-level services expressive power and flexibility is lost. In this paper we express visualization systems in a programming language that more naturally supports this kind of processing model. Lazy functional languages support fine-grained demand-driven processing, a natural form of streaming, and pipeline-like function composition for assembling applications. The technology thus appears well suited to visualization applications. Using surface extraction algorithms as illustrative examples, and the lazy functional language Haskell, we argue the benefits of clear and concise expression combined with fine-grained, demand-driven computation. Just as visualization provides insight into data, functional abstraction provides new insight into visualization.     

Two-stage optimization for machine learning workflow

Machine learning techniques play a preponderant role in dealing with massive amount of data and are employed in almost every possible domain. Building a high quality machine learning model to be deployed in production is a challenging task, from both, the subject matter experts and the machine learning practitioners.For a broader adoption and scalability of machine learning systems, the construction and configuration of machine learning workflow need to gain in automation. In the last few years, several techniques have been developed in this direction, known as AutoML.In this paper, we present a two-stage optimization process to build data pipelines and configure machine learning algorithms. First, we study the impact of data pipelines compared to algorithm configuration in order to show the importance of data preprocessing over hyperparameter tuning. The second part presents policies to efficiently allocate search time between data pipeline construction and algorithm configuration. Those policies are agnostic from the metaoptimizer. Last, we present a metric to determine if a data pipeline is specific or independent from the algorithm, enabling fine-grain pipeline pruning and meta-learning for the coldstart problem.     

Generating Semantic 3D Models of Underground Infrastructure

By combining semantic scene-graph markups with generative modeling, this framework retains semantic information late in the rendering pipeline. It can thus enhance visualization effects and interactive behavior without compromising interactive frame rates. Large geospatial databases are populated with the results of hundreds of person-years of surveying effort. Utility workers access these databases during fieldwork to help them determine asset location. Real-time rendering engines are highly advanced and optimized software toolkits that interactively display 3D information to users.To connect geospatial databases and rendering engines, we must transcode raw 2D geospatial data into 3D models suitable for standard rendering engines. Thus, transcoding isn't simply a one-to-one conversion from one format to another; we obtain 3D models from 2D information through procedural 3D modeling. Transcoding the geospatial database information's semantic attributes into visual primitives entails information loss. We must therefore find the right point in the pipeline to perform transcoding.     

Navi: Data Analysis System Powered by Natural Language Interaction

With the advent of the big data era, the significance of data analysis has increasingly come to the forefront, showcasing its ability to uncover valuable insights from vast datasets, thereby enhancing the decision-making process for users. Nonetheless, the data analysis workflow faces three dominant challenges: high coupling in the analysis workflow, a plethora of interactive interfaces, and a time-intensive exploratory analysis process. To address these challenges, we introduce with this paper Navi, a data analysis system powered by natural language interaction. Navi embraces a modular design philosophy that abstracts three core functional modules from mainstream data analysis workflows: data querying, visualization generation, and visualization exploration. This approach effectively reduces the coupling of the system. Meanwhile, Navi leverages natural language as a unified interactive interface to seamlessly integrate various functional modules through a task scheduler, ensuring their effective collaboration. Moreover, in order to address the challenges of exponential search space and ambiguous user intent in visualization exploration, we propose an automated approach for visualization exploration based on Monte Carlo tree search. In addition, a pruning algorithm and a composite reward function, both incorporating visualization domain knowledge, are devised to enhance the search efficiency and result quality. Finally, we validate the effectiveness of Navi through both quantitative experiments and user studies.     

As You Like It: Personalized Database Visualization Using a Visual Language

The Delaunay system supports a visual language that is specifically geared to the querying and visualization of databases. In this paper, we concentrate on the information visualization capabilities of the system. A distinctive feature of Delaunay is its full personalization capabilities: users can define their visualizations from scratch without limiting themselves to pre-defined visualization modes. Fine customization of the visualization is achieved by the availability of a visual alphabet of atomic graphical symbols and by the expressive power of the visual query language, which supports recursion.We describe the key components of the Delaunay system, namely its interface modules, which support advanced visualization techniques and principles, and its efficient constraint solver. The successful implementation of Delaunay demonstrates the feasibility of a powerful database system with which users can effectively specify a wide variety of visualizations supporting data and visualization exploration for different kinds of applications including graph visualization and business analysis.     

Visual signatures in video visualization

Video visualization is a computation process that extracts meaningful information from original video data sets and conveys the extracted information to users in appropriate visual representations. This paper presents a broad treatment of the subject, following a typical research pipeline involving concept formulation, system development, a path-finding user study, and a field trial with real application data. In particular, we have conducted a fundamental study on the visualization of motion events in videos. We have, for the first time, deployed flow visualization techniques in video visualization. We have compared the effectiveness of different abstract visual representations of videos. We have conducted a user study to examine whether users are able to learn to recognize visual signatures of motions, and to assist in the evaluation of different visualization techniques. We have applied our understanding and the developed techniques to a set of application video clips. Our study has demonstrated that video visualization is both technically feasible and cost-effective. It has provided the first set of evidence confirming that ordinary users can be accustomed to the visual features depicted in video visualizations, and can learn to recognize visual signatures of a variety of motion events.     

“Search, Show Context, Expand on Demand”: Supporting Large Graph Exploration with Degree-of-Interest

A common goal in graph visualization research is the design of novel techniques for displaying an overview of an entire graph. However, there are many situations where such an overview is not relevant or practical for users, as analyzing the global structure may not be related to the main task of the users that have semi-specific information needs. Furthermore, users accessing large graph databases through an online connection or users running on less powerful (mobile) hardware simply do not have the resources needed to compute these overviews. In this paper, we advocate an interaction model that allows users to remotely browse the immediate context graph around a specific node of interest. We show how Furnas' original degree of interest function can be adapted from trees to graphs and how we can use this metric to extract useful contextual subgraphs, control the complexity of the generated visualization and direct users to interesting datapoints in the context. We demonstrate the effectiveness of our approach with an exploration of a dense online database containing over 3 million legal citations.     

Time dependent processing in a parallel pipeline architecture

Pipeline architectures provide a versatile and efficient mechanism for constructing visualizations, and they have been implemented in numerous libraries and applications over the past two decades. In addition to allowing developers and users to freely combine algorithms, visualization pipelines have proven to work well when streaming data and scale well on parallel distributed-memory computers. However, current pipeline visualization frameworks have a critical flaw: they are unable to manage time varying data. As data flows through the pipeline, each algorithm has access to only a single snapshot in time of the data. This prevents the implementation of algorithms that do any temporal processing such as particle tracing; plotting over time; or interpolation, fitting, or smoothing of time series data. As data acquisition technology improves, as simulation time-integration techniques become more complex, and as simulations save less frequently and regularly, the ability to analyze the time-behavior of data becomes more important. This paper describes a modification to the traditional pipeline architecture that allows it to accommodate temporal algorithms. Furthermore, the architecture allows temporal algorithms to be used in conjunction with algorithms expecting a single time snapshot, thus simplifying software design and allowing adoption into existing pipeline frameworks. Our architecture also continues to work well in parallel distributed-memory environments. We demonstrate our architecture by modifying the popular VTK framework and exposing the functionality to the ParaView application. We use this framework to apply time-dependent algorithms on large data with a parallel cluster computer and thereby exercise a functionality that previously did not exist.     

Pipeline provenance for cloud-based big data analytics

Provenance is information about the origin and creation of data. In data science and engineering related with cloud environment, such information is useful and sometimes even critical. In data analytics, it is necessary for making data-driven decisions to trace back history and reproduce final or intermediate results, even to tune models and adjust parameters in a real-time fashion. Particularly, in cloud, users need to evaluate data and pipeline trustworthiness. In this paper, we propose a solution: LogProv, toward realizing these functionalities for big data provenance, which needs to renovate data pipelines or some of big data software infrastructure to generate structured logs for pipeline events, and then stores data and logs separately in cloud space. The data are explicitly linked to the logs, which implicitly record pipeline semantics. Semantic information can be retrieved from the logs easily since they are well defined and structured beforehand. We implemented and deployed LogProv in Nectar Cloud,^* associated with Apache Pig, Hadoop ecosystem, and adopted Elasticsearch to provide query service. LogProv was evaluated and empirically case studied. The results show that LogProv is efficient since the performance overhead is no more than 10%; the query can be responded within 1 second; the trustworthiness is marked clearly; and there is no impact on the data processing logic of original pipelines.     

基于OpenGL的空间管线的拼接显示与应用

空间管线是城市和矿山的重要基础设施。三维管线建模与可视化是构建三雏“数字城市”和“数字矿山”、实现城市现代化管理中不可或缺的重要组成部分。本文首先介绍管 线应用的现状,然后阐述了管线数据结构和相邻管线的拼接过程,提出了管线图形对象先存储拼接计算的结果。最后,介绍了基于Visual C＋＋和OpenGL技术所开发的系统。     

VDVR: verifiable visualization of projection-based data

Practical volume visualization pipelines are never without compromises and errors. A delicate and often-studied component is the interpolation of off-grid samples, where aliasing can lead to misleading artifacts and blurring, potentially hiding fine details of critical importance. The verifiable visualization framework we describe aims to account for these errors directly in the volume generation stage, and we specifically target volumetric data obtained via computed tomography (CT) reconstruction. In this case the raw data are the X-ray projections obtained from the scanner and the volume data generation process is the CT algorithm. Our framework informs the CT reconstruction process of the specific filter intended for interpolation in the subsequent visualization process, and this in turn ensures an accurate interpolation there at a set tolerance. Here, we focus on fast trilinear interpolation in conjunction with an octree-type mixed resolution volume representation without T-junctions. Efficient rendering is achieved by a space-efficient and locality-optimized representation, which can straightforwardly exploit fast fixed-function pipelines on GPUs.     

Prediction of individual learning curves across information visualizations

Confident usage of information visualizations is thought to be influenced by cognitive aspects as well as amount of exposure and training. To support the development of individual competency in visualization processing, it is important to ascertain if we can track users’ progress or difficulties they might have while working with a given visualization. In this paper, we extend previous work on predicting in real time a user’s learning curve—a mathematical model that can represent a user’s skill acquisition ability—when working with a visualization. First, we investigate whether results we previously obtained in predicting users’ learning curves during visualization processing generalize to a different visualization. Second, we study to what extent we can make predictions on a user’s learning curve without information on the visualization being used. Our models leverage various data sources, including a user’s gaze behavior, pupil dilation, and cognitive abilities. We show that these models outperform a baseline that leverages knowledge on user task performance so far. Our best performing model achieves good accuracies in predicting users’ learning curves even after observing users’ performance on a few tasks only. These results represent an important step toward understanding how to support users in learning a new visualization.     

埋地管道检测技术现状与展望

定期对埋地管道进行检测有利于管道的维护管理和完整性评价。对埋地管道常用的外检测和内检测技术进行了综述,包括用于管道缺陷检测的超声检测、涡流检测、漏磁检测和视觉检测,以及用于管道变形检测的通径检测法、超声波法、激光三角法、环形光投射成像法等。阐述了各检测方法的基本原理,总结了各自的优势与局限性,为检测技术的选择提供了有效参考。梳理了埋地管道检测技术的发展现状,特别是电磁超声、漏磁、视觉内检测技术的新进展,分析了管道检测的发展趋势,并给出了一些技术难点的解决思路。目前的研究热点表明,埋地管道检测技术正在朝着复杂化对象的检测新方法、缺陷定量化、检测数据可视化的方向发展。     

Albero: A Visual Analytics Approach for Probabilistic Weather Forecasting

Probabilistic weather forecasts are amongst the most popular ways to quantify numerical forecast uncertainties. The analog regression method can quantify uncertainties and express them as probabilities. The method comprises the analysis of errors from a large database of past forecasts generated with a specific numerical model and observational data. Current visualization tools based on this method are essentially automated and provide limited analysis capabilities. In this paper, we propose a novel approach that breaks down the automatic process using the experience and knowledge of the users and creates a new interactive visual workflow. Our approach allows forecasters to study probabilistic forecasts, their inner analogs and observations, their associated spatial errors, and additional statistical information by means of coordinated and linked views. We designed the presented solution following a participatory methodology together with domain experts. Several meteorologists with different backgrounds validated the approach. Two case studies illustrate the capabilities of our solution. It successfully facilitates the analysis of uncertainty and systematic model biases for improved decision‐making and process‐quality measurements.     

Selective Visualization of Vector Fields

In this paper, we present an approach to selective vector field visualization. This selective visualization approach consists of three stages: selectdon creation, selection processing and selective visualization mapping. It is described how selected regions, called selections, can be represented and created, how selections can be processed and how they can be used in the visualization mapping. Combination of these techniques with a standard visualization pipeline improves the visualization process and offers new facilities for visualization. Examples of selective visualization of fluid flow datasets are provided.