用于数据抽取的数据表末尾移动排序算法

在数据集成和分析处理中,数据抽取是一种常见操作,但是受参照完整性约束的限制,被参照的数据表必须优先抽取。本文提出的数据表末尾移动算法,解决了批量数据表排序问题,确保数据抽取过程中不违反参照完整性约束。     

基于转换技术的XML文档规范化及更新

XML已经成为互联网上数据表示、集成和转换的标准。作为一种半结构化数据，XML更新操作是扩展其查询能力的一个重要方面。为了保持原来的语义完整性，避免更新过程中的冗余和异常现象，作者提出了一种新的方法。该方法基于XML转换技术来规范XML文档，并基于规范化的XML，XNF来直接更新XML文档。该方法完成更新后，保持了XML的结构和原来的完整性约束。     

基于规范化关系模式生成XML的键

现实应用需要从关系数据库中提取数据并以XML的形式进行发布，需要利用定义在XML文档上的键检测发布后的XML文档是否满足原先关系数据库的语义及完整性约束。文章提出了生成XML键的方法，该方法由关系数据库中的键、外键约束构造规范化关系模式的约束概念层次图，并保持语义映射为有效的、完全的XML键约束。     

非确定性数据查询

数据库技术中尽管有完整性约束技术用来维护数据的确定性,但有些情况可能不满足完整性约束。本文提出了非确定性数据的确定性查询概念,给出的查询重写方法能有效地进行数据库SQL查询语句,它可以违反一系列的约束条件,重写查询去查找相应的与约束一致的数据。     

基于规则的关系数据库到本体的转换方法

提出了一种新的全自动的关系数据库到本体的转换方法,通过分析关系模式的主键、属性、引用关系、完整性约束和部分数据来创建本体,尽量保持了关系数据库的信息,并在构建的过程中对信息进行初步的集成和分类.系统实践证明,该方法可自动进行关系模式和数据到本体的等价转换,而且完成了对关系数据库中部分语义信息的辅助挖掘.     

数据库中域完整性的设计与应用

数据完整性约束在数据库设计过程中至关重要,是保障数据正确、一致的有效机制。其中,域完整性是保证数据完整性的重要约束之一,包括CHECK约束、默认值约束、默认值对象和规则等。笔者旨在深入探讨实现域完整性的方法和手段,详细介绍实现域完整性的不同方法和具体运用,从而使学习者能够掌握这些方法,实现数据库的完整性约束设计。     

时态数据完整性约束研究与实现

时态数据库的完整性是为了保证时态数据库中存储的时态数据的正确性。为了防止时态数据库中出现不符合时态模型语义的数据,需要研究时态数据库的时态完整性相关理论,为时态更新操作的正确执行提供理论支持。在时态关系数据模型的基础上,通过将传统的数据完整性约束机制扩展到时态数据库领域,给出了时态完整性约束的完整定义和在不同的情况下违反时态完整性约束的具体处理机制,为时态数据处理构件——TempDB实现时态数据完整性约束提供了理论基础和实现依据,这对时态数据库理论的进一步完善和相关的实现技术具有重要的参考价值。     

一种共享数据完整性验证方案的分析与改进

首先分析了一种云计算环境下公开可验证的共享数据完整性验证方案,指出其无法抵抗数据恢复攻击和完整性伪造攻击,从而存在数据泄露和完整性欺骗的风险。在此基础上提出一个安全性增强的改进方案并对其进行了安全性分析和性能开销分析,分析结果表明该方案保持了数据存储安全性和隐私保护性,并且能够高效实现共享数据的完整性验证,具备较高的安全性和验证效率。     

基于SQL Server数据库完整性约束的几点探讨

数据库的完整性约束是设计数据库的核心内容,一个数据库的完整性约束设计的好坏,直接影响到这个数据库的性能,同时也会影响数据库的开发,因此一个好的数据库需要严格考虑其完整性约束。本文对SQL Server 2000数据库中的完整性约束进行了探讨,并用T-SQL语句对数据库完整性约束进行了设计,从而实现了在数据库中存储数据的完整性、正确性和一致性。     

基于清洗规则和主数据的数据修复算法实现

为了有效地清洗数据,此前已经提出了很多的完整性约束规则,例如条件函数依赖、条件包含依赖.这些约束规则虽然可以侦测出错误的存在,但是不能有效地指导用户纠正错误.实际上,基于约束规则的数据修复可能最终得不到确定性的修复结果,相反会引入新的错误,因此很大程度上降低了数据修复的效率.针对以上不足,提出了一种有效的数据清洗框架:首先基于Editing Rules和Master Data对数据进行清洗操作,最终得到确定性的修复;然后依据条件函数依赖来修复遗漏的错误,此种修复结果是不确定的,但是相比之下该框架不仅可以有效地保证数据修复的精确性与唯一性,而且提高了数据修复的效率.     

A Taxonomy of Dirty Data

Today large corporations are constructing enterprise data warehouses from disparate data sources in order to run enterprise-wide data analysis applications, including decision support systems, multidimensional online analytical applications, data mining, and customer relationship management systems. A major problem that is only beginning to be recognized is that the data in data sources are often dirty. Broadly, dirty data include missing data, wrong data, and non-standard representations of the same data. The results of analyzing a database/data warehouse of dirty data can be damaging and at best be unreliable. In this paper, a comprehensive classification of dirty data is developed for use as a framework for understanding how dirty data arise, manifest themselves, and may be cleansed to ensure proper construction of data warehouses and accurate data analysis. The impact of dirty data on data mining is also explored.     

基于时效规则的数据修复方法

数据时效性是影响数据质量的重要因素,可靠的数据时效性对数据检索的精确度、数据分析结论的可信性起到关键作用.数据时效不精确、数据过时等现象给大数据应用带来诸多问题,很大程度上影响着数据价值的发挥.对于缺失了时间戳或者时间不准确的数据,精确恢复其时间戳是困难的,但可以依据一定的规则对其时间先后顺序进行还原恢复,满足数据清洗及各类应用需求.在数据时效性应用需求分析的基础上,首先明确了属性的时效规则相关概念,对属性的时效规则等进行了形式化定义;然后提出了基于图模型的时效规则发现以及数据时序修复算法;随后,对相关算法进行了实现,并在真实数据集上对算法运行效率、修复正确率等进行了测试,分析了影响算法修复数据正确率的一些影响因素,对算法进行了较为全面的分析评价.实验结果表明,算法具有较高的执行效率和较好的时效修复效果.     

Mixing numerical and categorical data in a Self-Organizing Map by means of frequency neurons

Even though Self-Organizing Maps (SOMs) constitute a powerful and essential tool for pattern recognition and data mining, the common SOM algorithm is not apt for processing categorical data, which is present in many real datasets. It is for this reason that the categorical values are commonly converted into a binary code, a solution that unfortunately distorts the network training and the posterior analysis. The present work proposes a SOM architecture that directly processes the categorical values, without the need of any previous transformation. This architecture is also capable of properly mixing numerical and categorical data, in such a manner that all the features adopt the same weight. The proposed implementation is scalable and the corresponding learning algorithm is described in detail. Finally, we demonstrate the effectiveness of the presented algorithm by applying it to several well-known datasets.     

零售业数据仓库中ETL技术的研究

数据仓库中的数据不是传统数据库中数据的简单堆积，它是一个复杂的容纳数据集成的系统工程，为决策者提供访问、分析及共享信息的能力，从而发挥数据仓库的真正功效。本文针对于零售业，阐述了在零售业数据仓库系统中ETL的流程并结合实际应用提出了在零售业数据仓库系统中的ETL过程应考虑的诸多问题及解决方案。     

VizRank: Data Visualization Guided by Machine Learning

Data visualization plays a crucial role in identifying interesting patterns in exploratory data analysis. Its use is, however, made difficult by the large number of possible data projections showing different attribute subsets that must be evaluated by the data analyst. In this paper, we introduce a method called VizRank, which is applied on classified data to automatically select the most useful data projections. VizRank can be used with any visualization method that maps attribute values to points in a two-dimensional visualization space. It assesses possible data projections and ranks them by their ability to visually discriminate between classes. The quality of class separation is estimated by computing the predictive accuracy of k-nearest neighbor classifier on the data set consisting of x and y positions of the projected data points and their class information. The paper introduces the method and presents experimental results which show that VizRank's ranking of projections highly agrees with subjective rankings by data analysts. The practical use of VizRank is also demonstrated by an application in the field of functional genomics.     

Privacy-aware data cleaning-as-a-service

Data cleaning is a pervasive problem for organizations as they try to reap value from their data. Recent advances in networking and cloud computing technology have fueled a new computing paradigm called Database-as-a-Service, where data management tasks are outsourced to large service providers. In this paper, we consider a Data Cleaning-as-a-Service model that allows a client to interact with a data cleaning provider who hosts curated, and sensitive data. We present PACAS: a Privacy-Aware data Cleaning-As-a-Service model that facilitates interaction between the parties with client query requests for data, and a service provider using a data pricing scheme that computes prices according to data sensitivity. We propose new extensions to the model to define generalized data repairs that obfuscate sensitive data to allow data sharing between the client and service provider. We present a new semantic distance measure to quantify the utility of such repairs, and we re-define the notion of consistency in the presence of generalized values. The PACAS model uses (X, Y, L)-anonymity that extends existing data publishing techniques to consider the semantics in the data while protecting sensitive values. Our evaluation over real data show that PACAS safeguards semantically related sensitive values, and provides lower repair errors compared to existing privacy-aware cleaning techniques.     

基于云数据中心的多源异构数据治理技术研究

目前常规的多源异构数据治理方法主要通过对数据属性进行判断,从而实现分区域数据清洗,由于缺乏对非线性数据的分析,导致治理性能不佳;对此,提出基于云数据中心的多源异构数据治理技术;采用关系型数据库中的ETL功能对数据进行清洗,对数据转换模式以及数据清洗规则进行定义;引入互信息系数对数据相关程度进行判定,并进行非线性数据相关性分析;以云数据中心作为载体,对多源异构数据治理体系进行构建;在实验中,对提出的数据治理技术进行了治理性能的检验;最终的实验结果表明,提出的数据治理技术具备较高的查准率,对云数据中心多源异构数据具备较为理想的数据治理效果。     

基于嵌入式MPP系统的数据操作策略研究

针对一种完全国产化的嵌入式MPP系统,通过抽象出系统的数据空间划分、确定系统的数据访问方式以及建立系统的数据映射机制,提出了与体系结构相适应的、与存储方式无关的数据操作策略。实验结果表明,该数据操作策略能够实现异构系统间高效、正确地数据交换。     

JADLib-科学计算数据高层I/O软件库

JADLib是为满足大规模复杂结构数据的存储与共享需求而研制的科学数据I/O软件库。其目标是设计并实现管理科学计算网格数据的数据存储模型,支持多种复杂结构数据的表示与操作,应用程序接口直观、易掌握,文件格式统一、可直接可视化,提供并行I/O、数居子集访问、压缩存储等高性能存储机制,支持多类科学计算程序跨平台使用,目前已应用到惯性约束聚变、高功率微波、计算流体力学、材料科学等多个研究领域中。实际应用表明,JADLib对于解决数值模拟软件数据存储及后处理分析所面临的I/O效率与组织管理问题具有很好的应用效果。     

Compressing heap data for improved memory performance

We introduce a class of transformations that modify the representation of dynamic data structures used in programs with the objective of compressing their sizes. Based upon a profiling study of data value characteristics, we have developed the common‐prefix and narrow‐data transformations that respectively compress a 32 bit address pointer and a 32 bit integer field into 15 bit entities. A pair of fields that have been compressed by the above compression transformations are packed together into a single 32 bit word. The above transformations are designed to apply to data structures that are partially compressible, that is, they compress portions of data structures to which transformations apply and provide a mechanism to handle the data that is not compressible. The accesses to compressed data are efficiently implemented by designing data compression extensions (DCX) to the processor's instruction set. We have observed average reductions in heap allocated storage of 25% and average reductions in execution time and power consumption of 30%. If DCX support is not provided the reductions in execution times fall from 30% to 18%. Copyright © 2006 John Wiley & Sons, Ltd.