基于数据库日志的变化数据捕获研究

变化数据捕获方法是数据集成基础设施的战略组成部分,不断推动ETL、EAI等技术的发展.许多数据库厂商都提供了自己的CDC(Change data capture)产品,但只限于针对本身的数据库系统,价格也比较昂贵.虽然通过扫描数据库日志文件可以捕获变化数据,但大多数数据库系统都不提供日志文件的内部格式而只是提供日志访问的程序接口,如Oracle,SQL Server和DB2等.这些提供的接口有的访问活动日志,有的访问稳定日志,有的访问归档日志,因此很难保证读取日志文件的可靠性.现有的研究主要是如何利用程序应用接口读取日志文件,忽略了对可靠性的分析.本文针对读取不同类型的日志文件的可靠性条件进行了分析,提出了可靠读取规则及读取算法；并提出了从日志文件中有效抽取变化数据算法,实验证明了可靠性分析模型.

Research of Log-based Change Data Capture

CDC(Change data capture) is a strategic component in the data integration infrastructure and compliments other technologies such ETL(Extract-Transform-Load) and EAI(Enterprise Application Integration).Many companies have home-grown CDC solutions that are typically limited in scope and are costly and difficult to maintain.By scanning the transactions log files one can capture the changes,but most database systems do not provide internal format of their log files.However few databases like Oracle,SQL Server and DB2 provide a programmed library to interface with their transaction log files.Some of these API(Application Program Interface) can read active redo logs,some read redo logs,and some read archive logs.In some cases it is difficult to ensure the reliability of reading log files.Existing researches are how to read log records by using API but ignore the analysis of reliability.We considered the reliability conditions of reading different log files and propose the algorithm of reading log records.And the algorithm is proposed to efficiently extract change data.The experiment shows the analysis model.