二维混合数据分布下相关性检测的新方法HY-COCA

混合数据分布是指数据分布的不同区域具有不同的特殊分布.例如销售额和地区两个属性之间,在销售额比较低的数值区间中,两者呈现近似相互独立的数据分布;而在销售额比较高的数值区间,二者呈现近似函数依赖的数据分布.现有检测数据相关性的研究专注于给出一个总体的二维相关性的度量,而无法检测出子区域的特殊相关性.在统计分析时,这类具有特殊相关性的子区域有更丰富的统计意义,值得引起重视.研究并提出了存在这类混合数据分布的情况下,检测数据相关性的新方法HY-COCA.该方法在熵相关系数的基础上,缩小了子区域的搜索空间,与Naive方法相比,降低了复杂度;同时HY-COCA还讨论了子区域的相关性差异判别与结果展示等问题.在生成的数据和测试基准数据上进行了实验,结果验证了方法的有效性.

HY-COCA:A Hybrid-data-distribution-aware Way to Detect Correlation over Bi-dimensional Data Space

Hybrid data distribution between two attributes means that different data sub-regions exhibit different correlated associations.For example,in a distribution between sale amounts and different cities,a semi-independent distribution is observed with lower sale amounts,but for higher sale amounts,the two attributes present soft functional depen-dency.Previous researches on auto detection of association focused on deducing an overall measure of association over two dimensional distributions.They were unable to address hybrid data distribution problem.In statistical analysis,such sub-regions with particular data associations are worth paying attention to.This paper proposed a new way,HY-COCA,to detect data associations globally and locally,finding those sub-regions with special data associations.We did experiments on both synthetic and benchmark data.Experimental results verify the effectiveness of HY-COCA.