Extracting logical structures from HTML tables

While HTML is mainly designed for the visual rendering of Web documents, XML is widely accepted as a standard format to process and manage information. In particular, it can embed the information of logical structures. However, in order to utilize XML, the logical structures of HTML tables should first be extracted and transformed into XML representations. This paper presents an efficient method for the process, which consists of two phases: area segmentation and structure analysis. The area segmentation cleans up tables and segments them into attribute and value areas by checking visual and semantic coherency. The hierarchical structure between attribute and value areas is then analyzed and transformed into an XML representation using a proposed table model. Experimental results with 1180 HTML tables show that the proposed method performs better than conventional methods, resulting in an average accuracy of 86.7%.     

HTML表格向XML的智能转换

XML已经成为处理与管理信息的标准格式,而HTML表格被广泛应用于Web。为了充分利用与管理HTML表格信息,需要将HTML表格转换成XML。提出一种有效的处理方法,该方法包含2个部分,即表格识别与结构转换。表格识别通过检查格式、语法及语义的特征将表格提取出来并分割成值域与属性域,使用预设的表格模板分析属性域与值域间的层次结构并将其转换成XML格式。通过 300多个表格的实验表明,所提出的方法要优于传统方法,结果的准确率达86.7%。     

A scalable hybrid approach for extracting head components from Web tables

We have established a preprocessing method for determining the meaningfulness of a table to allow for information extraction from tables on the Internet. A table offers a preeminent clue in text mining because it contains meaningful data displayed in rows and columns. However, tables are used on the Internet for both knowledge structuring and document design. Therefore, we were interested in determining whether or not a table has meaningfulness that is related to the structural information provided at the abstraction level of the table head. Accordingly, we: 1) investigated the types of tables present in HTML documents, 2) established the features that distinguished meaningful tables from others, 3) constructed a training data set using the established features after having filtered any obvious decorative tables, and 4) constructed a classification model using a decision tree. Based on these features, we set up heuristics for table head extraction from meaningful tables, and obtained an F-measure of 95.0 percent in distinguishing meaningful tables from decorative tables and an accuracy of 82.1 percent in extracting the table head from the meaningful tables.     

航班信息抽取规则的自动生成技术

在基于包装器的Web信息提取工作中,抽取规则占有重要的地位。由于网页经常改版,使得抽取规则需要不断更新,且手工生成抽取规则是一项费时费力的工作。为此,提出一种自动生成抽取规则的方法,通过扫描HTML源码,生成带语义信息的TABLE树,用以识别网页中的数据表格,并在此基础上利用贪心算法自动生成抽取规则。实验结果表明,该方法具有较高的准确率和F指数,且对于识别出的表格具有较高的规则生成率。     

网上表格数据到XML的自动转换

互联网上有大量信息采用HTML表格表示,由于HTML不描述数据的内容,机器不能理解和查询。论文利用HTML表格属性,在表格中插入冗余单元,使HTML表格规范化;对没有标志表头的HTML表格,采用格式化的信息的量化值识别网上表格的表头。在此基础上,提出了通过获取表格属性与值对应的语义层次,自动转换HTML表格数据为XML文挡的新方法。     

Turning tables [HTML tables]

Behind all the fancy tools that churn out volumes of messy HTML, there is a basic set of standardized HTML tags that all professional Webspinners must learn to master. It is important to understand these tags and how they work because despite recent advances in code-checking tools, debugging HTML often still comes down to hand-tweaking code with simple text editors like NotePad and BBEdit. The paper discusses HTML tables which are the fundamental building blocks for most of today's Web pages. It considers three basic tags to build a table     

Web表格信息抽取模型的设计与实现

Web表格作为一种简洁有效的数据信息表达方式,已广泛应用于Web页面中.现提出一种基于表格结构的Web表格信息抽取模型,该模型主要有表格定位模块、表格结构预处理模块和表格信息抽取与重构模块三个模块组成,根据Web表格的结构标记和自定义的启发式规则来抽取表格信息.实验结果表明该模型能够很好地应用于Web表格信息的抽取.     

Transforming arbitrary tables into logical form with TARTAR

The tremendous success of the World Wide Web is countervailed by efforts needed to search and find relevant information. For tabular structures embedded in HTML documents, typical keyword or link-analysis based search fails. The Semantic Web relies on annotating resources such as documents by means of ontologies and aims to overcome the bottleneck of finding relevant information. Turning the current Web into a Semantic Web requires automatic approaches for annotation since manual approaches will not scale in general. Most efforts have been devoted to automatic generation of ontologies from text, but with quite limited success. However, tabular structures require additional efforts, mainly because understanding of table contents requires the comprehension of the logical structure of the table on the one hand, as well as its semantic interpretation on the other. The focus of this paper is on the automatic transformation and generation of semantic (F-Logic) frames from table-like structures. The presented work consists of a methodology, an accompanying implementation (called TARTAR) and a thorough evaluation. It is based on a grounded cognitive table model which is stepwise instantiated by the methodology. A typical application scenario is the automatic population of ontologies to enable query answering over arbitrary tables (e.g. HTML tables).     

基于结构分析和实体识别的信息集成

针对海量的web数据，提出了一种基于文档结构分析和实体识别的web信息提取和集成方法，利用XML强大的数据描述能力，灵活组织集成的web文档信息内容．方法首先将半结构化的HTML文档转化成具有模式结构的XML文档，然后使用实体识别的技术对不同主题区域进一步抽取出格式良好的数据，最后将得到的多数据类型的信息集成到数据库中，以支持进一步的分析和查询．实验结果证明了该方法的实用和有效性．     

Web table discrimination with composition of rich structural and content information

A table is a well-organized and summarized knowledge expression for a domain. Therefore, it is of great importance to extract information from tables. However, many tables in Web pages are used not to transfer information but to decorate pages. One of the most critical tasks in Web table mining is thus to discriminate meaningful tables from decorative ones. The main obstacle of this task comes from the difficulty of generating relevant features for discrimination. This paper proposes a novel discrimination method using a composite kernel which combines parse tree kernels and a linear kernel. Because a Web table is represented as a parse tree by an HTML parser, it is natural to represent the structural information of a table as a parse tree. In this paper, two types of parse trees are used to represent structural information within and around a table. These two trees define the structure kernel that handles the structural information of tables. The contents of a Web table are manipulated by a linear kernel with content features. Support vector machines with the composite kernel distinguish meaningful tables from decorative ones with high accuracy. A series of experiments show that the proposed method achieves state-of-the-art performance.     

基于结构一致和特征学习的网页信息标签提取

网页信息指网页的正文、标题、发布时间、媒体等,每个信息都存在于HTML文档特定的标签中,自动获取这些标签可以实现在相同模板下的网页信息自动提取,对于大规模抓取网页内容有很大帮助。由于在相同模板下不同网页之间结构一致,网页信息有一定统计特征,提出了一种基于结构对比和特征学习的网页信息标签自动提取算法。该算法包含三个步骤：网页对比、内容识别和标签提取。在51个模块下对1?620个网页进行测试,实验结果表明,通过提取标签获取网页信息不仅速度快,而且抓取的内容更加准确。     

Web信息检索结构化排序函数与标引词加权技术

分析了当前Web信息检索的技术现状,指出检索效率不高的根本原因在于搜索引擎所采用的排序函数和标引词加权技术。介绍了传统的信息检索排序函数和标引词加权技术。分析了Web文档的特点,指出其主要形式HTML文档是一种结构化文档,结构由标签显式地定义,不同文档结构对检索性能的贡献不同。对本领域国内外学者的成果作了对比研究。最后探讨了Web信息检索排序函数及标引词加权技术的发展方向。     

基于相似度的网页标题抽取方法

目前网页标题的抽取方法大多结合HTML结构和标签特征进行抽取,但是这些方法并没有考虑标题与正文信息之间内容上的联系。该文提出一种基于相似度的网页标题抽取方法,该方法利用网页标题与正文信息之间的关系,通过计算语言“单位”之间的相似度和对应的权值,并引入HITS算法模型对权值进行调整,根据特定的选取方法抽取出真实标题。实验结果表明,该方法不仅对“非标准网页”的抽取达到满意的效果,而且对“标准网页”具有较高的泛化能力。     

灵活结构网页的正文提取

在Web数据挖掘中,由于网页大多都含有指向其他页面的超链接等噪音信息,为了减少噪音信息对Web数据挖掘效果的影响,有必要对网页进行净化处理,提取其中的正文,同时,现实中很多网页的代码结构不是特别规范,对此,提出一种对灵活结构网页适用的正文抽取算法。将网页用HTML标签分割成节点形式,找出其中含有正文内容的一个节点,以此节点为基础向前和向后进行余下正文内容的抽取。实验结果表明,本算法的适用性强、正确率较高。     

Web信息提取技术的研究及其在CSCW中的应用

分析基于XML的Web信息提取．讨论相关技术在Web信息抽取中的应用并建立相应的Web信息抽取模型,实现Web信息的自动提取。通过分析如何向网络上的不同网站发送HTTP请求数据包,处理响应信息,从而获得包含知识信息的HTML文档或者XML文档;并在Oracle公司的ADF框架下给出Web信息抽取模型的实现程序。     

一种基于未知结构网页抽取本体的方法

在Web上数据大多是结构化的,但事先并不熟知数据的结构,因此不能有效地查询感兴趣的数据.提出了一种独立于文本抽取本体的方法,其过程包括表的理解、数据集成和本体生成,其中表理解是搜寻定位兴趣表、识别及匹配属性和值,并形成记录;数据集成是匹配源记录和目标模式;本体卷积是将源记录的数据抽取到目标模式.结果表明这种方法可以通过已知的目标模式有效地抽取未知结构的数据.     

A Web page classification system based on a genetic algorithm using tagged-terms as features

The incredible increase in the amount of information on the World Wide Web has caused the birth of topic specific crawling of the Web. During a focused crawling process, an automatic Web page classification mechanism is needed to determine whether the page being considered is on the topic or not. In this study, a genetic algorithm (GA) based automatic Web page classification system which uses both HTML tags and terms belong to each tag as classification features and learns optimal classifier from the positive and negative Web pages in the training dataset is developed. Our system classifies Web pages by simply computing similarity between the learned classifier and the new Web pages. In the existing GA-based classifiers, only HTML tags or terms are used as features, however in this study both of them are taken together and optimal weights for the features are learned by our GA. It was found that, using both HTML tags and terms in each tag as separate features improves accuracy of classification, and the number of documents in the training dataset affects the accuracy such that if the number of negative documents is larger than the number of positive documents in the training dataset, the classification accuracy of our system increases up to 95% and becomes higher than the well known Naïve Bayes and k nearest neighbor classifiers.     

基于标记图的Web数据模型

本文详细探讨了一种新的Ｗｅｂ数据模型－标记图,给类格的形式化描述。     

基于正文特征及网页结构的主题网页信息抽取

Web正文信息抽取是信息检索、文本挖掘等Web信息处理工作的基础。在统计分析了主题网页的正文特征及结构特征的基础上,提出了一种结合网页正文信息特征及HTML标签特点的主题网页正文信息抽取方法。在将Web页面解析成DOM树的基础上,根据页面DOM树结构获取正文信息块,分析正文信息块块内噪音信息的特点,去除块内噪音信息。实验证明,这种方法具有很好的准确率及召回率。     

基于页面对比分析的数据提取

针对提供大规模数据查询的Web页面，提出了一种基于站点内页面对比分析的web数据提取方法。在对页面建树和分块的基础上对比分析获得页面数据块，然后利用同结构多页面对比和格式判断提取出数据，最后将数据存入到主据库中。该方法成功运用到多个信息提取系统中，实现了高效、准确的数据提取。