大数据导入数据库的方法与实现

针对多个Excel大数据文件需要进行处理时,存在查询关联困难,修改多个文件的记录时速度缓慢等问题,将多个Excel大数据文件合并导入到标准的数据库中以便进行其他操作.介绍了大数据文件导入数据库的原则和方法,并以VB 6.0为平台进行了实验,实现了多个Excel大数据文件导入SQL Server数据库的两种方法.实验表明,介绍的两种方法能完成预期目标,方法简单、有效.     

个性化DRM和数字水印技术

提出了一种视频的个性化的数字版权保护和数字水印技术,针对每个用户采用不同加密密钥和水印信息,提高了安全性和事后追查能力.把音视频文件分离为非关键数据文件和小数据量的关键数据文件,对关键数据进行个性化加密和嵌入数字水印来降低计算量和存储量问题.前端系统把多版本的加密关键数据文件和共同的非关键数据分别存放和发布,客户端播放器把两种数据流进行同步、解密和合成,然后解码播放.     

SA：一种有利于多属性范围查询的多维聚簇方法

一般来说.外存访问的数据文件中针对多属性的区域查询有两个改进其效率的方向.一个是在其上建立索引,另一个是在物理层按照某种规律重新安排记录.探讨如何通过第二种方法来提高范围查询的效率,即通过多维聚簇的方式得到数据文件中更好的记录的存储顺序.首先,细致分析了该问题,并针对该问题构造了一个数学模型,然后通过引入光谱算法(SA)的思想为解决该NP难问题提供了一种多项式时间内的近似解.最后通过实验来验证了该方法在矩形区域查询和单维范围查询方面的有效性.     

BASIC随机数据文件向数据库DBF文件的转换

BASIC和数据库(DBASE、F-OXBASE)都是交互性语言软件,功能丰富,使用方便,应用十分广泛.工作中有时需要将BASIC的数据文件转换为数据库的DBF数据文件.BASIC的顺序数据文件(用WRITE语句生成)可以直接进行这种转换,随机数据文件则不能.而BASIC的数据文件中,大量都是随机文件.为     

P2P网中基于文件分片的副本建立策略

在大规模P2P网络中,为了提高数据的可用性,可以通过在系统中多个节点上建立数据副本的方法来增加数据的冗余度.传统方法建立副本的过程中,数据资源通常是以完整的数据文件为单位在节点间进行拷贝.为了减小副本建立的开销,可采用一种基于文件分片的副本建立策略,将需要复制的数据文件先进行分片,然后把这些分片在一组选取好的候选节点间进行分发复制,能够充分利用网络带宽,明显减小建立副本所需的时间开销.     

多线程技术在航天测控网数据流分割中的应用

将多线程技术应用于航天测控网数据流的数据分割过程,解决多站多目标跟踪状态下数据流保存为数据文件的难题。首先分析测控网数据流模型,提出数据分割的概念;然后阐述引入多线程技术进行数据处理的必要性,提出利用多线程同步对象等待机制来实现数据分割、数据文件生成的方法,并给出程序的关键代码。应用和测试表明,该方法在生成数据文件的过程中,可实现对航天测控网数据流的无损分割,可靠性高,实时性较强。      

虚拟地震数据文件并行访问策略

针对石油勘探行业海量数据过大以及非结构化的特点导致地震数据合并处理时的效率低、消耗磁盘等问题, 提出对地震数据文件建立元数据, 将多个独立地震数据文件通过文件元数据合并方式生成一个逻辑上完整的地震数据文件即虚拟地震数据文件, 提高地震数据合并效率, 并根据地震数据并行访问模型实现对虚拟地震数据文件的IO访问.     

激光扫描点云数据的快速读取新方法

在反求工程中,随着测量技术的发展,点云数据文件越来越大.为了提高点云数据文件的读取效率,本文分析了虚拟内存及内存映射文件的工作原理,并在此基础上提出用内存映射文件来实现点云文件的快速读取.通过实验对比,内存映射文件相对传统I/O在文件读取效率上有着明显的提高.     

一种地形数据文件处理工具的开发

作者在研制三维虚拟环境的工作中,对于地形数据文件DEM接触很多。原始的一块地形数据文件大小居然是300 MB多,显然不适合计算机处理。为了生成自己需要的三维地形,我用VC++6.0和OpenGL编制了自己的一种地形数据处理工具,可以对DEM数据文件进行三维地形的预览、数据文件的切割、数据文件的粗化和数据文件的转化。 下面介绍工具功能及其程序设计。 工具的三大功能:一是可以进行地形的预览;二是可以进行地形的处理;三是可以进行文件格式的转化。软件界面     

词法分析工具在数据文件中的应用

词法分析是程序编译中的重要组成模块,它的任务是识别源程序中最小的语法单位.本文使用flex作为构建词法分析工具,把应用推广到数据文件分析中,并以生物大数据文件为例子,使用词法分析工具构建数据文件分析器,对文件中的数据进行检索和分析,展示词法分析工具的广阔应用.     

基于图像处理技术的工程曲线数字化及应用

工程设计中涉及大量的设计资料,设计资料的一部分是以工程曲线的形式存在.本文提出工程曲线的图片导入、图片校正、网格去除、图像细化和图像扫描等方法以实现工程曲线的数字化.进一步地,通过线性插值和抛物线插值方法实现曲线数据的检索与查询.     

Management of complex structural engineering objects in a relational framework

To structure the development of an integrated building design environment, the global representation of the design data may best be organized in terms of hierarchies of objects. In structural engineering design we deal with large sets of independent but interrelated objects. These objects are specified by data. For an engineering design data base the system must be able to model the objects composing the design as well as to manage effectively the design data. The data base management system therefore needs to have some knowledge of the intended use of the data, and must provide an abstraction mechanism to represent and manipulate objects. Much recent research in engineering data bases focuses on object management for specific tasks but gives little attention to the shareability of the underlying information. This paper describes an architecture for the management of complex engineering objects in a sharable, relational framework. Potential application of this approach to object management for structural engineering analysis and design is discussed.     

工程地质资料检索系统的设计和应用

铁路工程地质勘察资料数量非常庞大,单条线路的工程地质资料超过1 TB,其中大部分是文档、图像等非结构化数据,这些资料反映了岩土工程特点和地质条件,是工程设计阶段最重要的依据.在项目工期紧和质量要求高的大背景下,传统的资料查阅方式给工程设计人员造成相当大的困扰,急需设计实现一套快速、扩展性强、支持众多文件格式的资料检索系...     

A structured methodology for implementing engineering data management

One of the most significant positive effects on product and process development in recent years has been the application of data management techniques. Engineering data management or product data management is the most promising one. The implementation of engineering data management is heavily dependent on the engineering process and involves the technologies of management, engineering, and information. However, as there is no commonly acceptable approach and methodologies for implementing engineering data management, it’s implementation becomes a bottleneck.This paper presents a structured methodology for the implementation of engineering data management. The approach consists of a series of steps, from business and engineering process analysis, modeling and reengineering, through system design and modeling, to system realization.This research will facilitate engineering process improvement and the planning, design and implementation of engineering data management. Consequently, it will help increase product development capability and quality, reduce development cycle time and cost, and hence increase product marketability.     

Data-driven engineering design: A systematic review using scientometric approach

In the last two decades, data regarding engineering design and product development has increased rapidly. Big data exploration and mining offer numerous opportunities for engineering design; however, owing to the multitude of data sources and formats coupled with the high complexity of the design process, these techniques are yet to be utilised to the best of their full potential. In this study, a comprehensive assessment of the state-of-the-art data-driven engineering design (DDED) in the last 20 years was conducted. A scientometric approach was employed wherein first, a systematic article acquisition procedure was performed, where a dataset of 3339 articles related to engineering design and big data analytics applications were extracted from Web of Science (WoS) and Scopus. Thereafter, this dataset was reduced to a dataset of 366 articles based on concise data screening. The resulting articles were used to analyse the dynamics of research in DDED throughout the last 20 years and to detect the primary research topics related to DDED, the most influential authors, and the papers with the highest impact in the DDED domain. Furthermore, the co-occurrence network of keywords/keyphrases and co-authorship networks were constructed and analysed to reveal the interconnection of the research topics and the collaboration between the most prolific authors. Finally, an insight how big data analytics is being applied through product development activities to support decision-making in engineering design was presented.     

巨型计算机工程设计描述模型

巨型机并行设计环境是巨型机工程设计自动化的发展趋势,它能有机地集成多领域ＣＡＤ工具,统一管理数据,统筹安排设计人员协同完成巨型机工程设计任务。本文介绍的设计描述模型建立此类设计环境的基础。     

Shape mining: A holistic data mining approach for engineering design

Although the integration of engineering data within the framework of product data management systems has been successful in the recent years, the holistic analysis (from a systems engineering perspective) of multi-disciplinary data or data based on different representations and tools is still not realized in practice. At the same time, the application of advanced data mining techniques to complete designs is very promising and bears a high potential for synergy between different teams in the development process. In this paper, we propose shape mining as a framework to combine and analyze data from engineering design across different tools and disciplines. In the first part of the paper, we introduce unstructured surface meshes as meta-design representations that enable us to apply sensitivity analysis, design concept retrieval and learning as well as methods for interaction analysis to heterogeneous engineering design data. We propose a new measure of relevance to evaluate the utility of a design concept. In the second part of the paper, we apply the formal methods to passenger car design. We combine data from different representations, design tools and methods for a holistic analysis of the resulting shapes. We visualize sensitivities and sensitive cluster centers (after feature reduction) on the car shape. Furthermore, we are able to identify conceptual design rules using tree induction and to create interaction graphs that illustrate the interrelation between spatially decoupled surface areas. Shape data mining in this paper is studied for a multi-criteria aerodynamic problem, i.e. drag force and rear lift, however, the extension to quality criteria from different disciplines is straightforward as long as the meta-design representation is still applicable.     

树状建筑工程设计重用模型

为了减少建筑工程单一项目内的设计信息的冗余数据,进而减少设计中的重复工作量,提出了"五层树状模型"来重组信息.该模型基于设计重用理论,充分考虑了建筑工程信息的构成特点,依据重用属性提取重用单元形成基本元素,借助五层树状结构实现数据重构以获得最终图纸,提高了数据的存储和利用效率.基于该模型开发的程序在重用单元的定义、搜索和组装等方面实现了自动化,满足实际工程大规模数据处理的需要,大幅缩短了设计周期.     

工程数据库中的数据组织

本文在讨论工程数据结构特点的基础上，针对工程设计和工程信息管理系统，提出了一种在工程数据库中存储中描述工程数据的方法，并对常用的工程图形信息，设计原型和铁路确报信息等结构进行了论述和实践。     

一种基于设计过程的附加式工程数据库系统的研究与设计

针对传统ＣＡＤ系统在设计信息管理上的薄弱环节,提出了一种基于设计过程的附加式工程数据库系统,将工程数据库通过应用程序和数据接口附加在ＣＡＤ系统上,通过二者的有机结合,实现了设计过程中的数据动态实时交换和在线加载,达到了数据共享、重用以及集成化、结构化管理的目的。附加式工程数据库的应用增强了ＣＡＤ功能,提高了设计效率和质量。