Practical maintenance of evolving metadata for digital preservation: algorithmic solution and system support

Metadata (i.e., data describing about data) of digital objects plays an important role in digital libraries and archives, and thus its quality needs to be maintained well. However, as digital objects evolve over time, their associated metadata evolves as well, causing a consistency issue. Since various functionalities of applications containing digital objects (e.g., digital library, public image repository) are based on metadata, evolving metadata directly affects the quality of such applications. To make matters worse, modern data applications are often large-scale (having millions of digital objects) and are constructed by software agents or crawlers (thus often having automatically populated and erroneous metadata). In such an environment, it is challenging to quickly and accurately identify evolving metadata and fix them (if needed) while applications keep running. Despite the importance and implications of the problem, the conventional solutions have been very limited. Most of existing metadata-related approaches either focus on the model and semantics of metadata, or simply keep authority file of some sort for evolving metadata, and never fully exploit its potential usage from the system point of view. On the other hand, the question that we raise in this paper is “when millions of digital objects and their metadata are given, (1) how to quickly identify evolving metadata in various context? and (2) once the evolving metadata are identified, how to incorporate them into the system?” The significance of this paper is that we investigate scalable algorithmic solution toward the identification of evolving metadata and emphasize the role of “systems” for maintenance, and argue that “systems” must keep track of metadata changes pro-actively, and leverage on the learned knowledge in their various services.     

Developing IntegrityCatalog,a software system for managing integrity‐related metadata in digital repositories

Digital repositories must periodically check the integrity of stored objects to assure users of their correctness. Prior solutions calculate integrity metadata and require the repository to store it alongside the actual data objects. To safeguard and detect damage to this metadata, prior solutions rely on widely visible media (unaffiliated third parties) to store and provide back digests of the metadata to verify it is intact. However, they do not address recovery of the integrity metadata in case of damage or adversarial attack. We introduce IntegrityCatalog, a novel software system that can be integrated into any digital repository. It collects all integrity‐related metadata in a single component and treats them as first class objects, managing both their integrity and their preservation. We introduce a treap‐based persistent authenticated dictionary managing arbitrary length key/value pairs, which we use to store all integrity metadata, accessible simply by object name. Additionally, IntegrityCatalog is a distributed system that includes a network protocol that manages both corruption detection and preservation of this metadata, using administrator‐selected network peers with 2 possible roles. Verifiers store and offer attestations on digests and have minimal storage requirements, while preservers efficiently synchronize a complete copy of the catalog to assist in recovery in case of a detected catalog compromise on the local system. We present our approach in developing the prototype implementation, measure its performance experimentally, and demonstrate its effectiveness in real‐world situations. We believe the implementation techniques of our open‐source IntegrityCatalog will be useful in the construction of next‐generation digital repositories.     

Using distributed objects to build the Stanford digital libraryInfobus

For digital libraries to thrive, the providers of information processing services must be able to evolve their systems autonomously. However, as the complexity of their offerings increases, software tools more sophisticated than existing Web facilities are needed. Distributed object technology may be the answer. The availability of high-volume, increasingly sophisticated information is making the need for metadata facilities more urgent. Traditional, library-based approaches break down when used in an advanced digital library. More modular mechanisms are needed, and the CORBA system is one approach. Digital libraries are affected at a deep technical level by the widely differing user traditions of Web users and library patrons. The challenge and opportunity of digital libraries will be the synthesis of these traditions. The authors set out to create a technical infrastructure to support the construction of digital libraries. In their view, a digital library comprises widely distributed resources that can be maintained autonomously by different organizations and will not require adherence to uniform interfaces     

Long-term digital preservation: preserving authenticity and usability of 3-D data

Long-term digital preservation, the process of maintaining digital objects through time to ensure continued access, has become a crucial issue in recent years. Whilst the amount of digitised information is constantly increasing, so too is the pace of progress in information technology, resulting in obsolescence of the software and hardware required to access and view digital information. Despite many organisations recognising this threat and the resulting need for preservation action, more work is required to effectively address the issue. We present in this article a framework for the long-term digital preservation of 3-D data. This framework is based on two pertinent preservation practices, emulation and metadata which ensure that the authenticity and usability, respectively, of a preserved digital object remain intact through time. An evaluation of our framework is presented which illustrates the viability of our approach in retaining accessibility, authenticity and usability for future end users.     

A novel digital information service for federating distributed digital entities

We investigate the performance and the scalability metrics of a Digital Information Service framework that is used for unifying and federating online digital entities by retrieving and managing information located on the web. The Digital Information Service consists of tools and web services for supporting Cyberinfrastructure based scientific research. This system supports a number of existing online Web 2.0 research tools (social bookmarking, academic search, scientific databases, journal and conference content management systems) and aims to develop added-value community building tools that leverage the management and federation of digital entities and their metadata obtained from multiple services. We introduce a prototype implementation and present its evaluation. As the results indicate, the proposed system achieves federation and unification of digital entities coming from different sources with negligible processing overheads.     

State-of-the-art of long-term preservation in product lifecycle management

Providing access to digital information for the indefinite future is the intention of long-term digital preservation systems. One application domain that certainly needs to implement such long-term digital preservation processes is the design and engineering industry. In this industry, products are designed, manufactured, and operated with the help of sophisticated software tools provided by product lifecycle management (PLM) systems. During all PLM phases, including geographically distributed cross-domain and cross-company collaboration, a huge amount of heterogeneous digital product data and metadata is created. Legal and economic requirements demand that this product data has to be archived and preserved for a long-time period. Unfortunately, the software that is able to interpret the data will become obsolete earlier than the data since the software and hardware lifecycle is relatively short-lived compared to a product lifecycle. Companies in the engineering industry begin to realize that their data is in danger of becoming unusable while the products are in operation for several decades. To address this issue, different academic and industrial initiatives have been initiated that try to solve this problem. This article provides an overview of these projects including their motivations, identified problems, and proposed solutions. The studied projects are also verified against a classification of important aspects regarding scope and functionality of digital preservation in the engineering industry. Finally, future research topics are identified.     

网格环境下基于OAI的数字图书馆互操作机制

现有数字图书馆（DLs）互操作方案,在实现大规模数字图书馆集成方面都存在一定的局限性。该文将网格技术与OAI-PMH框架相结合,提出了一个数字图书馆网格（DL Grid）互操作框架,并对网格环境下元数据资源的发现、采集、重组等关键问题进行了分析和设计,初步实现了在集成的元数据基础上数字图书馆信息的共享。     

智能公话系统中Web Services整合平台研究与实现

根据智能公话系统访问数字增值业务的需要，提出一种利用Web Services技术和CORBA技术等实现的整合平台．Web Services整合平台可以动态实时地实现各种数字增值业务的整合．可以使服务提供商通过Internet远程注册来完成服务描述和服务工作流的增加、修改．通过调用存储在数据库中的服务元数据，来完成动态的Web SerVices调用，具有灵活，方便等特点．在简要介绍智能公话系统结构的基础上，阐述了Web Services整合平台的结构、设计、实现和性能分析。     

Design of an integrated data retrieval, analysis, and visualization system: Application in the hydrology domain

In this paper, we present a prototype system, an integrated data management system, which is capable of querying, retrieving, and visualizing datasets with heterogeneous formats and large sizes without requiring users to have any knowledge of any other specific software. Our system has three distinguished characteristics: (1) modular structure and simple architecture which make it easy and feasible for users to add new functions and features to the system, (2) a new search concept and method based on the bounding box and on dynamically delineated watershed boundary from GIS (Geographic Information System), and (3) no requirement on having any knowledge about or installation of any other complicated software. The architecture of our integrated data management system is based on a metadata approach, which consists of four components including a metadata mechanism and a Java-based application engine. The metadata mechanism in conjunction with the Java-based application engine allows users to access and retrieve diverse data formats and structures from many heterogeneous hydrological data sources. The visualization component of the system makes it possible for users to view their queried data first before spending time retrieving them. The extensible and integrative characteristics of our system are illustrated by an example in which new and unique functions for data merging and GIS-based data querying are added to the system. Although the data sources and applications shown in this prototype system are related to the field of hydrology, the ideas, approaches, and system architecture are not domain-specific, and can be used/applied to other fields as well.     

The design of the VERS encapsulated object experience with an archival information package

In this paper, the information package (IP) used within the Victorian Electronic Records Strategy (VERS) is described. The VERS IP was designed in 1999 to support the long-term preservation of digital information and has been implemented in government agencies and it is the basis of the Public Record Office Victoria's digital archive. Its key feature is the use of a self-documenting encapsulation. The encapsulation binds the content information to be preserved and its metadata into a single object that can survive independently of an archival system. The self-documentation ensures that the content can be manually extracted from the encapsulation using only the simplest tools. The VERS IP is compared with the IP defined in the Open Archival Information System (OAIS) and the Metadata Encoding and Transmission Standard (METS).     

Creating the first SCORM object

The creation of the first SCORM (Shareable Content Object Reference Model) object offers some challenges and difficulties which go beyond the facilities offered by content generation applications. In particular, the creation of really reusable, searchable learning objects requires a detailed consideration of metadata, where some institutional aspects may be unclear or not available. This work describes creation of a first learning object, from software tools installation to final packaging. It aims at a wider perspective than that offered by handbooks or user guides for content generation tools, generally poor or altogether deprived of suggestions on how to go about to achieve reusability, interoperability, durability and accessibility as conceived by the SCORM standard. Only free software and Internet publications are used as references. The creation of a simple SCORM package with the Reload Editor is described step by step, and the package created is then tested using Reload SCORM Player, allowing for the detection of some difficulties and alternatives of solution. Help available and some commented references are afterwards indicated. A list of suggestions finally emerges, to the purpose of solving beforehand most of the uncertainties, defining a consistent learning object creation scheme and reducing training time to master tools and metadata generation. As a conclusion, some limitations found along the work are pointed out, in particular the necessity of adopting or defining a LOM (Learning Object Metadata) application profile together with an institutional strategy to face metadata creation efficiently.     

Creating the first SCORM object

The creation of the first SCORM (Shareable Content Object Reference Model) object offers some challenges and difficulties which go beyond the facilities offered by content generation applications. In particular, the creation of really reusable, searchable learning objects requires a detailed consideration of metadata, where some institutional aspects may be unclear or not available. This work describes creation of a first learning object, from software tools installation to final packaging. It aims at a wider perspective than that offered by handbooks or user guides for content generation tools, generally poor or altogether deprived of suggestions on how to go about to achieve reusability, interoperability, durability and accessibility as conceived by the SCORM standard. Only free software and Internet publications are used as references. The creation of a simple SCORM package with the Reload Editor is described step by step, and the package created is then tested using Reload SCORM Player, allowing for the detection of some difficulties and alternatives of solution. Help available and some commented references are afterwards indicated. A list of suggestions finally emerges, to the purpose of solving beforehand most of the uncertainties, defining a consistent learning object creation scheme and reducing training time to master tools and metadata generation. As a conclusion, some limitations found along the work are pointed out, in particular the necessity of adopting or defining a LOM (Learning Object Metadata) application profile together with an institutional strategy to face metadata creation efficiently.     

PANGAEA—an information system for environmental sciences

PANGAEA is an information system for processing, long-term storage, and publication of georeferenced data related to earth science fields. Essential services supplied by PANGAEA are project data management and the distribution of visualization and analysis software. Organization of data management includes quality control and publication of data and the dissemination of metadata according to international standards. Data managers are responsible for acquisition and maintenance of data. The data model used reflect the information processing steps in the earth science fields and can handle any related analytical data. The basic technical structure corresponds to a three tiered client/server architecture with a number of comprehensive clients and middleware components controlling the information flow and quality. On the server side a relational database management system (RDBMS) is used for information storage. The web-based clients include a simple search engine (PangaVista) and a data mining tool (ART). The client used for maintenance of information contents is optimized for data management purposes. Analysis and visualization of metainformation and analytical data is supported by a number of software tools, which can either be used as ‘plug-ins’ of the PANGAEA clients or as standalone applications, distributed as freeware from the PANGAEA website. Established and well-documented software tools are the mini-GIS PanMap, the plotting tool PanPlot, and Ocean Data View (ODV) for the exploration of oceanographic data. PANGAEA operates on a long-term basis. The available resources are sufficient not only for the acquisition of new data and the maintenance of the system but also for further technical and organizational developments.     

A location model for smart environments

This paper presents a location model for location-aware and user-aware services in smart environments. It can be dynamically organized like a tree based on geographical containment, such as that in a user–room–floor–building hierarchy and each node in the tree can be constructed as an executable software component. The model is unique in existing approaches because it enables location-aware services to be managed without any database servers, it can be managed by multiple computers, and it can provide a unified view of the locations of not only physical entities and spaces, including users and objects, but also computing devices and services. A prototype implementation of this approach was constructed on a Java-based mobile agent system. This paper presents the rationale, design, implementation, and applications of the prototype system.     

Packaging models for the storage and distribution of complex digital objectsin archival information systems: a review of MPEG-21 DID principles

In recent years, the number of organizations making digital information available has increased dramatically. This evolution has encouraged the development of various approaches to represent complex digital objects. Approaches that are of specific relevance to the digital library and archiving community include the Metadata Encoding and Transmission Standard (METS), the Content Packaging XML Binding of the Instructional Management System (IMS) project, and the Sharable Content Object Reference Model (SCORM). The MPEG-21 Digital Item Declaration MPEG-21 DID) is yet another packaging approach that, so far, has received little attention in the digital library community. The primary goal of this article is to offer a method to compare these packaging approaches in the context of the needs of the digital library community and the rising digital preservation programs. The contribution of this paper is the definition of an integrated reference framework, based on both the ISO OAIS Reference Model and a set of additional significant properties that affect the quality, usability, encoding, and behavior of complex digital objects. This article also gives a brief insight into MPEG-21 DID and examines its potential relevance by showing the results obtained from applying the proposed reference framework. Published online: 8 April 2005