Semi-automatic photo annotation strategies using event based clustering and clothing based person recognition

Managing a large number of digital photos is a challenging task for casual users. Personal photos often don’t have rich metadata, or additional information associated with them. However, available metadata can play a crucial role in managing photos. Labeling the semantic content of photos (i.e., annotating them), can increase the amount of metadata and facilitate efficient management. However, manual annotation is tedious and labor intensive while automatic metadata extraction techniques often generate inaccurate and irrelevant results. This paper describes a semi-automatic annotation strategy that takes advantage of human and computer strengths. The semi-automatic approach enables users to efficiently update automatically obtained metadata interactively and incrementally. Even though automatically identified metadata are compromised with inaccurate recognition errors, the process of correcting inaccurate information can be faster and easier than manually adding new metadata from scratch. In this paper, we introduce two photo clustering algorithms for generating meaningful photo groups: (1) Hierarchical event clustering; and (2) Clothing based person recognition, which assumes that people who wear similar clothing and appear in photos taken in one day are very likely to be the same person. To explore our semi-automatic strategies, we designed and implemented a prototype called SAPHARI (Semi-Automatic PHoto Annotation and Recognition Interface). The prototype provides an annotation framework which focuses on making bulk annotations on automatically identified photo groups. The prototype automatically creates photo clusters based on events, people, and file metadata so that users can easily bulk annotation photos. We performed a series of user studies to investigate the effectiveness and usability of the semi-automatic annotation techniques when applied to personal photo collections. The results show that users were able to make annotations significantly faster with event clustering using SAPHARI. We also found that users clearly preferred the semi-automatic approaches.     

Context-Aware Person Identification in Personal Photo Collections

Identifying the people in photos is an important need for users of photo management systems. We present MediAssist, one such system which facilitates browsing, searching and semi-automatic annotation of personal photos, using analysis of both image content and the context in which the photo is captured. This semi-automatic annotation includes annotation of the identity of people in photos. In this paper, we focus on such person annotation, and propose person identification techniques based on a combination of context and content. We propose language modelling and nearest neighbor approaches to context-based person identification, in addition to novel face color and image color content-based features (used alongside face recognition and body patch features). We conduct a comprehensive empirical study of these techniques using the real private photo collections of a number of users, and show that combining context- and content-based analysis improves performance over content or context alone.     

Semantic analysis and retrieval in personal and social photo collections

Semantic understanding of images has been an important topic in the research community for a long time as it is an important prerequisite to build meaningful retrieval systems which are accessible by both users and automatic reasoning algorithms. Recently, especially with the growing trend to share photos online, the social aspect of image retrieval becomes more and more prevalent and image retrieval more and more focusses specifically on photos and their special characteristics, especially on information outside the image itself. Researchers are starting to explore how and why photos are shot, shared and used and try to incorporate this additional knowledge to aid image analysis and retrieval. Several survey papers have been written in the past reviewing works in the general field of image analysis and retrieval. However, the social aspect of image retrieval and the focus on digital photos has not sufficiently been addressed in these works. In this article we give an overview over the current research field of semantic photo understanding, annotation and retrieval. We review over 160 contributions in the field and identify trending topics and implications for future directions of research.     

Digital photo classification methodology for groups of photographers

Digital cameras have become an essential product when traveling or attending events. Because of its popularity and low cost, it is increasingly likely that more than one camera will be used at an event. The total number of photos captured is also increasing. Although the cost of digital photographs is low, managing numerous digital photos is burdensome to most users. Thus, an intelligent management tool for digital photos is required. In this paper, we propose novel clustering algorithms for concurrent digital photos obtained from multiple cameras. Since previous studies only considered a single user’s photo collection, they are not applicable to concurrent photos obtained from a group of photographers. To handle this situation, we define temporal/spatial combined clustering for the set of group photos taken from different cameras. If photos are submitted from a camera whose user has shown a preference between spatial and temporal clustering, we can obtain customized clustering output from other photo sets according to the reference clustering characteristics. We also propose unsupervised methods for general clustering output. Input concurrent photos are processed without a user’s true clusters, which can be a burden when the number of photos in the true clusters is huge. We tested our methods via more than one thousand photos taken by tourist groups. The final result was satisfactory compared to previous methods based on temporal (spatial) criteria only.     

Visual and textual fusion for semantically supervised region-based retrieval

This paper presents a unified annotation and retrieval framework, which integrates region annotation with image retrieval for performance reinforcement. To integrate semantic annotation with region-based image retrieval, visual and textual fusion is proposed for both soft matching and Bayesian probabilistic formulations. To address sample insufficiency and sample asymmetry in the annotation classifier training phase, we present a region-level multi-label image annotation scheme based on pair-wise coupling support vector machine (SVM) learning. In the retrieval phase, to achieve semantic-level region matching we present a novel retrieval scheme which differs from former work: the query example uploaded by users is automatically annotated online, and the user can judge its annotation quality. Based on the user’s judgment, two novel schemes are deployed for semantic retrieval: (1) if the user judges the photo to be well annotated, Semantically supervised Integrated Region Matching is adopted, which is a keyword-integrated soft region matching method; (2) If the user judges the photo to be poorly annotated, Keyword Integrated Bayesian Reasoning is adopted, which is a natural integration of a Visual Dictionary in online content-based search. In the relevance feedback phase, we conduct both visual and textual learning to capture the user’s retrieval target. Better annotation and retrieval performance than current methods were reported on both COREL 10,000 and Flickr web image database (25,000 images), which demonstrated the effectiveness of our proposed framework.     

本体推理在智能照片管理系统中的应用

现有的照片管理系统缺乏自动语义推理和扩展功能。为此,提出一种基于本体的智能照片管理系统。该系统以FamilyAlbum本体模型为知识框架,对照片进行多源信息的语义标注,并利用SWRL规则对系统中现有的语义标注进行自动推理,从而扩展出新的语义信息,为照片的智能管理提供有效支持。通过OntoAlbum原型系统的实现,验证了该方法的有效性。     

Robust event discovery from photo collections using Signature Image Bases (SIBs)

Analyzing personal photo albums for understanding the related events is an emerging trend. A reliable event recognition tool could suggest appropriate annotation of pictures, provide the context for single image classification and tagging, achieve automatic selection and summarization, ease organization and sharing of media among users. In this paper, a novel method for fast and reliable event-type classification of personal photo albums is presented. Differently from previous approaches, the proposed method does not process photos individually but as a whole, exploiting three main features, namely Saliency, Gist, and Time, to extract an event signature, which is characteristic for a specific event type. A highly challenging database containing more than 40.000 photos belonging to 19 diverse event-types was crawled from photo-sharing websites for the purpose of modeling and performance evaluation. Experimental results showed that the proposed approach meets superior classification accuracy with limited computational complexity.     

移动相册的复杂网络分析*

带有照相功能的移动终端的普及,产生了大量的照片,记录了用户的日常生活,这些照片附带了大量的多维元数据如时间、地点、作者、事件等,如何利用多维的元数据有效管理这些照片是一个研究热点。创新地提出利用复杂网络的方法分析和可视化移动相册,将照片看做一个节点,利用照片的多维元数据之间的相关性建立照片之间的边,这样所有在相册中的照片就组成了一个多维复杂加权网络。分析了照片复杂加权网络的统计特性,揭示了其小世界特性。利用照片复杂网络,不仅可以可视化照片之间的多维关系,而且可以快速检索相关照片。     

基于 TD_SCDMA 电子相框的照片分享方案

随着数码相机的普及,拍摄的数码照片呈几何级数增加,除了少部分会被冲洗成纸质照片以外,大部分数码照片都会被保存在电子存储设备中。电子相框是非常适合的存储设备,兼具展示数码照片的功能,近年来正在蓬勃发展。本文分析了电子相框发展的现状和业务需求,针对传统电子相框面临照片更新与共享的问题,提出了一种基于TD_SCDMA（Time Division-Synchronous Code Division Multiple Access）的照片分享方案,将电子相框接入UMTS（Universal Mobile Telecommunications System,通用移动通信系统）,采用电子相框统一服务平台实现了照片在手机、电脑和电子相框之间的无缝分享。     

Multi-modal and multi-scale photo collection summarization

With the proliferation of digital cameras and mobile devices, people are taking much more photos than ever before. However, these photos can be redundant in content and varied in quality. Therefore there is a growing need for tools to manage the photo collections. One efficient photo management way is photo collection summarization which segments the photo collection into different events and then selects a set of representative and high quality photos (key photos) from those events. However, existing photo collection summarization methods mainly consider the low-level features for photo representation only, such as color, texture, etc, while ignore many other useful features, for example high-level semantic feature and location. Moreover, they often return fixed summarization results which provide little flexibility. In this paper, we propose a multi-modal and multi-scale photo collection summarization method by leveraging multi-modal features, including time, location and high-level semantic features. We first use Gaussian mixture model to segment photo collection into events. With images represented by those multi-modal features, our event segmentation algorithm can generate better performance since the multi-modal features can better capture the inhomogeneous structure of events. Next we propose a novel key photo ranking and selection algorithm to select representative and high quality photos from the events for summarization. Our key photo ranking algorithm takes the importance of both events and photos into consideration. Furthermore, our photo summarization method allows users to control the scale of event segmentation and number of key photos selected. We evaluate our method by extensive experiments on four photo collections. Experimental results demonstrate that our method achieves better performance than previous photo collection summarization methods.     

Musical slideshow: boosting user experience in photo presentation

With the advance of digital acquisition and storage technology, people got used to take photos at will and to record almost any event with (many) photos. However, large amounts of photos without appropriate organization for access and consumption raise several potential problems, reducing the users’ quality of experience. Therefore, to facilitate an effective photo selection and organization process making the consumption process more enjoyable, it is useful to develop advanced image analysis and presentation tools, eventually integrating also some music content in a synchronized way, to further boost the photo presentation user experience. As people’s time is getting more precious and scarce, the availability of applications able to automatically generate a musical slideshow as a pleasant way to consume large amounts of photos is highly desirable. In this context, this paper describes the context and motivation for the development of a musical slideshow application, the architecture designed and the entire signal processing flow implemented. To assess the performance of this multimedia application, a user evaluation methodology was designed and applied with promising results, showing that the developed application is able to create entertaining and efficient photo presentation experiences.     

基于语义的照片检索的相关问题研究

随着数码产品的普及,数码照片的积累越来越多。为了更好的满足用户语义检索照片的需求,各国研究者做了很多相关工作。他们研究照片的本质和用户使用照片的习惯,使用这些知识来帮助检索和管理照片。分析和总结这些研究,主要介绍基于语义的数码照片检索研究工作中语义提取的方法、运用语义Web技术现状及其在照片管理中的应用。     

Photo annotation: a survey

Due to the large number of photos that are currently being generated, it is very important to have techniques to organize, search for, and retrieve such images. Photo annotation plays a key role in these mechanisms because it can link raw data (photos) to specific information that is essential for human beings to handle large amounts of content. However, the generation of photo annotation is still a difficult problem to solve as part of a well-known challenge called the semantic gap. In this paper, a literature review was conducted with the aim of investigating the most popular methods employed to produce photo annotations. Based on the papers surveyed, we identified that People (“Who?”), Location (“Where?”), and Event (“Where? When?”) are the most important features of photo annotation. We also established comparisons between similar photo annotation methods, highlighting key aspects of the most commonly used approaches. Moreover, we provide an overview of a general photo annotation process and present the main aspects of photo annotation representation comprising formats, context of usage, advantages and disadvantages. Finally, we discuss ways to improve photo annotation methods and present some future research guidelines.     

Using physical memorabilia as opportunities to move into collocated digital photo-sharing

The uptake of digital photos vs. print photos has altered the practice of photo-sharing. Print photos are easy to share within the home, but much harder to share outside of it. The opposite is true of digital photos. People easily share digital photos outside the home, e.g., to family and friends by e-mail gift-giving, and to social networks and the broader public by web publishing. Yet within the home, collocated digital photo-sharing is harder, primarily because digital photos are typically stored on personal accounts in desktop computers located in home offices. This leads to several consequences. (1) The invisibility of digital photos implies few opportunities for serendipitous photo-sharing. (2) Access control and navigation issues inhibit family members from retrieving photo collections. (3) Photo viewing is compromised as digital photos are displayed on small screens in an uncomfortable viewing setting.To mitigate some of these difficulties, we explore how physical memorabilia collected by family members can create opportunities that encourage social and collocated digital photo-sharing. First, we studied (via contextual interviews with 20 households) how families currently practice photo-sharing and how they keep memorabilia. We identified classes of memorabilia that can serve as memory triggers to family events, trips, and times when people took photos. Second, we designed Souvenirs, a photo-viewing system that exploits memorabilia as a social instrument. Using Souvenirs, a family member can meaningfully associate physical memorabilia with particular photo-sets. Later, any family member can begin their story-telling with others through the physical memento, and then enrich the story by displaying its associated photos simply by moving the memento close to the home's large-format television screen. Third, we re-examined our design premises by evoking household reactions to an early version of Souvenirs. Based on these interviews, we redesigned Souvenirs to better reflect the preferences and real practices of photo and memorabilia use in the home.     

家庭数码照片管理领域的本体建模研究*

数码相机的迅速普及对照片管理技术提出了严峻的挑战,传统的基于文本关键字的标注和检索系统已不能满足人们的需求。因此,基于语义的图像检索技术正在快速兴起,但语义知识的组织和表达问题始终未能得到有效解决。针对上述问题,提出了一种新的基于领域本体的照片管理方法。该方法的关键技术是对照片管理领域进行本体建模。实验表明了该方法的有效性。     

Leveraging multiple cues for recognizing family photos

Social relation analysis via images is a new research area that has attracted much interest recently. As social media usage increases, a wide variety of information can be extracted from the growing number of consumer photos shared online, such as the category of events captured or the relationships between individuals in a given picture. Family is one of the most important units in our society, thus categorizing family photos constitutes an essential step toward image-based social analysis and content-based retrieval of consumer photos. We propose an approach that combines multiple unique and complimentary cues for recognizing family photos. The first cue analyzes the geometric arrangement of people in the photograph, which characterizes scene-level information with efficient yet discriminative capability. The second cue models facial appearance similarities to capture and quantify relevant pairwise relations between individuals in a given photo. The last cue investigates the semantics of the context in which the photo was taken. Experiments on a dataset containing thousands of family and non-family pictures collected from social media indicate that each individual model produces good recognition results. Furthermore, a combined approach incorporating appearance, geometric and semantic features significantly outperforms the state of the art in this domain, achieving 96.7% classification accuracy.     

Pointwise and pairwise clothing annotation: combining features from social media

In this paper, we present effective algorithms to automatically annotate clothes from social media data, such as Facebook and Instagram. Clothing annotation can be informally stated as recognizing, as accurately as possible, the garment items appearing in the query photo. This task brings huge opportunities for recommender and e-commerce systems, such as capturing new fashion trends based on which clothes have been used more recently. It also poses interesting challenges for existing vision and recognition algorithms, such as distinguishing between similar but different types of clothes or identifying a pattern of a cloth even if it has different colors and shapes. We formulate the annotation task as a multi-label and multi-modal classification problem: (i) both image and textual content (i.e., tags about the image) are available for learning classifiers, (ii) the classifiers must recognize a set of labels (i.e., a set of garment items), and (iii) the decision on which labels to assign to the query photo comes from a set of instances that is used to build a function, which separates labels that should be assigned to the query photo, from those that should not be assigned. Using this configuration, we propose two approaches: (i) the pointwise one, called MMCA, which receives a single image as input, and (ii) a multi-instance classification, called M3CA, also known as pairwise approach, which uses pair of images to create the classifiers. We conducted a systematic evaluation of the proposed algorithms using everyday photos collected from two major fashion-related social media, namely pose.com and chictopia.com. Our results show that the proposed approaches provide improvements when compared to popular first choice multi-label, multi-modal, multi-instance algorithms that range from 20 % to 30 % in terms of accuracy.     

Algorithms for photo book authoring

Photo books are a convenient and attractive product for printing and storage of personal photographs. However, the creation of a photo book requires a lot of manual work and time. The automation of operations will attract new users and will make it possible to create more photo books. Algorithms should be sufficiently fast enough, tune in to the tastes and habits of the user, and ensure a quality of work that satisfies the user. We propose several adaptive algorithms for creating photo books, i.e., the recognition of normal quality photos based on active learning, clustering images by event, selection of the most attractive photos, automatic placement of images on the page, and creation of collages. Approaches are also proposed for the adjustment of algorithms to the tastes and habits of the user in the process of operation.     

A clustering method for concurrent photos obtained from multiple cameras using max-flow network model

With the popularization of digital cameras, the use of several cameras by group photographers at the same event is becoming common. Photographers can share their contents and even take pictures of each other. So it is becoming important to manage concurrent photos from multiple cameras in order to classify many accumulated photos into proper clusters. In this paper, we propose a novel photo clustering method based on the max-flow network algorithm, and we visualize a network graph for cluster verification. To apply our algorithm, input concurrent photos are used to create an edge-weighted graph structure. In order to transform the photo clustering problem into a graph partition one, first we need to construct an Augmented Concurrent photo Graph (ACG) and then rewrite our original problem in terms of the graph partition one using the min-cut max-flow network model. The previous methods dealt with photo clustering as a 1-D problem using a linear partition. But we consider clustering for concurrent group photos as a 2-D partition based on other users’ photo contents. Each photo is used to create a node and similarities between photos are used to create the edge weights (capacities) of the network. We partition the network into two subgraphs according to the min-cut, which represents the weakest edge connections between the photos. Using repeated graph partitions for each subgraph (sub-network), we can obtain suitable subgraphs corresponding to photo clusters. The graph construction or partition can be adjusted according to user preferences in order to obtain the intended results.