A clustering method for concurrent photos obtained from multiple cameras using max-flow network model

With the popularization of digital cameras, the use of several cameras by group photographers at the same event is becoming common. Photographers can share their contents and even take pictures of each other. So it is becoming important to manage concurrent photos from multiple cameras in order to classify many accumulated photos into proper clusters. In this paper, we propose a novel photo clustering method based on the max-flow network algorithm, and we visualize a network graph for cluster verification. To apply our algorithm, input concurrent photos are used to create an edge-weighted graph structure. In order to transform the photo clustering problem into a graph partition one, first we need to construct an Augmented Concurrent photo Graph (ACG) and then rewrite our original problem in terms of the graph partition one using the min-cut max-flow network model. The previous methods dealt with photo clustering as a 1-D problem using a linear partition. But we consider clustering for concurrent group photos as a 2-D partition based on other users’ photo contents. Each photo is used to create a node and similarities between photos are used to create the edge weights (capacities) of the network. We partition the network into two subgraphs according to the min-cut, which represents the weakest edge connections between the photos. Using repeated graph partitions for each subgraph (sub-network), we can obtain suitable subgraphs corresponding to photo clusters. The graph construction or partition can be adjusted according to user preferences in order to obtain the intended results.     

Event detection in real time on a personal camera phone image collection

Grouping photos of the same event together is extremely useful for the management of personal photo collections. However, most methods cannot be applied to the problem of online event detection in embedded devices because they do not consider hardware constraints or a user’s photo-taking behavior. In this paper, we propose an efficient and effective event detection algorithm for managing personal photo collections in camera phones or digital cameras. The proposed algorithm fuses time and location information, which is deemed the most important information for personal photo management, and works in real time in embedded devices. We model event occurrences in a user’s photo-taking behavior as a Poisson process by imposing certain constraints on calculating the elapsed time. Location information is incorporated into event detection when confidence in a decision based on the Poisson process is not high enough. The algorithm is user-centric because it provides the unique capabilities of accepting and adjusting to user feedback. Our experiment results show that the proposed event detection method has the potential to support emerging multimedia applications in embedded devices.     

基于透视投影下空间光照一致性分析的图像拼接篡改检测

将一个人的头像剪切并拼接到另一张照片中,是一种常见的图像篡改手段.如果将该合成照片用于敲诈勒索,会对社会带来严重危害.因此,用来检测图像篡改的图像取证技术具有重大意义.由于不同照片成像环境不同,拼接时很难做到不同人脸的光照绝对一致,因此可以通过光照是否一致检测篡改.以往光照估计方法基于平行投影的假设,利用照片投影光照进行光照一致性分析.实际上,相机针孔模型是透视投影,从而导致上述检测方法出现误差.针对这一问题,本文提出一种透视投影下物体空间光照估计算法,将各人脸姿态统一到相机坐标系下,估计各人脸相对于相机坐标系的空间光照,然后分析空间光照一致性.另外,根据人脸空间光照一致性约束可以优化出相机参数,并得到该参数下的等效焦距、人脸空间位置及重新透视投影的图像等空间信息.本文将空间光照的一致性和上述空间信息的合理性作为依据,对人脸图像进行拼接篡改检测.实验结果表明,相比于传统方法基于平行投影光照进行光照一致性分析,采用本文提出的方法得到的空间光照进行光照一致性分析具有更高的准确度,结合相关信息进行照片空间合理性分析的篡改检测方法具有更强的说服力.     

Intelligent photo clustering with user interaction and distance metric learning

Photo clustering is an effective way to organize albums and it is useful in many applications, such as photo browsing and tagging. But automatic photo clustering is not an easy task due to the large variation of photo content. In this paper, we propose an interactive photo clustering paradigm that jointly explores human and computer. In this paradigm, the photo clustering task is semi-automatically accomplished: users are allowed to manually adjust clustering results with different operations, such as splitting clusters, merging clusters and moving photos from one cluster to another. Behind users’ operations, we have a learning engine that keeps updating the distance measurements between photos in an online way, such that better clustering can be performed based on the distance measure. Experimental results on multiple photo albums demonstrated that our approach is able to improve automatic photo clustering results, and by exploring distance metric learning, our method is much more effective than pure manual adjustments of photo clustering.     

Photobook generation algorithms

Photobooks are comfortable and attractive solution for personal photo printing and storing. Photobook generation requires a lot of manual operations and takes a lot of time. Automation process will involve new users and allow making more photobooks. The algorithms should be quite fast, adjusted by user’s predilections and habits and ensure satisfied quality of work. We propose several adaptive photobook generation algorithms: photo quality recognition based on active learning, event-aware photo grouping, attractive photos selection, automatic photos layout through photobook pages, collage generation. We also suggest approaches to algorithms adjustment by user preferences on-the-fly.     

Point of interest mining with proper semantic annotation

Mining geo-tagged social photo media has received large amounts of attention from researchers recently. Points of interest (POI) mining from a collection of geo-tagged photos is one of these problems. POI mining refers to the processes of pattern recognition (namely clustering), extraction and semantic annotation. However, based on unsupervised clustering methods, many POIs might not be mined. Additionally, there is a great challenge for the proper semantic annotation to data clusters after clustering. In practice, there are many applications which require the accuracy of semantic annotation and high quality of pattern recognition such as POI recommendation. In this paper, we study POI mining from a collection of geo-tagged photos in combination with proper semantic annotation by using additional POI information from high coverage external POI databases. We propose a novel POI mining framework by using two-level clustering, random walk and constrained clustering. In random walk clustering step, we separate a large-scale collection of geo-tagged photos into many clusters. In the constrained clustering step, we continue to divide the clusters that include many POIs into many sub-clusters, where the geo-tagged photos in a sub-cluster associate with a particular POI. Experimental results on two datasets of geo-tagged Flickr photos of two cities in California, USA have shown that the proposed method substantially outperforms existing approaches that are adapted to handle the problem.     

试飞测试中摄像机标定方法研究

随着数码相机技术和摄影测量技术的发展,越来越多的数字摄像机应用于试飞测试中,而摄像机标定是其成功应用于飞行试验的关键之一。为了突破试飞测试中现有的仅以点特征作为控制,充分利用现有设备条件,更好地解决加装在飞机上的摄像机在飞行过程中的实时标定问题,采取内标定与实时外标定两步实现摄像机标定的方法。着重探讨了一种基于平行直线的摄像机内标定方法,详细论述了基于灭点约束和直线几何约束的摄像机标定解算模型,该方法在无控制点的情况下可得到每个摄像机的内方位元素、各项畸变改正系数和外方位角元素;并简要介绍了基于单片后方交会的实时外标定方法。实际数据的试验结果表明,该方法切实可行,能够获得精确、稳定的参数结果,有效减少了摄像机标定过程中所需布设的控制点数,从而提高了试飞测试中精确测量导弹运动轨迹、机翼变形测量等工作的可实施性。     

Estimating the camera direction of a geotagged image using reference images

Millions of smart phones and GPS-equipped digital cameras sold each year, as well as photo-sharing websites such as Picasa and Panoramio have enabled personal photos to be associated with geographic information. It has been shown by recent research results that the additional global positioning system (GPS) information helps visual recognition for geotagged photos by providing valuable location context. However, the current GPS data only identifies the camera location, leaving the camera viewing direction uncertain within the possible scope of 360°. To produce more precise photo location information, i.e. the viewing direction for geotagged photos, we utilize both Google Street View and Google Earth satellite images. Our proposed system is two-pronged: (1) visual matching between a user photo and any available street views in the vicinity can determine the viewing direction, and (2) near-orthogonal view matching between a user photo taken on the ground and the overhead satellite view at the user geo-location can compute the viewing direction when only the satellite view is available. Experimental results have shown the effectiveness of the proposed framework.     

Multi-modal and multi-scale photo collection summarization

With the proliferation of digital cameras and mobile devices, people are taking much more photos than ever before. However, these photos can be redundant in content and varied in quality. Therefore there is a growing need for tools to manage the photo collections. One efficient photo management way is photo collection summarization which segments the photo collection into different events and then selects a set of representative and high quality photos (key photos) from those events. However, existing photo collection summarization methods mainly consider the low-level features for photo representation only, such as color, texture, etc, while ignore many other useful features, for example high-level semantic feature and location. Moreover, they often return fixed summarization results which provide little flexibility. In this paper, we propose a multi-modal and multi-scale photo collection summarization method by leveraging multi-modal features, including time, location and high-level semantic features. We first use Gaussian mixture model to segment photo collection into events. With images represented by those multi-modal features, our event segmentation algorithm can generate better performance since the multi-modal features can better capture the inhomogeneous structure of events. Next we propose a novel key photo ranking and selection algorithm to select representative and high quality photos from the events for summarization. Our key photo ranking algorithm takes the importance of both events and photos into consideration. Furthermore, our photo summarization method allows users to control the scale of event segmentation and number of key photos selected. We evaluate our method by extensive experiments on four photo collections. Experimental results demonstrate that our method achieves better performance than previous photo collection summarization methods.     

基于 TD_SCDMA 电子相框的照片分享方案

随着数码相机的普及,拍摄的数码照片呈几何级数增加,除了少部分会被冲洗成纸质照片以外,大部分数码照片都会被保存在电子存储设备中。电子相框是非常适合的存储设备,兼具展示数码照片的功能,近年来正在蓬勃发展。本文分析了电子相框发展的现状和业务需求,针对传统电子相框面临照片更新与共享的问题,提出了一种基于TD_SCDMA（Time Division-Synchronous Code Division Multiple Access）的照片分享方案,将电子相框接入UMTS（Universal Mobile Telecommunications System,通用移动通信系统）,采用电子相框统一服务平台实现了照片在手机、电脑和电子相框之间的无缝分享。     

Concurrent photo sequence organization

Personal photo album organization is a highly demanding domain where advanced tools are required to manage large photo collections. In contrast to many previous works, that try to solve the problem of organizing a single user photo sequence, we present a new technique to account for the concurrent photo sequence organization problem, that is the problem of organizing multiple photo sequences taken during the same event. Given a set of sequences acquired at the same place during the same temporal window by several users using different cameras, our framework is intended to capture the evolution of the event and groups photos based on temporal proximity and visual content. The method automatically organizes the reference sequence in a tree capturing the event structure. Such a structure is then used to align the remaining photo sequences to the reference one. We tested our approach on the publicly available Gallagher dataset and on a new dataset we collected; this new dataset is composed of four photo sequences taken by four users at a public event. Results demonstrate the effectiveness of our method.     

Automated image-orientation detection: a scalable boosting approach

With the proliferation of digital cameras and self-publishing of photos, automatic detection of image orientation has become an important part of photo-management systems. In this paper, we present a novel system, based on combining the outputs of hundreds of classifiers trained with AdaBoost, to determine the upright orientation of an image. We thoroughly test our system on photos gathered from professional and amateur photo collections that have been taken with a variety of cameras (digital, film, camera phones). The test images include photos that are in color and black and white, realistic and abstract, and outdoor and indoor. As this system is intended for mass consumer deployment, efficiency in use and accessibility is paramount. Results show that the presented method surpasses similar methods based on Support Vector Machines, in terms of both accuracy and feasibility of deployment.

---

Semi-automatic photo annotation strategies using event based clustering and clothing based person recognition

Managing a large number of digital photos is a challenging task for casual users. Personal photos often don’t have rich metadata, or additional information associated with them. However, available metadata can play a crucial role in managing photos. Labeling the semantic content of photos (i.e., annotating them), can increase the amount of metadata and facilitate efficient management. However, manual annotation is tedious and labor intensive while automatic metadata extraction techniques often generate inaccurate and irrelevant results. This paper describes a semi-automatic annotation strategy that takes advantage of human and computer strengths. The semi-automatic approach enables users to efficiently update automatically obtained metadata interactively and incrementally. Even though automatically identified metadata are compromised with inaccurate recognition errors, the process of correcting inaccurate information can be faster and easier than manually adding new metadata from scratch. In this paper, we introduce two photo clustering algorithms for generating meaningful photo groups: (1) Hierarchical event clustering; and (2) Clothing based person recognition, which assumes that people who wear similar clothing and appear in photos taken in one day are very likely to be the same person. To explore our semi-automatic strategies, we designed and implemented a prototype called SAPHARI (Semi-Automatic PHoto Annotation and Recognition Interface). The prototype provides an annotation framework which focuses on making bulk annotations on automatically identified photo groups. The prototype automatically creates photo clusters based on events, people, and file metadata so that users can easily bulk annotation photos. We performed a series of user studies to investigate the effectiveness and usability of the semi-automatic annotation techniques when applied to personal photo collections. The results show that users were able to make annotations significantly faster with event clustering using SAPHARI. We also found that users clearly preferred the semi-automatic approaches.     

PhotoGeo: a photo digital library with spatial-temporal support and self-annotation

The recent popularity of digital cameras has posed a new problem: how to efficiently store and retrieve the very large number of digital photos captured and chaotically stored in multiple locations without any annotation. This paper proposes an infrastructure, called PhotoGeo, which aims at helping users with the people photo annotation, event photo annotation, storage and retrieval of personal digital photos. To achieve the desired objective, PhotoGeo uses new algorithms that make it possible to annotate photos with the key metadata to facilitate their retrieval, such as: the people who were shown in the photo (who); where it was captured (where); the date and time of capture (when); and the event that was captured. The paper concludes with a detailed evaluation of these algorithms.     

家庭数码照片管理领域的本体建模研究*

数码相机的迅速普及对照片管理技术提出了严峻的挑战,传统的基于文本关键字的标注和检索系统已不能满足人们的需求。因此,基于语义的图像检索技术正在快速兴起,但语义知识的组织和表达问题始终未能得到有效解决。针对上述问题,提出了一种新的基于领域本体的照片管理方法。该方法的关键技术是对照片管理领域进行本体建模。实验表明了该方法的有效性。     

An on-line learning method for face association in personal photo collection

Due to the widespread use of cameras, it is very common to collect thousands of personal photos. A proper organization is needed to make the collection usable and to enable an easy photo retrieval. In this paper, we present a method to organize personal photo collections based on “who” is in the picture. Our method consists in detecting the faces in the photo sequence and arranging them in groups corresponding to the probable identities. This problem can be conveniently modeled as a multi-target visual tracking where a set of on-line trained classifiers is used to represent the identity models. In contrast to other works where clustering methods are used, our method relies on a probabilistic framework; it does not require any prior information about the number of different identities in the photo album. To enable future comparison, we present experimental results on a public dataset and on a photo collection generated from a public face dataset.     

Time-Delayed Correlation Analysis for Multi-Camera Activity Understanding

We propose a novel approach to understanding activities from their partial observations monitored through multiple non-overlapping cameras separated by unknown time gaps. In our approach, each camera view is first decomposed automatically into regions based on the correlation of object dynamics across different spatial locations in all camera views. A new Cross Canonical Correlation Analysis (xCCA) is then formulated to discover and quantify the time delayed correlations of regional activities observed within and across multiple camera views in a single common reference space. We show that learning the time delayed activity correlations offers important contextual information for (i) spatial and temporal topology inference of a camera network; (ii) robust person re-identification and (iii) global activity interpretation and video temporal segmentation. Crucially, in contrast to conventional methods, our approach does not rely on either intra-camera or inter-camera object tracking; it thus can be applied to low-quality surveillance videos featured with severe inter-object occlusions. The effectiveness and robustness of our approach are demonstrated through experiments on 330 hours of videos captured from 17 cameras installed at two busy underground stations with complex and diverse scenes.     

Prediction of user navigation patterns by mining the temporal web usage evolution

Advances in the data mining technologies have enabled the intelligent Web abilities in various applications by utilizing the hidden user behavior patterns discovered from the Web logs. Intelligent methods for discovering and predicting user’s patterns is important in supporting intelligent Web applications like personalized services. Although numerous studies have been done on Web usage mining, few of them consider the temporal evolution characteristic in discovering web user’s patterns. In this paper, we propose a novel data mining algorithm named Temporal N-Gram (TN-Gram) for constructing prediction models of Web user navigation by considering the temporality property in Web usage evolution. Moreover, three kinds of new measures are proposed for evaluating the temporal evolution of navigation patterns under different time periods. Through experimental evaluation on both of real-life and simulated datasets, the proposed TN-Gram model is shown to outperform other approaches like N-gram modeling in terms of prediction precision, in particular when the web user’s navigating behavior changes significantly with temporal evolution.     

Sorting unorganized photo sets for urban reconstruction

In spite of advanced acquisition technology, consumer cameras remain an attractive means for capturing 3D data. For reconstructing buildings it is easy to obtain large numbers of photos representing complete, all-around coverage of a building; however, such large photos collections are often unordered and unorganized, with unknown viewpoints. We present a method for reconstructing piecewise planar building models based on a near-linear time process that sorts such unorganized collections, quickly creating an image graph, an initial pose for each camera, and a piecewise-planar facade model. Our sorting technique first estimates single-view, piecewise planar geometry from each photo, then merges these single-view models together in an analysis phase that reasons about the global scene geometry. A key contribution of our technique is to perform this reasoning based on a number of typical constraints of buildings. This sorting process results in a piecewise planar model of the scene, a set of good initial camera poses, and a correspondence between photos. This information is useful in itself as an approximate scene model, but also represents a good initialization for structure from motion and multi-view stereo techniques from which refined models can be derived, at greatly reduced computational cost compared to prior techniques.