Facing scalability: Naming faces in an online social network

Automatically naming faces in online social networks enables us to search for photos and build user face models. We consider two common weakly supervised settings where: (1) users are linked to photos, not to faces and (2) photos are not labeled but part of a user's album. The focus is on algorithms that scale up to an entire online social network. We extensively evaluate different graph-based strategies to label faces in both settings and consider dependencies. We achieve results on a par with a recent multi-person approach, but with 60 times less computation time on a set of 300K weakly labeled faces and 1.4 M faces in user albums. A subset of the faces can be labeled with a speed-up of over three orders of magnitude.     

Robust event discovery from photo collections using Signature Image Bases (SIBs)

Analyzing personal photo albums for understanding the related events is an emerging trend. A reliable event recognition tool could suggest appropriate annotation of pictures, provide the context for single image classification and tagging, achieve automatic selection and summarization, ease organization and sharing of media among users. In this paper, a novel method for fast and reliable event-type classification of personal photo albums is presented. Differently from previous approaches, the proposed method does not process photos individually but as a whole, exploiting three main features, namely Saliency, Gist, and Time, to extract an event signature, which is characteristic for a specific event type. A highly challenging database containing more than 40.000 photos belonging to 19 diverse event-types was crawled from photo-sharing websites for the purpose of modeling and performance evaluation. Experimental results showed that the proposed approach meets superior classification accuracy with limited computational complexity.     

Intelligent photo clustering with user interaction and distance metric learning

Photo clustering is an effective way to organize albums and it is useful in many applications, such as photo browsing and tagging. But automatic photo clustering is not an easy task due to the large variation of photo content. In this paper, we propose an interactive photo clustering paradigm that jointly explores human and computer. In this paradigm, the photo clustering task is semi-automatically accomplished: users are allowed to manually adjust clustering results with different operations, such as splitting clusters, merging clusters and moving photos from one cluster to another. Behind users’ operations, we have a learning engine that keeps updating the distance measurements between photos in an online way, such that better clustering can be performed based on the distance measure. Experimental results on multiple photo albums demonstrated that our approach is able to improve automatic photo clustering results, and by exploring distance metric learning, our method is much more effective than pure manual adjustments of photo clustering.     

Tile selection method based on error minimization for photomosaic image creation

Photomosaic images are composite images composed of many small images called tiles. In its overall visual effect, a photomosaic image is similar to the target image, and photomosaics are also called “montage art”. Noisy blocks and the loss of local information are the major obstacles in most methods or programs that create photomosaic images. To solve these problems and generate a photomosaic image in this study, we propose a tile selection method based on errorminimization. A photomosaic image can be generated by partitioning the target image in a rectangular pattern, selecting appropriate tile images, and then adding them with a weight coefficient. Based on the principles of montage art, the quality of the generated photomosaic image can be evaluated by both global and local error. Under the proposed framework, via an error function analysis, the results show that selecting a tile image using a global minimum distance minimizes both the global error and the local error simultaneously. Moreover, the weight coefficient of the image superposition can be used to adjust the ratio of the global and local errors. Finally, to verify the proposed method, we built a new photomosaic creation dataset during this study. The experimental results show that the proposed method achieves a lowmean absolute error and that the generated photomosaic images have a more artistic effect than do the existing approaches.     

Visualizing Social Photos on a Hasse Diagram for Eliciting Relations and Indexing New Photos

Social photos, which are taken during family events or parties, represent individuals or groups of people. We show in this paper how a Hasse diagram is an efficient visualization strategy for eliciting different groups and navigating through them. However, we do not limit this strategy to these traditional uses. Instead we show how it can also be used for assisting in indexing new photos.Indexing consists of identifying the event and people in photos. It is an integral phase that takes place before searching and sharing. In our method we use existing indexed photos to index new photos. This is performed through a manual drag and drop procedure followed by a content fusion process that we call ’propagation’. At the core of this process is the necessity to organize and visualize the photos that will be used for indexing in a manner that is easily recognizable and accessible by the user. In this respect we make use of an Object Galois Sub-Hierarchy and display it using a Hasse diagram. The need for an incremental display that maintains the user’s mental map also leads us to propose a novel way of building the Hasse diagram. To validate the approach, we present some tests conducted with a sample of users that confirm the interest of this organization, visualization and indexation approach. Finally, we conclude by considering scalability, the possibility to extract social networks and automatically create personalised albums.     

Semantic analysis and retrieval in personal and social photo collections

Semantic understanding of images has been an important topic in the research community for a long time as it is an important prerequisite to build meaningful retrieval systems which are accessible by both users and automatic reasoning algorithms. Recently, especially with the growing trend to share photos online, the social aspect of image retrieval becomes more and more prevalent and image retrieval more and more focusses specifically on photos and their special characteristics, especially on information outside the image itself. Researchers are starting to explore how and why photos are shot, shared and used and try to incorporate this additional knowledge to aid image analysis and retrieval. Several survey papers have been written in the past reviewing works in the general field of image analysis and retrieval. However, the social aspect of image retrieval and the focus on digital photos has not sufficiently been addressed in these works. In this article we give an overview over the current research field of semantic photo understanding, annotation and retrieval. We review over 160 contributions in the field and identify trending topics and implications for future directions of research.     

BFF: A tool for eliciting tie strength and user communities in social networking services

The use of social networking services (SNSs) such as Facebook has explosively grown in the last few years. Users see these SNSs as useful tools to find friends and interact with them. Moreover, SNSs allow their users to share photos, videos, and express their thoughts and feelings. However, users are usually concerned about their privacy when using SNSs. This is because the public image of a subject can be affected by photos or comments posted on a social network. In this way, recent studies demonstrate that users are demanding better mechanisms to protect their privacy. An appropriate approximation to solve this could be a privacy assistant software agent that automatically suggests a privacy policy for any item to be shared on a SNS. The first step for developing such an agent is to be able to elicit meaningful information that can lead to accurate privacy policy predictions. In particular, the information needed is user communities and the strength of users’ relationships, which, as suggested by recent empirical evidence, are the most important factors that drive disclosure in SNSs. Given the number of friends that users can have and the number of communities they may be involved on, it is infeasible that users are able to provide this information without the whole eliciting process becoming confusing and time consuming. In this work, we present a tool called Best Friend Forever (BFF) that automatically classifies the friends of a user in communities and assigns a value to the strength of the relationship ties to each one. We also present an experimental evaluation involving 38 subjects that showed that BFF can significantly alleviate the burden of eliciting communities and relationship strength.     

Photomosaic Generation and Representative Photomosaic Selection for Photograph Browsing

Photomosaic arranges many small photographs to represent a large image. The study applies the photomosaic to a photograph browser CAT. The implementation displays photomosaic while zooming out, and individual photographs while zooming in. Here, many photograph browsing software displays a set of photographs in the order of their timestamps. To maintain this order of photographs, the photomosaic-like image generation technique rstly arranges the given set of photographs in the order of their timestamps, and then retouches so that the set of photographs forms a photomosaic-like scene. This paper presents the technique for photomosaic generation, a user evaluation to discuss what kinds of photographs are preferable to be applied, and an automatic photomosaic selection technique.     

家庭数码照片管理领域的本体建模研究*

数码相机的迅速普及对照片管理技术提出了严峻的挑战,传统的基于文本关键字的标注和检索系统已不能满足人们的需求。因此,基于语义的图像检索技术正在快速兴起,但语义知识的组织和表达问题始终未能得到有效解决。针对上述问题,提出了一种新的基于领域本体的照片管理方法。该方法的关键技术是对照片管理领域进行本体建模。实验表明了该方法的有效性。     

视频相册系统

为了对视频数据进行有效的管理,提出了一种新的视频检索与浏览系统——视频相册系统。在该系统中,首先用相册生成方案挑选出用户数字视频库的一组代表性的关键帧;接着筛选出的关键帧被预训练的形状模板(如圆形、心形、扇形、邮票形等)所裁剪,最终被打印成册。当用户想浏览视频时,可事先浏览该视频相册,就像浏览普通相册一样,若用户想观看相册中某个关键帧所代表的视频片段,即可首先方便地用摄像手机等设备拍摄该关键帧,并通过无线网络(如蓝牙)把拍摄的图像传输给计算机终端;此后,视频相册系统采用基于自训练与Snakes轮廓进化的活动形状模型算法来定位关键帧在拍摄的图像中的轮廓位置,并纠正其成像畸变。最终,系统即可在视频数据库中自动找到与纠正后的关键帧最相似的一幅,并为用户回放其代表的视频片段。实验评测结果表明,该视频相册系统可在数字视频与模拟相册间建立有效的联系。     

Leveraging personal photos to inferring friendships in social network services

Social network services have become widely used as an important tool to share rich information; in such networks, making new friends is the most basic functionality to enable users to take advantage of their social networks. In this paper we look into personal photos as an additional source for social network analysis and analyze the potential of people tags in the photos for friend recommendations. We also propose a new compact data structure, collectively called Face Co-Occurrence Networks (FCON), which stores crucial and quantitative information about people’s appearance in photos. We discover strong associative relationships among people and recommend reliable social friends by utilizing FCON. Experimental results demonstrate the effectiveness and efficiency of our method for recommending friends in social network services.     

Cross-domain semantic transfer from large-scale social media

With the rapid development of location-based social networks (LBSNs), more and more media data are unceasingly uploaded by users. The asynchrony between the visual and textual information has made it extremely difficult to manage the multimodal information for manual annotation-free retrieval and personalized recommendation. Consequently the automated image semantic discovery of multimedia location-related user-generated contents (UGCs) for user experience has become mandatory. Most of the literatures leverage single-modality data or correlated multimedia data for image semantic detection. However, the intrinsically heterogeneous UGCs in LBSNs are usually independent and uncorrelated. It is hard to build correlation between textual information and visual information. In this paper, we propose a cross-domain semantic modeling method for automatic image annotation for visual information from social network platforms. First, we extract a set of hot topics from the collected textual information for image dataset preparation. Then the proposed noisy sample filtering is implemented to remove low-relevance photos. Finally, we leverage cross-domain datasets to discover the common knowledge of each semantic concept from UGCs and boost the performance of semantic annotation by semantic transfer. The comparison experiments on cross-domain datasets were conducted to demonstrate the superiority of the proposed method.     

A clustering method for concurrent photos obtained from multiple cameras using max-flow network model

With the popularization of digital cameras, the use of several cameras by group photographers at the same event is becoming common. Photographers can share their contents and even take pictures of each other. So it is becoming important to manage concurrent photos from multiple cameras in order to classify many accumulated photos into proper clusters. In this paper, we propose a novel photo clustering method based on the max-flow network algorithm, and we visualize a network graph for cluster verification. To apply our algorithm, input concurrent photos are used to create an edge-weighted graph structure. In order to transform the photo clustering problem into a graph partition one, first we need to construct an Augmented Concurrent photo Graph (ACG) and then rewrite our original problem in terms of the graph partition one using the min-cut max-flow network model. The previous methods dealt with photo clustering as a 1-D problem using a linear partition. But we consider clustering for concurrent group photos as a 2-D partition based on other users’ photo contents. Each photo is used to create a node and similarities between photos are used to create the edge weights (capacities) of the network. We partition the network into two subgraphs according to the min-cut, which represents the weakest edge connections between the photos. Using repeated graph partitions for each subgraph (sub-network), we can obtain suitable subgraphs corresponding to photo clusters. The graph construction or partition can be adjusted according to user preferences in order to obtain the intended results.     

TouchTone: Interactive Local Image Adjustment Using Point‐and‐Swipe

Recent proliferation of camera phones, photo sharing and social network services has significantly changed how we process our photos. Instead of going through the traditional download‐edit‐share cycle using desktop editors, an increasing number of photos are taken with camera phones and published through cellular networks. The immediacy of the sharing process means that on‐device image editing, if needed, should be quick and intuitive. However, due to the limited computational resources and vastly different user interaction model on small screens, most traditional local selection methods can not be directly adapted to mobile devices. To address this issue, we present TouchTone, a new method for edge‐aware image adjustment using simple finger gestures. Our method enables users to select regions within the image and adjust their corresponding photographic attributes simultaneously through a simple point‐and‐swipe interaction. To enable fast interaction, we develop a memory‐ and computation‐efficient algorithm which samples a collection of 1D paths from the image, computes the adjustment solution along these paths, and interpolates the solutions to entire image through bilateral filtering. Our system is intuitive to use, and can support several local editing tasks, such as brightness, contrast, and color balance adjustments, within a minute on a mobile device.     

Gender differences in Facebook self-presentation: An international randomized study

Facebook is a popular social network that can be used for self-presentation. In the current study we examined gender differences in Facebook self-presentation by evaluating components of profile and cover photos. We used evolutionary psychology—a theory which holds many assumptions regarding gender differences—to draw hypotheses. In order to eliminate the pitfalls of self-reported data, we analyzed public data presented in Facebook pages of a random representative international sample of 500 Facebook users. As hypothesized, profile photos on Facebook differed according to gender. Males’ photos accentuated status (using objects or formal clothing) and risk taking (outdoor settings), while females’ photos accentuated familial relations (family photos) and emotional expression (eye contact, smile intensity and lack of sunglasses). Cover photos, however, did not show most of these gender differences, perhaps since they serve only as a supplement to the self-presentation that appears in the profile photos. These findings demonstrate that evolutionary theory rooted in the past can help us understand new social tools of the future.     

A spatio-temporal semantic model for multimedia database systemsand multimedia information systems

As more information sources become available in multimedia systems, the development of abstract semantic models for video, audio, text, and image data is becoming very important. An abstract semantic model has two requirements: it should be rich enough to provide a friendly interface of multimedia presentation synchronization schedules to the users and it should be a good programming data structure for implementation in order to control multimedia playback. An abstract semantic model based on an augmented transition network (ATN) is presented. The inputs for ATNs are modeled by multimedia input strings. Multimedia input strings provide an efficient means for iconic indexing of the temporal/spatial relations of media streams and semantic objects. An ATN and its subnetworks are used to represent the appearing sequence of media streams and semantic objects. The arc label is a substring of a multimedia input string. In this design, a presentation is driven by a multimedia input string. Each subnetwork has its own multimedia input string. Database queries relative to text, image, and video can be answered via substring matching at subnetworks. Multimedia browsing allows users the flexibility to select any part of the presentation they prefer to see. This means that the ATN and its subnetworks can be included in multimedia database systems which are controlled by a database management system (DBMS). User interactions and loops are also provided in an ATN. Therefore, ATNs provide three major capabilities: multimedia presentations, temporal/spatial multimedia database searching, and multimedia browsing     

Real-time video photomosaics with optimized image set and GPU

We propose a real-time approach to automatically generate photomosaic videos from a set of optimized images by taking advantage of CUDA GPU acceleration. Our approach divides an input image into smaller cells—usually rectangular cells—and replaces each cell with a small image of an appropriate color pattern. Photomosaics require a large set of tile images with a variety of patterns to create high-quality digital mosaic images. Because a large database of images requires longer processing time and larger storage space for searching patterns from the database, this requirement causes problems in developing a real-time system or mobile applications that have limited resources. This paper deals with a real-time video photomosaics using genetic feature selection method for building an optimized image set and taking advantage of CUDA to accelerate pattern searching that minimizes computation cost.     

基于多尺度注意力半监督学习的老照片划痕修复

老照片由于长时间的磨损或保存不当,会出现照片的划痕损伤。随着深度学习在图像重建中的应用,基于深度学习方法能够在纹理修复的基础上获取图像的语义信息并预测语义内容,使老照片修复的整体效果更加符合客观事实,但利用深度学习进行老照片划痕修复缺乏学习所需数据集。提出一种基于半监督学习的老照片划痕自动修复的方法,创建划痕合成数据集SynOld用于网络训练,同时搜集真实的划痕老照片用于训练和测试,将合成数据集和真实老照片加入网络学习,两者共享网络参数,并通过鉴别器来区分网络生成图像与真实图像。对于合成数据集有监督的分支采用均方差损失、感知损失和对抗损失约束训练,对于真实老照片无监督的分支采用总变差损失控制训练。实验结果表明,相比于多尺度特征注意力网络的监督学习方法,该方法在合成数据集SynOld和真实老照片上都具有较好的修复效果。     

Automatic rule generation using crowdsourcing for better relationship type discovery

With the increasing popularity of information sharing and the growing number of social network users, relationship management is one of the key challenges which arise in the context of social networks. One particular relationship management task aims at identifying relationship types that are relevant between social network users and their contacts. Manually identifying relationship types is one possible solution, however it is a time-consuming and tedious task that requires constant maintenance. In this paper, we present a rule-based approach that sets the focus on published photos as a valuable source to identify relationship types. Our approach automatically generates relevant relationship discovery rules based on a crowdsourcing methodology that constructs useful photo datasets. Knowledge is first retrieved from these datasets and then used to create relationship discovery rules. The obtained set of rules is extended using a number of predefined common sense rules and then personalized using a rule mining algorithm. Experimental results demonstrate the correctness and the efficiency of the generated sets of rules to identify relationship types.     

Searching Activity Trajectories with Semantics

Journal of Computer Science and Technology - With the widespread use of smart phones and mobile Internet, social network users have generated massive geo-tagged tweets, photos and videos to form...