Photobook generation algorithms

Photobooks are comfortable and attractive solution for personal photo printing and storing. Photobook generation requires a lot of manual operations and takes a lot of time. Automation process will involve new users and allow making more photobooks. The algorithms should be quite fast, adjusted by user’s predilections and habits and ensure satisfied quality of work. We propose several adaptive photobook generation algorithms: photo quality recognition based on active learning, event-aware photo grouping, attractive photos selection, automatic photos layout through photobook pages, collage generation. We also suggest approaches to algorithms adjustment by user preferences on-the-fly.     

Digital photo classification methodology for groups of photographers

Digital cameras have become an essential product when traveling or attending events. Because of its popularity and low cost, it is increasingly likely that more than one camera will be used at an event. The total number of photos captured is also increasing. Although the cost of digital photographs is low, managing numerous digital photos is burdensome to most users. Thus, an intelligent management tool for digital photos is required. In this paper, we propose novel clustering algorithms for concurrent digital photos obtained from multiple cameras. Since previous studies only considered a single user’s photo collection, they are not applicable to concurrent photos obtained from a group of photographers. To handle this situation, we define temporal/spatial combined clustering for the set of group photos taken from different cameras. If photos are submitted from a camera whose user has shown a preference between spatial and temporal clustering, we can obtain customized clustering output from other photo sets according to the reference clustering characteristics. We also propose unsupervised methods for general clustering output. Input concurrent photos are processed without a user’s true clusters, which can be a burden when the number of photos in the true clusters is huge. We tested our methods via more than one thousand photos taken by tourist groups. The final result was satisfactory compared to previous methods based on temporal (spatial) criteria only.     

A personal perspective on photowork: implicit human–computer interaction for photo collection management

In the age of digital photography, the amount of photos we have in our personal collections has increased substantially along with the effort needed to manage these new, larger collections. This issue has already been addressed in various ways: from organization by meta-data analysis to image recognition and social network analysis. We introduce a new, more personal perspective on photowork that aims at understanding the user and his/her subjective relationship to the photos. It does so by means of implicit human–computer interaction, that is, by observing the user’s interaction with the photos. In order to study this interaction, we designed an experiment to see how people behave when manipulating photos on a tablet and how this implicitly conveyed information can be used to aid photo collection management.     

Event detection in real time on a personal camera phone image collection

Grouping photos of the same event together is extremely useful for the management of personal photo collections. However, most methods cannot be applied to the problem of online event detection in embedded devices because they do not consider hardware constraints or a user’s photo-taking behavior. In this paper, we propose an efficient and effective event detection algorithm for managing personal photo collections in camera phones or digital cameras. The proposed algorithm fuses time and location information, which is deemed the most important information for personal photo management, and works in real time in embedded devices. We model event occurrences in a user’s photo-taking behavior as a Poisson process by imposing certain constraints on calculating the elapsed time. Location information is incorporated into event detection when confidence in a decision based on the Poisson process is not high enough. The algorithm is user-centric because it provides the unique capabilities of accepting and adjusting to user feedback. Our experiment results show that the proposed event detection method has the potential to support emerging multimedia applications in embedded devices.     

Automatic tag expansion using visual similarity for photo sharing websites

In this paper we present an automatic photo tag expansion method designed for photo sharing websites. The purpose of the method is to suggest tags that are relevant to the visual content of a given photo at upload time. Both textual and visual cues are used in the process of tag expansion. When a photo is to be uploaded, the system asks for a couple of initial tags from the user. The initial tags are used to retrieve relevant photos together with their tags. These photos are assumed to be potentially content related to the uploaded target photo. The tag sets of the relevant photos are used to form the candidate tag list, and visual similarities between the target photo and relevant photos are used to give weights to these candidate tags. Tags with the highest weights are suggested to the user. The method is applied on Flickr (). Results show that including visual information in the process of photo tagging increases accuracy with respect to text-based methods.     

一种基于 Android 平台的安全图片传输技术

随着移动终端设备的进步与发展以及互联网的高速发展,人们的生活越来越依赖于互联网和移动终端设备。为了保护企业的商业机密以及用户的个人隐私,文章通过限制接收方对图片的控制权,并且在一定时段内自动销毁未授权用户接收的图片,使发送方的隐私权得到保护。基于移动互联的客户-服务器系统,充分利用服务器的下载上传功能,使其一次只能下载一张照片,从而不断替换掉已有的照片,并利用 Android 系统自带的 handler 多线程的延时功能,实现了照片的定时显示,使接收方在能够看见发送方的图片的同时却不能控制发送方的图片,保护发送方的隐私。使用过程简单明了,不需要过于复杂的操作。     

Estimating the camera direction of a geotagged image using reference images

Millions of smart phones and GPS-equipped digital cameras sold each year, as well as photo-sharing websites such as Picasa and Panoramio have enabled personal photos to be associated with geographic information. It has been shown by recent research results that the additional global positioning system (GPS) information helps visual recognition for geotagged photos by providing valuable location context. However, the current GPS data only identifies the camera location, leaving the camera viewing direction uncertain within the possible scope of 360°. To produce more precise photo location information, i.e. the viewing direction for geotagged photos, we utilize both Google Street View and Google Earth satellite images. Our proposed system is two-pronged: (1) visual matching between a user photo and any available street views in the vicinity can determine the viewing direction, and (2) near-orthogonal view matching between a user photo taken on the ground and the overhead satellite view at the user geo-location can compute the viewing direction when only the satellite view is available. Experimental results have shown the effectiveness of the proposed framework.     

Exploring Issues of User Model Transparency and Proactive Behaviour in an Office Environment Control System

It is important that systems that exhibit proactive behaviour do so in a way that does not surprise or frustrate the user. Consequently, it is desirable for such systems to be both personalised and designed in such a way as to enable the user to scrutinise her user model (part of which should hold the rules describing the behaviour of the system). This article describes on-going work to investigate the design of a prototype system that can learn a given user’s behaviour in an office environment in order to use the inferred rules to populate a user model and support appropriate proactive behaviour (e.g. turning on the user’s fan under appropriate conditions). We explore the tension between user control and proactive services and consider issues related to the design of appropriate transparency with a view to supporting user comprehensibility of system behaviour. To this end, our system enables the user to scrutinise and possibly over-ride the ‘IF-THEN’ rules held in her user model. The system infers these rules from the context history (effectively a data set generated using a variety of sensors) associated with the user by using a fuzzy-decision-tree-based algorithm that can provide a confidence level for each rule in the user model. The evolution of the system has been guided by feedback from a number of real-life users in a university department. A questionnaire study has yielded supplementary results concerning the extent to which the approach taken meets users’ expectations and requirements.     

Semi-automatic photo annotation strategies using event based clustering and clothing based person recognition

Managing a large number of digital photos is a challenging task for casual users. Personal photos often don’t have rich metadata, or additional information associated with them. However, available metadata can play a crucial role in managing photos. Labeling the semantic content of photos (i.e., annotating them), can increase the amount of metadata and facilitate efficient management. However, manual annotation is tedious and labor intensive while automatic metadata extraction techniques often generate inaccurate and irrelevant results. This paper describes a semi-automatic annotation strategy that takes advantage of human and computer strengths. The semi-automatic approach enables users to efficiently update automatically obtained metadata interactively and incrementally. Even though automatically identified metadata are compromised with inaccurate recognition errors, the process of correcting inaccurate information can be faster and easier than manually adding new metadata from scratch. In this paper, we introduce two photo clustering algorithms for generating meaningful photo groups: (1) Hierarchical event clustering; and (2) Clothing based person recognition, which assumes that people who wear similar clothing and appear in photos taken in one day are very likely to be the same person. To explore our semi-automatic strategies, we designed and implemented a prototype called SAPHARI (Semi-Automatic PHoto Annotation and Recognition Interface). The prototype provides an annotation framework which focuses on making bulk annotations on automatically identified photo groups. The prototype automatically creates photo clusters based on events, people, and file metadata so that users can easily bulk annotation photos. We performed a series of user studies to investigate the effectiveness and usability of the semi-automatic annotation techniques when applied to personal photo collections. The results show that users were able to make annotations significantly faster with event clustering using SAPHARI. We also found that users clearly preferred the semi-automatic approaches.     

A designerly critique on enchantment

To develop the concept of user experience in HCI, McCarthy et al. introduce the notion of enchantment in interaction design. They describe five sensibilities that support exploration and evaluation in design for enchantment. In this paper, we discuss design for enchantment in light of our approach to design for interaction, called design for meaningful mediation. Based on our experiences from case studies, we argue that ‘considering the whole person with feelings, desires and anxieties’, one of the sensibilities McCarthy et al. formulate, influences the desirability and realisation of the other four sensibilities. By way of case studies, we show how we explored the link between ‘the whole person’ and desired interaction experience in a designerly way. We place enchantment in a context of other interaction experiences and demonstrate possible design techniques relevant to design for interaction experiences, including enchantment.     

Intelligent photo clustering with user interaction and distance metric learning

Photo clustering is an effective way to organize albums and it is useful in many applications, such as photo browsing and tagging. But automatic photo clustering is not an easy task due to the large variation of photo content. In this paper, we propose an interactive photo clustering paradigm that jointly explores human and computer. In this paradigm, the photo clustering task is semi-automatically accomplished: users are allowed to manually adjust clustering results with different operations, such as splitting clusters, merging clusters and moving photos from one cluster to another. Behind users’ operations, we have a learning engine that keeps updating the distance measurements between photos in an online way, such that better clustering can be performed based on the distance measure. Experimental results on multiple photo albums demonstrated that our approach is able to improve automatic photo clustering results, and by exploring distance metric learning, our method is much more effective than pure manual adjustments of photo clustering.     

Easy on that trigger dad: a study of long term family photo retrieval

We examine the effects of new technologies for digital photography on people’s longer term storage and access to collections of personal photos. We report an empirical study of parents’ ability to retrieve photos related to salient family events from more than a year ago. Performance was relatively poor with people failing to find almost 40% of pictures. We analyze participants’ organizational and access strategies to identify reasons for this poor performance. Possible reasons for retrieval failure include: storing too many pictures, rudimentary organization, use of multiple storage systems, failure to maintain collections and participants’ false beliefs about their ability to access photos. We conclude by exploring the technical and theoretical implications of these findings.     

Adaptation and user expertise modelling in AthosMail

This article describes the User Model component of AthosMail, a speech-based interactive e-mail application developed in the context of the EU project DUMAS. The focus is on the system’s adaptive capabilities and user expertise modelling, exemplified through the User Model parameters dealing with initiative and explicitness of the system responses. The purpose of the conducted research was to investigate how the users could interact with a system in a more natural way, and the two aspects that mainly influence the system’s interaction capabilities, and thus the naturalness of the dialogue as a whole, are considered to be the dialogue control and the amount of information provided to the user. The User Model produces recommendations of the system’s appropriate reaction depending on the user’s observed competence level, monitored and computed on the basis of the user’s interaction with the system. The article also discusses methods for the evaluation of adaptive user models and presents results from the AthosMail evaluation.The research was done while the author was affiliated with the University of Art and Design Helsinki as the scientific coordinator of the DUMAS project.     

Concurrent photo sequence organization

Personal photo album organization is a highly demanding domain where advanced tools are required to manage large photo collections. In contrast to many previous works, that try to solve the problem of organizing a single user photo sequence, we present a new technique to account for the concurrent photo sequence organization problem, that is the problem of organizing multiple photo sequences taken during the same event. Given a set of sequences acquired at the same place during the same temporal window by several users using different cameras, our framework is intended to capture the evolution of the event and groups photos based on temporal proximity and visual content. The method automatically organizes the reference sequence in a tree capturing the event structure. Such a structure is then used to align the remaining photo sequences to the reference one. We tested our approach on the publicly available Gallagher dataset and on a new dataset we collected; this new dataset is composed of four photo sequences taken by four users at a public event. Results demonstrate the effectiveness of our method.     

An ontological modelling of user requirements for personalised information provision

The knowledge economy offers opportunity to a broad and diverse community of information systems users to efficiently gain information and know-how for improving qualifications and enhancing productivity in the work place. Such demand will continue and users will frequently require optimised and personalised information content. The advancement of information technology and the wide dissemination of information endorse individual users when constructing new knowledge from their experience in the real-world context. However, a design of personalised information provision is challenging because users’ requirements and information provision specifications are complex in their representation. The existing methods are not able to effectively support this analysis process. This paper presents a mechanism which can holistically facilitate customisation of information provision based on individual users’ goals, level of knowledge and cognitive styles preferences. An ontology model with embedded norms represents the domain knowledge of information provision in a specific context where users’ needs can be articulated and represented in a user profile. These formal requirements can then be transformed onto information provision specifications which are used to discover suitable information content from repositories and pedagogically organise the selected content to meet the users’ needs. The method is provided with adaptability which enables an appropriate response to changes in users’ requirements during the process of acquiring knowledge and skills.     

基于语义的照片检索的相关问题研究

随着数码产品的普及,数码照片的积累越来越多。为了更好的满足用户语义检索照片的需求,各国研究者做了很多相关工作。他们研究照片的本质和用户使用照片的习惯,使用这些知识来帮助检索和管理照片。分析和总结这些研究,主要介绍基于语义的数码照片检索研究工作中语义提取的方法、运用语义Web技术现状及其在照片管理中的应用。     

Large-scale user modeling with recurrent neural networks for music discovery on multiple time scales

The amount of content on online music streaming platforms is immense, and most users only access a tiny fraction of this content. Recommender systems are the application of choice to open up the collection to these users. Collaborative filtering has the disadvantage that it relies on explicit ratings, which are often unavailable, and generally disregards the temporal nature of music consumption. On the other hand, item co-occurrence algorithms, such as the recently introduced word2vec-based recommenders, are typically left without an effective user representation. In this paper, we present a new approach to model users through recurrent neural networks by sequentially processing consumed items, represented by any type of embeddings and other context features. This way we obtain semantically rich user representations, which capture a user’s musical taste over time. Our experimental analysis on large-scale user data shows that our model can be used to predict future songs a user will likely listen to, both in the short and long term.

     

Hyperlinking reality via camera phones

A novel user interface concept for camera phones, called “Hyperlinking Reality via Camera Phones”, that we present in this article, provides a solution to one of the main challenges facing mobile user interfaces, that is, the problem of selection and visualization of actions that are relevant to the user in her current context. Instead of typing keywords on a small and inconvenient keypad of a mobile device, a user of our system just snaps a photo of her surroundings and objects in the image become hyperlinks to information. Our method commences by matching a query image to reference panoramas depicting the same scene that were collected and annotated with information beforehand. Once the query image is related to the reference panoramas, we transfer the relevant information from the reference panoramas to the query image. By visualizing the information on the query image and displaying it on the camera phone’s (multi-)touch screen, the query image augmented with hyperlinks allows the user intuitive access to information.     

移动相册的复杂网络分析*

带有照相功能的移动终端的普及,产生了大量的照片,记录了用户的日常生活,这些照片附带了大量的多维元数据如时间、地点、作者、事件等,如何利用多维的元数据有效管理这些照片是一个研究热点。创新地提出利用复杂网络的方法分析和可视化移动相册,将照片看做一个节点,利用照片的多维元数据之间的相关性建立照片之间的边,这样所有在相册中的照片就组成了一个多维复杂加权网络。分析了照片复杂加权网络的统计特性,揭示了其小世界特性。利用照片复杂网络,不仅可以可视化照片之间的多维关系,而且可以快速检索相关照片。