On the impact of knowledge extraction and aggregation on crowdsourced annotation of visual artworks

Cultural heritage institutions more and more provide online access to their collections. Collections containing visual artworks need detailed and thorough annotations of the represented visual objects (e.g. plants or animals) to enable human access and retrieval. To make these suitable for access and retrieval, visual artworks need detailed and thorough annotations of the visual classes. Crowdsourcing has proven a viable tool to cater for the pitfalls of automatic annotation techniques. However, differently from traditional photographic image annotation, the artwork annotation task requires workers to possess the knowledge and skills needed to identify and recognise the occurrences of visual classes. The extent to which crowdsourcing can be effectively applied for artwork annotation is still an open research question. Based on a real-life case study from Rijksmuseum Amsterdam, this paper investigates the performance of a crowd of workers drawn from the CrowdFlower platform. Our contributions include a detailed analysis of crowd annotations based on two annotation configurations and a comparison of these crowd annotations with the ones from trusted annotators. In this study we apply a novel method for the automatic aggregation of local (i.e. bounding box) annotations, and we study how different knowledge extraction and aggregation configurations affect the identification and recognition aspects of artwork annotation. Our work sheds new light on the process of crowdsourcing artwork annotations, and shows how techniques that are effective for photographic image annotation cannot be straightforwardly applied to artwork annotation, thus paving the way for new research in the area.     

Cross-language transfer of semantic annotation via targeted crowdsourcing: task design and evaluation

Modern data-driven spoken language systems (SLS) require manual semantic annotation for training spoken language understanding parsers. Multilingual porting of SLS demands significant manual effort and language resources, as this manual annotation has to be replicated. Crowdsourcing is an accessible and cost-effective alternative to traditional methods of collecting and annotating data. The application of crowdsourcing to simple tasks has been well investigated. However, complex tasks, like cross-language semantic annotation transfer, may generate low judgment agreement and/or poor performance. The most serious issue in cross-language porting is the absence of reference annotations in the target language; thus, crowd quality control and the evaluation of the collected annotations is difficult. In this paper we investigate targeted crowdsourcing for semantic annotation transfer that delegates to crowds a complex task such as segmenting and labeling of concepts taken from a domain ontology; and evaluation using source language annotation. To test the applicability and effectiveness of the crowdsourced annotation transfer we have considered the case of close and distant language pairs: Italian–Spanish and Italian–Greek. The corpora annotated via crowdsourcing are evaluated against source and target language expert annotations. We demonstrate that the two evaluation references (source and target) highly correlate with each other; thus, drastically reduce the need for the target language reference annotations.     

Facilitating Collocated Crowdsourcing on Situated Displays

Online crowdsourcing enables the distribution of work to a global labor force as small and often repetitive tasks. Recently, situated crowdsourcing has emerged as a complementary enabler to elicit labor in specific locations and from specific crowds. Teamwork in online crowdsourcing has been recently shown to increase the quality of output, but teamwork in situated crowdsourcing remains unexplored. We set out to fill this gap. We present a generic crowdsourcing platform that supports situated teamwork and provide experiences from a laboratory study that focused on comparing traditional online crowdsourcing to situated team-based crowdsourcing. We built a crowdsourcing desk that hosts three networked terminal displays. The displays run our custom team-driven crowdsourcing platform that was used to investigate collocated crowdsourcing in small teams. In addition to analyzing quantitative data, we provide findings based on questionnaires, interviews, and observations. We highlight 1) emerging differences between traditional and collocated crowdsourcing, 2) the collaboration strategies that teams exhibited in collocated crowdsourcing, and 3) that a priori team familiarity does not significantly affect collocated interaction in crowdsourcing. The approach we introduce is a novel multi-display crowdsourcing setup that supports collocated labor teams and along with the reported study makes specific contributions to situated crowdsourcing research.     

Competition matters! Self-efficacy,effort, and performance in crowdsourcing teams

Advances in information technologies (IT) have enabled organizations to seek solutions for their business problems from beyond their own workforce through digital crowdsourcing platforms. In the most common form of crowdsourcing, teams that offer solutions compete for rewards. Thus, a question of interest is whether competition is a key crowdsourcing characteristic that influences how teams allocate their effort and achieve desired performance. Motivated by this question, we investigate how competition moderates the relationship between self-efficacy and effort using comprehensive, time-variant data collected from crowdsourcing teams that completed a project under competitive and non-competitive conditions. Under competitive conditions, self-efficacy shows a positive effect on effort, which in turn, affects performance positively. Whereas, under noncompetitive conditions, self-efficacy has a negative effect on effort and subsequently on performance. Our results also show a recursive relationship between self-efficacy and performance, in which performance subsequently affects self-efficacy positively. Thus, inducing a sense of competition through competitive reward structures and IT-based “gaming elements” helps improve team effort and subsequent performance. We also tested for mediation of team motivation in the self-efficacy and effort relationship, and we found that motivation partially mediates the relationship. Based on our findings, implications for both theory and practice are discussed.     

Part and Attribute Discovery from Relative Annotations

Part and attribute based representations are widely used to support high-level search and retrieval applications. However, learning computer vision models for automatically extracting these from images requires significant effort in the form of part and attribute labels and annotations. We propose an annotation framework based on comparisons between pairs of instances within a set, which aims to reduce the overhead in manually specifying the set of part and attribute labels. Our comparisons are based on intuitive properties such as correspondences and differences, which are applicable to a wide range of categories. Moreover, they require few category specific instructions and lead to simple annotation interfaces compared to traditional approaches. On a number of visual categories we show that our framework can use noisy annotations collected via “crowdsourcing” to discover semantic parts useful for detection and parsing, as well as attributes suitable for fine-grained recognition.     

An impactful crowdsourcing intermediary design - a case of a service imagery crowdsourcing system

Crowdsourcing is usually implemented using an intermediary organization or directly coordinated by solution seekers. For those using an IT-based crowdsourcing intermediary, the important factors of intermediaries that impact the crowdsourcing outcome are yet unclear. In addition, little research was conducted on how to design a crowdsourcing intermediary that can address the combined challenges of considering different cognitive demand levels of solution providers’ contributions, combining contributions and evaluating contributions. This paper identifies three important factors and provides a novel design of crowdsourcing intermediary to cope with these challenges. This study uses a case that focuses on how to assist small and medium businesses (SMBs) to develop their service imageries in triggering service innovation and designing their service experiences as to fulfill the desired outcomes of customers. Through the case, the benefits of our crowdsourcing intermediary design are demonstrated and justified. The three important factors are the crowdsourcing intermediary knowledge base, generative networks and empowerment of crowd members. This study shows that the crowdsourcing process can facilitate achieving a higher chance of attaining creative solutions for SMBs’ innovation problems when the three factors are well incorporated and managed within the crowdsourcing intermediary design. This study also presents a novel design of crowdsourcing intermediary that can address the combined challenges of coping with different cognitive demand levels of crowd members and combining and evaluating crowd members’ contributions, in order to attain impactful crowdsourcing outcome.     

Information Visualization Evaluation Using Crowdsourcing

Visualization researchers have been increasingly leveraging crowdsourcing approaches to overcome a number of limitations of controlled laboratory experiments, including small participant sample sizes and narrow demographic backgrounds of study participants. However, as a community, we have little understanding on when, where, and how researchers use crowdsourcing approaches for visualization research. In this paper, we review the use of crowdsourcing for evaluation in visualization research. We analyzed 190 crowdsourcing experiments, reported in 82 papers that were published in major visualization conferences and journals between 2006 and 2017. We tagged each experiment along 36 dimensions that we identified for crowdsourcing experiments. We grouped our dimensions into six important aspects: study design & procedure, task type, participants, measures & metrics, quality assurance, and reproducibility. We report on the main findings of our review and discuss challenges and opportunities for improvements in conducting crowdsourcing studies for visualization research.     

时空众包数据管理技术研究综述

近年来,众包为传统数据管理提供了一种通过汇聚群体智慧求解问题的新模式,并成为当前数据库领域的研究热点之一.特别是随着移动互联网技术与共享经济模式的快速发展,众包技术已融入到各类具有时空数据的应用场景中,例如各类O2O(Online-To-Offline)应用,实时交通监控与动态物流管理等.简言之,这种应用众包技术处理时空数据的方式称为时空众包数据管理.本文对近期在时空众包数据管理方面的研究工作进行综述,首先阐述了时空众包的概念,解释了其与传统众包技术的关系,并介绍了各类典型的时空众包应用;随后描述了时空众包应用平台的工作流程及其任务特点;然后讨论了时空众包数据管理的三项核心研究问题和三类应用技术.最后,本文总结了时空众包数据管理技术的研究现状并展望了其未来潜在研究方向,为相关研究人员提供有价值的参考.     

Crowdsourcing privacy preferences in context-aware applications

Developers of context-aware applications are faced with a tough challenge: powerful privacy controls are essential to maintain user trust, but they are also hard to use and not adequate in all situations. To address this tradeoff, we present Super-Ego, a crowdsourcing framework for privacy management of location information in ubiquitous environment. We study how crowdsourcing can be used to predict the user’s privacy preferences for different location on the basis of the general user population. The crowdsourcing methods are evaluated in a 2-week user study in which we tracked the locations of 30 subjects and asked them to provide privacy preferences for the locations they had visited. Our results show that by employing simple methods for semantic analysis of locations and by profiling the user’s privacy inclination, our methods can accurately predict the privacy preferences for 80 % of the user’s locations. By employing semi-automatic decision strategies, which ask the user to decide regarding the privacy of some of the locations, the accuracy rate raises to 90 %.     

Deep Generative Crowdsourcing Learning with Worker Correlation Utilization

Traditional supervised learning requires the groundtruth labels for the training data, which can be difficult to collect in many cases. In contrast, crowdsourcing learning collects noisy annotations from multiple non-expert workers and infers the latent true labels through some aggregation approach. In this paper, we notice that existing deep crowdsourcing work does not sufficiently model worker correlations, which is, however, shown to be helpful for learning by previous non-deep learning approaches. We propose a deep generative crowdsourcing learning approach to incorporate the strengths of Deep Neural Networks (DNNs) and exploit worker correlations. The model comprises a DNN classifier as a prior and an annotation generation process. A mixture model of workers'' capabilities within each class is introduced into the annotation generation process for worker correlation modeling. For adaptive trade-off between model complexity and data fitting, we implement fully Bayesian inference. Based on the natural-gradient stochastic variational inference techniques developed for the Structured Variational AutoEncoder (SVAE), we combine variational message passing for conjugate parameters and stochastic gradient descent for DNN parameters into a unified framework for efficient end-to-end optimization. Experimental results on 22 real crowdsourcing datasets demonstrate the effectiveness of the proposed approach.     

Perceptions of virtual reward systems in crowdsourcing games

The gaming approach to crowdsourcing is a major way to foster engagement and sustained participation. Also known as crowdsourcing games, players contribute their effort to tackle problems and receive enjoyment in return. As in any game, a fundamental mechanism in crowdsourcing games is its virtual reward system. This paper investigates how virtual reward systems evoke intrinsic motivation, perceived enjoyment and output quality in the context of crowdsourcing games. Three mobile applications for crowdsourcing location-based content were developed for an experimental study. The Track version offered a points-based reward system for actions such as contribution of content. The Badge version offered different badges for collection while the Share version served as a control which did not have any virtual reward system. For each application, participants performed a series of tasks after which a questionnaire survey was administered. Results showed that Badge and Track enhanced enjoyment emotionally, cognitively and behaviorally. They also increased perceptions of the quality of outputs when compared to Share. As well, they better satisfied the motivational needs for autonomy and competence than Share. Interestingly, there were also significant differences in how Badge and Track were perceived.     

Crowdsourcing intelligent design

Design intelligence, namely, artificial intelligence to solve creative problems and produce creative ideas, has improved rapidly with the new generation artificial intelligence. However, existing methods are more skillful in learning from data and have limitations in creating original ideas different from the training data. Crowdsourcing offers a promising method to produce creative designs by combining human inspiration and machines’ computational ability. We propose a crowdsourcing intelligent design method called ‘flexible crowdsourcing design’. Design ideas produced through crowdsourcing design can be unreliable and inconsistent because they rely solely on selection among participants’ submissions of ideas. In contrast, the flexible crowdsourcing design method employs a cultivation procedure that integrates the ideas from crowd participants and cultivates these ideas to improve design quality at the same time. We introduce a series of studies to show how flexible crowdsourcing design can produce original design ideas consistently. Specifically, we will describe the typical procedure of flexible crowdsourcing design, the refined crowdsourcing tasks, the factors that affect the idea development process, the method for calculating idea development potential, and two applications of the flexible crowdsourcing design method. Finally, it summarizes the design capabilities enabled by crowdsourcing intelligent design. This method enhances the performance of crowdsourcing design and supports the development of design intelligence.     

Provable Protection against Web Application Vulnerabilities Related to Session Data Dependencies

Web applications are widely adopted and their correct functioning is mission critical for many businesses. At the same time, Web applications tend to be error prone and implementation vulnerabilities are readily and commonly exploited by attackers. The design of countermeasures that detect or prevent such vulnerabilities or protect against their exploitation is an important research challenge for the fields of software engineering and security engineering. In this paper, we focus on one specific type of implementation vulnerability, namely, broken dependencies on session data. This vulnerability can lead to a variety of erroneous behavior at runtime and can easily be triggered by a malicious user by applying attack techniques such as forceful browsing. This paper shows how to guarantee the absence of runtime errors due to broken dependencies on session data in Web applications. The proposed solution combines development-time program annotation, static verification, and runtime checking to provably protect against broken data dependencies. We have developed a prototype implementation of our approach, building on the JML annotation language and the existing static verification tool ESC/Java2, and we successfully applied our approach to a representative J2EE-based e-commerce application. We show that the annotation overhead is very small, that the performance of the fully automatic static verification is acceptable, and that the performance overhead of the runtime checking is limited.     

利用标注者相关性的深度生成式众包学习

传统监督学习需要训练样本的真实标记信息,而在很多情况下,真实标记并不容易收集.与之对比,众包学习从多个可能犯错的非专家收集标注,通过某种融合方式估计样本的真实标记.注意到现有深度众包学习工作对标注者相关性建模不足,而非深度众包学习方面的工作表明,标注者相关性建模利用有助于改善学习效果.提出一种深度生成式众包学习方法,以...     

Ethical norms and issues in crowdsourcing practices: A Habermasian analysis

Crowdsourcing practices have generated much discussion on their ethics and fairness, yet these topics have received little scholarly investigation. Some have criticized crowdsourcing for worker exploitation and for undermining workplace regulations. Others have lauded crowdsourcing for enabling workers' autonomy and allowing disadvantaged people to access previously unreachable job markets. In this paper, we examine the ethics in crowdsourcing practices by focusing on three questions: (a) What ethical issues exist in crowdsourcing practices? (b) are ethical norms emerging or are issues emerging that require ethical norms? and, more generally, (c) how can the ethics of crowdsourcing practices be established? We answer these questions by engaging with Jürgen Habermas' discourse ethics theory to interpret findings from a longitudinal field study (from 2013 to 2016) involving key crowdsourcing participants (workers, platform organizers, and requesters) of three crowdsourcing communities. Grounded in this empirical study, we identify ethical concerns and discuss the ones for which ethical norms have emerged as well as others which remain unresolved and problematic in crowdsourcing practices. Furthermore, we provide normative considerations of how ethical concerns can be identified, discussed, and resolved based on the principles of discourse ethics.     

供需匹配中的非诚信行为预防

供需匹配问题可以通过社交网络(Social Network,SN)下的众包模式得到解决。但由于实际应用中的非合作约束,以及社交网络的隐私保护机制,众包的参与者具有通过非诚信行为获利的动机与条件。这类行为会影响公平性原则,并将导致网络中信任链的崩塌,最终使得整个众包模式的供需匹配规则失效。为解决众包供需匹配方法中的非诚信问题,考虑通过分布式公开记账的方式来确保成员如实汇报个体的行为与状态,并通过核对公开的信息来寻找两类非诚信者。此外,设计基于信誉的惩罚机制来对抗非诚信行为,并最终通过理论分析证明了此机制的有效性与可行性。在此机制下,众包参与者的最优策略便是保证诚实。     

Gamifying collective human behavior with gameful digital rhetoric

This paper presents a design framework called Gameful Digital Rhetoric that offers a set of design frames for designing meaningful digital rhetoric that guides collective human behavior in ubiquitous social digital services, such as crowdsourcing. The framework is extracted from our experiences with building and developing crowdsourcing case studies. From a video game perspective, the paper has categorized our experiences into seven design frames to encourage collective human activity. This approach is different from traditional gamification, as it focuses more on the semiotic aspect of virtuality in the video games, not game mechanics; it helps to enhance the current meaning of the real world for changing human attitude and behavior through various socio-cultural and psychological techniques. Therefore, it is possible to discuss respective design frames for enhancing crowdsourcing by incrementally adding new digital rhetoric. The paper also presents how Gameful Digital Rhetoric allows us to guide collective human behavior in Collectivist Crowdsourcing; the design is explained through a scenario-based and experiment-based analyses. The paper then discusses how to design collective human behavior with Gameful Digital Rhetoric and how to identify the design’s potential pitfalls. Our approach offers useful insights into the design of future social digital services that influence collective human behavior     

三方众包市场中的发包方-平台博弈机制设计

众包(crowdsourcing)通常涉及到目标各不相同的多个参与者.设计有效的众包机制,使得各个参与者在竞争中实现共赢,是众包理论研究中的基本问题之一.当前,众包机制设计通常基于发包方-标注者直接进行交互的两方博弈模型.而现实应用中,发包方与标注者之间往往通过平台进行交互,从而构成三方博弈下的众包市场.其中的发包方-平台博弈机制设计是过往众包研究中未曾涉及的全新问题.将三方众包市场建模为不完全信息博弈,并证明该博弈问题的Nash均衡可通过在线学习来最小化发包方和平台的累计遗憾而达到.在单发包方情形下,证明经典的EXP3算法对于发包方的最优性,并基于反事实遗憾最小化技术为平台设计了有效策略.同时,将单发包方情形下发包方和平台策略拓展到多发包方情形下并给出理论分析.合成及真实数据集上的实验验证了该方法的有效性.     

Face Annotation Using Transductive Kernel Fisher Discriminant

Face annotation in images and videos enjoys many potential applications in multimedia information retrieval. Face annotation usually requires many training data labeled by hand in order to build effective classifiers. This is particularly challenging when annotating faces on large-scale collections of media data, in which huge labeling efforts would be very expensive. As a result, traditional supervised face annotation methods often suffer from insufficient training data. To attack this challenge, in this paper, we propose a novel Transductive Kernel Fisher Discriminant (TKFD) scheme for face annotation, which outperforms traditional supervised annotation methods with few training data. The main idea of our approach is to solve the Fisher's discriminant using deformed kernels incorporating the information of both labeled and unlabeled data. To evaluate the effectiveness of our method, we have conducted extensive experiments on three types of multimedia testbeds: the FRGC benchmark face dataset, the Yahoo! web image collection, and the TRECVID video data collection. The experimental results show that our TKFD algorithm is more effective than traditional supervised approaches, especially when there are very few training data.