Collusion-Proof Result Inference in Crowdsourcing

在传统的 crowdsourcing,工人们被期望提供独立答案给任务以便保证答案的差异。然而,最近的研究证明人群不是许多独立工人,但是相反工人们与对方一起交流并且协作。与小努力追求更多的报酬,一些工人可以共谋勾结提供重复答案,它将损坏聚集的结果的质量。尽管如此,就在 crowdsourcing 的结果推理上的串通的否定影响而言有很少努力。在这份报纸,我们特殊在公共平台为一般 crowdsourcing 任务担心防串通的结果推理问题。到那个目的,我们设计一个度量标准,工人表演变化率,由在移开重复答案前后计算吝啬的工人表演的差别识别共谋勾结的答案。然后,我们把串通察觉结果合并到存在结果推理方法甚至与串通行为的出现保证聚集的结果的质量。与真实世界、合成的数据集,我们进行了我们的途径的评估的一个广泛的集合。试验性的结果与最先进的方法比较表明我们的途径的优势。     

Efficient task assignment for spatial crowdsourcing: A combinatorial fractional optimization approach with semi-bandit learning

Spatial crowdsourcing has emerged as a new paradigm for solving problems in the physical world with the help of human workers. A major challenge in spatial crowdsourcing is to assign reliable workers to nearby tasks. The goal of such task assignment process is to maximize the task completion in the face of uncertainty. This process is further complicated when tasks arrivals are dynamic and worker reliability is unknown. Recent research proposals have tried to address the challenge of dynamic task assignment. Yet the majority of the proposals do not consider the dynamism of tasks and workers. They also make the unrealistic assumptions of known deterministic or probabilistic workers’ reliabilities. In this paper, we propose a novel approach for dynamic task assignment in spatial crowdsourcing. The proposed approach combines bi-objective optimization with combinatorial multi-armed bandits. We formulate an online optimization problem to maximize task reliability and minimize travel costs in spatial crowdsourcing. We propose the distance-reliability ratio (DRR) algorithm based on a combinatorial fractional programming approach. The DRR algorithm reduces travel costs by 80% while maximizing reliability when compared to existing algorithms. We extend the DRR algorithm for the scenario when worker reliabilities are unknown. We propose a novel algorithm (DRR-UCB) that uses an interval estimation heuristic to approximate worker reliabilities. Experimental results demonstrate that the DRR-UCB achieves high reliability in the face of uncertainty. The proposed approach is particularly suited for real-life dynamic spatial crowdsourcing scenarios. This approach is generalizable to the similar problems in other areas in expert systems. First, it encompasses online assignment problems when the objective function is a ratio of two linear functions. Second, it considers situations when intelligent and repeated assignment decisions are needed under uncertainty.     

A context-aware approach for trustworthy worker selection in social crowd

Crowdsourcing applications like Amazon Mechanical Turk (AMT) make it possible to address many difficult tasks (e.g., image tagging and sentiment analysis) on the internet and make full use of the wisdom of crowd, where worker quality is one of the most crucial issues for the task owners. Thus, a challenging problem is how to effectively and efficiently select the high quality workers, so that the tasks online can be accomplished successfully under a certain budget. The existing methods on the crowd worker selection problem mainly based on the quality measurement of the crowd workers, those who have to register on the crowdsourcing platforms. With the connect of the OSNs and the crowdsourcing applications, the social contexts like social relationships and social trust between participants and social positions of participants can assist requestors to select one or a group of trustworthy crowdsourcing workers. In this paper, we first present a contextual social network structure and a concept of Strong Social Component (SSC), which emblems a group of workers who have high social contexts values. Then, we propose a novel index for SSC, and a new efficient and effective algorithm C-AWSA to find trustworthy workers, who can complete the tasks with high quality. The results of our experiments conducted on four real OSN datasets illustrate that the superiority of our method in trustworthy worker selection.     

基于活跃度的众包工作者信誉模型

针对现有众包系统不能有效地控制众包交互过程中工作者的活跃积极性和任务完成质量的问题,提出了一种基于活跃度的工作者信誉模型来实现众包平台的质量控制。该模型改进了平均信誉模型,从工作者活跃度和历史信誉值的角度提出了活跃因子和历史因子的概念。首先根据众包工作者最近30 d内参与众包活动的天数计算工作者的活跃因子;然后根据历史因子计算众包工作者的历史信誉值;最后根据计算出来的活跃因子和历史信誉值计算基于活跃度的工作者信誉值,以衡量众包工作者的工作能力。理论分析和测试实验结果表明：与平均信誉模型相比,根据基于活跃度的工作者信誉模型选取的众包工作者在任务完成质量上提高了4.95%,在任务完成时间上减少了25.33%;与基于证据理论信任模型相比,在任务完成质量上提高了6.63%,在任务完成时间上减少了25.11%。实验结果表明,基于活跃度的工作者信誉模型在实际众包项目中能够有效提高众包任务的完成质量,减少众包任务的完成时间。     

An Analysis of the Use of Qualifications on the Amazon Mechanical Turk Online Labor Market

Several human computation systems use crowdsourcing labor markets to recruit workers. However, it is still a challenge to guarantee that the results produced by workers have a high enough quality. This is particularly difficult in markets based on micro-tasks, where the assessment of the quality of the results needs to be done automatically. Pre-selection of suitable workers is a mechanism that can improve the quality of the results achieved. This can be done by considering worker’s personal information, worker’s historical behavior in the system, or through the use of customized qualification tasks. However, little is known about how requesters use these mechanisms in practice. This study advances present knowledge in worker pre-selection by analyzing data collected from the Amazon Mechanical Turk platform, regarding the way requesters use qualifications to this end. Furthermore, the influence of using customized qualification tasks in the quality of the results produced by workers is investigated. Results show that most jobs (93.6%) use some mechanism for the pre-selection of workers. While most workers use standard qualifications provided by the system, the few requesters that submit most of the jobs prefer to use customized ones. Regarding worker behavior, we identified a positive and significant correlation between the propensity of the worker to possess a particular qualification, and both the number of tasks that require this qualification, and the reward offered for the tasks that require the qualification, although this correlation is weak. To assess the impact that the use of customized qualifications has in the quality of the results produced, we have executed experiments with three different types of tasks using both unqualified and qualified workers. The results showed that, generally, qualified workers provide more accurate answers, when compared to unqualified ones.     

How many crowdsourced workers should a requester hire?

Recent years have seen an increased interest in crowdsourcing as a way of obtaining information from a potentially large group of workers at a reduced cost. The crowdsourcing process, as we consider in this paper, is as follows: a requester hires a number of workers to work on a set of similar tasks. After completing the tasks, each worker reports back outputs. The requester then aggregates the reported outputs to obtain aggregate outputs. A crucial question that arises during this process is: how many crowd workers should a requester hire? In this paper, we investigate from an empirical perspective the optimal number of workers a requester should hire when crowdsourcing tasks, with a particular focus on the crowdsourcing platform Amazon Mechanical Turk. Specifically, we report the results of three studies involving different tasks and payment schemes. We find that both the expected error in the aggregate outputs as well as the risk of a poor combination of workers decrease as the number of workers increases. Surprisingly, we find that the optimal number of workers a requester should hire for each task is around 10 to 11, no matter the underlying task and payment scheme. To derive such a result, we employ a principled analysis based on bootstrapping and segmented linear regression. Besides the above result, we also find that overall top-performing workers are more consistent across multiple tasks than other workers. Our results thus contribute to a better understanding of, and provide new insights into, how to design more effective crowdsourcing processes.     

基于Pareto分布的众包工人欺骗行为处理方法

由于众包的组织模式自由松散,致使众包工人在完成任务的过程中存在欺骗行为。如何识别工人的欺骗行为并降低其影响,从而保障众包任务的完成质量,已经成为众包领域的研究热点之一。通过对任务结果的评估与分析,针对众包工人统一型欺骗行为,提出了一种基于广义Pareto分布（GPD）的权重设置算法（WSABG）。该算法对GPD进行极大似然估计,并用二分法逼近似然函数的零点以计算出尺度参数σ和形状参数ε。算法中定义了新的权重公式,并利用众包工人完成当前任务的反馈数据赋予每位工人一个绝对影响权重,最终设计出了基于GPD的众包工人权重设置框架。所提算法可以解决任务结果数据之间差异性小且容易集中在两极的问题。以烟台大学学生评教数据为实验数据集,提出了区间转移矩阵的概念,证明了WSABG算法的有效性和优势。     

基于树分解的空间众包最优任务分配算法

随着配备高保真传感器的移动设备的普及以及无线网络资费的迅速下降,空间众包成为一种新型的问题解决框架,被用于将位置相关的任务（如路况报告,食品配送）分配给工人（配备智能设备并愿意完成任务的人）。本文研究空间众包中最优任务分配问题,关键在于设计出将每个任务分配给最合适的工人的任务分配策略,以使得完成的总任务数目最大化,而所有的工人可以在完成所分配的任务后,在预期最晚工作时间之前返回起点。找到全局最优分配是一个棘手的问题,因为该问题不等于单个工人的最佳分配的简单累加。本文注意到,仅有部分工人存在任务依赖,因此本文利用树分解技术将工人分割成独立的集合,并提出一种带启发式的深度优先搜索算法,该算法可以快速地更新启发函数界限,从而高效的对不可能成为最优解分配方案尽早地剪枝。实验表明,本文所提出的方法是非常有效的,可以很好地解决最优任务分配问题。     

Evolutionary approach for crowdsourcing quality control

Crowdsourcing is widely used for solving simple tasks (e.g. tagging images) and recently, some researchers (Kittur et al., 2011 [9] and Kulkarni et al., 2012 [10]) propose new crowdsourcing models to handle complex tasks (e.g. article writing). In both type of crowdsourcing models (for simple and complex tasks), voting is a technique that is widely used for quality control [9]. For example, 5 workers are asked to write 5 outlines for an article, and another 5 workers are asked to vote for the best outline among the 5 outlines. However, we argue that voting is actually a technique that selects a high quality answer from a set of answers. It does not directly enhance answer quality. In this paper, we propose a new quality control approach for crowdsourcing that can incrementally improve answer quality. The new approach is based upon two principles – evolutionary computing and slow intelligence, which help the crowdsourcing system to propagate knowledge among workers and incrementally improve the answer quality. We perform explicitly 2 experimental case studies to show the effectiveness of the new approach. The case study results show that the new approach can incrementally improve answer quality and produce high quality answers.     

软件众包工人能力动态度量算法

针对现有软件众包平台对工人能力考虑不足，导致分配给工人的任务完成质量低下的问题，提出了一种软件众包工人能力动态度量算法（ADM），实现工人能力的动态度量。首先，基于静态技能覆盖率，实现工人初始能力的计算；其次，对于工人历史完成的单个任务，综合任务复杂度、任务完成质量及任务开发时效，实现开发能力的计算，并根据时间因子计算随时间衰减的开发能力；然后，根据所有历史完成任务的时间先后顺序，实现能力度量值的动态更新；最后，基于历史任务技能覆盖率，计算工人对于待分配任务的开发能力。实验结果表明，与用户可靠性度量算法相比，所提出的能力动态度量算法具有较好的合理性与有效性，使能力度量吻合度平均值最高达到90.5%，能有效指导任务分配。     

面向不可信服务器的空间众包隐私保护研究

随着互联网技术的迅速发展以及智能移动设备的普遍使用,空间众包的使用愈加广泛.用户发布空间任务,空间众包平台将会雇佣工作者为其分配任务并执行.该类方法需要通过智能设备获取用户位置数据和工作者位置数据,容易泄露位置隐私,严重威胁了用户和工作者的隐私安全.针对个人位置隐私泄露的问题,本文提出了一种采用地理不可区分性对不可信服...     

Toward Maximizing the Quality of Results of Dependent Tasks Computed Unreliably

This paper studies the problem of maximizing the number of correct results of dependent tasks computed unreliably. We consider a distributed system composed of a reliable server that coordinates the computation of a massive number of unreliable workers. Any worker computes correctly with probability p < 1. Any incorrectly computed task corrupts all dependent tasks. The goal is to determine which tasks should be computed by the (reliable) server and which by the (unreliable) workers, and when, so as to maximize the expected number of correct results, under a constraint d on the computation time. This problem is motivated by distributed computing applications that persist partial results of computations for future use in other computations and that want to ensure that the persisted results are of high quality. We show that this optimization problem is NP-hard. Then we study optimal scheduling solutions for the mesh with the tightest deadline. We present combinatorial arguments that describe all optimal solutions for two ranges of values of worker reliability p, when p is close to zero and when p is close to one.     

基于活跃时间分组的软件众包工人选择机制

针对现有的软件众包工人选择机制对工人间协同开发考虑不足的问题，在竞标模式的基础上提出一种基于活跃时间分组的软件众包工人选择机制。首先，基于活跃时间将众包工人划分为多个协同开发组；然后，根据组内工人开发能力和协同因子计算协同工作组权重；最后，选定权重最大的协同工作组为最优工作组，并根据模块复杂度为每个任务模块从该组内选择最适合的工人。实验结果表明，该机制相比能力优先选择方法在工人平均能力上仅有0.57%的差距，同时因为保证了工人间的协同而使项目风险平均降低了32%，能有效指导需多人协同进行的众包软件任务的工人选择。     

Auction-based crowdsourcing supporting skill management

Crowdsourcing is a promising approach for enterprises to maintain a flexible workforce that is able to solve parts of business processes formerly processed in-house. Companies perceive crowdsourcing as a concept that allows receiving solutions quickly and at little cost. Similar to cloud computing where computing power is provided on demand, the crowd promises a flexible on-demand workforce. However, businesses realize that these benefits entail a lack of quality control. The main difference compared to traditional approaches in business process execution is that tasks or activities cannot be directly assigned to employees but are posted to the crowdsourcing platform. Its members can choose deliberately which tasks to book and work on. In fact, crowdsourcing is heavily affected by the loose-coupling of workers to crowdsourcers and the dynamics of the environment. Hence, it remains a major challenge to guarantee high-quality processing of tasks within the prescribed time limit. A further obstacle for adoption of crowdsourcing in enterprises is the fact that it is hard to specify a fair monetary reward in advance. The concepts introduced in this work allow to smoothly integrate new workers, to keep them motivated, and to help them develop and improve skills needed in the system. We present a crowdsourcing marketplace that matches complex tasks, requiring multiple skills, to suitable workers. The key to ensuring high quality lies in skilled members whose capabilities can be estimated correctly. To that end, we present auction mechanisms that help to correctly estimate workers and to evolve skills that are needed in the system. Crowdsourcers do not need to predefine exact prices but only maximum prices they are willing to pay since the actual rewards for tasks are formed by supply and demand. Extensive experiments show that our approach leads to improved crowdsourcing, in most cases.     

Task assignment for social-oriented crowdsourcing

Crowdsourcing has become an efficient measure to solve machine-hard problems by embracing group wisdom, in which tasks are disseminated and assigned to a group of workers in the way of open competition. The social relationships formed during this process may in turn contribute to the completion of future tasks. In this sense, it is necessary to take social factors into consideration in the research of crowdsourcing. However, there is little work on the interactions between social relationships and crowdsourcing currently. In this paper, we propose to study such interactions in those social-oriented crowdsourcing systems from the perspective of task assignment. A prototype system is built to help users publish, assign, accept, and accomplish location-based crowdsourcing tasks as well as promoting the development and utilization of social relationships during the crowdsourcing. Especially, in order to exploit the potential relationships between crowdsourcing workers and tasks, we propose a “worker-task” accuracy estimation algorithm based on a graph model that joints the factorized matrixes of both the user social networks and the history “worker-task” matrix. With the worker-task accuracy estimation matrix, a group of optimal worker candidates is efficiently chosen for a task, and a greedy task assignment algorithm is proposed to further the matching of worker-task pairs among multiple crowdsourcing tasks so as to maximize the overall accuracy. Compared with the similarity based task assignment algorithm, experimental results show that the average recommendation success rate increased by 3.67%; the average task completion rate increased by 6.17%; the number of new friends added per week increased from 7.4 to 10.5; and the average task acceptance time decreased by 8.5 seconds.     

Motivating participation and improving quality of contribution in ubiquitous crowdsourcing

Ubiquitous crowdsourcing, or the crowdsourcing of tasks in settings beyond the desktop, is attracting interest due to the increasing maturity of mobile and ubiquitous technology, such as smartphones and public displays. In this paper we attempt to address a fundamental challenge in ubiquitous crowdsourcing: if people can contribute to crowdsourcing anytime and anyplace, why would they choose to do so? We highlight the role of motivation in ubiquitous crowdsourcing, and its effect on participation and performance. Through a series of field studies we empirically validate various motivational approaches in the context of ubiquitous crowdsourcing, and assess the comparable advantages of ubiquitous technologies' affordances. We show that through motivation ubiquitous crowdsourcing becomes comparable to online crowdsourcing in terms of participation and task performance, and that through motivation we can elicit better quality contributions and increased participation from workers. We also show that ubiquitous technologies' contextual capabilities can increase participation through increasing workers' intrinsic motivation, and that the in-situ nature of ubiquitous technologies can increase both participation and engagement of workers. Combined, our findings provide empirically validated recommendations on the design and implementation of ubiquitous crowdsourcing.     

Task selection in spatial crowdsourcing from worker’s perspective

With the progress of mobile devices and wireless broadband, a new eMarket platform, termed spatial crowdsourcing is emerging, which enables workers (aka crowd) to perform a set of spatial tasks (i.e., tasks related to a geographical location and time) posted by a requester. In this paper, we study a version of the spatial crowdsourcing problem in which the workers autonomously select their tasks, called the worker selected tasks (WST) mode. Towards this end, given a worker, and a set of tasks each of which is associated with a location and an expiration time, we aim to find a schedule for the worker that maximizes the number of performed tasks. We first prove that this problem is NP-hard. Subsequently, for small number of tasks, we propose two exact algorithms based on dynamic programming and branch-and-bound strategies. Since the exact algorithms cannot scale for large number of tasks and/or limited amount of resources on mobile platforms, we propose different approximation algorithms. Finally, to strike a compromise between efficiency and accuracy, we present a progressive algorithms. We conducted a thorough experimental evaluation with both real-world and synthetic data on desktop and mobile platforms to compare the performance and accuracy of our proposed approaches.     

Assessment of crowdsourcing and gamification loss in user-assisted object segmentation

There has been a growing interest in applying human computation – particularly crowdsourcing techniques – to assist in the solution of multimedia, image processing, and computer vision problems which are still too difficult to solve using fully automatic algorithms, and yet relatively easy for humans. In this paper we focus on a specific problem – object segmentation within color images – and compare different solutions which combine color image segmentation algorithms with human efforts, either in the form of an explicit interactive segmentation task or through an implicit collection of valuable human traces with a game. We use Click’n’Cut, a friendly, web-based, interactive segmentation tool that allows segmentation tasks to be assigned to many users, and Ask’nSeek, a game with a purpose designed for object detection and segmentation. The two main contributions of this paper are: (i) We use the results of Click’n’Cut campaigns with different groups of users to examine and quantify the crowdsourcing loss incurred when an interactive segmentation task is assigned to paid crowd-workers, comparing their results to the ones obtained when computer vision experts are asked to perform the same tasks. (ii) Since interactive segmentation tasks are inherently tedious and prone to fatigue, we compare the quality of the results obtained with Click’n’Cut with the ones obtained using a (fun, interactive, and potentially less tedious) game designed for the same purpose. We call this contribution the assessment of the gamification loss, since it refers to how much quality of segmentation results may be lost when we switch to a game-based approach to the same task. We demonstrate that the crowdsourcing loss is significant when using all the data points from workers, but decreases substantially (and becomes comparable to the quality of expert users performing similar tasks) after performing a modest amount of data analysis and filtering out of users whose data are clearly not useful. We also show that – on the other hand – the gamification loss is significantly more severe: the quality of the results drops roughly by half when switching from a focused (yet tedious) task to a more fun and relaxed game environment.     

考虑工人路径的多智能体强化学习空间众包任务分配方法

针对工人和任务进行匹配是空间众包研究的核心问题之一,但已有的方法通常会忽略工人路径对任务分配结果产生的影响.传统的任务分配方法存在计算速度慢、适用范围小和协作效果不突出等问题.对此,从空间众包平台的角度出发研究面向路网的空间众包任务分配问题,以任务完成时间最短为目标,提出考虑工人路径规划的基于多智能体强化学习的QMIX-A*算法,缩短任务的平均完成时间,进而提高用户的满意度.大量的数值仿真研究验证了QMIX-A*的有效性和稳定性,为空间众包服务平台的任务分配与路径优化策略的选择提供决策支持.