Competition matters! Self-efficacy,effort, and performance in crowdsourcing teams

Advances in information technologies (IT) have enabled organizations to seek solutions for their business problems from beyond their own workforce through digital crowdsourcing platforms. In the most common form of crowdsourcing, teams that offer solutions compete for rewards. Thus, a question of interest is whether competition is a key crowdsourcing characteristic that influences how teams allocate their effort and achieve desired performance. Motivated by this question, we investigate how competition moderates the relationship between self-efficacy and effort using comprehensive, time-variant data collected from crowdsourcing teams that completed a project under competitive and non-competitive conditions. Under competitive conditions, self-efficacy shows a positive effect on effort, which in turn, affects performance positively. Whereas, under noncompetitive conditions, self-efficacy has a negative effect on effort and subsequently on performance. Our results also show a recursive relationship between self-efficacy and performance, in which performance subsequently affects self-efficacy positively. Thus, inducing a sense of competition through competitive reward structures and IT-based “gaming elements” helps improve team effort and subsequent performance. We also tested for mediation of team motivation in the self-efficacy and effort relationship, and we found that motivation partially mediates the relationship. Based on our findings, implications for both theory and practice are discussed.     

Crowdsourcing contests: understanding the effect of competitors’ participation history on their performance

Crowdsourcing contests have become increasingly important and prevalent with the ubiquity of the Internet. Designing efficient crowdsourcing contests is not possible without the deep understanding of the factors affecting individuals’ continuous participation and their performance. Prior studies have mainly focused on identifying the effect of task-specific, environment-specific, organisation-specific, and individual-specific factors on individuals’ performance in crowdsourcing contests. And to our knowledge, there are no or very few studies on evaluating the effect of individuals’ participation history on their performance. This paper aims to address this research gap using a data set from TopCoder. This study derives competitors’ participation history factors, such as participation frequency, participation recency, winning frequency, winning recency, tenure, and last performance to construct models depicting effects of these factors on competitors’ performance in online crowdsourcing contests. The research findings demonstrate that most of competitors’ participation history factors have significant effect on their performance. This paper also indicates that competitors’ participation frequency and winning frequency moderate the relationship between last performance and performance, and relationship between tenure and performance positively. On the other hand, individuals’ participation recency and winning recency moderate relationship between last performance and performance negatively, but have no significant effect on the relationship between tenure and performance.     

Collusion-Proof Result Inference in Crowdsourcing

在传统的 crowdsourcing,工人们被期望提供独立答案给任务以便保证答案的差异。然而,最近的研究证明人群不是许多独立工人,但是相反工人们与对方一起交流并且协作。与小努力追求更多的报酬,一些工人可以共谋勾结提供重复答案,它将损坏聚集的结果的质量。尽管如此,就在 crowdsourcing 的结果推理上的串通的否定影响而言有很少努力。在这份报纸,我们特殊在公共平台为一般 crowdsourcing 任务担心防串通的结果推理问题。到那个目的,我们设计一个度量标准,工人表演变化率,由在移开重复答案前后计算吝啬的工人表演的差别识别共谋勾结的答案。然后,我们把串通察觉结果合并到存在结果推理方法甚至与串通行为的出现保证聚集的结果的质量。与真实世界、合成的数据集,我们进行了我们的途径的评估的一个广泛的集合。试验性的结果与最先进的方法比较表明我们的途径的优势。     

Solvers’ participation in crowdsourcing platforms: Examining the impacts of trust,and benefit and cost factors

Organizations are increasingly crowdsourcing their tasks to unknown individual workers, i.e., solvers. Solvers' participation is critical to the success of crowdsourcing activities. However, challenges exist in attracting solvers to participate in crowdsourcing. In this regard, prior research has mainly investigated the influences of benefit factors on solvers’ intention to participate in crowdsourcing. Thus, there is a lack of understanding of the cost factors that influence actual participation behavior, in conjunction with the benefits. Additionally, the role of trust in the cost-benefit analysis remains to be explored. Motivated thus, based on social exchange theory and context-related literature, we develop a model to explain the impacts of benefit and cost factors as well as trust on solver participation behavior in crowdsourcing. The model was tested using survey and archival data from 156 solvers on a large crowdsourcing platform. As hypothesized, monetary reward, skill enhancement, work autonomy, enjoyment, and trust were found to positively affect solvers’ participation in crowdsourcing, while cognitive effort negatively affects their participation. In addition, it was found that monetary reward positively affects trust (trust partially mediates its effect on participation behavior), while loss of knowledge power negatively affects trust. The theoretical contributions and practical implications of the study are discussed.     

Motivating participation in crowdsourcing contests: The role of instruction-writing strategy

Task instructions are seeker-generated content aiming to disclose information and persuade solvers to participate in crowdsourcing contests. There are various writing strategies for task instructions, with different levels of informativeness and persuasiveness for solvers. This work examines how requirement-oriented and reward-oriented strategies affect solver participation. The empirical results reveal a U-shaped relationship between the usage of restrictive words and the number of participants. In contrast, an inverted U-shaped relationship is found between terminology usage and participation. Emphasizing intrinsic and extrinsic rewards through issue-relevant information statements and emotional appeals also proves to be effective in motivating solvers’ involvement.     

Communitition: The Tension between Competition and Collaboration in Community‐Based Design Contests

Following the concepts of crowdsourcing, co‐creation or open innovation, companies are increasingly using contests to foster the generation of creative solutions. Currently, online idea and design contests are enjoying a resurgence through the usage of new information and communication technologies. These virtual platforms allow users both to competitively disclose their creative ideas to corporations and also to interact and collaborate with like‐minded peers, communicating, discussing and sharing their insights and experiences, building social networks and establishing a sense of community. Little research has considered that contest communities both promote and benefit from simultaneous co‐operation and competition and that both types of relationships need to be emphasized at the same time. In this article, it is argued that the firm‐level concept of co‐opetition might also be relevant for an innovation's success on the individual level within contest communities. Our concept of ‘communitition’ should include the elements of competitive participation without disabling the climate for co‐operation, as numerous user discussions and comments improve the quality of submitted ideas and allow the future potential of an idea to shine through the so‐called ‘wisdom of the crowd’.     

Crowdsourcing intelligent design

Design intelligence, namely, artificial intelligence to solve creative problems and produce creative ideas, has improved rapidly with the new generation artificial intelligence. However, existing methods are more skillful in learning from data and have limitations in creating original ideas different from the training data. Crowdsourcing offers a promising method to produce creative designs by combining human inspiration and machines’ computational ability. We propose a crowdsourcing intelligent design method called ‘flexible crowdsourcing design’. Design ideas produced through crowdsourcing design can be unreliable and inconsistent because they rely solely on selection among participants’ submissions of ideas. In contrast, the flexible crowdsourcing design method employs a cultivation procedure that integrates the ideas from crowd participants and cultivates these ideas to improve design quality at the same time. We introduce a series of studies to show how flexible crowdsourcing design can produce original design ideas consistently. Specifically, we will describe the typical procedure of flexible crowdsourcing design, the refined crowdsourcing tasks, the factors that affect the idea development process, the method for calculating idea development potential, and two applications of the flexible crowdsourcing design method. Finally, it summarizes the design capabilities enabled by crowdsourcing intelligent design. This method enhances the performance of crowdsourcing design and supports the development of design intelligence.     

Journalists as Crowdsourcerers: Responding to Crisis by Reporting with a Crowd

Widespread adoption of new information communication technologies (ICTs) is disrupting traditional models of news production and distribution. In this rapidly changing media landscape, the role of the journalist is evolving. Our research examines how professional journalists within a rural community impacted by Hurricane Irene successfully negotiated a new role for themselves, transforming their journalistic practice to serve in a new capacity as leaders of an online volunteer community. We describe an emergent organization of media professionals, citizen journalists, online volunteers, and collaborating journalistic institutions that provided real-time event coverage. In this rural context, where communications infrastructure is relatively uneven, this ad hoc effort bridged gaps in ICT infrastructure to unite its audience. In this paper, we introduce a new perspective for characterizing these information-sharing activities: the “human powered mesh network” extends the concept of a mesh network to include human actors in the movement of information. Our analysis shows how journalists played a key role in this network, and facilitated the movement of information to those who needed it. These findings also note a contrast between how HCI researchers are designing crowdsourcing platforms for news production and how crowdsourcing efforts are forming during disaster events, suggesting an alternative approach to designing for emergent collaborations in this context.     

基于多Agent的众包任务推荐系统建模与仿真

为了让众包平台用户更方便准确地搜寻到合适任务,促进其能力水平提升,解决众包任务推荐动态性等问题,提出了一种基于多Agent的众包任务推荐系统。首先,基于众包平台建立多Agent任务推荐模型,提出了模型设计思路与模型框架,并进一步阐述了各Agent功能、相互作用关系与相关算法;其次,提出众包用户能力水平提升相关算法;最后,利用NetLogo仿真软件进行验证。结果表明,众包任务推荐系统可对用户能力水平的提升起到促进作用,证明了在众包平台引入推荐系统的必要性。并且分析了多Agent技术可提升推荐系统的动态性、智能性与灵活性等整体性能,促进了众包平台数据的管理与维护。     

An impactful crowdsourcing intermediary design - a case of a service imagery crowdsourcing system

Crowdsourcing is usually implemented using an intermediary organization or directly coordinated by solution seekers. For those using an IT-based crowdsourcing intermediary, the important factors of intermediaries that impact the crowdsourcing outcome are yet unclear. In addition, little research was conducted on how to design a crowdsourcing intermediary that can address the combined challenges of considering different cognitive demand levels of solution providers’ contributions, combining contributions and evaluating contributions. This paper identifies three important factors and provides a novel design of crowdsourcing intermediary to cope with these challenges. This study uses a case that focuses on how to assist small and medium businesses (SMBs) to develop their service imageries in triggering service innovation and designing their service experiences as to fulfill the desired outcomes of customers. Through the case, the benefits of our crowdsourcing intermediary design are demonstrated and justified. The three important factors are the crowdsourcing intermediary knowledge base, generative networks and empowerment of crowd members. This study shows that the crowdsourcing process can facilitate achieving a higher chance of attaining creative solutions for SMBs’ innovation problems when the three factors are well incorporated and managed within the crowdsourcing intermediary design. This study also presents a novel design of crowdsourcing intermediary that can address the combined challenges of coping with different cognitive demand levels of crowd members and combining and evaluating crowd members’ contributions, in order to attain impactful crowdsourcing outcome.     

Crowdsourcing tasks to social networks in BPEL4People

Human-interactions are a substantial part of today’s business processes. In service-oriented systems this has led to specifications such as WS-HumanTask and BPEL4People which aim at standardizing the interaction protocol between software processes and humans. These specifications received considerable attention from major industry players due to their extensibility and interoperability. Recently, crowdsourcing has emerged as a new paradigm for leveraging a human workforce using Web technologies. We argue that crowdsourcing techniques and platforms could benefit from XML-based standards such as WS-HumanTask and BPEL4People as these specifications allow for extensibility and cross-platform operation. However, most efforts to model human interactions using BPEL4People focus on relatively static role models for selecting the right person to interact with. Thus, BPEL4People is not well suited for specifying and executing processes involving crowdsourcing of tasks to online communities. Here, we extend BPEL4People with non-functional properties that allow to cope with the inherent dynamics of crowdsourcing processes. Such properties include human capabilities and the level of skills. We discuss the formation of social networks that are particularly beneficial for processing extended BPEL4People tasks. Furthermore, we present novel approaches for the automated assignment of tasks to a social group. The feasibility of our approach is shown through a proof of concept implementation of various concepts as well as simulations and experiments to evaluate our ranking and selection approach.     

Factors affecting participation of solvers in crowdsourcing: an empirical study from China

Crowdsourcing comprises a variety of creative contests, and its success is closely related to the quantity and quality of solvers. The research model of factors influencing the quantity and quality of solvers with respect to contest arrangement attributes and market competition situation has been developed in this paper, and the model has been tested with data from a crowdsourcing website in China. The results show that higher awards, easier tasks, longer duration and lower competition intensity lead to a higher number of solvers. Higher awards, longer duration and higher difficulty level of tasks lead to higher ability level of winners, but competition intensity and market price for other competing projects do not show significant correlation with the ability level of winners.     

软件众包参与度影响因素分析及预测模型

随着"众包"这种商业模式的快速发展, 越来越多的互联网公司选择以"众包"的形式发布软件任务. 然而, 软件任务因其高门槛、高复杂度、长周期等特性, 面临着严重的低参与度问题. 本文结合全球最大的软件众包平台TopCoder的数据, 对软件众包任务的参与度进行研究. 首先, 使用多元回归分析了影响软件众包参与度的因素; 接着, 综合数据挖掘领域的多种分类预测算法, 探讨软件众包参与度的预测模型. 希望通过本实证研究, 为发包方、众包平台降低软件众包的低参与风险提供参考.     

An evaluation methodology for crowdsourced design

In recent years, the “power of the crowd” has been repeatedly demonstrated and various Internet platforms have been used to support applications of collaborative intelligence in tasks ranging from open innovation to image analysis. However, crowdsourcing applications in the fields of design research and creative innovation have been much slower to emerge. So, although there have been reports of systems and researchers using Internet crowdsourcing to carry out generative design, there are still many gaps in knowledge about the capability and limitations of the technology. Indeed the process models developed to support traditional commercial design (e.g. Pugh’s Total Design, Agile, Double-Diamond etc.) have yet to be established for Crowdsourced Design (cDesign). As a contribution to the development of such a general model this paper proposes a cDesign framework to support the creation of crowdsourced design activities. Within the cDesign framework the effective evaluation of design quality is identified as a key component that not only enables the leveraging of a large, virtual workforce’s creative activities but is also fundamental to almost all iterative optimisation processes. This paper reports an experimental investigation into two different Crowdsourced design evaluation approaches; free evaluation and ‘Crowdsourced Design Evaluation Criteria’ (cDEC). The results are benchmarked against a ‘manual’ evaluation carried out by a panel of experienced designers. The results suggest that the cDEC approach produces design rankings that correlate strongly with the judgements of an “expert panel”. The paper concludes that cDEC assessment methodology demonstrates how Crowdsourcing can be effectively used to evaluate, as well as generate, new design solutions.     

Collaborating with Bounty Hunters: How to Encourage White Hat Hackers’ Participation in Vulnerability Crowdsourcing Programs through Formal and Relational Governance

Although crowdsourcing studies have provided insights into the factors motivating solvers, how to collaborate with white hat hackers in vulnerability crowdsourcing is still unclear. Drawing from information systems (IS) governance literature and psychological contract theory, this study proposed a research model to explore the effects of formal and relational governance mechanisms on solvers’ participation. Data from 242 programs and 6095 white hat hackers were collected and analyzed through a negative binomial regression. Results show that outcome control, expressing commitment, and prompt communication can increase the submission quantity of vulnerabilities. Interestingly, behavior control has an inverted U-shaped relationship with submission quantity.     

Participation recommendation system for crowdsourcing contests

We propose a novel participation recommendation approach for crowdsourcing contests including probabilistic modeling of contest participation and winner determination. Our method estimates the winning and participation probability of each worker and offers ranked lists of recommended contests. Since there is only one winner in most contests, standard recommendation techniques fail to estimate the accurate winning probability using only the extremely sparse winning information of completed contests. Our solution is to utilize contest participation information and features of workers and contests as auxiliary information. We use the concept of a transfer learning method for matrices and a feature-based matrix factorization method. Experiments conducted using real crowdsourcing contest datasets show that the use of auxiliary information is crucial for improving the performance of contest recommendation, and also reveal several important common skills.     

The role of rating feedback and its implications for solver submission behavior in crowdsourcing contests

In crowdsourcing contests, seekers face a decision about how to strategically provide feedback to solvers, thereby impacting solvers’ submission behavior. Despite considerable interest in seeker feedback, extant literature lacks a thorough understanding of it. This study, in particular, examines how rating feedback can encourage the submission behavior of solvers from three aspects: the level of rating, the dispersion of rating, and the first rating timing. Based on a sample of 37,881 unique observations from 99design.com, empirical analysis indicates that the level of rating has a curvilinear effect on a solver's submission behavior, suggesting that seekers can benefit the most from a moderate level of ratings in crowdsourcing contests. This study further finds that solvers who receive a large dispersion of ratings are more likely to contribute submissions in a contest. Finally, results show that providing a lower position of first rating is positively related to submission behavior. These results contribute some implications in both theory and practice.     

Customized crowds and active learning to improve classification

Traditional classification algorithms can be limited in their performance when a specific user is targeted. User preferences, e.g. in recommendation systems, constitute a challenge for learning algorithms. Additionally, in recent years user’s interaction through crowdsourcing has drawn significant interest, although its use in learning settings is still underused.In this work we focus on an active strategy that uses crowd-based non-expert information to appropriately tackle the problem of capturing the drift between user preferences in a recommendation system. The proposed method combines two main ideas: to apply active strategies for adaptation to each user; to implement crowdsourcing to avoid excessive user feedback. A similitude technique is put forward to optimize the choice of the more appropriate similitude-wise crowd, under the guidance of basic user feedback.The proposed active learning framework allows non-experts classification performed by crowds to be used to define the user profile, mitigating the labeling effort normally requested to the user.The framework is designed to be generic and suitable to be applied to different scenarios, whilst customizable for each specific user. A case study on humor classification scenario is used to demonstrate experimentally that the approach can improve baseline active results.     

社会网络下分配众包任务的真实机制

众包平台上出现了越来越多的宏任务,而这些宏任务需要工人的专业技能和团队的集体贡献。社会网络为社会工作者之间的合作提供了一个可用的平台。事实上,很少有研究关注众包工人之间的社会网络。在社会网络下的众包任务分配问题是NP难问题,并且社会网络中会存在参与者为了提高自己的效用而谎报要价的情况,因此提出一种社会网络下分配众包任务的真实机制（TMC-SN）。在社会网络下的众包任务分配问题被模拟成一个拍卖,其中任务请求者是买家,工人是卖家,众包平台充当拍卖者。为了找出最合适的团队,TMC-SN从边际贡献和团队凝聚力两个方面来衡量工人对团队的适应性。理论分析证明,TMC-SN具有真实性、个体理性、预算平衡等经济属性。实验结果表明,TMC-SN在社会福利方面具有一定的优势,并且能够提升工人的效用。     

Providing real-time assistance in disaster relief by leveraging crowdsourcing power

Crowdsourcing platforms for disaster management have drawn a lot of attention in recent years due to their efficiency in disaster relief tasks, especially for disaster data collection and analysis. Although the on-site rescue staff can largely benefit from these crowdsourcing data, due to the rapidly evolving situation at the disaster site, they usually encounter various difficulties and have requests, which need to be resolved in a short time. In this paper, aiming at efficiently harnessing crowdsourcing power to provide those on-site rescue staff with real-time remote assistance, we design and develop a crowdsourcing disaster support platform by considering three unique features, viz., selecting and notifying relevant off-site users for individual request according to their expertise; providing collaborative working functionalities to off-site users; improving answer credibility via “crowd voting.” To evaluate the platform, we conducted a series of experiments with three-round user trials and also a System Usability Scale survey after each trial. The results show that the platform can effectively support on-site rescue staff by leveraging crowdsourcing power and achieve good usability .