《Perspectives on Crowdsourcing Annotations forNatural Language Processing1自然语言处理的众包标注透视》由会员分享，可在线阅读，更多相关《Perspectives on Crowdsourcing Annotations forNatural Language Processing1自然语言处理的众包标注透视（22页珍藏版）》请在金锄头文库上搜索。
1、Perspectives on Crowdsourcing Annotations for Natural Language Processing1Aobo WangCong Duy Vu Hoang Min-Yen Kan Computing 1, 13 Computing Drive National University of Singapore Singapore 117417 wangaobo,hcdvu,kanmycomp.nus.edu.sgJuly 24, 20101The authors gratefully acknowledge the support of the Ch
2、ina-Singapore Institute of DigitalMedias support of this work by the “Co-training NLP systems and Language Learners” grant R 252-002-372-490.AbstractCrowdsourcing has emerged as a new method for obtaining annotations for training models for machine learning. While many variants of this process exist
3、, they largely differ in their method of motivating subjects to contribute and the scale of their appli- cations. To date, however, there has yet to be a study that helps a practitioner to decide what form an annotation application should take to best reach its objectives within theconstraints of a
4、project. We first provide a faceted analysis of existing crowdsourc- ing annotation applications. We then use our analysis to discuss our recommendations on how practitioners can take advantage of crowdsourcing and discuss our view on potential opportunities in this area.0.1IntroductionIt is an acce
5、pted tradition in natural language processing (NLP) to use annotated cor- pora to obtain machine learned models for performing many tasks: machine trans- lation, parsing, and summarization. Given that machine learners can only performtasks as good as their input annotation, much work in annotation c
6、entered on defininghigh quality standards that were reliable and reproducible, and finding appropriately trained personnel to carry out such tasks. The Penn Treebank and WordNet are prob- ably the most visible examples in this community. Even now, this high quality route continues to be used in sist
7、er projects in extending these resources to other languages: HowNet (Dong and Dong, 2006) and EuroWordNet (Jacquin et al., 2007). An alternative to high quality annotation is to make use of quantity and the rulethat redundancy in large data would act to filter out noise. The emergence of the Web mad
8、e this a real possibility, where raw monolingual and parallel corpora, term counts and user generated content enabled the mining of large amounts of statistical data to train NLP models. In Web 2.0, it is also clear that the Web has made people avail- able as resources to take advantage of. This tre
9、nd reaches one logical conclusion when the web serves to network human service providers with those seeking their services. Although this process is described by many different terms, we use the term crowd- sourcing throughout this paper. Crowdsourcing is a strategy that combines the effort of the p
10、ublic to solve one problem or produce one particular thing. “Crowdsourcing” has been used in the popular press to emphasize that the workers need not be experts but laymen or amateurs. While human subjects can be used to provide data or services in many forms, we limit our attention in this work on
11、annotations for data useful to NLP tasks, and do not focus on the distributed nature of crowdsourcing. Crowdsourcing takes many forms that require different forms of motivation to achieve the end goal of annotation. In Games with a Purpose (hereafter, GWAP), the main motivator is fun (von Ahn and Da
12、bbish, 2008a,b). Annotation tasks are de- signed to provide entertainment to the human subject over the course of short sessions.In Amazon Mechanical Turk (MTurk), the main motivator is profit. Providers create and list batches of small jobs termed Human Intelligence Tasks (HITs) on Amazons Mechanic
13、al Turk website, which may be done by the general public. Workers who ful-fill these tasks get credited in micropayments. While certainly not the only paid labor sourcing environment, Mechanical Turks current ubiquity make “MTurk” a useful la- bel to refer to this and other forms of computer mediate
14、d labor. Wisdom of the Crowds (WotC) is another form of crowdsourcing. WotC deployments allow members of the general public to collaborate to build a public resource, or to predict event outcomes orto estimate difficult to guess quantities. Wikipedia, the most well-known fielded WotC application, ha
15、sdifferentmotivatorsthathavechangedovertime. Initially, altruismandindirect benefit were factors: people contributed articles to Wikipedia to help others but also to build a resource that would ultimately help themselves. As Wikipedia matured, prestige of being a regular contributor or editor also s
16、lowed the ranks of contributors from the crowd to a more stable formalized group (Suh et al., 2009). It is important to recognize that these different motivators crucially shape each form of crowdsourcing, changing key characteristics. Equally important is to note that the space of possible motivations and dimensions of crowdsourcing have not been fully1explored. Given raw linguistic data, what vehicle for annotation would be most fruitful to pursue? Thus far, there has been no systematic
