Crowdsourcing at a large scale raises a variety of open challenges:
We believe tackling such problems will be key to taking crowdsourcing to the next level – from its uptake by early adopters today, to its future as how the world’s work gets done.
- How do we programmatically measure, incentivize and improve the quality of work across thousands of workers answering millions of questions daily?
- As the volume, diversity and complexity of crowdsourcing tasks increase, how do we scale the hiring, training and evaluation of workers?
- How do we design effective elastic marketplaces for more skilled work?
- How do we adapt models for long-term, sustained contributions rather than ephemeral participation of workers?
To advance the research and practice in crowdsourcing at scale, our workshop invites position papers tackling such issues of scale. In addition, we are organizing a shared task challenge regarding how to best aggregate crowd labels on large crowdsourcing datasets released by Google and CrowdFlower.
Tatiana Josephy (
Matthew Lease (@mattlease), University of Texas at Austin
Praveen Paritosh (@heuristicity), Google
Omar Alonso, Microsoft
Ed Chi, Google
Lydia Chilton, University of Washington
Matt Cooper, oDesk
Peng Dai, Google
Benjamin Goldenberg, Yelp
David Huynh, Google
Panos Ipeirotis, Google/NYUChris Lintott, Zooniverse/GalaxyZoo
Greg Little, oDesk
Stuart Lynn, Zooniverse/GalaxyZoo
Stefano Mazzocchi, Google
Rajesh Patel, MicrosoftMike Shwe, Google
Rion Snow, Twitter
Maria Stone, Microsoft
Alexander Sorokin, CrowdFlower
Jamie Taylor, Google
Tamsyn Waterhouse, Google
Patrick Philips, LinkedInSanga Reddy Peerreddy, SetuServ