CrowdBenchmark – An extensive framework to assess the quality of workers and answers

In recent years, crowdsourcing becomes a promising methodology to overcome human-intensive computational tasks such as item labeling and natural language processing. Its benefits vary from unlimited labor resources of user community to cost-effective business models. The core problem then becomes how to aggregate the answer set given by workers. In that, the estimated quality of workers and aggregated results is provided. Many techniques have been proposed to tackle this problem such as majority voting, expectation maximization (EM), supervised learning, etc. However, each technique has a specific usage scenario and distinct performance characteristics. It is difficult to compare these techniques in the same playground.

The goal of this project is to develop a benchmark which helps users to choose the best-suited technique for a specific usage scenario as well as provides an extensive insight into existing techniques. In order to achieve this goal, there are some challenges to deal with:

• How to simulate the crowd and questions?
• How to find the factors that affect the performance of aggregation techniques?
• How to compare given techniques in a fair manner?