We welcome and thank Esra for sharing her CHI 2024 paper!
What’s your name?
Hi, my name is Esra de Groot.
Citation:
Esra Cemre Su de Groot and Ujwal Gadiraju. 2024. “Are we all in the same boat?” Customizable and Evolving Avatars to Improve Worker Engagement and Foster a Sense of Community in Online Crowd Work. In Proceedings of the CHI Conference on Human Factors in Computing Systems (CHI ’24), May 11–16, 2024, Honolulu, HI, USA. ACM, New York, NY, USA, 26 pages. https://doi.org/10.1145/3613904.3642429
TL;DR (if you had to describe this work in one sentence, what would you tell someone?):
Our work has investigated how customizable and evolving avatars and fostering a sense of group identification among crowd workers can impact their experience and task-related outcomes in crowd work.
What problem is this research addressing, and why does it matter?
To build ground-truth data and training sets for the development and testing of artificial intelligence and machine learning models, human intelligence tasks are distributed at scale on online crowdsourcing platforms such as Amazon Mechanical Turk and Prolific. This work often consists of monotonous and boring task batches, which can cause decreased motivation and low engagement among workers. Moreover, workers often work in isolation and complex work environments and are not connected to their remote peers. Together, these factors may contribute to a poor work experience for workers, leading to high drop-out rates, low task quality, and decreased worker well-being.
How did you approach this problem?
We were interested in investigating how we can improve worker experience and task-related outcomes for workers who individually work on monotonous task batches. From prior research, we know that using gamification, conversational interfaces, avatar customization, and community identification can contribute to facilitating engagement in online crowd work. We then combined these different aspects to create gamified evolving customizable avatars, which serve as a worker’s digital identity. On completing a set of 4 tasks each time in a batch, workers unlock a new level that comes with new editable features to further personalize their avatar. After workers completed their tasks, they entered the worker community space, which was designed to induce a feeling of group identification among crowd workers. In the worker community space, workers saw their own avatars surrounded by the avatars of other workers and a brief expression of how the task made them feel. Our goal was to create a lightweight and privacy-sensitive method to induce a feeling of group identification. To investigate the impact of the evolving avatars and fostering group identification on worker experience and task-related outcomes, we conducted a randomized controlled trial spanning five conditions and two different task types (a credibility analysis task and an information finding task).
What were your key findings?
Evolving customizable worker avatars can increase worker retention (i.e., the number of tasks completed), especially for the credibility analysis task. While we did not find an impact of the worker community space on group identification among crowd workers, our exploratory analysis showed that crowd workers who identify strongly as being crowd workers show greater intrinsic motivation, subjective engagement, and perceived workload. As our results differed between our two tasks, we explored how the two tasks were perceived differently by the crowd workers. This second exploratory analysis revealed that workers showed greater intrinsic motivation, greater subjective engagement, and lower perceived workload for the credibility analysis task. Our findings suggest that task-specific characteristics can amplify and/or mitigate the impact of (gamified) interventions.
What is the main message you’d like people to take away?
Although the size of the impact might depend on task-specific characteristics, evolving and customizable worker avatars can improve worker retention. This finding might be particularly interesting for designing tasks that require some training time for the worker.
As workers with a strong sense of group identification show greater intrinsic motivation and subjective engagement, fostering group identification among crowd workers may contribute to sustainable crowd work. Future research can focus on finding lightweight methods to foster this sense of group identification among crowd workers and how to leverage a healthy sense of group identification (as greater group identification may come with greater perceived workload).
What led / inspired you to carry out this research?
Earlier work on avatar customization in conversational crowdsourcing inspired us to conduct this study. Besides, I have a general interest in promoting well-being, which is why I find it important to foster good work experiences among workers. I believe work should be an enjoyable experience and not detrimental to (mental) well-being.
What kind of skills / background would you say one needs to perform this type of research?
To perform this type of research I would say one needs a general interest in human behavior. Regarding skills, one needs to know how to set up a randomized controlled trial and perform appropriate statistical tests to analyze the results. Apart from that, one needs (the willingness to learn) the technical skills to develop and deploy the experimental interface for the participants.
Besides these aspects, I would highly recommend researchers preregister the study setup and statistical methods for this type of research (for instance on platforms like OSF). I believe this is great practice, however not yet very common in the field of computer science (compared to other fields such as psychology).
Any further reading you recommend?
- Qiu, S., Bozzon, A., Birk, M. V., & Gadiraju, U. (2021). Using worker avatars to improve microtask crowdsourcing. Proceedings of the ACM on Human-Computer Interaction, 5(CSCW2), 1-28.
- Kobayashi, M., Arita, S., Itoko, T., Saito, S., & Takagi, H. (2015, February). Motivating multi-generational crowd workers in social-purpose work. In Proceedings of the 18th ACM Conference on Computer Supported Cooperative Work & Social Computing (pp. 1813-1824).
Your biography:
I am a PhD candidate in Computer Science at Delft University of Technology. My main research revolves around human-AI decision-making, with a specific focus on developing and using human-AI decision-making tools to promote adolescents’ mental well-being. To develop this decision-making tool, I also conduct studies related to understanding users’ needs and investigating ways to improve engagement.
Please refer to my personal website for an overview of my work and if you want to get in contact! 🙂