CHI NL Read: Metaphorical Agent Representation in Conversational Crowdsourcing

Hello everyone! We are very excited to to have Ji-Youn Jung showcase her recent CHI 2022 Paper with us, titled “Great Chain of Agents: The Role of Metaphorical Representation of Agents in Conversational Crowdsourcing“.

[Reading time: 4 min.]

Lead author’s name:

Ji-Youn Jung

Citation:

Ji-Youn Jung, Sihang Qiu, Alessandro Bozzon, and Ujwal Gadiraju. 2022. Great Chain of Agents: The Role of Metaphorical Representation of Agents in Conversational Crowdsourcing. In CHI Conference on Human Factors in Computing Systems (CHI ’22), April 29-May 5, 2022, New Orleans, LA, USA. ACM, New York, NY, USA, 22 pages. https://doi.org/10.1145/3491102.3517653

TL;DR (if you had to describe this work in one sentence, what would you tell someone?):

Adopting the Great Chain of Being metaphorical framework, we investigate the role of metaphorical representations of conversational agents spanning from god, human, animal, plant, and inorganic objects in the context of crowdsourcing worker engagement.

What problem is this research addressing, and why does it matter?

While numerous non-human metaphors are used in conversational agents (e.g., Clippy (Microsoft Office) or Animal Crossing (Nintendo)), previous research has heavily focused on investigating human metaphors and the level of anthropomorphism. We fill this research gap by adopting the Great Chain of Being framework and reporting empirical findings. 

How did you approach this problem?

We sampled the metaphors (Fig 1) from the Great Chain of Being levels through a pilot study. Afterward, we designed a chatbot that collects human-intelligent tasks (HITs) with two different data types: Image Classification (IC) and Information Finding (IF). It was a 6 (agent metaphor) × 2 (task type) between-subject study. Participants interacted with one of five metaphors or a control condition (Fig 2) and either an IC or IF task. After finishing the task, participants answered self-report questions about user engagement, cognitive task load, intrinsic motivation, and trust.

Five avatars used during the experiment from each metaphor condition are shown. The first avatar is for the God condition. The God condition is shown as a yellow and golden light, where orange lines of lights are beamed from the center of a circle. The second avatar is for the Human condition. It shows a woman wearing a headset with a mic. The third avatar is for the Animal condition. Overall, the dog has an orange color white spot in the middle of its face. The fourth avatar is for the Plant condition. A drawing of two avocados is displayed, one of them showing a cross-section with a brown seed. The final avatar is for the Inorganic Object condition. It is a book wide open with some text written on it.
Figure 1
Four screenshots with four different experimental conditions are shown on the worker interface. The first two screenshots show Image Classification tasks, and the last two screenshots show the Information Finding task. Within each task type, there are two different conditions shown, one without an agent avatar (control condition) and one with an agent having an avatar.
Figure 2

What were your key findings?

Our results (Table 1) showed a trade-off between adopting different metaphors from the Great Chain of Being framework. An Animal condition and a Control condition with no representation outperform others in improving worker engagement and intrinsic motivation without affecting the cognitive task load. The Inorganic Object metaphor defected worker engagement and motivation. However, the Object metaphor significantly lowered the cognitive task load that participants experienced. God and Plant metaphor showed no positive effect. Interestingly, the Human metaphor did not outperform animal or control conditions, as it gave a heavier task load on the workers compared to the control and Inorganic Object conditions (see table below). 

A table with a summarized view of significant results. Results that are considered more positive are colored in light blue, while negative results are colored in light red. Trust and Accuracy measurements were omitted since they did not vary significantly across the experimental conditions.
Table 1

What is the main message you’d like people to take away?

Designers have been intuitively attaching non-human metaphors to agents to communicate the computing system’s ability (e.g., Clippy). These real-world use cases show clear examples of how users can metaphorically understand an agent when the agent is in non-human form. This phenomenon implies endless possibilities in today’s proliferation of feminine yet disembodied agents.

Any further reading you recommend?

  • Sihang Qiu, Ujwal Gadiraju, and Alessandro Bozzon. 2020. Improving worker engagement through conversational microtask crowdsourcing. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems. 1–12. https://doi.org/10.1145/3313831.3376403 
  • Pranav Khadpe, Ranjay Krishna, Li Fei-Fei, Jeffrey T. Hancock, and Michael S. Bernstein. 2020. Conceptual Metaphors Impact Perceptions of Human-AI Collaboration. In Proceedings of the ACM on Human-Computer Interaction 4, CSCW2 (2020).  https://doi.org/10.1145/3415234
  • George Lakoff and Mark Johnson. 2008. Metaphors we live by. University of Chicago press.

Your biography

I’m a soon-to-graduate master’s student at TU Delft studying Design for Interaction in Industrial Design faculty. I am currently looking for a Ph.D. position to continue my passion for Human-Computer Interaction. My research interests include Human-AI collaboration, Conversational User Interface (CUI), and Natural Language Processing (NLP). I recently researched intersectional impact by connecting Conversational Agent identity design and conceptual metaphor theory. In addition, I also investigate the effect of user trust in decision support systems.

Ji-Youn’s website: jiyounjung.com 

CHI NL Read takes place around once a month, where board members Lisa and Abdo invite a member of CHI NL to showcase a recent research paper they published to the wider SIGCHI community and world 🌍. One of the ideas of CHI NL Read is to make research a bit more accessible to those outside of academic HCI.

CHI Nederland (CHI NL) is celebrating its 25th year anniversary this year, and we have much in store to acknowledge this occasion. Stay tuned!