WorkAboutResume
Laptop with robot face and eye tracking device
Human-Robot Interaction Research Project
Understanding the effect of eye gaze mirroring on perception of robot comfort
February - April 2022

My Role

User Testing Lead
Literature review, experiment design, survey development, eye tracking setup, data analysis

Team

(2) Researchers (including me)(2) Software Developers

Context

Academic Research Project (Carnegie Mellon Robotics Institute)

TL;DR

Research Goal

What is the effect of eye gaze mirroring on comfort in human-robot social interaction?
As part of an academic research project within Carnegie Mellon’s Robotics Institute, our team sought to better understand the relationship between eye gaez and human comfort with a robot. The team focused on eye gaze mirroring due to some core hypotheses informed by a literature review:
  • Eye gaze mirroring will increase human comfort - While not too much research has been conducted on eye gaze mirroring in human-robot social interaction, eye gaze is a main nonverbal cue. Eye gaze mirroring is prevalent in human-to-human interaction, so we hypothesized that gaze mirroring in human-robot interaction would similarly make it feel more comfortable.

Project Highlights

We analyzed the data from our user testing sessions, presented our findings, and wrote a research paper about the work. Some learnings included:
  • Blink frequency, as measured by the eye tracking classes, appeared higher when eye gaze mirroring was present suggesting more information processing.
  • According to a comment sentiment analysis, interactions with eye gaze mirroring were described with more positive words (22% positive) than interactions without mirroring (7% positive).

Project Process

Research process diagram

Understanding the Problem Space

To familiarize ourselves with HRI eye gaze literature and ensure our research was novel, our team conducted a thorough literature review of 32 academic references.
Literature Review
Hypothesis Development
Literature Review

What research has already been about eye gaze in the human-robot interaction (HRI) field? What questions are still unanswered?

For an academic research project, we wanted to ensure our research contributions would be novel to the field of human-robot interaction. We conducted a thorough literature review reading articles about robot mimicry, eye gaze/eye tracking, nonverbal communication, robot trust, and eye tracking technologies.
Through our research, we learned that eye gaze mirroring happens naturally in human-to-human conversations. For this reason, we hypothesized that eye gaze mirroring in human-to-robot interactions would make the interaction more natural and make humans, therefore, feel more comfortable.
A collection of articles including in a literature review

We learned...

The friendliest robot had a round eye shape with large irises
(Onuki, Tomomi, et al., 2013)
Robots received more favorable impressions if it imitated the participant’s behavior
(Shimada, Michihiro, et al., 2008)
Directed gaze during small talk was more accepted than a random gaze
(Babel, Franziska, et al., 2021)
Nonverbal cues seem to convey an expression of internal state
(Breazeal, Cynthia, et al, 2005)
Research Question
How does mirroring the eye gaze patterns of a human partner effect comfort level during interaction?

Testing the Hypothesis with Users

To set up our user tests, I led the experiment design, task, and survey development. Two software developers worked on programming the robotic eye gaze to mirror human eye gaze. We recruited 10 participants for the user tests.
Experiment Design
Survey Design
User Testing

How can we test and measure the effect of eye gaze mirroring on human-robot comfort?

To test our hypothesis that human comfort level will increase with a robot that mirrors their eye gaze, we defined variables and how to measure them. Due to a limitations with the operability of the eyes on robots we had access to, we decided to develop our own “robot” interface using Python so we had full control over the eye gaze.
Subject Design: Within Subjects
Independent Variable (IV): Gaze pattern
Dependent Variable (DV): Comfort level
An image showing objective metrics in a red box and subjective metrics in a blue box

Study Protocol

A visual showing the outline of the study protocol
An image of the study with a participant wearing eye tracking glasses and looking at the robot face on the computer screen

Task/Script Development

Using Voiceflow, our team developed a mock interview script to “Wizard of Oz” conversation between the robot and the human. All of the questions in the interview script were open-ended and not organized by category in order to decrease any confounds that might result between trials.
A zoomed out image of the voiceflow script used to create the script for the participant
Research Question

How should we design the robot interface? What eye gaze pattern should we follow?

The design of our robotic interface consisted of two main components:
  1. PupilCore Interface - connected to the eye tracking glasses and builds the gaze model for objective metrics
  2. Robot Animation - created using pygame and consists of 2 images overlayed and the “eye” image moving based on polar coordinate of the input
A GIF of the control condition robot that does not track eye gaze

Control Condition

Robot has default eye gaze pattern, pupil core glasses record user’s gaze pattern
A GIF of the experimental condition where the robot faces mirrors the user's gaze

Experimental Condition

Robot mirrors the user’s gaze based on the pupil core’s tracking, pupil core glasses also record user’s gaze pattern

Questionnaire Design

To assess human comfort with the robot, after each condition, participants were given a questionnaire with eight 5-point likert scale questions. Five of the questions corresponded to the Robotic Social Attributes Scale (RoSAS) about comfort. To manage bias, 4 questions were framed positively and 4 were framed negatively.
To assess human comfort with the robot, after each condition, participants were given a questionnaire with eight 5-point likert scale questions. Five of the questions corresponded to the Robotic Social Attributes Scale (RoSAS) about comfort. To manage bias, 4 questions were framed positively and 4 were framed negatively.
  • Describe their experience interacting with the robot
  • Explain why they felt the robot was or was not engaged
  • Explain why they felt comfortable or uncomfortable interacting with the robot
A screenshot of the post-study survey asking participants likert scale questions about their interactions with the robot

Analyzing the Data & Identifying Insights

I owned the analysis of both survey data, comment sentiment analysis, and eye tracking data including performing statistical analyses. All data analysis was performed in Excel.
Data Analysis
Statistical Tests

Results

We recruited 10 participants for the pilot study with the following breakdown:
  • Ages 22-30
  • 6 Male, 4 Female (self-identified)
  • MHCI, MBA, and MRSD programs
  • Interested in roles in UX design, UX research, robotics, and product management
A donut pie chart breakdown of the academic fields of participants

Were any of the results statistically significant?

Subjective Metrics
I performed two-sample paired t-tests for all 8 questions between the 2 conditions. None of the results between the experimental and control were found to be statistically significant (p < 0.05).
Given the small sample size, I then performed sign tests for all 8 questions and had the following two findings:
  • Participants rated the experimental condition as more sociable (p = .031) and more engaged (p = .035) than the control condition
  • Participants perceived the control as listening better than the experimental (p = .035)
A bar graph with the impact of eye gaze mirroring on positive comments in control versus experimental
Objective Metrics
Using objective metric data from the PupilLabs eye tracking glasses, I performed two-sample paired t-tests for all 4 metrics between the conditions. None of the results between the experimental and control were found to be statistically significant (p < .05).
Sign tests could not be used on this dataset because it contained ordinal data.
A bar graph depicting differences in positive and negative interaction descriptions for control and experimental groups

Findings

Eye gaze mirroring shows promise
A majority of the positive statements (comfortable, engaged, sociable) did see a rise in agreement for the experimental condition and a majority of the negative statements (awkward, dismissive, strange) saw a decrease in agreement for the experimental condition.
A bar graph depicting the differences in eye gaze mirroring comfort level between control and experimental
A bar graph depicting differences in positive and negative interaction descriptions for control and experimental groups
Eye gaze mirroring was described positively
A sentiment analysis was conducted for participants responses to open-ended questions. When asked to describe their interactions with the robot, participants used more positive words after the experimental condition than they did after the control and more negative words after the control than they did after the experimental.
Eye gaze mirroring was perceived as ‘natural’
In their comments, participants noted a difference in experience in the experimental versus control condition. This positive difference was more pronounced for participants who saw the control condition before the experimental condition.
Quotes from participants describing the control and experimental conditions
A bar graph depicting differences in blink frequency for control versus experimental groups
Blink frequency was higher when gaze mirroring was present
While not statistically significant, the blink frequency appeared higher when eye gaze mirroring was present for a majority of the participants. Blinking typically indicates higher workload to update working memory, potentially indicating that the experimental condition kept participants more engaged.
Nonverbal behavior impacts engagement
Participants commented about other robot attributes besides eye gaze. Among the nonverbal communication methods mentioned were:
  • Content of the questions, which related to psychological safety
  • Tone of voice
  • Absence of body language (head nodding, mouth movement)
Quotes about non-verbal behavior of robot

Project Outcome

To present our findings, our team wrote an academic research paper about the project.
View Paper

Study Limitations & Confounds

  • Novelty Effect - participants may have been excited to use a robot
  • 2D robot interface versus a 3D robot may have impacted perceived comfort
  • Topic of conversation impacted comfort, which may have confounded our results

User Benefits

  • The research begins to consider how humans could be made more comfortable when interacting with social robots, which will become increasingly prevalent
  • Suggests interesting potential research comparing another nonverbal behavior (i.e. tone, body language) in addition to eye gaze mirroring
  • Suggests interesting potential research about which contexts, based on psychological distancing and social proxemics, eye gaze mirroring would be beneficial in
Back to Work