Morphing facial technology sheds light on the boundaries of self-recognition

Facial recognition is a critical part of self-image and social interactions. In an era of advanced digital technology, we face intriguing questions about communication and identity. How does altering our facial identity affect our sense of “self” and our interactions with others?

These are questions Dr. Shunichi Kasahara, a researcher in the Cybernetic Humanity Studio at the Okinawa Institute of Science and Technology (OIST) is investigating, using real-time morphing of facial images (turning our faces into someone else’s and vice versa). The studio was established in 2023 as a platform for joint research between OIST and Sony Computer Science Laboratories, Inc.

Dr. Kasahara and his collaborators have investigated the dynamics of face recognition using motor-visual synchrony—the coordination between a person’s physical movements and the visual feedback they receive from those movements.

They found that whether we influence the movement of our self-image or not, levels of identification with our face remain consistent. Therefore, our sense of agency, or subjective feelings of control, do not impact our level of identification with our self-image. Their results have been published in Scientific Reports.

The effect of agency on perceptions of identity

With psychological experiments using displays and cameras, the scientists investigated where the “self-identification boundary” is and what impacts this boundary. Participants were seated and asked to look at screens showing their faces gradually changing.

At some point, the participants could notice a change in their facial identity and were asked to press a button when they felt that the image on the screen was no longer them. The experiment was done in both directions: the image changing from self to other and other to self.

“It’s like watching your face in a mirror as you move it and you identify yourself, but your face slowly changes up to a point and you realize this is no longer you,” Dr. Kasahara explained.

The researchers examined how three movement conditions affect the facial boundary: synchronous, asynchronous, and static. They hypothesized that if the motions are synchronized, participants would identify with the images to a greater extent.

Surprisingly, they found that whether movements were synchronized or not, their facial identity boundaries were similar. Additionally, participants were more likely to identify with static images of themselves than images with their faces moving.

Interestingly, the direction of morphing—whether from self to other or other to self—influenced how participants perceived their own facial boundaries: participants were more likely to identify with their facial images when these images morphed from self to other rather than from other to self. Overall, the results suggest that a sense of agency of facial movements does not significantly impact our ability to judge our facial identity.

“Consider the example of deepfakes, which are essentially a form of asynchronous movement. When I remain still but the visual representation moves, it creates an asynchronous situation. Even in these deepfake scenarios, we can still experience a feeling of identity connection with ourselves,” Dr. Kasahara explained.

“This suggests that even when we see a fake or manipulated version of our image, for example, someone else using our face, we might still identify with that face. Our findings raise important questions about our perception of self and identity in the digital age.”

How does identity impact perceptions of control?

What about the other way around? How does our sense of identity impact our sense of agency? Dr. Kasahara published a paper in collaboration with Professor of Psychology at Rikkyo University, Dr. Wen Wen, who specializes in research on our sense of agency. They investigated how recognizing oneself through facial features might affect how people perceive control over their own movements.

During experiments, participants observed either their own face or another person’s face on a screen and could interact and control the facial and head movements. They were asked to observe the screen for about 20 seconds while moving their faces and changing their facial expressions.

The motion of the face was controlled either only by their own facial and head motion or by an average of the participant’s and the experimenter’s motion (full control vs. partial control). Thereafter, they were asked “how much did you feel that this face looks like you?” and “how much control did you feel over this presented face?”

Again, the main findings were intriguing: participants reported a higher sense of agency over the “other face” rather than the “self-face.” Additionally, controlling someone else’s face resulted in more variety of facial movements than controlling one’s own face.

“We gave the participants a different face, but they could control the facial movements of this face—similar to deepfake technology, where AI can transfer movement to other objects. This AI technology allows us to go beyond the conventional experience of simply looking into a mirror, enabling us to disentangle and investigate the relationship between facial movements and visual identity,” Dr. Kasahara stated.

“Based on previous research, one might expect that if I see my own face, I will feel more control over it. Conversely, if it’s not my face, I might expect to feel less control because it’s someone else’s face. That’s the intuitive expectation. However, the results are the opposite—when people see their own face, they report a lower sense of agency.

“Conversely, when they see another person’s face, they’re more likely to feel a sense of agency.” These surprising results challenge what we thought we knew about how we see ourselves in images.

Dr. Kasahara emphasized that the acceptance of technology in society plays a crucial role in technological advancements and human evolution.

“The relationship between technology and human evolution is cyclical; we evolve together. But concerns about certain computer technology may lead to restrictions. My goal is to help foster acceptance within society and update our understanding of ‘the self’ in relation to human-computer integration technology.”