Emo, equipped with cameras in its eyes and 26 motors beneath its blue flexible silicone skin mimicking human facial muscles, can power its expressions. It predicts upcoming smiles 839 milliseconds before humans smile, synchronizing its expression with theirs. Emo can also anticipate expressions like sadness, anger, and surprise.
While advancements in AI have accelerated language communication with robots, non-verbal cues have lagged behind. Researchers at Columbia University have developed a robot capable of observing facial expressions, predicting human smiles 0.9 seconds in advance based on subtle facial changes, and responding with a smile. The related study was published on March 27 in Science Robotics.
Artificial intelligence can mimic human language, but robots still struggle to replicate complex non-verbal cues and speech behaviors crucial to communication. Humanoid robots can rely on voice for communication, but expressing through facial movements presents a dual challenge: driving a robot face with rich, flexible expressions poses physical challenges, and determining what expressions to generate to make the robot appear genuine, natural, and timely.
Emo, a robot developed by researchers at Columbia University's Creative Machines Lab, aims to predict and replicate human facial expressions using artificial intelligence models and high-resolution cameras.
According to the team led by Professor Hod Lipson from Columbia University's Department of Mechanical Engineering, they propose a solution to mitigate both these challenges by training robots to predict future facial expressions and simultaneously execute them alongside humans. The Emo robot, developed by Lipson's team, utilizes AI models and high-resolution cameras to forecast human facial expressions and attempt to mimic them.
Researchers claim that the robot can predict an upcoming smile 839 milliseconds before a human smiles and replicate it in real-time using its model. Yuhang Hu, a doctoral student at Columbia University's Creative Machines Lab and the lead author of the paper, explains that there's a brief period before a smile fully forms where the corners of the mouth start to turn up and the eyes begin to slightly crinkle. Emo is capable of capturing these subtle changes on people's faces to predict facial expressions. The researchers demonstrated this capability using a robot face with 26 degrees of freedom.
The robot comprises 26 motors and utilizes positional control. Three motors govern the movement of the neck along three axes. Twelve motors control the upper face, including the eyeballs, eyelids, and eyebrows. Eleven motors control the mouth and jaw.
Emo's eyes are equipped with cameras, and beneath its blue flexible silicone skin lie 26 motors, akin to human facial muscles, powering the robot's facial expressions. Among these, three motors control the movement of the neck along three axes. Twelve motors control the upper face, including the eyeballs, eyelids, and eyebrows. Eleven motors govern the mouth and jaw. The robot employs two neural networks: one to observe human faces and predict expressions, and another to study how to generate expressions on the robot's face. The first neural network is trained using videos from video-sharing platforms, while the second neural network enables the robot to train by watching its own expressions through a real-time camera feed.
"When it pulls all these muscles, it knows what its face is going to look like," said Lipson. "It's a bit like a person smiling in front of a mirror, even with their eyes closed, knowing what their face is going to look like."
Researchers hope this technology will make human-robot interaction more lifelike. They believe that robots must first learn to anticipate and mimic human expressions before evolving into more spontaneous, self-driven expressive communication.
In addition to smiling, Emo can also predict expressions like sadness, anger, and surprise. However, Emo cannot yet replicate all human expressions, as it only has 26 facial "muscles." In the future, researchers will need to broaden the range of expressions the robot can make. They also aim to train the robot to respond to what people say, rather than simply mimicking another person. Furthermore, researchers are integrating language communication into Emo using large models, enabling Emo to answer questions and engage in dialogue.
