Visiting the Venice Biennale exhibition remotely with the iCub3 robot

Over the past few decades, technological advances have opened new and exciting possibilities for both remote tourism and the teleoperation of robotic systems. This allowed computer scientists to develop increasingly sophisticated systems that allow humans to virtually visit remote locations in immersive ways.

Researchers at the Italian Institute of Technology (IIT) in Genoa have recently introduced an innovative avatar-based system that allows a human operator to virtually navigate and move around in a real-world environment remotely and in real-time, as if the operator were physically there. This system, introduced in a paper pre-published on arXiv and presented in a video posted on YouTube, is based on the humanoid robot iCub3, also created at IIT. The team successfully used it to visit the Venice Biennale, a renowned art and architecture exhibition in Venice, in Italy.

“My lab, the Artificial and Mechanical Intelligence lab at IIT, has several research axes, and one of those is what we call telexistence,” Daniele Pucci, coordinator of the research project, told Tech Xplore. “The primary goal of this research is to make physical avatars for humans via humanoid robots. This technology will allow us to exist physically and act effectively in a remote place thanks to a robotic (humanoid) body.”

The term “telexistence” entails allowing humans to feel like they are somewhere else, while also being able to interact with their surrounding environment physically, as if they actually existed in this remote location. This is typically achieved using a combination of virtual reality (VR) technology and physical robots.

Pucci and his colleagues have been trying to develop telexistence systems for several years. Their recent paper builds on a study they carried out in 2017 and on the research efforts that followed.

“In September 2018, we showed a preliminary avatar system using the iCub 2.0,” Pucci explained. “The iCub 2.0, however, could not play the role of an avatar for humans effectively, as its height (about one meter) and its arms’ limited workspace did not allow the robot to be effective in an environment made for adult humans. For instance, the robot had difficulties in reaching objects on a table.”

To improve their previous system, in 2019 Pucci and their colleagues started working on a new version of iCub, the iCub3, which is 1.2m in height, has an improved actuation and integrates more advanced sensors. These characteristics make it a better avatar for the team’s telexistence system than its predecessors.

“Our work also has three philanthropic objectives,” Pucci said. “Firstly, the COVID-19 pandemic taught us that advanced telepresence systems might become necessary very quickly across different fields, including healthcare and logistics. Secondly, avatars may allow people with severe physical disabilities to work and accomplish tasks in the real world via the robotic body. This may be an evolution of rehabilitation and prosthetics technologies. Lastly, we may imagine a future for tourism, where physical avatars are distributed around the world and we can access them from our home and visit Tokyo, New York, or New Delhi remotely just by wearing a headset and few other wearable technologies.”

The iCub3-based Avatar System introduced by Pucci and his colleagues virtually transports a human operator to a remote place, allowing it to move, grasp or manipulate objects, speak and make facial expressions in this place. The system has two key components: the operator technology and the avatar (iCub3).

The user’s movements, words and facial expressions are carried out by the iCub3 robot, which is deployed in the place he/she is virtually visiting. In addition, users receive constant visual, auditory, haptic and touch-related feedback from the remote environment, through a series of wearable devices, integrated in the operator technology.

“The operator technology consists of devices created at IIT and off-the-shelf products that have been integrated into the system,” Pucci explained. “The IIT devices, developed in the context of iFeel (a start-up project by our lab), include a set of wearable technologies to track the human motion and to generate haptic feedback. The iFeel technology allows us to monitor the motion of the operator arms, for instance, and then project it onto the avatar. Analogously, it allows us to emulate the sense of touch: if someone hugs the avatar, motors emulate a sense of touch on the human operator.”

In addition to the wearable technology developed by the IIT’s spinoff, the system’s operator hardware integrates a series of off-the-shelf products, including headsets, haptic gloves, and treadmills. A huge part of Pucci and his colleague’s recent study was to develop algorithms and software architectures that can efficiently integrate the data gathered by both these off-the-shelf products and iFeel devices, while also producing outputs that are sent to the iCub3 avatar.

“Our system’s second component is the avatar, namely the iCub3 humanoid robot,” Pucci said. “The iCub3 robot is 25 cm taller than the previous iCub versions, thus representing a more adequate platform to interact within a human environment. Its balance and locomotion are more robust and able to emulate human movements and physical interaction better.”

As it is bigger than its predecessors, iCub3 weighs 52kg, approximately 20 kg more than iCub2. In addition, the motors in its legs are more powerful than those in previous iCub robots, allowing the robot to move faster and more efficiently.

“The iCub3 platform also has different actuation mechanics, as it is no longer based on cable-driven joints,” Pucci explained. “In terms of sensors, it has an additional depth camera and force sensing of the latest generation, withstanding higher robot weight. Lastly, iCub3 has a higher capacity battery, which is located within the torso assembly instead of being included in a rigidly attached backpack.”

To evaluate their avatar system and demonstrate its potential, the researchers used it to remotely visit the Italian Pavilion at the 17^th International Architecture Exhibition at the Venice Biennale. Using a standard fiber optic connection, they allowed a user located in their campus in Genoa to walk real-time through the exhibition in Venice.

“To the best of our knowledge, this is the first time that an avatar system transports the operator locomotion, manipulation, voice, and face expressions to a legged humanoid avatar while receiving visual, auditory, haptic and touch feedbacks,” Pucci said. “The use of a system to allow remote tourism in Venice is also unique to us.”

In the future, the avatar system could allow users to virtually travel to a wide range of cities or art exhibitions in the world. The system was also entered into the ANA Avatar XPRIZE, a competition for teams building robotic avatars with a total prize of 10 million dollars, and is among the semifinalists. Pucci and his colleagues are thus currently working to further improve the platform for the final stages of the competition.

“Another plan for our future work is related to the so-called Metaverse,” Pucci added. “We believe that much of the technology and algorithms we developed could be useful for controlling and receiving information from digital avatars that populate the Metaverse. So, we now also plan to investigate how to apply our results in this new fascinating direction, hoping that investors will support us on this journey.”

© 2022 Science X Network