A software package to ease the use of neural radiance fields in robotics research

Neural radiance fields (NeRFs) are advanced machine learning techniques that can generate three-dimensional (3D) representations of objects or environments from two-dimensional (2D) images. As these techniques can model complex real-world environments realistically and in detail, they could greatly support robotics research.

Most existing datasets and platforms for training NeRFs, however, are designed to be used offline, as they require the completion of a pose optimization step that significantly delays the creation of photo realistic representations. This has so far prevented most roboticists from using these techniques to test their algorithms on physical robots in real-time.

A research team at Stanford University recently introduced NerfBridge, a new open-source software package for training NeRF algorithms that could ultimately enable their use in online robotics experiments, This package, introduced in a paper pre-published on arXiv, is designed to effectively bridge ROS (the robot operating system), a renowned software library for robotics applications, and Nerfstudio, an open-source library designed to train NeRFs in real-time.

“Recently members of my lab, the Stanford Multi-robot Systems Lab, have been excited about exploring applications of Neural Radiance Fields (NeRFs) in robotics, but we found that right now there isn’t an easy way to use these methods with an actual robot, so it’s impossible to do any real experiments with them,” Javier Yu, the first author of the paper, told Tech Xplore. “Since the tools didn’t exist, we decided to build them ourselves, and out of that engineering push to see how NeRFs work on robots we got a nice tool that we think will be useful to a lot of folks in the robotics community.”

NeRFs are sophisticated techniques based on artificial neural networks that were first introduced by the computer graphics research community. They essentially create detailed maps of the world by training a neural network to reconstruct the 3D geometry and color of the scene captured in a photograph or 2D image.

“The problem of mapping from images is one that we in the robotics community have been working on for a long time and NeRFs offer a new perspective on how to approach it,” Yu explained. “Typically, NeRFs are trained in an offline fashion where all of the images are gathered ahead of time, and then the NeRF of the scene is trained all at once. In robotics, however, we want to use the NeRF directly for tasks like navigation and so the NeRF is not useful if we only get it when we arrive at our destination. Instead, we want to build the NeRF incrementally (online) as the robot explores its environment. This is exactly the problem that NerfBridge solves.”

NerfBridge, the package introduced by Yu and his colleagues, utilizes images captured by the sensors and cameras integrated in physical robots. These images are continuously streamed into Nerfstudio’s powerful NeRF training library, enabling the creation of NeRFs that are constantly updating themselves and improving as the robot captures new images of its surroundings.

To demonstrate the potential of their method, Yu and his colleagues used it to train a NeRF based on images captured by a camera mounted on a quadrotor, a drone with four rotors, as it flew around in both indoor and outdoor environments. Their results were remarkable, highlighting the value of NerfBridge for facilitating the use of NeRFs in robotics research.

This promising method could thus soon be used by other researchers to train NERFs and test their algorithms on physical robots as they navigate their surrounding environment. Meanwhile, Yu and his colleagues plan to explore additional strategies that could broaden the use of NeRFs in robotics.

“Ultimately, we hope that NerfBridge will lower the barrier of entry for other researchers to start looking at applications of NeRFs in robotics, and to test their new algorithms on robots in the real world,” Yu added. “Moving forward from NerfBridge, we are going to be looking into methods for improving NeRF training when images come streamed from a robot and demonstrating the concrete advantages of using NeRF-based maps for other tasks in robotics like localization and navigation.”

© 2023 Science X Network