New technology uses AI and virtual reality to monitor safety of bridges and buildings

Monitoring the structural health of the nation’s aging buldings and bridges is vital to keeping people safe and helping prevent tragedies such as the Surfside condominium collapse in 2021.

That’s why University of Central Florida researchers have developed four new inventions that use artificial intelligence and virtual reality to improve the structural health monitoring of buildings, bridges, roads and other civil structures.

“Structural health monitoring is an area of need internationally,” says Necati Catbas, a Lockheed Martin St. Laurent Professor in UCF’s Department of Civil, Environmental and Construction Engineering. “It’s almost like human health monitoring. As we get older, monitoring our health becomes very, very critical.”

Catbas, who lead the development of the structural health monitoring technologies, says civil infrastructure systems in developed countries are aging but these new technologies can help.

“By better understanding their conditions, we can anticipate risks and better prioritize infrastructure investments,” he says.

Catbas says that traditional monitoring methods involve onsite visual inspection, which can be both time-consuming and costly with manual inspections and can create road and bridge traffic closures. In addition to time and expense, sites with aging or damaged structures can pose dangers to those at the site, even if they wear personal protective equipment.

Catbas and his research team developed the technologies to help address these issues.

“I am very lucky to have collaborated with many people who have expertise in structural health monitoring over the years, and I have to acknowledge their contribution,” he says. “It’s not a one-person effort.”

Monitoring structural health using computer vision and augmented/virtual reality

One invention Catbas and his team developed employs computer vision, while another uses augmented reality (AR) and virtual reality (VR).

He says computer vision can complement sensors and visual inspection of structural health, and that it is very practical because it doesn’t require access structures such as bridges, buildings, or towers.

“We can use the camera, and by analyzing the images, we can extract meaningful information about these bridges and buildings,” he says.

The technology, a comprehensive structural health monitoring system, enables inspectors to safely view and accurately assess the load-worthiness and serviceability of structures without having to be onsite.

Catbas says that the UCF invention uses cameras stationed on and around a structure, like a bridge, to collect image and location data related to the structure’s use. In the bridge example, the data relates to vehicles crossing it. The data can include the vertical or horizontal displacement of girders caused by their movement, vibrational effects and velocity. While the cameras continually monitor the site, computer vision software processes and analyzes the collected data, providing system users with a safety assessment that includes information about structural changes and weaknesses, as well as immediate damage.

The second invention that the team developed is an immersive visualization system that uses VR and AR to analyze structures via “virtual visits.” VR provides a completely computer-simulated environment, while AR generates or overlays content onto actual views of a real-world environment.

“With this technology, you can virtually bring experts to disaster areas, such as buildings and bridges, like after a hurricane,” Catbas says. “I can virtually be on a damaged bridge in Florida discussing decisions with colleagues who might be in California.”

Like the first invention, the visualization system provides damage detection and load-carrying information about a structure using cameras and sensors. Additionally, it employs other tools such as robots, unmanned aerial vehicles (UAVs) or drones, LiDAR scanners and infrared thermography cameras. With its visualization platform, the technology provides the collected data and images via a user interface and sophisticated computer graphics. The result is a real-time view of a site and the ability to interact and communicate with people from different locations: onsite, across the country, and even globally.

Enhancing inspections and structural damage diagnostics using artificial intelligence

Two other inventions developed by Catbas and his team incorporate AI. First, the collective intelligence framework technology blends human-centric AI with mixed reality to help fast-track inspection processes and keep costs down while ensuring accuracy. With this invention, an inspector standing outside a damaged building could wear a headset and/or use a hand-held device integrated with the technology.

The inspector uses the items to scan the damaged areas, which the system analyzes in real-time, saving the inspector from having to perform manual measurements. It then calculates or assesses the building’s condition, thus speeding the inspection process. During the assessment, the inspector interacts with the AI and can adjust its defect and detection boundaries. The system uses the inspector’s changes to retrain the AI model so that the AI’s accuracy improves over time. A major advantage of the invention is its ability to combine the professional judgment of an inspector/engineer with the AI’s analytical power.

The other invention, the generative adversarial network (GAN), enables a more proactive approach to managing and maintaining the health and safety of structures. It uses AI to predict damage and minimize the need for data collection from many structures.

“Instead of putting sensors and devices on all structures, we can collect data from just a few of them,” Catbas says.

He explained that collecting useful data from sensors about damaged structures is expensive and challenging.

“There is not enough data from damaged areas to train detection models,” he says. “Yet, machine learning (ML) and deep learning (DL) algorithms used with AI yield better, more accurate output using big data sets. As a solution to the data scarcity in civil structural health monitoring applications, the invention takes data collected from structures. It uses model variants of the GAN architecture to generate large, accurate synthetic data samples to train damage diagnostics systems.

“Then, by using AI, we can better understand what’s going on with other similar structures and more effectively decide how to respond,” he says.

The technology can predict the dynamic response of a structure change before damage conditions occur. It’s also possible to create potential future conditions of structures, such as generating data showing what a healthy bridge’s response would be after damage compared to the response of an unhealthy bridge.

Catbas says that the inventions can be used independently or together.

Upcoming projects

Catbas says that his team’s future research plans include a framework for smart and resilient communities to withstand extreme events.

“It enhances community resilience by providing valuable insights for disaster preparedness, resource allocation and evacuation planning,” he says. “The framework improves emergency management by enabling informed decision-making during crises.”

They are also developing a “digital twin” of infrastructure assets, like the way NASA uses replicas of spacecraft components.

“They have those components on the ground, and if something happens, they work with these replicas,” he says. “So, this twin, in a sense, allows us to collect data simultaneously and work on different structure scenarios using predictive analysis.”