How One Guy’s AI Tracked the Chinese Spy Balloon Across the US

Earlier this month, entrepreneur Corey Jaskolski pulled out a pen and drew his best guess at what the surveillance balloon shot down by a US jet would have looked like from space. Then he fed the sketch and “a gob” of recent satellite images from the area where the balloon was taken down into algorithms developed by his image and video detection startup Synthetatic, and waited.

Within two minutes, he says, the algorithms found the 200-foot-tall balloon off the coast of South Carolina. “I couldn’t believe it,” Jaskolski says. Nor could his wife when he excitedly showed her his results. But when he estimated the altitude of the balloon in the image it was around 57,000 feet—matching the height at which the balloon was spotted by a US spy plane—and social media sightings from 20 minutes before the image was taken appeared to confirm he had found it.

Jaskolski dug in, poring over wind models and social media sightings to feed his software, called RAIC (rapid automatic image categorization), new swathes of satellite data from the company Planet Labs. The tool is designed to make it possible to search large image collections for objects of interest using a single example image.

“We drew a big arc across time and space and started searching that,” Jaskolski says. Having found the balloon once, Synthetiatic’s software could be trained with a real image of the balloon to further guide its search.

Over the next several days, Jaskolski put RAIC to work. The company has since compiled six sightings of the balloon (five confirmed, one still being investigated) on its satellite imagery and has used wind data to estimate how it moved between those points. “We can draw a 1-kilometer-wide track across the whole of the United States and just follow the balloon,” he says. “We have a track from where it entered from Canada, all the way to South Carolina, where it got popped, with six points along that arc.”

Jaskolski’s stratospheric scavenger hunt may have been made possible by smart software, but it also required human expert knowledge. His initial drawing of the craft looked more like a technicolor snowman—stacked red, green, and blue circles. The aim was to mimic the way satellites generally capture different wavelengths of light using separate sensors that aren’t always synced in time, creating multiple disjointed views of objects. And it throws up false positives.

But the ability to map a surveillance balloon’s path with such clarity could be a game changer for national security, says Arthur Holland Michel, senior fellow at the Carnegie Council and author of a book on drones and surveillance. “The combination of AI with satellite imagery is undoubtedly a very powerful technology for surveillance and espionage and counterespionage,” he says. 

Holland Michel also points out that satellite imagery and AI have their limitations. The method by which Synthetatic first found the balloon—using a drawing—could result in false positives if the object of interest was something more complex or less publicly documented, such as a tank. “Things often look a bit weird and unfamiliar from above,” he says.

“There’s undoubted potential there,” Holland Michel says, “but it’s easy to think this combination of satellites and AI is an all-seeing capability that will lay everything bare.” It’s useful in certain cases, like the balloon, he says, but likely not all scenarios.

That’s something Jaskolski acknowledges—but he also considers the project an example of how human expertise and grunt work can be elevated by AI. “This human-machine collaboration is my idea of how AI works today,” he says. “And it’s definitely how we build our product.” The tool is currently used for humanitarian purposes, including by the UN World Food Program to find flood victims.

The pursuit of the balloon isn’t over just because Jaskolski has managed to track it across the United States. He says the process is “resource-intensive” because the software isn’t perfect and turns up many potential sightings that have to be whittled down by people. “But we’d like to still continue to track it,” he says. “Whether we go all the way back to China or not, we feel like we solved a technical problem at least. We’d be crazy not to try.”