Rise of Open-Source Intelligence Tests U.S. Spies

The intelligence shop had just eight analysts and used only publicly available information, not spy satellites and secret agents. It belonged to multinational chemicals company

Dow Inc.,

not to any government.

“I’m leading an intelligence center that accurately predicted the invasion of Ukraine without any access to sensitive sources,” said

John Robert,

Dow’s director of global intelligence and protection, whose unit helps the company manage business risk and employee safety.

Supercharged by the Ukraine war, the rise of open-source intelligence, or OSINT, which comprises everything from commercial satellite imagery to social-media posts and purchasable databases, poses revolutionary challenges for the Central Intelligence Agency and its sister spy agencies, according to former senior officials who spent decades working in those agencies’ classified spaces.

Dow is just one of a fast-growing number of companies, nonprofit groups and countries transforming publicly available data into intelligence for strategic and economic advantage. China has the largest, most focused effort, while U.S. spy agencies, with deeply ingrained habits of operating in the shadows, have been slow to adapt to a world in which much of what is important isn’t secret, according to dozens of officials and many studies.

The CIA is simultaneously dealing with a closely related challenge: It is pivoting from two decades focused on terrorism toward spying on a new primary intelligence target, China. But some officials say the technological tsunami facing U.S. intelligence agencies poses a more fundamental challenge than merely swapping priorities.

“The agency is used to running this way and that, depending on what the demand of the day is,” said

Paul Kolbe,

a former CIA officer who directs the Intelligence Project at Harvard Kennedy School’s Belfer Center. But the growing dominance of open-source data represents a uniquely difficult test, particularly for the CIA, he said.

There will always be some tasks only secretive agencies with classified data and covert human sources can do, current and former officials say. The CIA obtained elements of Russian President

Vladimir Putin

‘s war plan for invading Ukraine, and warned U.S. allies in Kyiv. At the end of July, it tracked and killed al Qaeda leader

Ayman al-Zawahiri,

ending a 21-year hunt.

But by some estimates, more than 80% of what a U.S. president or military commander needs to know comes from OSINT, and not from foreign agents, spy satellites or expensive eavesdropping platforms.

That means, officials said, that the CIA and other agencies need to give priority to vetting and sifting through troves of OSINT that ranges from YouTube videos to publicly posted genetic databases—or else risk missing the next threat or looming global crisis. And they need to do so faster than U.S. rivals, principally China.

Threats to U.S. security are considerable and growing, according to interviews with many officials and numerous studies.

U.S. intelligence agencies could miss signs of the next global pandemic that are hiding in plain sight. China could leap ahead on technologies such as quantum computing and artificial intelligence while using public data to identify U.S. intelligence officers.

And the $90 billion U.S. intelligence community could see its role diminished, as private companies generate more intelligence insights that a U.S. president and his top advisers would want to know.

The nonprofit C4ADS used shipping databases, satellite imagery, property records and other public data to trace the source of Russian GPS “spoofing” that disguised Russian ships’ locations to a defense facility near one of Mr. Putin’s dachas, according to a November presentation at the Harvard Intelligence Project. A U.K. firm, 3AI, used artificial intelligence and public information to estimate the cost to Russian companies of the Ukraine war and subsequent economic sanctions at $372 billion—50% larger than indicated by stock markets and equity researchers. Bellingcat, the investigative website, used phone and travel data to identify three operatives from Moscow’s FSB intelligence service it said attempted to kill Russian opposition politician

Alexei Navalny.

“I don’t worry about the [intelligence community] going away. I worry about it mattering,” said

Robert Cardillo,

who served in several senior U.S. intelligence roles. Government policy makers, he said, could rely less on traditional intelligence briefings and more on open-source products, which are generally cheaper and easier to access. “I worry about customers voting with their mouse click.”

A congressionally mandated report on OSINT, reviewed by The Wall Street Journal, found that the U.S. government is already behind China and other competitors.

Adversaries’ use of publicly available data “outpaces ours to fully harness the power of OSINT for National Security needs,” said the report, completed in February by MITRE, a federally funded, nonprofit research organization. It hasn’t been made public.

China puts a premium on OSINT and has an estimated 100,000 analysts tasked with scouring scientific and technical developments globally, mostly in the U.S., according to research by

William Hannas

of Georgetown University. The system for gathering such intelligence is centrally directed but “functions at all levels in separate but interlocking organizations,” Mr. Hannas and

Huey-Meei Chang

wrote in a 2021 paper.

“We don’t have a comparable effort,” said

Jason Matheny,

former head of the U.S. Intelligence Advanced Research Projects Activity and the president of Rand Corp., also a federally funded research organization. “It really is an immense enterprise in China.”

Beijing’s authoritarian rulers don’t face legal and ethical quandaries that U.S. and European spy agencies confront when sifting through public information that might contain individuals’ private data, according to the officials and studies.

Senior U.S. intelligence officials say they recognize the challenges and are making significant changes. CIA Director

William Burns

gave higher priority to accelerated outreach to the private sector and academia, after an internal review he had ordered found the agency wasn’t capitalizing on such partnerships. Last year, Mr. Burns established new agency units focused on China and on technology and transnational threats. He frequently notes that nearly a third of the CIA’s employees work in technology, science and related fields.

Analysts generally praise Mr. Burns and Director of National Intelligence

Avril Haines

for attempting to reorient the huge U.S. intelligence apparatus. But OSINT initiatives remain disjointed, underfunded and underprioritized, according to the officials and studies.

“It’s not as if we’re not using it,” said retired Army Lt. Gen.

Robert Ashley,

former director of the Pentagon’s Defense Intelligence Agency. “We can’t do this at scale because it’s not funded.”

At the CIA, efforts to give OSINT a more central role have repeatedly been stymied by a culture that has necessarily revolved around highly classified information and secret operations, according to former agency officials and others.

Culture change “is happening, but it’s happening slowly” in the U.S. intelligence community, said Rep.

Adam Schiff

(D., Calif.), chairman of the House Intelligence Committee. Meanwhile, “the world is moving at lightning speed.”

U.S. intelligence officials “have a significant bias against the use of open-source intelligence,” said the MITRE report, while pointedly noting that the limited changes so far aren’t enough.

In a recent LinkedIn post,

Jennifer Ewbank,

the CIA’s deputy director for digital innovation, said that the agency’s Open Source Enterprise, which she oversees, “has undergone a dramatic transformation in recent years” and that publicly available information will be the agency’s initial go-to intelligence source in the near future.

In interviews, senior intelligence officials said the CIA is integrating OSINT more into its analysis and operations on China and other topics. Used correctly, they said, it will enable the agency to focus spies, satellites and other intelligence-gathering tools on what is truly secret, such as the hidden intentions of foreign adversaries.

One official cautioned that U.S. intelligence can’t simply take OSINT products from outsiders and feed them into analysis destined for the White House or other government agencies. It must first vet the sourcing and methodology.

Yet basic questions, including which agency should take the lead on OSINT—or whether a new agency should be created—are unsettled.

Ms. Haines has commissioned a series of studies on how U.S. intelligence should handle open-source data, including where the effort should be centered, officials said. No final decisions have been announced.

The Office of the Director of National Intelligence “is working to position the intelligence community to most effectively leverage open-source intelligence, a valuable and increasingly critical component of our national-security mission,” said spokeswoman

Nicole de Haay.

—Byron Tau contributed to this article.

Write to Warren P. Strobel at [email protected]