The Costly Pursuit of Self-Driving Cars Continues On. And On. And On.

It was seven years ago when Waymo discovered that spring blossoms made its self-driving cars get twitchy on the brakes. So did soap bubbles. And road flares.

New tests, in years of tests, revealed more and more distractions for the driverless cars. Their road skills improved, but matching the competence of human drivers was elusive. The cluttered roads of America, it turned out, were a daunting place for a robot.

The wizards of Silicon Valley said people would be commuting to work in self-driving cars by now. Instead, there have been court fights, injuries and deaths, and tens of billions of dollars spent on a frustratingly fickle technology that some researchers say is still years from becoming the industry’s next big thing.

Now the pursuit of autonomous cars is undergoing a reset. Companies like Uber and Lyft, worried about blowing through their cash in pursuit of autonomous technology, have tapped out. Only the most deep pocketed outfits like Waymo, which is a subsidiary of Google’s parent company Alphabet, auto industry giants, and a handful of start-ups are managing to stay in the game.

The tech and auto giants could still toil for years on their driverless car projects. Each will spend an additional $6 billion to $10 billion before the technology becomes commonplace — sometime around the end of the decade, according to estimates from Pitchbook, a research firm that tracks financial activity. But even that prediction might be overly optimistic.

“This is a transformation that is going to happen over 30 years and possibly longer,” said Chris Urmson, an early engineer on the Google self-driving car project before it became the Alphabet business unit called Waymo. He is now chief executive of Aurora, the company that acquired Uber’s autonomous vehicle unit.

So what went wrong? Some researchers would say nothing — that’s how science works. You can’t entirely predict what will happen in an experiment. The self-driving car project just happened to be one of the most hyped technology experiments of this century, occurring on streets all over the country and run by some of its most high-profile companies.

That hype drew billions of dollars of investments, but it set up unrealistic expectations. In 2015, the electric carmaker Tesla’s billionaire boss, Elon Musk, said that fully functional self-driving cars were just two years away. More than five years later, Tesla cars offered simpler autonomy designed solely for highway driving. Even that has been tinged with controversy after several fatal crashes (which the company blamed on misuse of the technology).

Perhaps no company experienced the turbulence of driverless car development more fitfully than Uber. After poaching 40 robotics experts from Carnegie Mellon University and acquiring a self-driving truck start-up for $680 million in stock, the ride-hailing company settled a lawsuit from Waymo, which was followed by a guilty plea from a former executive accused of stealing intellectual property. A pedestrian in Arizona was also killed in a crash with one of its driverless cars. In the end, Uber essentially paid Aurora to acquire its self-driving unit.

But for the most deep-pocketed companies, the science, they hope, continues to advance one improved ride at a time. In October, Waymo reached a notable milestone: It launched the world’s first “fully autonomous” taxi service. In the suburbs of Phoenix, Ariz., anyone can now ride in a minivan with no driver behind the wheel. But that does not mean the company will immediately deploy its technology in other parts of the country.

Dmitri Dolgov, who recently took over as Waymo’s co-chief executive after the departure of John Krafcik, an automobile industry veteran, said the company considers its Arizona service a test case. Based on what it has learned in Arizona, he said, Waymo is building a new version of its self-driving technology that it will eventually deploy in other geographies and other kinds of vehicles, including long-haul trucks.

The suburbs of Phoenix are particularly well suited to driverless cars. Streets are wide, pedestrians are few and there is almost no rain or snow. Waymo supports its autonomous vehicles with remote technicians and roadside assistance crews who can help get cars out of a tight spot, either via the internet or in person.

“Autonomous vehicles can be deployed today, in certain situations,” said Elliot Katz, a former lawyer who counseled many of the big autonomous vehicle companies before launching a start-up, Phantom Auto, that provides software for remotely assisting and operating self-driving vehicles when they get stuck in difficult positions. “But you still need a human in the loop.”

Self-driving tech is not yet nimble enough to reliably handle the variety of situations human drivers encounter each day. They can usually handle suburban Phoenix, but they can’t duplicate the human chutzpah needed for merging into the Lincoln Tunnel in New York or dashing for an offramp on Highway 101 in Los Angeles.

“You have to peel back every layer before you can see the next layer” of challenges for the technology, said Nathaniel Fairfield, a Waymo software engineer who has worked on the project since 2009, in describing some of the distractions faced by the cars. “Your car has to be pretty good at driving before you can really get it into the situations where it handles the next most challenging thing.”

Like Waymo, Aurora is now developing autonomous trucks as well as passenger vehicles. No company has deployed trucks without safety drivers behind the wheel, but Mr. Urmson and others argue that autonomous trucks will make it to market faster than anything designed to transport regular consumers.

Long-haul trucking does not involve passengers who might not be forgiving of twitchy brakes. The routes are also simpler. Once you master one stretch of highway, Mr. Urmson said, it is easier to master another. But even driving down a long, relatively straight highway is extraordinarily difficult. Delivering dinner orders across a small neighborhood is an even greater challenge.

“This is one of the biggest technical challenges of our generation,” said Dave Ferguson, another early engineer on the Google team who is now president of Nuro, a company focused on delivering groceries, pizzas and other goods.

Mr. Ferguson said that many thought self-driving technology would improve like an internet service or a smartphone app. But robotics is a lot more challenging. It was wrong to claim anything else.

“If you look at almost every industry that is trying to solve really really difficult technical challenges, the folks that tend to be involved are a little bit crazy and little bit optimistic,” he said. “You need to have that optimism to get up everyday and bang your head against the wall to try to solve a problem that has never been solved, and it’s not guaranteed that it ever will be solved.”

Uber and Lyft aren’t entirely giving up on driverless cars. Even though it may not help the bottom line for a long time, they still want to deploy autonomous vehicles by partnering with the companies that are still working on the technology. Lyft now says autonomous rides could arrive by 2023.

“These cars will be able to operate on a limited set of streets under a limited set of weather conditions at certain speeds,” said Jody Kelman, the executive of Lyft. “We will very safely be able to deploy these cars, but they won’t be able to go that many places.”