One year of OpenSearch: Grading AWS’ open source effort

If breaking up is hard to do, as Neil Sedaka sang years ago, forking an open source project is even harder. Amazon Web Services forked ElasticSearch in 2021, citing the need to “ensure open source versions of [ElasticSearch and Kibana]”. Which is a fine sentiment, but hard to pull off. Even though the right to fork is arguably the most essential right afforded by an open source license, the success of a fork is hardly guaranteed.

And so, roughly a year after AWS forked ElasticSearch, how is it doing? To find out, I talked with Carl Meadows, director of product management for OpenSearch. Since launching, OpenSearch has seen two major releases, a burgeoning community and erstwhile AWS enemy Oracle offering OpenSearch as a managed service. In other words, while there’s much more progress to be made, things are moving in the right direction, and along the way, AWS is getting a deep tutorial in how to engage with open source.

Open source: A question of community

In open source, we like to talk about “the community,” as if it’s some looming presence, but the reality is that most open source projects have no community at all. Not in the sense of active contributors to a project, anyway. ElasticSearch is the same. The company, Elastic, has for years done most of the development, which allowed it to make unilateral licensing changes.

SEE: Hiring kit: Back-end Developer (TechRepublic Premium)

In striking out on its OpenSearch fork, AWS declared its intent to create a “community-driven” alternative: The word “community” was used 18 times in the blog post announcing the fork. I worked at AWS during the formation of OpenSearch and can attest both to the good intentions of the people involved to build this community-driven fork but also to the difficulty in achieving this. For any company, it can be easier to say “community” than to actually hand over control to a community. After all, it is the company, not some nebulous community, that is ultimately responsible to customers and partners.

Even so, despite some fits and starts, one year in there are definite signs of community health in OpenSearch, as Meadows has written. Over 300 external contributions, at least one external maintainer, 18 associated community projects, 33 OpenSearch projects, companies publishing ElasticSearch-to-OpenSearch migration guides and tens of millions of downloads. Not too shabby.

It’s easy, however, to point to vanity metrics, and much harder to establish enterprise credibility thereby. In this area, OpenSearch has some heavyweight members of its community that suggest big bets being placed on the project. Take, for example, Adobe’s decision to switch from ElasticSearch to OpenSearch for Adobe Commerce. Or Oracle’s decision to offer OpenSearch as a managed service, which follows other such offerings from Instaclustr, Aiven and more.

All of which is critical, it turns out, to the AWS game plan.

Community creates the flywheel

Spend enough time talking to an Amazonian and you’ll invariably hear them mention “the flywheel.” Another way of saying this is “virtuous cycle.” In terms of Amazon’s retail site, the flywheel starts with lower prices, which drives more buyers, which attracts more third-party sellers on its site, which helps Amazon more efficiently use its fulfillment centers, which drives lower prices, which… You get the picture.

When I asked Meadows how AWS measures OpenSearch success, his response was immediate and completely flywheel-oriented: “Really, broad adoption is the most important thing.” And the reason? You don’t get customers paying for someone to relieve them of managing the “undifferentiated heavy lifting” — a favorite phrase among Amazonians — of an open source project if no one is using that project.

SEE: Metaverse cheat sheet: Everything you need to know (free PDF) (TechRepublic)

“The Amazon ElasticSearch Service, as it was called, was broadly successful due to the ubiquity of the open-source ElasticSearch project,” Meadows said. “When we set out to do OpenSearch, we didn’t have to open source it or try to make it community-led. But we really felt that having a real community-led project in the long-term, where it’s going to build the broadest adoption and the most sustainable ecosystem, is the flywheel that actually powers our main service.”

Even so, this is a significant change for AWS. The company has long built managed services for open source projects, but this is perhaps the first time that the company has also undertaken the undifferentiated heavy lifting of managing an open source project in this way. AWS is used to competing with the creators of open source projects, but inviting others to compete with itself? That feels new.

According to Meadows, however new it may be, the company feels it’s essential, and also that it can hold its own.

“We feel confident that on the AWS side, we’ve built a great managed service, so we’re going to be fine,” he said.

Oracle, Aiven, Instaclustr and others can build rival services: That just makes it safer for customers to swim the OpenSearch waters, as it were.

“If there are more people using this, it’s going to create a flywheel back into the product to make it better,” Meadows said.

This might be Open Source 101 at an experienced open source company like Red Hat, but at AWS it feels really new — and a heck of a lot of work, which I pointed out to Meadows. He agreed.

“It is a lot of work,” he said, “But to win this type of bet, [customers and partners] have to have a good amount of trust that we’re going to stay the course and not get bored and go do something else.”

He’s right, but this is a very different AWS than the one that came out swinging at Elastic in 2019 with lots of talk about what others should be doing in open source, without actually doing the same. Now AWS, through OpenSearch, is practicing what it somewhat sanctimoniously used to preach, and it seems to be working. This is good for AWS customers and partners well beyond those reliant on OpenSearch, because it suggests a new era for open source at AWS might be dawning.

Disclosure: I work for MongoDB, and used to work for AWS, but the views expressed herein are mine.