[PROPOSAL] Release OpenSearch 3.0

[andrross]

In the nearly 3 years since we released OpenSearch 2.0 we have delivered significant features and innovations while adhering to semver. Now that Lucene 10 has been released, we should take this opportunity to release OpenSearch 3.0. We will update Lucene, make key changes to enable future modularity, remove legacy cruft, and establish a modern JDK baseline to unlock the latest language improvements and allow for compatibility with future JDKs.

Meta Issues

• OpenSearch: [META] Breaking changes in 3.0 OpenSearch#3351
• Dashboards: @ashwin-pc / @AMoo-Miki to create a tracking issue in OpenSearch-Dashboards
• Clients: [PROPOSAL] Support OpenSearch 3.0 opensearch-clients#78

Timeline

This proposal is to tentatively target 2.19 (to be released by February 19, 2025) as the last minor version of the 2.x line. That would result in a 3.0 release around April 2025. A more specific timeline, including the rest of 2025, can be found here: #252

FAQ

How Can I Help?

Please provide any feedback you have! For project-wide concerns like timeline and maintenance policies please comment directly on this issue. If you have suggestions about changes to a specific component, feel free to comment on an issue (or open a new one) in the corresponding repository.

What is the impact to users?

For end users that interface with the system through dashboards, the REST API, or any language client, the impact will be minimal provided no deprecated features are being used. Our adherence to semver means that anything removed from the API must have been deprecated in the prior major version before removal. Users are able to inspect response headers and deprecation logs for warnings about usage of deprecated features prior to upgrading.

For operators and system admins, upgrading may require updates to the deployment environment (particularly if not using the bundled JDK). Any operator that is hosting custom plugins will need to make code changes to the custom plugin to make it compatible.

Are in-place updates possible?

Yes. In-place updates are defined as either a rolling update (deactivate a host, update the software, and restart it, repeating for all nodes in a cluster) or a blue-green update (add new nodes to the cluster running the new software version, then remove the old nodes). OpenSearch 3.0 will be wire-compatible with the last minor version of the 2.x line and therefore capable of an in-place upgrade from that version to any 3.x version.

Any index data that was written with OpenSearch 1.x (or ES 7.x) must be re-indexed into OpenSearch 2.x prior to upgrading, as OpenSearch and Lucene only support reading data written by the previous major version. See the compatibility documentation for more details.

What about OpenSearch 1.3?

It will become end-of-life in open-source and the OpenSearch Project will no longer release updates.

What about OpenSearch 2.x?

The OpenSearch Project will continue to maintain the last 2.x release with regularly scheduled patch releases containing security fixes or other critical updates. This support will last until a 4.0 release happens, per the current maintenance policy.

What about clients?

Clients provide a super-set of APIs in order to make them compatible with multiple major versions. Existing clients should work with a 3.0 version, provided no previously-deprecated APIs are called by the end user. See the OpenSearch meta issue detailing breaking changes, including removal of previously-deprecated APIs.

[dblock]

I opened opensearch-project/opensearch-clients#78 for client support.

[peterzhuamazon]

Thanks @andrross ,

Based on this issue we have also proposed the new release schedule of 2025 here:

• [PROPOSAL] OpenSearch Release Schedule for Year 2025 #252

Please take a look and provide feedback. Thanks!

[abunday]

My question is if 1.3 is going to be deprecated, what will be the pathway for people using Elasticsearch currently who wish to migrate to OpenSearch? Currently, users are required to migrate from Elasticsearch to OpenSearch 1.3 before they can upgrade to 2.x so OpenSearch 1.3 is on the critical path from an Elasticsearch to OpenSearch migration.

[peternied]

The pathway onto the latest version of OpenSearch is via OpenSearch Migrations [1] [2]. See the project for details on the full version and feature matrix that are supported. If you notice an important absence please scan the projects open issues or open a new issue [3].

• [1] Documentation https://opensearch.org/docs/latest/migration-assistant/
• [2] GitHub Repository https://github.com/opensearch-project/opensearch-migrations
• [3] Migration Issues https://github.com/opensearch-project/opensearch-migrations/issues

[rursprung]

IMHO opensearch-project/OpenSearch#13187 should really be resolved before a 3.x release should happen: we should use 3.x to disentangle the plugin releases from the core release. not requiring the lock-step upgrades anymore would be a huge help. while everyone would still have to do a plugin upgrade/release for 3.0.0 this chance could/should be used to declare the plugins as being semver compatible with 3.x, thus not explicitly requiring a new plugin release anymore unless a new feature is being added or a dependency in the plugin needs to be updated.

the technical basis for this is already integrated, but since every plugin i know uses the gradle plugin for the integration i'm not aware of a single plugin using the new semver feature. thus i guess it's also completely untested (in real-world scenarios; of course there's technical test coverage).

with this, the plugin version numbers can then also be disentangled from the OpenSearch release numbers (i.e. all the plugins could also release a 3.0.0 version - since realistically they're all on 2.x right now - but then don't need to release 3.1.0 when OS 3.1.0 comes out; also, the plugins can go to a standard semver based 3-part number rather than the current 4-part number which isn't semver at all)

[dblock]

@rursprung are you proposing not to ship a distribution with plugins at all (a breaking change)?

[rursprung]

@rursprung are you proposing not to ship a distribution with plugins at all (a breaking change)?

i wouldn't go that far, though it is certainly an option. what i'm suggesting is that (1) the technical basis for separate releases is in place (AFAIK the only thing missing is the support in this gradle plugin), (2) plugins start using semver compatibility ranges (at least all plugins under this org + it should be a very strong recommendation for any 3rd party plugin, which are the ones profiting the most of this) and (3) plugins start using their own semver versioning scheme.
with this you can then still build a distribution with all plugins, it might just not contain OpenSearch 3.1.0 with plugin A 3.1.0.0, plugin B 3.1.0.0, etc. but instead it might be OpenSearch 3.1.0 with plugin A 3.0.0 (semver compatible with OS >= 3.0) and plugin B 3.7.4 (the plugin had some non-breaking releases and is declares compatibility with OS 3.1.0). and there might be a 3rd party plugin C which bumped to 4.0.0 because there was a breaking change there (e.g. in the config) which is compatible with OS 3.1.0. the only thing which shouldn't happen is major releases of bundled plugins since then that'd imply a breaking change for the users of the distribution.

right now, all plugins need to release in lock-step with the core. once they're just using semver ranges you're not forced to release everything all the time. so that should make it much easier to create a new release of OS since theoretically you can just release a new core and take the previous plugin versions and bundle them up (unless there was a breaking change in the core).

the only advantages of a bundle distribution i can see is that (a) users don't need to install common plugins (like security; though wasn't that supposed to move into core as part of 3.0?) and (b) it's a guarantee that these plugins are working well together since they've (hopefully) been tested together.
for the first of these points: i don't know which plugins see how much usage, but if security (the only truly important plugin which everyone should be using) were in core then i think that most/all other plugins should be a simple ./bin/opensearch-plugin install away.
for the second: if semver is used correctly this shouldn't be an issue and you can still have a test campaign using the latest compatible release of each plugin to be covered (regardless of whether you end up publishing them as a bundle or just posting the test campaign results somewhere).

full disclosure: i'm anyway using the min distribution and manually adding the plugins i need - this reduces the size drastically and also makes vulnerability scanners happier if there are fewer dependencies in the whole package.

[andrross]

@rursprung Is the breaking change here that plugins start using a 3-part number that is compatible with semver? Let me know if I'm wrong, but if we do that for the 3.0 release then I think we've got all the pieces in place for plugins to use their own semver versioning scheme even if in the initial releases we bump them all in lock-step like we do today.

[rursprung]

@rursprung Is the breaking change here that plugins start using a 3-part number that is compatible with semver? Let me know if I'm wrong, but if we do that for the 3.0 release then I think we've got all the pieces in place for plugins to use their own semver versioning scheme even if in the initial releases we bump them all in lock-step like we do today.

yes, the breaking part would be the version number (unless you also stop publishing the bundled distribution). but i fear that if we don't do the semver compatibility right from the get-go it'll just be ignored (like it's being ignored now) and won't ever be implemented/used 🙁