-
Notifications
You must be signed in to change notification settings - Fork 136
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[BUG] k-NN score discrepancy after update to 2.17 #2296
Comments
assigning it to @0ctopus13prime |
@0ctopus13prime has already raised a PR for this. |
5 tasks
Fix PR : #2389 |
github-project-automation
bot
moved this from 2.19.0
to ✅ Done
in Vector Search RoadMap
Jan 17, 2025
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Describe the bug
After updating from version 2.11 to 2.17 on AWS OpenSearch (I know it's a large step, but bear with me), we noticed a significant score variation for newly indexed documents. For documents already in the index, the kNN score remains consistent across OpenSearch versions. However, if you re-index the same document, the score drops dramatically (e.g., from 0.72 to 0.12).
Upon further investigation using local OpenSearch versions, we identified that the issue arises when updating from 2.15 to 2.17.
The root cause appears to be related to the
index.knn.space_type
setting, which is documented here as deprecated. When we use the new approach of settingspace_type
directly in the field, the issue is resolved. However, this solution requires re-indexing all our documents, which is a significant drawback.I couldn't find any mention of the deprecation or removal of support for this setting in the release notes. Is this the expected behavior?
Related component
Indexing
To Reproduce
index.knn.space_type
setting, don't use thespace_type
option in the field and index some documentsExtra:
space_type
option in the field instead ofindex.knn.space_type
and index the same documentsExpected behavior
It was expected that the minor version update would not change the behavior of
index.knn.space_type
Additional Details
Plugins
All plugins enabled by default from AWS OpenSearch
Host/Environment:
The text was updated successfully, but these errors were encountered: