Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

vector search: hide vector index for sqlite planner #1678

Merged
merged 4 commits into from
Aug 23, 2024

Conversation

sivukhin
Copy link
Contributor

Context

PR #1594 tried to solve issue with sqlite planner using vector index for some optimizations (for example in SELECT COUNT(*) FROM t queries).

Turned out hacks to avoid usage of vector indices in planner were not enough. Actually, bUnordered field set by sqlite3 during index analysis process and manipulating with it can be dangerous and error-prone (as sqlite3 can reset it at any moment).

So, this PR fixes this more robustly:

  1. opMask is set to 0 for vector index in whereLoopAddBtreeIndex which must additionally prevent sqlite to use index in any operations (>, >=, =, IN) (actually, it shouldn't happen because index has expression - but still - this is another guard)
  2. whereIsCoveringIndex return false for vector index
  3. Loop over indices in query planner skips vector indices completely
  4. All places where optimizer prevented to use bUnordered indices were extended to have additional check for idxIsVector field

@@ -307,6 +321,8 @@ do_execsql_test vector-f64-index {
SELECT * FROM vector_top_k('t_f64_idx', vector64('[1,2]'), 2);
} {1 2}

reset_db
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We currently have some issue and test without db reset starting to generate generic sqlite error in some unexpected places.

This is really weird but I don't understand why it happens and how we can fix it.
Last time I investigated this issue I noticed that SELECT over libsql_vector_meta_shadow table with condition over name field sometimes returns nothing but if you will force full scan of the table - then everything will be fine.

I suspect that issue arise when split of libsql_vector_meta_shadow B-Tree happens. But still not sure why we have this issue...

CC @penberg

@sivukhin sivukhin changed the title Hide vector index for sqlite planner vector search: hide vector index for sqlite planner Aug 19, 2024
@sivukhin sivukhin marked this pull request as ready for review August 19, 2024 16:27
@sivukhin sivukhin requested review from penberg and haaawk August 19, 2024 16:27
@sivukhin sivukhin added this pull request to the merge queue Aug 23, 2024
Merged via the queue into main with commit 2f15a0e Aug 23, 2024
19 checks passed
@sivukhin sivukhin deleted the hide-vector-index-for-sqlite-planner branch August 23, 2024 09:05
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants