Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Remove search calls to fetch max seq no for each shard in doc level monitor. Use first call to fetch data from a shard to determinte max seq no. #1409

Closed
eirsep opened this issue Feb 6, 2024 · 0 comments
Assignees
Labels
performance Make it fast!

Comments

@eirsep
Copy link
Member

eirsep commented Feb 6, 2024

Current behaviour

  • If there are n indices with m shards each, we first run n*m search calls to simply fetch the max sequence number
    for each shard and then perform the searches which fetch docs from each shard based on last seen seq no. and max seq no.

Optimized behaviour proposed

Change first search request from each shard to fetch all docs in descending order of seq no with filter on `seqno >
That way we are fetching docs and also able to note the sequence number correctly within the same call and avoid the first batch of search calls which simply help determine the sequence number

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
performance Make it fast!
Projects
None yet
Development

No branches or pull requests

1 participant