Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Necessary fixes for _bulk endpoint #1202

Merged
merged 9 commits into from
Jan 20, 2025
Merged

Necessary fixes for _bulk endpoint #1202

merged 9 commits into from
Jan 20, 2025

Conversation

mieciu
Copy link
Member

@mieciu mieciu commented Jan 20, 2025

For our V2 ingest processor, we need better implementation of _bulk API, which can handle more situations. Some shortcuts we took earlier, are no longer a viable option.

Changes:

  • We used to take bulk payload lines and just process them in pairs (action/metadata line followed by document) which doesn't have to be the case. There can always be a delete call which isn't followed by any document. So the whole BulkForEach had to be rewritten.
  • Extending NDJSON type validation, so that single-line JSON is also a valid NDJSON 😉
  • Adding some tests

When loading sample datasets and having operational Kibana, every single request ends with code 200 🎊

@mieciu mieciu marked this pull request as ready for review January 20, 2025 15:04
@mieciu mieciu requested a review from a team as a code owner January 20, 2025 15:04
@mieciu
Copy link
Member Author

mieciu commented Jan 20, 2025

IT run https://github.com/QuesmaOrg/quesma/actions/runs/12871349933

@mieciu
Copy link
Member Author

mieciu commented Jan 20, 2025

Had to retry previous one and it succeeded, looks like a flakiness, weird...
https://github.com/QuesmaOrg/quesma/actions/runs/12871859960

@mieciu mieciu added this pull request to the merge queue Jan 20, 2025
Merged via the queue into main with commit 7f158aa Jan 20, 2025
5 checks passed
@mieciu mieciu deleted the fix-bulk-api branch January 20, 2025 16:17
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants