Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Local Storage Provider runs out of memory when it encounters big files #672

Closed
rzvoncek opened this issue Oct 25, 2023 · 0 comments · Fixed by #671
Closed

Local Storage Provider runs out of memory when it encounters big files #672

rzvoncek opened this issue Oct 25, 2023 · 0 comments · Fixed by #671
Assignees
@rzvoncek
Labels
bug Something isn't working done Issues in the state 'done'

Comments

@rzvoncek
Copy link
Contributor

rzvoncek commented Oct 25, 2023 •
edited by adejanovski
Loading

Project board link

It turns out we had a very naiive implementation of the local storage provider. It was doing f.write(other_f.read()).
This is no good because f.read() reads everything into memory.

A fix needs to make all interaction with files properly chunked.
@rzvoncek rzvoncek added the bug Something isn't working label Oct 25, 2023
@rzvoncek rzvoncek self-assigned this Oct 25, 2023
@rzvoncek rzvoncek moved this to Ready in K8ssandra Oct 25, 2023
@adejanovski adejanovski added the ready Issues in the state 'ready' label Oct 25, 2023
@rzvoncek rzvoncek mentioned this issue Oct 25, 2023
Merged
@rzvoncek rzvoncek moved this from Ready to Ready For Review in K8ssandra Oct 25, 2023
@adejanovski adejanovski added ready-for-review Issues in the state 'ready-for-review' and removed ready Issues in the state 'ready' labels Oct 25, 2023
@github-project-automation github-project-automation bot moved this from Ready For Review to Done in K8ssandra Oct 25, 2023
@adejanovski adejanovski added done Issues in the state 'done' and removed ready-for-review Issues in the state 'ready-for-review' labels Oct 25, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@rzvoncek rzvoncek

Labels
bug Something isn't working done Issues in the state 'done'
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

[Local Storage] Prevent OOM errors by processing files by chunks thelastpickle/cassandra-medusa
2 participants
@rzvoncek @adejanovski