[Local Storage] Prevent OOM errors by processing files by chunks (#671) #2067

Job	Run time
debian-build (focal)	5m 49s
build (3.7)	1m 37s
build (3.8)	1m 56s
build (3.9)	1m 41s
build (3.10)	1m 33s
build (3.11)	1m 53s
k8ssandra-CreateSingleMedusaJob	10m 54s
integration-tests (3.9, local, 2.2.19)	10m 51s
integration-tests (3.9, local, 3.11.11)	10m 50s
integration-tests (3.9, local, 4.0.0)	10m 47s
integration-tests (3.9, s3, 2.2.19)	10m 52s
integration-tests (3.9, s3, 4.0.0)	10m 51s
integration-tests (3.9, gcs, 2.2.19)	10m 53s
integration-tests (3.9, gcs, 4.0.0)	12m 28s
integration-tests (3.9, minio, 2.2.19)	10m 49s
integration-tests (3.9, minio, 4.0.0)	10m 51s
integration-tests (3.9, azure, 2.2.19)	10m 52s
integration-tests (3.9, azure, 4.0.0)	10m 50s
integration-tests (3.9, azure-hierarchical, 2.2.19)	10m 50s
integration-tests (3.9, azure-hierarchical, 4.0.0)	10m 49s
integration-tests (3.11, 4.0.0, gcs, false)	10m 50s
integration-tests (3.10, 4.0.0, gcs, false)	10m 49s
integration-tests (3.8, 4.0.0, gcs, false)	10m 50s
integration-tests (3.7, 4.0.0, gcs, false)	10m 52s
publish-docker-master	1s
publish-debian	1s
publish-docker	1s
publish-pypi	1s
	3h 31m 21s

Provide feedback