Write performance disparity between Rust / Python

[a10y]

Connected to the sample file in #1749, specifically the 500MB A0.small.50.vortex file.

Loading the file into memory and writing it back out via the VortexFileWriter is really snappy in Rust.

On Python, doing the same load runs for several minutes without completing:

import vortex as vx
arr = vx.io.read_path("A0.small.50.vortex")
vx.io.write_path(arr, "A0.small.50-out.vortex")    #... hangs for a really long time

[a10y]

Ah ok, apparently we compress by default in the write path in Python.

@danking I'm wondering if we should revisit this default setting? Or perhaps find a better way to decide if we should re-compress?

[danking]

Defaulting to compress=False seems to me unexpected behavior if you're coming from Parquet or ORC.

What is the tree display for this file? I suppose if it's incompressible data, then the sampling compressor will always receive a PrimitiveArray which it will attempt to compress.

This also seems like a problem with the compressor. Maybe we should add this file as a compressor benchmark?

[a10y]

The file is compressible, it's just that the user has already run vx.compress() on it before trying to write it.

Perhaps we can store something in python-land that acknowledges that an array has already been compressed so that when we pass it to vx.io.write_path it doesn't re-compress it needlessly