Compression speed too slow

[a10y]

Using the sample file provided in #1749, Vortex compression is nearly ~10x slower than the equivalent Parquet compression:

A couple of thoughts

• This schema is large (14,000 columns) and deeply nested (Struct(Struct(List) * 14,000))
• Tree search for best encodings is probably a large part of this
• The search could be trivially parallelized
• The compression step for structs/chunks could similarly be parallelized trivially
• There is probably a certain amount of wasteful work being done in the single-threaded case that we should fix. Need to dig into a full profile to get a solid breakdown

[robert3005]

I am curious if pyarrow parallelizes parquet compression and if it does it would useful to see without parallelization.

[a10y]

It appears to also be single-threaded