Why not serde

[Tudyx]

I'm just curious why you have defined your own trait (ArrowDeserialize/ArrowSerialize.) instead of trying to add a new serde "Data formats". Please could you explain me if you have the time.

[ncpenke]

Thanks for the question @Tudyx. I'm a huge fan of serde and we had a longer section dedicated to this in the README. I'll look to add that back.

The two main goals of this crate are:

1. Map rust types to arrow2 schema at compile-time.
2. Provide a straight forward mechanism to perform this conversion.

For 1, if we use serde, we either need to infer the schema in a separate pass, which this crate does. Or we need to infer it while performing the operation. For deserialization, we could do this (and inferring the schema is already something arrow2 already does for formats like JSON btw). For serialization, we would need an alternate mechanism to specify the desired schema.

For this crate, we wanted compile-time generation of the arrow2 schema to not require the user to come up with another way to specify the schema. This is where the ArrowField trait, which is the third trait that works in conjunction with ArrowDeserialize and ArrowSerialize comes in. Generating implementations for the ArrowField trait requires a proc-macro, so we're already breaking some of the abstraction serde provides.

If we have compile-time mapping, the remaining problem is mapping between rust types and arrow2 data structures that correspond to our schema. We could use serde for this, but the serde data model is an additional layer of abstraction we don't need. The conversions that need to be performed is more similar to From or TryFrom than serde::Deserialize or serde::Serialize i.e. we're not converting to/from an opaque wire-format, we're just transforming from in-memory data structures to another in-memory data structures. The easiest way to do this was via the ArrowSerialize and ArrowDeserialize traits and it would also allow users to directly specify the arrow type conversions.

Honestly, if the compile-time schema was not needed, then it would make a lot more sense to try to use serde to avoid all the proc-macro work. But it's my belief that the compile-time schema drastically improves the usability of this crate, which is why we took this path.

Hope that makes sense!

[Tudyx]

That make a lot of sense! Thank you for your very complete answer