Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support preserving unknown fields in ProtoBuf format #2860

Open
wants to merge 6 commits into
base: dev
Choose a base branch
from

Conversation

xiaozhikang0916
Copy link
Contributor

Introducing new annotation @ProtoUnknownFields and built-in type ProtoMessage, to store any fields with proto ids that are undefined in current message class.

In my understanding, both ktx-serialization and protocol buffer are designed in strong typing, all messages used in code should be well defined in kt class or proto message. Therefore I would like to keep ProtoMessage and ProtoField minimun published. Not like JsonObject in json format, users cannot retrieve detailed data from it.

Close #2655

@xiaozhikang0916 xiaozhikang0916 changed the base branch from master to dev November 18, 2024 04:04
@xiaozhikang0916
Copy link
Contributor Author

@sandwwraith Hi, please take a look here?

@sandwwraith
Copy link
Member

@xiaozhikang0916 Sure, when I'll have time

@sandwwraith sandwwraith self-assigned this Nov 22, 2024
Copy link
Contributor

@pdvrieze pdvrieze left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The structure you use to represent protobuf content is somewhat complex, especially considering the protobuf wire format. You probably want to think a bit more how to represent protobuf data where the wire based information permits that. In particular you may want to distinguish primitives, "blobs", packed arrays. You may also allow users to observe a protobuf with provided metadata.

/**
* A data representation of a protobuf field in [ProtoField.data], without the field number and wire type.
*/
internal sealed interface ProtoContentHolder {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Given that protobuf is a recursive format, it would make sense for ProgoMessage and ProtoContentHolder to be the same. It may also make sense to have support for proto primitives (with fixed size) rather than storing that data as bytearray.

@xiaozhikang0916
Copy link
Contributor Author

@pdvrieze Actually I don't want to decode the protobuf wire data into a tree-structure type like what JsonObject do, neither provide retreiving method of the content.

The raw content in protobuf wire data is quite ambiguous that, you can even not possible to tell whether a VARINT 1 is 1 or -1 or true, or tell a LEN field is string or sub-message. The only way to parse wire data content correctly and explictly is to parse with message definition, or with a concrete type in kotlin.

Base on it, ProtoMessage here is not designed for retrieving content from undefined fields, but for holding it only.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Supporting protobuf preserving unknown fields
3 participants