Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

New types and traits for (co-existing!) eager serialization #8642

Open
wants to merge 4 commits into
base: main
Choose a base branch
from

Conversation

teh-cmc
Copy link
Member

@teh-cmc teh-cmc commented Jan 10, 2025

This introduces SerializedComponentBatch, which will become the main type we use to carry user data around internally.

/// The serialized contents of a [`ComponentBatch`] with associated [`ComponentDescriptor`].
///
/// This is what gets logged into Rerun:
/// * See [`ComponentBatch`] to easily serialize component data.
/// * See [`AsComponents`] for logging serialized data.
///
/// [`AsComponents`]: [crate::AsComponents]
#[derive(Debug, Clone)]
pub struct SerializedComponentBatch {
    pub array: arrow::array::ArrayRef,

    // TODO(cmc): Maybe Cow<> this one if it grows bigger. Or intern descriptors altogether, most likely.
    pub descriptor: ComponentDescriptor,
}

The goal is to keep the ComponentBatch trait isolated at the edge, where it is used as a means of easily converting any data into arrow arrays, instead of simultaneously being used as a means of transporting data around through the internals.
ComponentBatch is here to stay, if only for its conversion capabilities.

This opens a lot of opportunities of improvements in terms of DX, UX and future features (e.g. generics).

The two code paths will co-exist for the foreseeable future, until all archetypes have been made eager.

Copy link

github-actions bot commented Jan 10, 2025

Web viewer built successfully. If applicable, you should also test it:

  • I have tested the web viewer
Result Commit Link Manifest
7d74213 https://rerun.io/viewer/pr/8642 +nightly +main

Note: This comment is updated whenever you push a commit.

Copy link

github-actions bot commented Jan 10, 2025

Latest documentation preview deployed successfully.

Result Commit Link
7d74213 https://landing-k2akvml6q-rerun.vercel.app/docs

Note: This comment is updated whenever you push a commit.

@teh-cmc teh-cmc added enhancement New feature or request 🦀 Rust API Rust logging API 🍏 primitives Relating to Rerun primitives include in changelog labels Jan 10, 2025
@teh-cmc teh-cmc marked this pull request as ready for review January 10, 2025 16:01
@emilk emilk self-requested a review January 10, 2025 17:40
@@ -34,7 +33,37 @@ pub trait AsComponents {
//
// NOTE: Don't bother returning a CoW here: we need to dynamically discard optional components
// depending on their presence (or lack thereof) at runtime anyway.
fn as_component_batches(&self) -> Vec<ComponentBatchCowWithDescriptor<'_>>;
#[deprecated(since = "0.22.0", note = "use as_serialized_batches instead")]
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's worth to also change the docstring to DEPRECATED - DO NOT USE or similar.

///
/// # Fallibility
///
/// There are very few ways in which serialization can fail, all of which are very rare to hit
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Still worth mentioning a few of the ways it can fail (e.g. overflowing an i32)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request include in changelog 🍏 primitives Relating to Rerun primitives 🦀 Rust API Rust logging API
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants