feat(sdk): Add LinkedChunk::subscribe_as_vector

[Hywan]

This patch implements LinkedChunk::<Item, Gap>::as_vector,
which returns a AsVector.

AsVector is a type that transforms a
Vec<Update<Item, Gap>> into a Vec<VectorDiff<Item>> —this type—. Basically, it
helps to consume a LinkedChunk<CAP, Item, Gap> as if it was an
eyeball::ObservableVector<Item>.

How this type transforms Update into VectorDiff? There is no internal
buffer of kind eyeball_im::ObservableVector<Item>, which could have been
used to generate the VectorDiffs. They are computed manually.

The only buffered data is pairs of ChunkIdentifier and ChunkLength.
The following rules must be respected:

• A chunk of kind ChunkContent::Gap has a length of 0,
• A chunk of kind ChunkContent::Items has a length equals to its number
  of items,
• The pairs must be ordered exactly like the chunks in LinkedChunk, i.e.
  the first pair must represent the first chunk, the last pair must
  represent the last chunk.

The only thing this algorithm does is maintaining the pairs:

• Update::NewItemsChunk and Update::NewGapChunk are inserting a new
  pair with a chunk length of 0 at the appropriate index,
• Update::RemoveChunk is removing a pair,
• Update::PushItems is increasing the length of the appropriate pair by
  the number of new items, and is potentially emitting VectorDiff,
• Update::DetachLastItems is decreasing the length of the appropriate pair
  by the number of items to be detached; no VectorDiff is emitted,
• Update::StartReattachItems and Update::EndReattachItems are
  respectively muting or unmuting the emission of VectorDiff by
  Update::PushItems.

The only VectorDiff that are emitted are VectorDiff::Insert or
VectorDiff::Append because a LinkedChunk is append-only.

VectorDiff::Append is an optimisation when numerous VectorDiff::Inserts
have to be emitted at the last position.

VectorDiff::Insert need an index. To compute this index, the algorithm
will iterate over all pairs to accumulate each chunk length until it finds
the appropriate pair (given by Update::PushItems::position_hint). This
is the offset. To this offset, the algorithm adds the position's index of
the new items (still given by Update::PushItems::position_hint). This is
the index. This logic works for all cases as long as pairs are maintained
according to the rules hereinabove.

That's a pretty memory compact and computation efficient way to map a
Vec<Update<Item, Gap>> into a Vec<VectorDiff<Item>>. The larger
the LinkedChunk capacity is, the
fewer pairs the algorithm will have to handle, e.g. for 1'000 items and a
LinkedChunk capacity of 128, it's only 8 pairs, be 256 bytes.

Most of the patch is documentation and tests, don't be impressed 🙂.

---

• Address [meta] EventCache storage #3280

[codecov]

Codecov Report

Attention: Patch coverage is 82.53968% with 22 lines in your changes are missing coverage. Please review.

Project coverage is 82.95%. Comparing base (3517334) to head (39a6a2e).
Report is 56 commits behind head on main.

❗ Current head 39a6a2e differs from pull request most recent head 52c0614

Please upload reports for the commit 52c0614 to get more accurate results.

Files	Patch %	Lines
...trix-sdk/src/event_cache/linked_chunk/as_vector.rs	75.94%	19 Missing ⚠️
...tes/matrix-sdk/src/event_cache/linked_chunk/mod.rs	93.33%	2 Missing ⚠️
...matrix-sdk/src/event_cache/linked_chunk/updates.rs	94.11%	1 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #3392      +/-   ##
==========================================
- Coverage   82.98%   82.95%   -0.03%     
==========================================
  Files         246      247       +1     
  Lines       25022    25116      +94     
==========================================
+ Hits        20764    20835      +71     
- Misses       4258     4281      +23     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[Hywan]

After a long discussion with @bnjbvr, it appears that having a Stream may create difficulties when integrating with the EventCache. I'm working on renaming AsVectorSubscriber to AsVector, and use take instead of poll_next (so no more Stream) to have a serial API. Not a big deal, the PR can still be reviewed for the moment.

[bnjbvr]

Sweet, the algorithm seems to make sense, and the tests increase confidence in it. Good job!

A few questions, and I'd like to make sure we can still clear() a whole linked chunk before merging.

[poljar]

This mostly makes sense to me, I'm not that familiar with what you're trying to achieve and don't have time to do a deep dive as of now.

I left some nits, and it would probably be wise to get approval from @bnjbvr due to the above, but feel free to move forward if you disagree.

[bnjbvr]

Thanks! An amazing chunk of work 🥁

[bnjbvr]

Instead of doing this assert once at the end, could we do it in apply_and_assert_eq? Iterate on all the items of the linked chunk to reconstruct a Vector from it, and compare it against the accumulator \o/

[Hywan]

I think it's robust enough right now. The idea of comparing a rebuilt vector is to ensure the VectorDiffs we expect are the correct ones. Having this once at the end seems fine to me.