Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Declarative RLP Encoding/Decoding #7975

Draft
wants to merge 115 commits into
base: master
Choose a base branch
from
Draft
Changes from 2 commits
Commits
Show all changes
115 commits
Select commit Hold shift + click to select a range
d3a3232
Initial Rlp + Writer
emlautarom1 Dec 19, 2024
9a90ccb
Rename `Sequence` -> `List`
emlautarom1 Dec 19, 2024
30dda91
Initial `RlpReader`
emlautarom1 Dec 19, 2024
d3d83f9
Initial `ReadList`
emlautarom1 Dec 19, 2024
65f0891
Multiple `ReadList`
emlautarom1 Dec 19, 2024
b4656cd
Restructure into separate files
emlautarom1 Dec 19, 2024
a63b64f
Use `Rlp.Read` API
emlautarom1 Dec 19, 2024
6735d29
Add `HeterogeneousList` test
emlautarom1 Dec 19, 2024
74cf2fd
Add overload for `Action`
emlautarom1 Dec 19, 2024
2ef4c4d
Test `UnknownLengthList`
emlautarom1 Dec 19, 2024
b3dddcb
Rename converter
emlautarom1 Dec 19, 2024
8b19e01
Use custom Exception
emlautarom1 Dec 19, 2024
2c67165
Add `Action` overload
emlautarom1 Dec 19, 2024
52e1f7b
Test for invalid readings
emlautarom1 Dec 19, 2024
7262407
Support long lists (+55 bytes)
emlautarom1 Dec 19, 2024
e4c2e63
Implement interface on `ReadOnlySpanConverter`
emlautarom1 Dec 20, 2024
35b9203
Make `Rlp[Reader|Writer]` symmetric
emlautarom1 Dec 20, 2024
e0ec1ce
Support ref struct
emlautarom1 Dec 20, 2024
d3a2e5b
Reorder tests
emlautarom1 Dec 20, 2024
67a69df
Annotate as scoped
emlautarom1 Dec 20, 2024
dc3798b
Initial `Choice` implementation
emlautarom1 Dec 20, 2024
ffa62f4
Cleanup test
emlautarom1 Dec 20, 2024
e626fc9
Restructure internals
emlautarom1 Dec 20, 2024
6f3efbc
Test for deep `Choice` (backtracking)
emlautarom1 Dec 20, 2024
2d2eeda
Move `IRlpConverter`
emlautarom1 Dec 20, 2024
67abc2b
Demo user-defined record support
emlautarom1 Dec 20, 2024
8f4b02a
Remove versions
emlautarom1 Dec 20, 2024
9db2ebb
Use `ref struct` over `Interface`
emlautarom1 Dec 20, 2024
2223845
Consistent error
emlautarom1 Dec 20, 2024
51fd038
Split tests from library
emlautarom1 Dec 20, 2024
5fad6ef
Initial source generator for `RlpSerializable`
emlautarom1 Dec 20, 2024
c22f1df
Implement `Write`
emlautarom1 Dec 20, 2024
70c78e2
Test for Source Generated instances
emlautarom1 Dec 20, 2024
f11ff9a
Move record
emlautarom1 Dec 20, 2024
38cbb3d
No need to copy `using` directives
emlautarom1 Dec 20, 2024
f284c28
Remove `Derived` namespace
emlautarom1 Dec 23, 2024
6e91e0d
Update docs
emlautarom1 Dec 23, 2024
ec2fdbc
Rename `List` to `Sequence`
emlautarom1 Dec 23, 2024
f158463
Add support for `List` collection
emlautarom1 Dec 23, 2024
6677222
Test for equivalence in collections with explicit
emlautarom1 Dec 23, 2024
e32a33c
Add explicit converters for IntXX
emlautarom1 Dec 23, 2024
f665779
Reduce duplication
emlautarom1 Dec 23, 2024
2e75a7d
Add `Dictionary` collection
emlautarom1 Dec 23, 2024
aa6b4bd
Extend generator to support Generics (WIP)
emlautarom1 Dec 23, 2024
fcde607
Support multi-param generics
emlautarom1 Dec 23, 2024
416a45b
Add Generic writer Action
emlautarom1 Dec 23, 2024
837c2e4
Rewrite Generics
emlautarom1 Dec 23, 2024
f484b8b
Add recursive record test
emlautarom1 Dec 23, 2024
599da39
Enable `nullable`
emlautarom1 Dec 23, 2024
3eff5ea
Fix `ReadBytes`
emlautarom1 Dec 23, 2024
b7889ed
Test for all base Integer types
emlautarom1 Dec 23, 2024
d41ecd3
Initial README
emlautarom1 Dec 23, 2024
03325a0
Make `Read` calls static
emlautarom1 Dec 24, 2024
914505a
Add `context` overload
emlautarom1 Dec 24, 2024
7dbc2dc
Remove allocations
emlautarom1 Dec 24, 2024
15eb6b6
Use `static` on `Write`
emlautarom1 Dec 24, 2024
c42eb50
Use `static` on `Read`
emlautarom1 Dec 24, 2024
ed0d88b
Return `byte[]` instead of `ReadOnlySpan<byte>` in overload
emlautarom1 Dec 24, 2024
88a8cc4
Use `static` on `Write`
emlautarom1 Dec 24, 2024
a596ab0
Fix typo
emlautarom1 Dec 24, 2024
df3ffc9
Add overloads for `TContext`
emlautarom1 Dec 24, 2024
42b4401
Use `context` overload when possible
emlautarom1 Dec 24, 2024
917a7b8
Remove duplicated overloads
emlautarom1 Dec 24, 2024
88e7bac
Remove "context" TODO
emlautarom1 Dec 24, 2024
1bc2d38
Remove allocating overload
emlautarom1 Dec 24, 2024
fb254e9
More `static` usage
emlautarom1 Dec 24, 2024
21a18b9
Use "context" overload in Generator
emlautarom1 Dec 24, 2024
4740466
Annotate as `scoped`
emlautarom1 Dec 26, 2024
5705da1
Optimize `StringRlpConverter`
emlautarom1 Dec 26, 2024
cc58776
Rename `FastRlp` to `FluentRlp`
emlautarom1 Dec 26, 2024
2456c40
Remove allocations in Dictionary
emlautarom1 Dec 26, 2024
868d272
Remove unused method
emlautarom1 Dec 26, 2024
69c8a92
Introduce "context" overload for `Reader`
emlautarom1 Dec 26, 2024
f0191d1
Remove allocations in `Dictionary`
emlautarom1 Dec 26, 2024
1011df9
Remove allocations in `List`
emlautarom1 Dec 26, 2024
63c3ed9
Use field Encoding
emlautarom1 Dec 26, 2024
0c2807b
Add `ArrayRlpConverter`
emlautarom1 Dec 26, 2024
410dfc7
Merge branch 'master' into feature/declarative-rlp
emlautarom1 Dec 26, 2024
efc25bd
Remove allocations
emlautarom1 Dec 26, 2024
aae0100
Add `TupleRlpConverter`
emlautarom1 Dec 26, 2024
c7a32d1
Support `Tuple` in source generators
emlautarom1 Dec 26, 2024
3dc5b65
Add support for `Representation`
emlautarom1 Dec 26, 2024
fd9fc30
Formatting
emlautarom1 Dec 26, 2024
9f5d236
Merge branch 'master' into feature/declarative-rlp
emlautarom1 Dec 26, 2024
8341a0e
Update docs
emlautarom1 Dec 27, 2024
3e1330d
Throw on trailing bytes
emlautarom1 Dec 30, 2024
4f68cb3
Reduce usage of `StringBuilder`
emlautarom1 Dec 30, 2024
ac98b21
Remove unused code
emlautarom1 Dec 30, 2024
79dc13f
Use format string
emlautarom1 Dec 30, 2024
c9ba2eb
Merge `Array` with generics
emlautarom1 Dec 30, 2024
e43f13c
Merge branch 'master' into feature/declarative-rlp
emlautarom1 Dec 30, 2024
db5408a
Formatting
emlautarom1 Dec 30, 2024
38eb0f0
Merge remote-tracking branch 'origin/feature/declarative-rlp' into fe…
emlautarom1 Dec 30, 2024
31726b6
Fix `0`
emlautarom1 Jan 2, 2025
149145b
Remove signed integer requirement
emlautarom1 Jan 2, 2025
dd57f80
Add `BytesRead`
emlautarom1 Jan 2, 2025
40cfd25
Include user `using` statements
emlautarom1 Jan 2, 2025
f419316
Fix nested Generics support
emlautarom1 Jan 2, 2025
d3592d8
Initial benchmark
emlautarom1 Jan 2, 2025
a52d923
Prefer `sizeof(T)` over `Marshal.SizeOf<T>`
emlautarom1 Jan 2, 2025
606a260
Formatting
emlautarom1 Jan 2, 2025
abf143d
Merge branch 'master' into feature/declarative-rlp
emlautarom1 Jan 2, 2025
2a9bee7
File encoding
emlautarom1 Jan 2, 2025
d0dfb12
Merge remote-tracking branch 'origin/feature/declarative-rlp' into fe…
emlautarom1 Jan 2, 2025
bcaa83f
Remove unused import
emlautarom1 Jan 2, 2025
783acc6
Avoid boxing due to interface default bodies
emlautarom1 Jan 3, 2025
6dc21f4
Add `MemoryDiagnoser`
emlautarom1 Jan 3, 2025
1c8f472
Use `IBufferWriter` instead of `byte[]`
emlautarom1 Jan 6, 2025
b2e5951
Use `IBufferWriter` instead of `byte[]`
emlautarom1 Jan 6, 2025
9622a7b
Merge remote-tracking branch 'origin/feature/declarative-rlp' into fe…
emlautarom1 Jan 7, 2025
11bf0c7
Merge remote-tracking branch 'origin/feature/declarative-rlp' into fe…
emlautarom1 Jan 7, 2025
92bc902
Merge remote-tracking branch 'origin/feature/declarative-rlp' into fe…
emlautarom1 Jan 7, 2025
7e03ac3
Add support for `Optional`
emlautarom1 Jan 10, 2025
04ac6b2
Use `sizeof` instead of `Marshal.SizeOf<T>` in `RlpReader`
emlautarom1 Jan 10, 2025
3250b15
More tests for `Optional`
emlautarom1 Jan 10, 2025
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
56 changes: 46 additions & 10 deletions src/Nethermind/Nethermind.Serialization.FluentRlp/Rlp.cs
Original file line number Diff line number Diff line change
Expand Up @@ -3,31 +3,27 @@

using System;
using System.Buffers;
using System.Runtime.CompilerServices;
using System.Diagnostics;

namespace Nethermind.Serialization.FluentRlp;

public static class Rlp
{
public static ReadOnlyMemory<byte> Write(RefRlpWriterAction action)
public static byte[] Write(RefRlpWriterAction action)
=> Write(action, static (ref RlpWriter w, RefRlpWriterAction action) => action(ref w));

public static ReadOnlyMemory<byte> Write<TContext>(TContext ctx, RefRlpWriterAction<TContext> action)
public static byte[] Write<TContext>(TContext ctx, RefRlpWriterAction<TContext> action)
where TContext : allows ref struct
{
var lengthWriter = RlpWriter.LengthWriter();
action(ref lengthWriter, ctx);
var buffer = new ArrayBufferWriter<byte>(lengthWriter.Length);
var contentWriter = RlpWriter.ContentWriter(buffer);
var bufferWriter = new FixedArrayBufferWriter<byte>(lengthWriter.Length);
var contentWriter = RlpWriter.ContentWriter(bufferWriter);
action(ref contentWriter, ctx);
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

When is it used? Do we write the data twice?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

When is what used? We "write" the data twice to first compute the length (LengthWriter), and then we actually write the bytes into a buffer (ContentWriter).

Copy link
Contributor

@Scooletz Scooletz Jan 7, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is costly. There might be a way to write I think. Paprika does it. In Paprika, there's enough room left at the beginning to encode any length, then once the encoding is done, the length is written at the beginning and the position is adjusted. This would require a copy of the memory though as it would leave a gap at the beginning. This might be still cheaper if you copy the memory once more instead of going through the serialization two times?

https://github.com/NethermindEth/Paprika/blob/61c326a39e942ab9c550f6a6b9b5df59e67c0f0f/src/Paprika/Merkle/ComputeMerkleBehavior.cs#L750-L757

Copy link
Contributor Author

@emlautarom1 emlautarom1 Jan 7, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This could work but I think it would be quite a redesign if it were to be adapted to all paths, not only here at the top level. Recursive calls in WriteSequence would need to be adjusted to first immediately write the bytes and then move them around to deal with writing the length first.

I don't have an immediate answer as to what approach is cheaper. Moving the data around sounds more expensive (to me at least) than the current approach but it might we worth exploring.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Moving once vs writing twice 👀 Clearly, I'm just asking as I don't have a clear answer. Also, in Paprika there are just a few cases where it's needed as majority of the written RLP is really short (beside branches, where it's applied).


return buffer.WrittenMemory;
return bufferWriter.Buffer;
}

public static T Read<T>(ReadOnlyMemory<byte> source, RefRlpReaderFunc<T> func) where T : allows ref struct
=> Read(source.Span, func);

[OverloadResolutionPriority(1)]
public static T Read<T>(ReadOnlySpan<byte> source, RefRlpReaderFunc<T> func)
where T : allows ref struct
{
Expand All @@ -37,3 +33,43 @@ public static T Read<T>(ReadOnlySpan<byte> source, RefRlpReaderFunc<T> func)
return result;
}
}

/// <remarks>
/// The existing <see cref="ArrayBufferWriter{T}"/> performs various bound checks and supports resizing buffers
/// which we don't need for our use case.
/// </remarks>
internal class FixedArrayBufferWriter<T> : IBufferWriter<T>
{
private readonly T[] _buffer;
private int _index;

public T[] Buffer => _buffer;

/// <summary>
/// Creates an instance of an <see cref="FixedArrayBufferWriter{T}"/>, in which data can be written to,
/// with a fixed capacity specified.
/// </summary>
/// <param name="capacity">The capacity of the underlying buffer.</param>
public FixedArrayBufferWriter(int capacity)
{
_buffer = new T[capacity];
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ArrayPool?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

By default I think we should go with a plain array to match the ArrayBufferWriter behavior. At the end of the day, the RLP static class is like a "safe default" API.

If we want more control over the buffers we use we can write a custom IBufferWriter as you suggested earlier while using RlpReader and RlpWriter directly.

_index = 0;
}

public void Advance(int count)
{
_index += count;
}

public Memory<T> GetMemory(int sizeHint = 0)
{
Debug.Assert(_buffer.Length > _index);
return _buffer.AsMemory(_index);
}

public Span<T> GetSpan(int sizeHint = 0)
{
Debug.Assert(_buffer.Length > _index);
return _buffer.AsSpan(_index);
}
}
Loading