Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

S3 collab cache #1028

Merged
merged 16 commits into from
Dec 3, 2024
Merged

S3 collab cache #1028

merged 16 commits into from
Dec 3, 2024

Conversation

Horusiath
Copy link
Collaborator

@Horusiath Horusiath commented Nov 28, 2024

This PR reimplements DiskCache to store encoded collabs in S3-compatible storage rather than in Postgres. Postgres is used only as a fallback for compatibility with an existing storage model.

TODOs:

  • Use S3 for collab snapshot data as well.
  • We need to decide if we really want to store EncodedCollab format in S3. IMO it's useless - it's heavily reliant on bincode (which is not version tolerant) and all of the fields (namely: state_vector) can be resolved from doc_state directly instead. We could keep a straight up yrs Doc state instead.
  • Added ZSTD compression of S3 objects.
  • Added a threshold (S3_COLLAB_THRESHOLD, 2000 bytes), so that any collab data smaller than that threshold will still land in Postgres. PS: All new snapshots are always stored in S3 regardless of size.

@Horusiath Horusiath changed the title [WIP] S3 collab cache S3 collab cache Dec 2, 2024
@appflowy appflowy merged commit 9ff6f1c into main Dec 3, 2024
10 checks passed
@appflowy appflowy deleted the a3-collab-cache branch December 3, 2024 05:08
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants