Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Enhancement] Enable dictionary to improve zstd compress performance #9132

Open
1 task done
ferrirW opened this issue Jan 15, 2025 · 0 comments
Open
1 task done

[Enhancement] Enable dictionary to improve zstd compress performance #9132

ferrirW opened this issue Jan 15, 2025 · 0 comments

Comments

@ferrirW
Copy link
Contributor

ferrirW commented Jan 15, 2025

Before Creating the Enhancement Request

  • I have confirmed that this should be classified as an enhancement rather than a bug/feature.

Summary

When the message body is too large or has fixed characteristics, using dictionaries can further improve the compression rate and thus improve the throughput.

Motivation

The compressed message body is more than 10KB, but with dictionaries it can be reduced to about 5KB

Describe the Solution You'd Like

  1. design a dictionary management mechanism between client and broker
  2. add the attribute of whether to enable dictionary and which dictionary to use for compression/decompression
  3. implement a new ZstdDictCompressor interface and compression and decompression based on dictionary-related configurations

Describe Alternatives You've Considered

The question that can be discussed is how to manage dictionaries
There may be several choices

  1. The dictionary file is passed in as the client configuration
  2. The dictionary is dynamically trained, generated, and managed by the broker
  3. The training and management of the dictionary are handled by a third-party component

Additional Context

No response

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

1 participant