Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

2.5.0 #81

Merged
merged 135 commits into from
Jan 8, 2025
Merged
Changes from 1 commit
Commits
Show all changes
135 commits
Select commit Hold shift + click to select a range
5d5a8ce
KeyRange::empty
marvin-j97 Sep 21, 2024
521e84c
import
marvin-j97 Sep 21, 2024
a782200
import
marvin-j97 Sep 21, 2024
eeb51a1
#51
marvin-j97 Sep 21, 2024
64d6c94
Merge branch '2.1.0' into 2.2.0
marvin-j97 Sep 27, 2024
888b11f
also use full index for L1
marvin-j97 Sep 28, 2024
22b33fd
Merge branch 'main' into 2.2.0
marvin-j97 Oct 9, 2024
369ab94
Merge branch 'main' into perf/l0-full-index
marvin-j97 Oct 22, 2024
d7eae1e
Merge branch 'main' into perf/l0-full-index
marvin-j97 Oct 26, 2024
df33c04
Merge branch 'main' into perf/l0-full-index
marvin-j97 Oct 26, 2024
d18b4cb
fix bench
marvin-j97 Nov 2, 2024
6b34e13
refactor: bench
marvin-j97 Nov 2, 2024
58635c7
add test case
marvin-j97 Nov 2, 2024
3f1482f
closes #62
marvin-j97 Nov 2, 2024
36baa11
Merge branch 'main' into perf/lazy-range-eval
marvin-j97 Nov 2, 2024
417d243
Merge branch 'main' into perf/lazy-range-eval
marvin-j97 Nov 2, 2024
a53fce2
Merge branch 'main' into perf/l0-full-index
marvin-j97 Nov 2, 2024
124b9ff
change size tiered base size to 64M
marvin-j97 Nov 6, 2024
224558c
perf: make Memtable::get_highest_seqno O(1)
marvin-j97 Nov 10, 2024
b1d1419
add test cases
marvin-j97 Nov 10, 2024
dc4b4c6
Merge branch 'main' into perf/lazy-range-eval
marvin-j97 Nov 10, 2024
0c3df6b
Merge branch 'main' into perf/lazy-range-eval
marvin-j97 Nov 13, 2024
ef20ff4
Merge branch 'main' into perf/l0-full-index
marvin-j97 Nov 13, 2024
9043abd
Merge branch 'main' into perf/l0-full-index
marvin-j97 Nov 17, 2024
cbbc9e9
Merge branch 'main' into perf/lazy-range-eval
marvin-j97 Nov 20, 2024
dd19d96
remove unused memtable clear method
marvin-j97 Nov 20, 2024
5cf470d
Revert "remove unused memtable clear method"
marvin-j97 Nov 20, 2024
f5e2ccb
fix: Memtable::clear
marvin-j97 Nov 20, 2024
540089e
Merge pull request #76 from fjall-rs/perf/lazy-range-eval
marvin-j97 Nov 21, 2024
cec2878
Merge remote-tracking branch 'origin/main'
marvin-j97 Nov 21, 2024
7e911ba
Merge branch 'main' into perf/l0-full-index
marvin-j97 Nov 22, 2024
0e02551
Merge branch 'main' into perf/l0-full-index
marvin-j97 Nov 22, 2024
50badf7
fix: range upper bound
marvin-j97 Nov 22, 2024
3339646
fix: workaround for 1.74 rust
marvin-j97 Nov 22, 2024
f808294
fix(full block index): can't debug assert len
marvin-j97 Nov 22, 2024
d2cedba
reimplement segment verify
marvin-j97 Nov 23, 2024
5de0290
recover L0/L1 with full block index
marvin-j97 Nov 23, 2024
4be47cc
stage missing file
marvin-j97 Nov 23, 2024
9671f3d
refactor
marvin-j97 Nov 23, 2024
fa766d0
closes #51
marvin-j97 Nov 23, 2024
9674bd0
wip
marvin-j97 Nov 26, 2024
f04b7a8
rename
marvin-j97 Nov 26, 2024
489ffd4
refactor
marvin-j97 Nov 26, 2024
06f105e
Merge pull request #80 from fjall-rs/perf/l0-full-index
marvin-j97 Nov 26, 2024
8d96736
fix: leveled compaction
marvin-j97 Nov 26, 2024
d543dad
filter out 0 compactions
marvin-j97 Nov 26, 2024
9527036
wip
marvin-j97 Nov 26, 2024
f376442
Merge pull request #82 from fjall-rs/compaction/leveled/level_base_size
marvin-j97 Nov 26, 2024
95bdeb0
preliminary parallel compactions
marvin-j97 Nov 26, 2024
2c8da7d
Merge remote-tracking branch 'origin/main' into leveled-parallel
marvin-j97 Nov 27, 2024
4dff1d2
refactor
marvin-j97 Nov 27, 2024
85a8151
add comments
marvin-j97 Nov 27, 2024
9081447
refactor: rename
marvin-j97 Nov 27, 2024
afd3287
Merge branch 'main' into 2.5.0
marvin-j97 Nov 28, 2024
00604fb
Merge branch '2.5.0' into leveled-parallel
marvin-j97 Nov 28, 2024
f49d476
clippy
marvin-j97 Nov 28, 2024
e661a02
clippy
marvin-j97 Nov 28, 2024
bcdbff1
change L0 bloom filter FPR on flush as well
marvin-j97 Nov 29, 2024
ba438e6
Merge branch 'main' into 2.5.0
marvin-j97 Nov 29, 2024
58c87d3
Merge branch 'main' into leveled-parallel
marvin-j97 Nov 29, 2024
b2492b4
Merge branch '2.5.0' into leveled-parallel
marvin-j97 Nov 29, 2024
27b773b
take compacted bytes into account
marvin-j97 Nov 30, 2024
c47e194
Merge branch 'main' into 2.5.0
marvin-j97 Nov 30, 2024
05beaa8
Merge branch '2.5.0' into leveled-parallel
marvin-j97 Nov 30, 2024
6dd9abf
Update memtable.rs
marvin-j97 Nov 30, 2024
d562c36
Merge branch '2.5.0' into leveled-parallel
marvin-j97 Nov 30, 2024
ced984a
Merge branch 'main' into 2.5.0
marvin-j97 Nov 30, 2024
87f642e
Merge branch '2.5.0' into leveled-parallel
marvin-j97 Nov 30, 2024
5734108
Merge branch 'main' into 2.5.0
marvin-j97 Dec 1, 2024
77259f8
Merge branch '2.5.0' into leveled-parallel
marvin-j97 Dec 1, 2024
e054c8c
Merge branch 'main' into 2.5.0
marvin-j97 Dec 2, 2024
d9c6c8e
Merge branch '2.5.0' into leveled-parallel
marvin-j97 Dec 2, 2024
168d4e0
better zero copy support from types that implement Into<Slice>
carlsverre Dec 4, 2024
12413c3
Merge branch 'main' into 2.5.0
marvin-j97 Dec 4, 2024
d726767
Merge branch '2.5.0' into leveled-parallel
marvin-j97 Dec 4, 2024
03c68c7
Merge branch 'main' into 2.5.0
marvin-j97 Dec 4, 2024
faa3251
Merge branch '2.5.0' into leveled-parallel
marvin-j97 Dec 4, 2024
2d675ec
wip
marvin-j97 Dec 4, 2024
315d558
remove some ? in compaction worker
marvin-j97 Dec 4, 2024
c7ebaec
replace another ? in compaction worker
marvin-j97 Dec 4, 2024
37bf57d
wip
marvin-j97 Dec 4, 2024
d3e2f77
Improve HiddenSet ergonomics and fix a bug in the leveled compaction …
carlsverre Dec 4, 2024
d98c0eb
pass hidden set directly to pick_minimal_computation, but still keep …
carlsverre Dec 4, 2024
186fd9f
Update hidden_set.rs
marvin-j97 Dec 4, 2024
5c5fe26
Merge branch 'leveled-parallel' into leveled-parallel
marvin-j97 Dec 4, 2024
ec3e1b9
update value-log
marvin-j97 Dec 4, 2024
891011c
Update mod.rs
marvin-j97 Dec 4, 2024
1f6cbd3
Merge pull request #88 from carlsverre/leveled-parallel
marvin-j97 Dec 4, 2024
eeab262
cleanup
marvin-j97 Dec 5, 2024
27e7b4d
refactor
marvin-j97 Dec 5, 2024
46f2d83
fmt
marvin-j97 Dec 5, 2024
c219ff2
wip
marvin-j97 Dec 5, 2024
0cd43e6
refactor
marvin-j97 Dec 5, 2024
69b4038
clippy
marvin-j97 Dec 5, 2024
d4246ae
wip
marvin-j97 Dec 5, 2024
95ccb2b
refactor
marvin-j97 Dec 5, 2024
1ee2272
remove unneeded struct
marvin-j97 Dec 5, 2024
dde29cf
wip
marvin-j97 Dec 5, 2024
5806a14
refactor
marvin-j97 Dec 5, 2024
85a3ff7
refactor
marvin-j97 Dec 5, 2024
38f3648
Merge pull request #83 from fjall-rs/leveled-parallel
marvin-j97 Dec 5, 2024
3dcaf4f
Merge branch 'main' into 2.5.0
marvin-j97 Dec 7, 2024
fb98138
compaction: remove overshoot checking again
marvin-j97 Dec 7, 2024
4ada35b
add merge bench
marvin-j97 Dec 7, 2024
4c2754c
adjust bench
marvin-j97 Dec 7, 2024
26c6072
add mvcc stream bench
marvin-j97 Dec 7, 2024
f4b745c
wip
marvin-j97 Dec 7, 2024
5c02628
catch another ? in compaction worker, #87
marvin-j97 Dec 7, 2024
f46b6fe
fix(MultiWriter): make sure KV versions cannot span segments
marvin-j97 Dec 9, 2024
ffed629
doc
marvin-j97 Dec 9, 2024
5dca9e5
refactor
marvin-j97 Dec 9, 2024
615ce00
fix
marvin-j97 Dec 9, 2024
96ead47
remove dbg log
marvin-j97 Dec 10, 2024
a9c883e
doc: update internal docs
marvin-j97 Dec 11, 2024
8332b1a
refactor: segment writer
marvin-j97 Dec 12, 2024
88575ef
move module
marvin-j97 Dec 14, 2024
b9c1bcd
perf: upgrade crossbeam-skiplist to skip heap allocation in get()
marvin-j97 Dec 14, 2024
2d29e55
perf: simplify segment point read fast path
marvin-j97 Dec 14, 2024
164adb5
revert version change
marvin-j97 Dec 14, 2024
bd52297
Merge remote-tracking branch 'origin/main' into 2.5.0
marvin-j97 Dec 14, 2024
122c77c
Merge branch 'main' into 2.5.0
marvin-j97 Dec 14, 2024
296e64f
Merge pull request #85 from carlsverre/main
marvin-j97 Dec 14, 2024
26847b3
2.5.0-pre.0
marvin-j97 Dec 14, 2024
ab28434
perf: remove heap allocation in snapshot point read path
marvin-j97 Dec 14, 2024
9113cb4
perf: specialize Segment reader for snapshot point reads
marvin-j97 Dec 17, 2024
8175914
revert crossbeam-skiplist for now
marvin-j97 Dec 20, 2024
72f69fd
refactor: simplify get_internal_entry
marvin-j97 Dec 20, 2024
13d7252
leveled compaction: only consider best trivial move
marvin-j97 Dec 20, 2024
540013d
fix(leveled compaction): trivial moves
marvin-j97 Dec 22, 2024
a89bf4c
reenable size filter
marvin-j97 Dec 22, 2024
000fa08
refactor
marvin-j97 Dec 23, 2024
f4e5e58
change logging
marvin-j97 Dec 27, 2024
0198ada
refactor
marvin-j97 Jan 6, 2025
e3e264d
update value log
marvin-j97 Jan 7, 2025
abddb10
2.5.0
marvin-j97 Jan 7, 2025
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
closes #62
  • Loading branch information
marvin-j97 committed Nov 2, 2024
commit 3f1482f577f81d65a990eda6cf47d41944f9e645
35 changes: 14 additions & 21 deletions src/segment/range.rs
Original file line number Diff line number Diff line change
@@ -19,7 +19,8 @@ use std::sync::Arc;
pub struct Range {
block_index: Arc<TwoLevelBlockIndex>,

is_initialized: bool,
lo_initialized: bool,
hi_initialized: bool,

pub(crate) range: (Bound<UserKey>, Bound<UserKey>),

@@ -45,7 +46,8 @@ impl Range {
);

Self {
is_initialized: false,
lo_initialized: false,
hi_initialized: false,

block_index,

@@ -75,9 +77,13 @@ impl Range {
Some(start)
}
};

if let Some(key) = start_key.cloned() {
self.reader.set_lower_bound(key);
}

self.lo_initialized = true;

Ok(())
}

@@ -107,19 +113,8 @@ impl Range {
if let Some(key) = end_key.cloned() {
self.reader.set_upper_bound(key);
}
Ok(())
}

fn initialize(&mut self) -> crate::Result<()> {
// TODO: can we skip searching for lower bound until next is called at least once...?
// would make short ranges 1.5-2x faster (if cache miss) if only one direction is used
self.initialize_lo_bound()?;

// TODO: can we skip searching for upper bound until next_back is called at least once...?
// would make short ranges 1.5-2x faster (if cache miss) if only one direction is used
self.initialize_hi_bound()?;

self.is_initialized = true;
self.hi_initialized = true;

Ok(())
}
@@ -129,8 +124,8 @@ impl Iterator for Range {
type Item = crate::Result<InternalValue>;

fn next(&mut self) -> Option<Self::Item> {
if !self.is_initialized {
if let Err(e) = self.initialize() {
if !self.lo_initialized {
if let Err(e) = self.initialize_lo_bound() {
return Some(Err(e));
};
}
@@ -182,16 +177,14 @@ impl Iterator for Range {

impl DoubleEndedIterator for Range {
fn next_back(&mut self) -> Option<Self::Item> {
if !self.is_initialized {
if let Err(e) = self.initialize() {
if !self.hi_initialized {
if let Err(e) = self.initialize_hi_bound() {
return Some(Err(e));
};
}

loop {
let entry_result = self.reader.next_back()?;

match entry_result {
match self.reader.next_back()? {
Ok(entry) => {
match self.range.start_bound() {
Bound::Included(start) => {