Increase default dbcache to ∞

[luke-jr]

Bitcoin Knots already detects low-memory situations and flushes caches when there's only 10 MB left. It would be nice to just leave the UTXO cache unbound for optimal performance.

But is 10 MB enough? Or will it cause problems?

Testing requested :)

Just set dbcache=1048576 and let me know if you run into issues (maybe re-sync too)

[tchjntr]

Currently testing this on a laptop with the following specs:

• CPU: Intel Core i7-1165G7 @ 2.80GHz, 4 Cores, 8 Threads
• RAM: 32GB Crucial DDR4-3200 (2 x 16GB)
• SSD: Samsung 980 PRO PCIe 4.0 NVMe 1TB
• OS: Windows 11 Pro (Version 10.0.22631 Build 22631)

Started a full IBD from scratch with Bitcoin Knots v26.1.knots20240325 and the following options in bitcoin.conf:

assumevalid=0
blockfilterindex=1
blocksonly=1
coinstatsindex=1
connect=[IP address of my main Bitcoin full node in my home network]
dbcache=1048576
discover=0
dns=0
dnsseed=0
listen=0
peerblockfilters=1
peerbloomfilters=1
server=1
txindex=1
v2transport=1

Will provide an update after it syncs or if there are issues.

UPDATE 05/10/2024:

IBD took about 33 hours and 19 minutes.

Start: 2024-05-09T09:49:53Z
Finished syncing to tip: 2024-05-10T18:47:35Z (Block 842887)

This was done in one go with no interruptions.

[chrisguida]

@luke-jr do i need to set any of the above options from @tchjntr other than dbcache=1048576? does assumevalid matter?

[luke-jr]

@luke-jr do i need to set any of the above options from @tchjntr other than dbcache=1048576? does assumevalid matter?

Shouldn't matter

[chrisguida]

Cool, syncing

[chrisguida]

Hmm I was almost done syncing and needed to shut down gracefully. I ran sudo shutdown now and the kernel showed a message indicating that it would wait for 30 minutes before killing bitcoind. I waited 5-10 minutes, so it took at least this long to flush the utxo cache to disk. I stepped away for an hour or so.

I turned the system back on and it started rolling forward from block 1 :/

My system has about 13GB ram, so I'm not sure how the cache is 20GB. Is it using swap?

$ free -m
               total        used        free      shared  buff/cache   available
Mem:           13942        1985         554           8       11401       11620
Swap:          15336           7       15329

[luke-jr]

The log shows it was killed, so the rolling forward is expected at that point I guess. :( I suppose that indicates maybe this requires benchmarking the disk and making sure we never cache more than it can write in N=30 minutes.

That free output is from while it said it had 20 GB cached??? That doesn't make sense :/

[chrisguida]

The log shows it was killed, so the rolling forward is expected at that point I guess.

I guess people need to keep in mind that flushing to disk will take a loooong time heh

I suppose that indicates maybe this requires benchmarking the disk and making sure we never cache more than it can write in N=30 minutes.

Yeah, unless we expect users to know not to use sudo shutdown now. Or unless 30 minutes is not a common setting for this. I'm just using the nix-bitcoin default system configuration (mostly).

That free output is from while it said it had 20 GB cached??? That doesn't make sense :/

No, the free output is from now. I didn't check it while the cache was full. I'm just wondering if you know whether bitcoind swaps or not. I suppose this is managed automatically by the system, so the answer is probably yes.

Another question, why is the cache 20GB? Isn't the entire utxoset like 12GB?

[chrisguida]

I was slightly afraid this would happen given that I kept seeing this message...

May 28 12:53:01 nix-um350 bitcoind[88208]: [coindb] Node is still in IBD, rescheduling chainstate disk sync...

How are we able to put off flushing to disk indefinitely? I guess the cache stores more than just the current utxoset?

[luke-jr]

I'm just wondering if you know whether bitcoind swaps or not. I suppose this is managed automatically by the system, so the answer is probably yes.

The only reason this could have worked, is because Knots checks free memory often and flushes if it's getting low. So it would be a bug to activate swap (aside from the OS deciding swap is optimal anyway). It might be good to have further testing monitor swap usage and desktop usability while it syncs.

Another question, why is the cache 20GB? Isn't the entire utxoset like 12GB?

Size on disk is not necessarily size in memory. I would expect them to be closer, but I haven't ever compared. Perhaps this is another area things could be optimised? Obvious differences (coins.h class Coin) are that we store the height (in memory and disk) even when it's not strictly needed, and we only compress the UTXO set on disk.

May 28 12:53:01 nix-um350 bitcoind[88208]: [coindb] Node is still in IBD, rescheduling chainstate disk sync...

How are we able to put off flushing to disk indefinitely?

The log message is about the final disk sync after IBD completes. It might still sync periodically before that. Maybe comment on bitcoin#15218 that the message is confusing, so it gets addressed.

I guess the cache stores more than just the current utxoset?

I don't think it does.

[1ma]

I guess the cache stores more than just the current utxoset?

I don't think it does.

Yesterday I did a full IBD on my local network and I happened to run watch du -h -d 1 . inside the datadir while Knots was doing the final flush (I had 32GB RAM so the process didn't need to flush once).

What I saw during the flush is that the chainstate dir kept growing and shrinking continuously, like 3.1GB -> 2.7GB -> 3.2 GB -> 3.7GB -> 2.9GB -> ... and so forth until it completed the flush (in about 15min) and reached the final size.

Based on that it seemed like the dbcache holds some sort of log data structure that then applies to the chainstate sequentially, not the latest utxo set directly 🤔