Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

accounts-db: initial implementation #40

Closed
wants to merge 54 commits into from
Closed

accounts-db: initial implementation #40

wants to merge 54 commits into from

Conversation

0xNineteen
Copy link
Contributor

@0xNineteen 0xNineteen commented Oct 19, 2023 •
edited
Loading

@0xNineteen 0xNineteen changed the title 19/bank snapshot logic: code to deserialize bank-fields + accountsdb-fields metadata Oct 19, 2023
@0xNineteen 0xNineteen changed the title snapshot logic: code to deserialize bank-fields + accountsdb-fields metadata snapshot logic: code to deserialize bank/accountsdb metadata Oct 19, 2023
@0xNineteen 0xNineteen changed the title snapshot logic: code to deserialize bank/accountsdb metadata snapshot logic: deserialize bank/accountsdb metadata Oct 19, 2023
@0xNineteen 0xNineteen marked this pull request as ready for review October 19, 2023 21:58
@0xNineteen 0xNineteen linked an issue Dec 6, 2023 that may be closed by this pull request
Initial AccountsDB implementation #53
Closed
20 tasks
@0xNineteen 0xNineteen mentioned this pull request Dec 6, 2023
Closed
20 tasks
@0xNineteen 0xNineteen changed the title snapshot logic: deserialize bank/accountsdb metadata snapshot: deserialize bank/accountsdb metadata Dec 6, 2023
@0xNineteen 0xNineteen changed the title snapshot: deserialize bank/accountsdb metadata accounts-db: deserialize bank/accountsdb metadata Dec 7, 2023
0xNineteen and others added 7 commits December 15, 2023 15:23
@0xNineteen
accounts-db: accounts files (#42)
4dab91a 
`build.zig`
- make it easy to add new executables with all the required modules

`src/core/accounts_file.zig`
- logic for parsing accounts out of a file

`pubkey.zig`
- removed `cached_str` optimization so we can directly case bytes to the pubkey struct (used when parsing account files)

`src/cmd/snapshot_utils.zig` 
- snapshot utils for: 
  - dumping the `accounts_db.bincode` from the full bank metadata (the full bank metadata is quite large, and we only need the accounts_db fields  to correctly parse the append_vecs) 
  - dump accounts found in append vec into a .csv format (_note_: current approach contains duplicate accounts (ie, multiple slots))
@0xNineteen
accounts-db: loading, indexing, validating accounts (#54)
d68b680 
`src/common/accounts_db.zig`
- loading account files + indexing pubkeys 
- verification of snapshot data

`src/common/merkle_tree.zig`
- simple merkle tree hash calculation 

`src/core/account.zig`
- logic for hashing account

`test_data`
- new local snapshot .zstd (same size as genesis.bin)
- matching genesis.bin 
- both used in the accounts-db test

### dev notes: 
- snapshot data / load / verification can take a long time because the mainnet snapshots are so big - so we use the validator + the `solana-ledger-tool` to create smaller local snapshots and load from those
    - _note:_ when doing this the accounts-hash will use the current slot in the hash calculation (because the local slot < feature activation slot) fyi
@0xNineteen
accounts-db: sysvar parsing (#56)
acee776 
`bitvec.zig` / `bloom.zig`
- move BitVecConfig from bloom.zig to bitvec.zig 

`clock.zig` -renamed-> `time.zig`

`accounts_db.zig`
- reorg functions to get accounts better `getAccount` (pubkey), `getAccountInner` (file_id, offset) `getAccountType` (get account + cast with data type)
- read and validate status_cache code `validateStatusCache` 

`snapshot_fields.zig` 
- add error enums (required for status cache deser) 

`sysvars.zig`
- add common sysvar pubkeys + structs 

`test_data`
- add an example status_cache to read from for tests
@0xNineteen
accounts-db: support incremental snapshots (#58)
45c1262 
`bincode.zig`
- fix optional EOF fields parsing (we dont need this anymore)

`accounts_db.zig`
- main functions changes: 
  - `loadFromSnapshot` now only takes accounts_db fields 
  - `computeAccountHashesAndLamports`: computes the accounts hash + accounts-delta hash depending on validating a full snapshot or incremental 
  - `validateLoadFromSnapshot`: only validates the accounts-db fields now 
  - add full/incremental snapshot path parsing - needed for finding snapshots + matching incremental snapshots 
  - `unpackZstdTarBall` - decompresses and untars a snapshot 
  - `main` - the full flow of loading the accounts-db and validating (will turn into full validator code) 

`bank.zig`
- rn just includes bank validation against genesis 

`snapshot_fields.zig`
- changes to properly parse incremental snapshot fields + include additional tests 
- `validateStatusCache`

`test_data`
- use different snapshot files (still small - just needed to build a fresh set during testing)
@0xNineteen
lint and tests passing - small fixes
b9a17ff
@0xNineteen
fix tests
0c11f6d
@0xNineteen
fix lint
b00532f
@0xNineteen 0xNineteen changed the title accounts-db: deserialize bank/accountsdb metadata accounts-db: initial implementation Dec 15, 2023
0xNineteen
0xNineteen commented Dec 15, 2023
View reviewed changes
src/core/accounts_db.zig Outdated Show resolved Hide resolved
0xNineteen and others added 28 commits January 1, 2024 15:28
@0xNineteen
mainnet snapshot loaded and verified in ~20min
7c26d22
@0xNineteen
tests passing
8aedfdd
@0xNineteen
Merge branch 'main' of https://github.com/Syndica/sig into 19/bank
64a2ef8
@0xNineteen
fix logger errs
9955975
@0xNineteen
fix tests
83a3d47
@0xNineteen
fix lint
244ac7b
@0xNineteen
write account logic
17f7325
@0xNineteen
read/write benchmarks
3db8bfc
@0xNineteen
uncomment read benchmark
f30c8af
@0xNineteen
writes are fast now
e039b5d
@0xNineteen
add disk based indexing logic
654784d
@0xNineteen
improve disk based logic (now uses alloc)
91ef726
@0xNineteen
disk indexing working
41e4520
@0xNineteen
use disk indexes on snapshot load
40568e9
@0xNineteen
fix disk refs before trying out hashmap
2216d60
@ultd
resolved conflicts
025158e
@0xNineteen
hash map for account index
0565672
@0xNineteen
add fast map implementation
12b49bb
@0xNineteen
fast snapshot loading with the fast hashmap
89cd9c9
@0xNineteen
Merge branch '19/bank' of https://github.com/Syndica/sig into 19/bank
5f56d08
@0xNineteen
remove c hashmap lib
1e7f9ca
@0xNineteen
parallel untar + zstd decompression
afadc9b
@0xNineteen
reader interface
6a7e60e
@0xNineteen
parallel snapshot unpack working
efe0e16
@0xNineteen
progress report - zstd c impl + some other things (will clean up next…
d1e8ce7 
… commit)
@0xNineteen
reorg + cleanup
5e3d956
@0xNineteen
commit before benchmarks
a0d975c
@0xNineteen
update
5dc667a
@0xNineteen 0xNineteen closed this Feb 29, 2024
@0xNineteen 0xNineteen deleted the 19/bank branch March 3, 2024 08:38
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees

@0xNineteen 0xNineteen

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Initial AccountsDB implementation
2 participants
@0xNineteen @ultd