Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Validation loss #1864

Open
wants to merge 77 commits into
base: sd3
Choose a base branch
from
Open
Changes from 1 commit
Commits
Show all changes
77 commits
Select commit Hold shift + click to select a range
5b19bda
Add validation loss
rockerBOO Nov 5, 2023
33c311e
new ratio code
rockerBOO Nov 5, 2023
3de9e6c
Add validation split of datasets
rockerBOO Nov 5, 2023
a93c524
Update args to validation_seed and validation_split
rockerBOO Nov 5, 2023
c892521
Add process_batch for train_network
rockerBOO Nov 5, 2023
e545fdf
Removed/cleanup a line
rockerBOO Nov 5, 2023
9c591bd
Remove unnecessary subset line from collate
rockerBOO Nov 5, 2023
569ca72
Set grad enabled if is_train and train_text_encoder
rockerBOO Nov 7, 2023
b558a5b
val
gesen2egee Mar 9, 2024
78cfb01
improve
gesen2egee Mar 10, 2024
923b761
Update train_network.py
gesen2egee Mar 10, 2024
47359b8
Update train_network.py
gesen2egee Mar 10, 2024
a51723c
fix timesteps
gesen2egee Mar 11, 2024
7d84ac2
only use train subset to val
gesen2egee Mar 11, 2024
befbec5
Update train_network.py
gesen2egee Mar 11, 2024
63e58f7
Update train_network.py
gesen2egee Mar 11, 2024
a6c41c6
Update train_network.py
gesen2egee Mar 11, 2024
bd7e229
fix
gesen2egee Mar 13, 2024
5d7ed0d
Merge remote-tracking branch 'kohya-ss/dev' into val
gesen2egee Mar 13, 2024
d05965d
Update train_network.py
gesen2egee Mar 13, 2024
b5e8045
fix control net
gesen2egee Mar 16, 2024
086f600
Merge branch 'main' into val
gesen2egee Apr 10, 2024
36d4023
Update config_util.py
gesen2egee Apr 10, 2024
229c5a3
Update train_util.py
gesen2egee Apr 10, 2024
3b251b7
Update config_util.py
gesen2egee Apr 10, 2024
459b125
Update config_util.py
gesen2egee Apr 10, 2024
89ad69b
Update train_util.py
gesen2egee Apr 11, 2024
fde8026
Update config_util.py
gesen2egee Apr 11, 2024
31507b9
Remove unnecessary is_train changes and use apply_debiased_estimation…
gesen2egee Aug 2, 2024
1db4951
Update train_db.py
gesen2egee Aug 4, 2024
6816217
Update train_db.py
gesen2egee Aug 4, 2024
96eb74f
Update train_db.py
gesen2egee Aug 4, 2024
b9bdd10
Update train_network.py
gesen2egee Aug 4, 2024
3d68754
Update train_db.py
gesen2egee Aug 4, 2024
a593e83
Update train_network.py
gesen2egee Aug 4, 2024
f6dbf7c
Update train_network.py
gesen2egee Aug 4, 2024
aa850aa
Update train_network.py
gesen2egee Aug 4, 2024
cdb2d9c
Update train_network.py
gesen2egee Aug 4, 2024
3028027
Update train_network.py
gesen2egee Oct 4, 2024
dece2c3
Update train_db.py
gesen2egee Oct 4, 2024
05bb918
Add Validation loss for LoRA training
hinablue Dec 27, 2024
62164e5
Change val loss calculate method
hinablue Dec 27, 2024
64bd531
Split val latents/batch and pick up val latents shape size which equa…
hinablue Dec 28, 2024
cb89e02
Change val latent loss compare
hinablue Dec 28, 2024
8743532
val
gesen2egee Mar 9, 2024
449c1c5
Adding modified train_util and config_util
rockerBOO Jan 2, 2025
7f6e124
Merge branch 'gesen2egee/val' into validation-loss-upstream
rockerBOO Jan 3, 2025
d23c732
Merge remote-tracking branch 'hina/feature/val-loss' into validation-…
rockerBOO Jan 3, 2025
7470173
Remove defunct code for train_controlnet.py
rockerBOO Jan 3, 2025
534059d
Typos and lingering is_train
rockerBOO Jan 3, 2025
c8c3569
Cleanup order, types, print to logger
rockerBOO Jan 3, 2025
fbfc275
Update text for train/reg with repeats
rockerBOO Jan 3, 2025
58bfa36
Add seed help clarifying info
rockerBOO Jan 3, 2025
6604b36
Remove duplicate assignment
rockerBOO Jan 3, 2025
0522070
Fix training, validation split, revert to using upstream implemenation
rockerBOO Jan 3, 2025
695f389
Move get_huber_threshold_if_needed
rockerBOO Jan 3, 2025
1f9ba40
Add step break for validation epoch. Remove unused variable
rockerBOO Jan 3, 2025
1c0ae30
Add missing functions for training batch
rockerBOO Jan 3, 2025
bbf6bbd
Use self.get_noise_pred_and_target and drop fixed timesteps
rockerBOO Jan 6, 2025
f4840ef
Revert train_db.py
rockerBOO Jan 6, 2025
1c63e7c
Cleanup unused code and formatting
rockerBOO Jan 6, 2025
c64d1a2
Add validate_every_n_epochs, change name validate_every_n_steps
rockerBOO Jan 6, 2025
f885029
Fix validate epoch, cleanup imports
rockerBOO Jan 6, 2025
fcb2ff0
Clean up some validation help documentation
rockerBOO Jan 6, 2025
742bee9
Set validation steps in multiple lines for readability
rockerBOO Jan 6, 2025
1231f51
Remove unused train_util code, fix accelerate.log for wandb, add init…
rockerBOO Jan 8, 2025
556f3f1
Fix documentation, remove unused function, fix bucket reso for sd1.5,…
rockerBOO Jan 8, 2025
9fde0d7
Handle tuple return from generate_dataset_group_by_blueprint
rockerBOO Jan 8, 2025
1e61392
Revert bucket_reso_steps to correct 64
rockerBOO Jan 8, 2025
d6f158d
Fix incorrect destructoring for load_abritrary_dataset
rockerBOO Jan 8, 2025
264167f
Apply is_training_dataset only to DreamBoothDataset. Add validation_s…
rockerBOO Jan 9, 2025
4c61adc
Add divergence to logs
rockerBOO Jan 12, 2025
2bbb40c
Fix regularization images with validation
rockerBOO Jan 12, 2025
0456858
Fix validate_every_n_steps always running first step
rockerBOO Jan 12, 2025
ee9265c
Fix validate_every_n_steps for gradient accumulation
rockerBOO Jan 12, 2025
25929dd
Remove Validating... print to fix output layout
rockerBOO Jan 12, 2025
b489082
Disable repeats for validation datasets
rockerBOO Jan 12, 2025
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Set validation steps in multiple lines for readability
rockerBOO committed Jan 6, 2025

Verified

This commit was signed with the committer’s verified signature.
rockerBOO Dave Lage
commit 742bee9738e9d190a39f5a36adf4515fa415e9b7
8 changes: 6 additions & 2 deletions train_network.py
Original file line number Diff line number Diff line change
@@ -1251,7 +1251,11 @@ def remove_model(old_ckpt_name):
# log empty object to commit the sample images to wandb
accelerator.log({}, step=0)

validation_steps = min(args.max_validation_steps, len(val_dataloader)) if args.max_validation_steps is not None else len(val_dataloader)
validation_steps = (
min(args.max_validation_steps, len(val_dataloader))
if args.max_validation_steps is not None
else len(val_dataloader)
)

# training loop
if initial_step > 0: # only if skip_until_initial_step is specified
@@ -1689,7 +1693,7 @@ def setup_parser() -> argparse.ArgumentParser:
"--validate_every_n_steps",
type=int,
default=None,
help="Run validation on validation dataset every N steps if a validation dataset is available / 検証データセットが利用可能な場合は、Nステップごとに検証データセットの検証を実行します"
help="Run validation on validation dataset every N steps. By default, validation will only occur every epoch if a validation dataset is available / 検証データセットの検証をNステップごとに実行します。デフォルトでは、検証データセットが利用可能な場合にのみ、検証はエポックごとに実行されます"
)
parser.add_argument(
"--validate_every_n_epochs",