Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add rank_stabilized for networks #1870

Open
wants to merge 2 commits into
base: sd3
Choose a base branch
from

Conversation

rockerBOO
Copy link
Contributor

@rockerBOO rockerBOO commented Jan 10, 2025

https://arxiv.org/abs/2312.03732
https://huggingface.co/blog/damjan-k/rslora

Screenshot 2025-01-09 at 22-08-13 A Rank Stabilization Scaling Factor for Fine-Tuning with LoRA - 2312 03732v1 pdf
Screenshot 2025-01-09 at 22-08-00 A Rank Stabilization Scaling Factor for Fine-Tuning with LoRA - 2312 03732v1 pdf

Inspired by KohakuBlueleaf/LyCORIS#172

Usage

network_args=["rank_stabilized=True"]
--network_args rank_stabilized=True

Maybe rs_lora would be better to replicate what is on LyCORIS but I thought the full name would be clearer when viewing configuration.

@recris
Copy link

recris commented Jan 10, 2025

Does the scale in apply_max_norm_regularization also have to be changed to work correctly?

@rockerBOO
Copy link
Contributor Author

@recris good point, there is a scale value in there. I will update it and test.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants