Train Multi face with train_dreambooth.py on SD model #3175

[hosein-moayedi]

Describe the bug

Hi there!
I have an issue with training the SD model with train_dreambooth.py.
I am doing these steps:

Step 1:
Downloaded the Stable Diffusion from hugging face and using from train_dreambooth.py script to start to train my own face with some parameters and special SEED.

Step 2:
After training, I saved the result as a .ckpt file on my local.

Step 3:
Using that result model (.ckpt file that I saved in step 2) as a pre-trained model on my train_dreambooth.py parameter and give the dir of my friend's face to train that on my model with different SEED.

Step 4:
After finishing the training process I saved the new result as a .ckpt file and start using from that and give prompts to generate an image of faces.

Problem:
When I give the friend's name to generate an image from his face, I will see the right image of his face without any issues but when I try to write the prompt of generate an image from my face, I will see the generated image was 90% look like of my friend's face and 10% look like me!

Note: "I defined my face name as a special uuid version 4 and my friend's face has a different uuid"

I think the last train is overwriting on my previous train!!

How can I fix this issue?

Reproduction

this first train params:

!accelerate launch train_dreambooth.py
--pretrained_model_name_or_path="runwayml/stable-diffusion-v1-5"
--pretrained_vae_name_or_path="stabilityai/sd-vae-ft-mse"
--output_dir=$OUTPUT_DIR
--revision="fp16"
--with_prior_preservation --prior_loss_weight=0.1
--seed=1337 \ ---->>>> Different SEED
--resolution=512
--train_batch_size=1
--train_text_encoder
--mixed_precision="fp16"
--use_8bit_adam
--gradient_accumulation_steps=1
--learning_rate=1e-7
--lr_scheduler="constant"
--lr_warmup_steps=80
--num_class_images=120
--sample_batch_size=4
--max_train_steps=800
--save_interval=10000
--save_sample_prompt="05b62e68-8513-4779-9309-5643f27f288a"
--concepts_list="concepts_list.json"

The second training params:

!accelerate launch train_dreambooth.py
--pretrained_model_name_or_path="path of .ckpt result from last train."
--pretrained_vae_name_or_path="stabilityai/sd-vae-ft-mse"
--output_dir=$OUTPUT_DIR
--revision="fp16"
--with_prior_preservation --prior_loss_weight=0.1
--seed=9385 . ---->>>> Different SEED
--resolution=512
--train_batch_size=1
--train_text_encoder
--mixed_precision="fp16"
--use_8bit_adam
--gradient_accumulation_steps=1
--learning_rate=1e-7
--lr_scheduler="constant"
--lr_warmup_steps=80
--num_class_images=120
--sample_batch_size=4
--max_train_steps=800
--save_interval=10000
--save_sample_prompt="05b62e68-8513-4779-9309-5643f27f288a"
--concepts_list="concepts_list.json"

Logs

No response

System Info

Nvidia Tesla T4
RAM: 12 Gig
CUDA: 11.3
CPU: 2 Cores

[hosein-moayedi]

@ShivamShrirao ShivamShrirao

[ShivamShrirao]

It will overwrite, you can't train sequentially. Try training both together with very different looking instance tokens and prompts. Training multiple subjects doesn't mostly work, you can just try and experiment with various settings.

[hosein-moayedi]

@ShivamShrirao
Thanks for the replay :)
Can you help me more?

Can you suggest to me the best solution for this:

I am creating a mobile app that has these steps:

1. New users should register on my app with their information.
2. They should send 10 images of his face to my back-end server and I should generate a new UUID (version 4) specific for that user as a special prompt and start to train my main model with his face images.

Now users can generate any AI images with their faces just by calling their UUID prompts with style.

I want to do this for all users that register their information at different times and I want to generate the face of all my users with one model and using from just user UUID.

First question)
How can I do that?

Second question)
has a bad effect on the main model in the future after training faces of all my users?

[hosein-moayedi]

And other question
If we can not train the model sequentially, So how some merges of Stable diffusion like the open-journey model, Realistic-Vision model, or Deliberate model that are stable diffusion based trained with special instances or data to get a better result in a special type of image content?!

Also can you tell me what is the main difference between these files? Are you suggest me to using from Lora replace than dream booth or no (like bad result reason)?

train_dreambooth_lora.py
train_text_to_image_lora.py