Separate inference 720p video with 24G VRAM

[narrowsnap]

Add VAE encoder for reference.

Reduce inference VRAM by separate process:

1. Run text_encoder and save text embeding.
2. Run VAE encoder if reference_path in prompt.(optional)
3. Run STDiT with saved text embeding and save latents.
4. Run VAE decoder with saved latents.

[FrankLeeeee]

I am aware of your motivation, but can you add more doc to tell other users why, when and how to run inference separately so that they can feel more guided?

[narrowsnap]

Done.

[FrankLeeeee]

This will by default use 2 GPUs, can you make this configurable via bash argument as well?

[tpc2233]

I tried you 24gb vram code @narrowsnap but im getting the issue on inference_stdit.py with caption_embs=caption_embs, caption_emb_masks=caption_emb_masks where fails with AttributeError: 'NoneType' object has no attribute 'encode' during inference. Rest of steps seams ok.

[narrowsnap]

caption_emb_masks

Are you update the code of RFLOW?(opensora/schedulers/rf/__init__.py)

[tpc2233]

caption_emb_masks

Are you update the code of RFLOW?(opensora/schedulers/rf/init.py)

Yes, i even tried git clone your fork, but no luck. This one right?
https://github.com/narrowsnap/Open-Sora/blob/main/opensora/schedulers/rf/__init__.py

[narrowsnap]

caption_emb_masks

Are you update the code of RFLOW?(opensora/schedulers/rf/init.py)

Yes, i even tried git clone your fork, but no luck. This one right? https://github.com/narrowsnap/Open-Sora/blob/main/opensora/schedulers/rf/__init__.py

This is wrong! You need to use feature/720p_for_16g branch.

[tpc2233]

This is wrong! You need to use feature/720p_for_16g branch.

Sorry that is what i meant. Yes using from there:
https://github.com/narrowsnap/Open-Sora/blob/feature/720p_for_16g/opensora/schedulers/rf/__init__.py

It gets:
TypeError: sample() got an unexpected keyword argument 'caption_embs'
scripts/separate_inference/inference_stdit.py FAILED
Resulting on
FileNotFoundError: [Errno 2] No such file or directory: './samples/samples/2024-08-15/00002/0_0_latents.pt'

Also, many thanks for the quick replies, much appreaciated

[narrowsnap]

This is wrong! You need to use feature/720p_for_16g branch.

Sorry that is what i meant. Yes using from there: https://github.com/narrowsnap/Open-Sora/blob/feature/720p_for_16g/opensora/schedulers/rf/__init__.py

It gets: TypeError: sample() got an unexpected keyword argument 'caption_embs' scripts/separate_inference/inference_stdit.py FAILED Resulting on FileNotFoundError: [Errno 2] No such file or directory: './samples/samples/2024-08-15/00002/0_0_latents.pt'

Also, many thanks for the quick replies, much appreaciated

What is the command you used?

[tpc2233]

This is wrong! You need to use feature/720p_for_16g branch.

Sorry that is what i meant. Yes using from there: https://github.com/narrowsnap/Open-Sora/blob/feature/720p_for_16g/opensora/schedulers/rf/__init__.py
It gets: TypeError: sample() got an unexpected keyword argument 'caption_embs' scripts/separate_inference/inference_stdit.py FAILED Resulting on FileNotFoundError: [Errno 2] No such file or directory: './samples/samples/2024-08-15/00002/0_0_latents.pt'
Also, many thanks for the quick replies, much appreaciated

What is the command you used?

From root on the fork, im just running: bash ./scripts/separate_inference.sh

[narrowsnap]

This is wrong! You need to use feature/720p_for_16g branch.

Sorry that is what i meant. Yes using from there: https://github.com/narrowsnap/Open-Sora/blob/feature/720p_for_16g/opensora/schedulers/rf/__init__.py
It gets: TypeError: sample() got an unexpected keyword argument 'caption_embs' scripts/separate_inference/inference_stdit.py FAILED Resulting on FileNotFoundError: [Errno 2] No such file or directory: './samples/samples/2024-08-15/00002/0_0_latents.pt'
Also, many thanks for the quick replies, much appreaciated

What is the command you used?

From root on the fork, im just running: bash ./scripts/separate_inference.sh

I can successfully run it. According to the error you showed, I suggest you check if there is caption_embs in your code.[opensora/schedulers/rf/init.py line 45]

[tpc2233]

I can successfully run it. According to the error you showed, I suggest you check if there is caption_embs in your code.[opensora/schedulers/rf/init.py line 45]

Made work:) solution was delete all the instalations and cond env and use only your fork to intall. After you said was working, i tried delete the rf/init and still got the same issue, so i think was some type of caching or something referecing to the original instalation. After all re-installs. Worked. many thanks for quick replies and help @narrowsnap Great work.

[Luke100000]

Hi, is it possible to further squash VRAM usage to get it running on 12GB? :)
Right now, the T5 encoder has the highest spike. Running it on CPU (only the text encoder) allows me to generate a 720p 3s video. And smaller gens would fit into 8GB just fine as well.