Process only control net (reference_frames)?

[moon47usaco]

I really love this script.

I would like to modify it to not superimpose images from input_dir with images from reference_frames_dir.

I would like it to only process images from reference_frames_dir.

I am using the prompt to create a character then using multi control net to animate the character. I have three directories of animated frames for pose, depth and canny.

With the way the script is so far I can set the superimpose alpha to 1 but the images from input_dir still affect the output.
If the images in the input_dir are white or completely transparent png the characters cloths are drawn white.
I would really just like them ignored completely to generate the first animation with 1 loop controlled only by the prompt and control net.

The plan is to use the output of that one loop of animated character turning back on the input_dir to process another batch with 10 loops.

I can post images to explain better my work flow when I have some time.

Thank you for this script,

R

[moon47usaco]

Here is an image to help describe workflow.
Also a gif of manually generated frames from txt2img, the desired result.

You can see how using dummy images sometimes makes the pose use the image pose rather than the CN info. See ghost feet in Alpha =1 image.

If the script would not interfere with the pose, using dummy images would be fine. Otherwise it would be better to ignore the input frames and only use the txt2img prompt for the CN input.

[fishslot]

I have found the same problem, it seems that even when denoising is 1 the input image still affects the output image. One possible way to avoid this is to use a mask that covers the whole input image, then set the mask content to latent noise and make the denoising 1.

[moon47usaco]

I will give it a try, thank you. =]

[moon47usaco]

Not sure what you mean by "then set the mask content to latent noise"

[moon47usaco]

Hmm... I have used a set of 10 white images in the mask input.
The pose is correct now but there is some interference still as the outfit and front/back orientation are flipping.
To me that tells me it is still adding too much extra information, that is changing the final image.

I think what I need is a script like this but for processing from the txt2img prompt.

Looking through you scrip I understand generally how the images are gathered and the superimposing step but I can not figure out how they are routed into control net.

[fishslot]

It's a common error for SD img2img to suddenly appear with a backside view during a frame of a video, and control net can't fix this completely. This was suppressed by adding words like "face to viewer" to prompt and "from behind" to negative prompt. From personal experience, the best way to curb this is to tag all backside views in the training set when training the model, or not use any backside images at all.

[moon47usaco]

All true. Less concerned with the backwards pose and more the style changes in the uniform. I think in this case the extra info is confusing the Ai and this is why the outfit changes so much and the backwards pose comes in. Compared to the first gif that I did with txt2img or even by hand with img2img.

Again what I really need is to batch the control net directories straight through txt2img.

In the mean time, I think I will use the white input images for now, it seems the most consistent. I can build a good data set from that and color the uniform other ways afterwords.

Thank you agin for this script. I would have to do this by hand otherwise. I would not want to do that for more than 10 frames though. =]

[moon47usaco]

Update on the status. For my record really.

After training a more consistent character first loop: