All language subtitles for SEGUIMIENTO (David Browne)

Hello there. So, recently I've been

playing around with the W video

generation model here.

Uh, trying a few things out uh

particularly with face tracking in the

last week or so.

I made a video a few days ago. I got

some uh pretty good results. I think I

got a lot of questions in the comments

section about how to do this, where to

find the workflow, that sort of thing.

So, I thought I would do a bit of a

video tutorial here to go over some of

that. Okay. So, here is one version of

the workflow that I'm using.

So, here's the output

that we're going to be uh building here.

And that's being driven by this video

here. Let me just turn this audio off.

Just uh moving my head around a bit.

Looking really goofy.

It tracks very well. Um, the eyes and

the eyebrows, they don't always match

up. I mean, it's coherent in the video

itself. Like, it looks like she's moving

very naturally, but if you compare it

directly with what's going on in the

video here, it it doesn't quite match

up.

So, overall, this uh workflow,

it's just a template workflow from the

the basics here. You got uh essential

stuff here.

Image, video.

This is the one I'm using here.

This is a

image to video workflow

that uses a control video.

So you would load in a reference image

here

and a control video that has the motion

that you want here.

And in the workflow as it is right now

it's passing the image through here.

But that's not what we want to do. Now

by default it's got this canny edge

detector uh control net.

This is not what I'm using in my

workflow.

using a pose estimator instead.

So, what I'll do is I'll recreate what's

going on here

in the demo over here so you can make

some sense of this.

Okay. So, here's the reference image,

a control video.

So, right now it's going through this uh

Cany edge detector. Let me just make

sure some of these settings are okay.

So, it's going to produce a video that's

81 frames long,

720 x 720. We don't want that. Let's

make it uh 480 by 720. So, it's going to

go a bit quicker.

16 frames per second. Let's put that up

to 30.

So, we should see the detection on here.

Now, this is not what we want to use,

but it's still interesting to see what

will happen.

So, let's run this.

Oh, yeah. And while that's run,

okay, maybe I won't run that.

Let's try 41.

Too much memory.

It's doing the poses as well.

So, we can just expand this out. What

it's doing here is it's uh drawing some

edges around the frames of the video.

Got these edge detection frames

that's going to process this over here.

Let's get this out of the way.

So, we're just doing 41 frames right

now. So, this is going to be like a

second and a half.

All right. So, there she is.

It's looking pretty good. That should

match about the first second of what's

going on here.

Now, because it's using uh edge

detection,

it's

not really ideal. This is not really the

ideal way to do

um

sort of a face animation.

I mean it can be helpful. It can be

useful. You can do it this way.

But this one over here is definitely the

better result. This is the pose

detector.

So it's going to give you this kind of

thing.

It's going to estimate the pose for each

frame in the video.

I want to feed that over here.

Now, when we run this again, this should

match up a lot better with what's going

on over here.

Well,

it will be better, but really it's just

a different way of doing kind of the

same thing.

Really the best approach would be to be

layering these and and using both of

them at the same time. To be honest, I'm

not sure how to do that yet. I know in

normal image generation when I'm using

stable diffusion XL or something like

that, I'll usually stack a few of these

like the the canny edge, the pose, uh a

depth pass,

uh normal map pass, and and I'll get

into that in a minute.

Right. So, this is done now. And I think

yeah, the the head motion is a lot

better. It's a lot closer to what's

going on over here.

But there are other ways of doing it

too. So there's also uh you could use

depth,

a normal map,

and there are other ones as well. They

all have slightly different outcomes and

you would use them for different

situations.

So, let's go through some of them right

now and see what they look like.

So, I happen to know that this one here

will give pretty good results, but

they're also kind of strange results.

So, let's have a look at that.

So, this is going to extract

uh sort of 3D information from the scene

that has to do with uh sort of the

incident angle.

You if you uh know what a normal map is,

you'll you'll know what I'm talking

about. My expectation is that it's

probably going to result in her face

being a bit distorted,

maybe more closely resembling my face.

But it probably will transfer my uh

eyebrows and my eyes a bit better. So

maybe some of those micro expressions.

We'll find out. Well, I got this one

processing, too. So, we can have a look

at this. This is going to be a depth

map.

So after we finish uh after we finish

trying this one, we can have a look at

this one to see how that looks when we

want to use it over here.

So it's a it's a depth map.

I got to say I'm a bit uh pleasantly

surprised.

I've done this in the past and it has uh

distorted the face a lot more, but

actually this is working out really

well.

Just have a look over here.

That's doing a pretty good job.

So, we can try out the depth map as

well.

See what that one does.

So, like I was saying, um what I'm used

to doing with image generation is you

have all of these different uh filters

and you can control the strength that

they're going to apply to the image

generation.

So, you don't have to choose between

which one you want to use. you have like

a a little bit of this, a little bit of

that, a lot of this, a little bit of

that, or or whatever the proportions

happen to be, and you can sort of

fine-tune how much you want to process.

But right now, I haven't figured out how

to do that with video.

All right, so using the depth map,

um, it's b she's basically not following

my head movements at all.

And probably most of the motion that's

being generated here is just all on on

the part of the model making up some

some stuff.

So it it's going to adhere the motion to

sort of be within the figure ground

relationship of the depth map. But

that's probably the only thing that's

really going to be happening with this

if I had to guess.

So this seems to be the way to go.

That is the uh DW pose estimator.

Yeah, much better.

So, that's essentially the workflow I'm

using here.

Um, additionally, um,

for those interested, I thought I might

talk a bit about how, uh, I created this

image.

Now, this is a very elementary

workflow for doing image generation with

stable diffusion XL using the juggernaut

XL checkpoint which is excellent

and over here I have uh Laura

a custom model that I've trained on my

character so that I can always generate

images of the same character

my prompt over here and This this here

is the keyword I'm using that's

triggering this Laura to sort of

activate so it knows to generate my

character specifically.

So this is what we're generating here.

Just some settings in the

sampler. This is sort of rendering I'm

calling it.

Here's our output image. Now I'm going

to do a This isn't really necessary

because the output on its own is

actually really good, but a lot of the

time this may not be the case. You might

get some distortions in the face. It's

fairly typical for this to happen.

So there's a good reason to do a face

detailing pass afterwards. So this is

the the first output and this is the

output with the face detailer. Like here

they look almost identical. So this is

not a great example actually but

this is useful sometimes and over here

I'm just saving it

but let me go through a new generation

here so we can have a look at this I

just move through the seed number

load the checkpoint

it's going to pass through to the

sampler

right so there she

Now we're going to do some face

detailing.

All right. Again, it's almost

indistinguishable. So, probably this is

not necessary right now. The the

checkpoint is very good and the Laura is

pretty well trained. So, this whole step

is probably not necessary.

Let's try giving her a space suit.

So, for training the Laura, I have a

folder of about I don't know 250 images

or so. Uh just character images.

So, to do the training, I used Koya SS.

That's it here.

There's a whole lot of instructions on

how to work with this.

It worked very well.

I think it took about I want to say

about 3 hours to do the training.

Now, typically

training a Laura like this for a

character, I don't know. I guess you

need 30 or 50 images or so or something

like that. So, I I was using about 250,

which is probably overkill. It's way too

many,

but it seems to work pretty well.

Getting uh very good consistent results.

All right, lastly, uh so I've just

generated uh an image here. Let's bring

that back over to the workflow here.

Try that one out.

We'll do

Make this one longer.

So, this could take a minute or two.

It's reprocessing the pose.

It's probably loading the checkpoint

now.

Okay. Okay. So, while this is loading,

because this is going to take a minute

or two,

there are two different versions of this

model. There's a 14 billion parameter

model and a 1.3 billion model. So, this

one takes a lot more processing power.

And then even within that, there are

different versions for 720p and 480p.

This Laura here apparently increases the

speed

by reducing the number of steps and the

CFG required to do the sampling.

I'm not really sure how that works.

I just know that it does.

All right, this is almost done.

So for me here, I'm making an image

that's or video that's 720x 480 uh

vertical and 121 frames. And that's

taking about 2 minutes maybe.

I had to guess maybe a minute and a

half. It's pretty snappy.

30 frames per second.

All right. So, here's the video that's

come out of this.

And that's worked out pretty well.

Now, you can set the resolution quite a

bit higher, as much as I don't know,

about 1280

by 720 with this model. And you can make

it about as long as, I don't know, maybe

210 frames. 209, let's say,

and it will still work, but that's about

the limit of of what I've been able to

do anyways. And it will take a lot

longer to do with those kinds of

settings.

Well, that's all for this video. I hope

that was interesting, uh, helpful.

Uh, let me know in the comments.

[Music]