Lips Syncing anyone?

[jcottier]

Hello,
I have some cool technology that I would like to share. It is about lips syncing. I am writing about it in my blog http://www.ovogame.com/Blog/?p=8. I need to know if it is something that might interest people in the indie devs community.

thanks

JC

[Anthony Flack]

Neat! I assume it's using some variant of voice recognition to analyse the sound waves and assign the proper lip shapes.

I have full lip syncing in my game. In my case all the lip-sync data was done by hand (several hundred lines worth), and for anyone who wants to add lip-sync to their games (and why not, it looks great) - I reckon $150 is totally worth it, to avoid the several days of pain I went through. Even if the results were a little rough and you had to go over it afterwards and tweak it in places, it would still be worth it.

[TimS]

The demo does not run for me... I get an error saying:

Location: .\SysPC\MVVertexBufferSys.cpp(263):

Abort / Retry / Ignore

I'm quite interested to see how it stacks up against Papagayo (http://www.lostmarble.com/papagayo/index.shtml) and similar stuff. If it's really doing all the work for you it'd be quite worth the license fee to me!

-Tim

[Polycount Productions]

Hah, this was really fun. I recorded my own test.wav - and it worked! At first attempt my Mic recording volume was too low... but after adjusting the volume I managed to get the lips sync my voice. I'm impressed, nice job :)

[Polycount Productions]

Btw - you can get this lip sync stuff for $35 (if you give credits/stuff to this page).

[jcottier]

I recorded my own test.wav - and it worked! At first attempt my Mic recording volume was too low... but after adjusting the volume I managed to get the lips sync my voice.

Yes, I forget to tell you that you could create your own samples for testing.

JC

[zoombapup]

Sounds like a cool feature to me.. so yeah, I'd go for a license, wether I used it or not :)

In fact, it would be cool for a "commander" character I've got a plan for :)

[zoombapup]

Hmm.. so its kind of more like a "lip flapping" than lip sync.. i.e. some filter on the sound to take peeks and fit them into a sequence of frames.

But still fun :)

Does it require fmod?

[zoombapup]

Oh, btw.. can it work on a realtime stream?

[soniCron]

Hmm.. so its kind of more like a "lip flapping" than lip sync.. i.e. some filter on the sound to take peeks and fit them into a sequence of frames. I don't understand the distinction between the two? This technology is commonly referred to as automatic lip syncing...

[zoombapup]

lip sync to me, means automatically generating the shape of the mouth from phoneme recognition within the audio.

This is more of a filter that extracts peaks within the audio and changes a variable based on that (which then selects which sprite to show).

One is basically a hell of a lot harder than the other :)

[soniCron]

I guess I misunderstood this guy's technology, then. :o I was under the impression it was doing frequency analysis and adjusting frames accordingly. I didn't realize it was a simple on/off type thing. 'Spose I should have downloaded the demo, huh? ;) Sorry for the misunderstanding! :)

[jcottier]

> The demo does not run for me... I get an error saying:
Shame. Try to update your graphic drivers... Might fix it.

> so its kind of more like a "lip flapping" than lip sync..
No it is not. The 2D demo is not showing the full potential. It is a cut down version of the 3D. I still have a lot of work to make it work. The output is a percentage of few phonemes. So there is nice blinding and no peaks. The "peaks" are just an artifact of the 2D demo.

> can it work on a real-time stream?
I know some similar stuff performing in real time but the lips sync is a little bit dodgy (well it is c..p in comparison ). In my implementation I make the choice of slow but more preferment analysis. The demo is in debug, the progress bar is showing the time it takes to perform the extraction of the data. The final data is about a couple of Ks for small samples. In the final product, the lips data extraction will be an offline process. I might add some parameters for you to tweak but I would like to avoid it. So 100% automatic.

> Does it require fmod?
Nope. The playback of the audio is completely independent from the lips sync libs. The only things you need is to tell me how far you are in your audio sample. For example with FMOD you call FMOD_Channel_GetPosition. You have something similar with Direct Sound (GetCurrentPosition I think). Then I have a function that return some values for each phonemes. Then you do as you wish with these values: (select peaks for 2D), or control some animations or perform a morph target on your 3D models.

JC

[zoombapup]

Ah gotcha, yes, the 2d demo doesnt quite do it justice then. You can get a very similar effect with peak meter filtering for a similar effect.

Having phoneme blends would be FAAAAR more useful and well worth the cost.

[Rainer Deyke]

Neat. I wish I could use it, but at the moment I don't have the budget for voice acting.

[MikeRozak]

If you're looking for lip syncing using speech recognition, download 3D Outside the Box, http://www.mxac.com.au/m3d. It comes with a wave editor that will use SR to identify phonemes, which can then be used to lip sync.

It works okay, but it does a one-to-one mapping of phoneme to mouth shape. (It even has a talking 2D mouth option.) This isn't entirely correct. For a good description of how lip syncing should be done, there's a book I have in my collection... "Stop smiling" or something like that. I can look up the name and author if anyone wants it.

The wave editor saves the phonemes into a RIFF chunk in the wave. I've never heard of any standards, so I made up my own; a sequence of structures with a float time-stamp and 4 WCHARs for the phoneme. If there is a standard, please tell me, and I'll impliment it.

The 3D editor has a head model, and with a little bit of work you can create an animation, piping the wave file with lip-sync information to the head. The mouth of the head will lip sync from the wave file.

-- Edit

I remembered the name: Stop staring: Facial modelling and animation done right, by Jason Opia.

Also, the wave editor has a nifty voice disguise utility that's useful for stretching the budget.

[jcottier]

mine doesn't requieres any editing and is not discreet throught time: meaning you can ask for phonemes values at any time and you will have a nice blending values. Basically it is smooth and doesn't snap from one phonems to an other. You will need to wait for the 3d version to see demo.

JC

[zoombapup]

Cool, more options are always worth having.. definitely will get both of your products when available.

Not that I particularly need lip sync, I just like having an arsenal of tools for all eventualities.

[jcottier]

I am please to announce that TalkToMe SDK is now released.
You can read the news on my blog (http://www.ovogame.com/Blog/?p=8).

This is the link to the TalkToMe website:
www.ovogame.com/TalkToMe.

Feel free to test it and give me feedback. I hope this will be usefull for the indie dev community.

JC