Results 1 to 19 of 19

Thread: Lips Syncing anyone?

  1. #1
    Senior Member
    Join Date
    Jul 2006
    Location
    France
    Posts
    1,388

    Default Lips Syncing anyone?

    Hello,
    I have some cool technology that I would like to share. It is about lips syncing. I am writing about it in my blog http://www.ovogame.com/Blog/?p=8. I need to know if it is something that might interest people in the indie devs community.

    thanks

    JC
    Last edited by jcottier; 11-05-2006 at 03:36 PM. Reason: update to the newest post

  2. #2
    Senior Member
    Join Date
    Jul 2004
    Posts
    2,179

    Default

    Neat! I assume it's using some variant of voice recognition to analyse the sound waves and assign the proper lip shapes.

    I have full lip syncing in my game. In my case all the lip-sync data was done by hand (several hundred lines worth), and for anyone who wants to add lip-sync to their games (and why not, it looks great) - I reckon $150 is totally worth it, to avoid the several days of pain I went through. Even if the results were a little rough and you had to go over it afterwards and tweak it in places, it would still be worth it.
    Anthony
    www.squashysoftware.com
    A slave to the ideal

  3. #3
    Senior Member
    Join Date
    Feb 2005
    Location
    Chicago, IL
    Posts
    674

    Default

    The demo does not run for me... I get an error saying:

    Location: .\SysPC\MVVertexBufferSys.cpp(263):

    Abort / Retry / Ignore
    I'm quite interested to see how it stacks up against Papagayo and similar stuff. If it's really doing all the work for you it'd be quite worth the license fee to me!

    -Tim
    Nightmare Adventures: The Witch's Prison is now available on PC, Mac, iPad, and iPhone.

  4. #4
    Moderator
    Join Date
    Jan 2006
    Location
    Finland
    Posts
    1,422

    Default

    Hah, this was really fun. I recorded my own test.wav - and it worked! At first attempt my Mic recording volume was too low... but after adjusting the volume I managed to get the lips sync my voice. I'm impressed, nice job

  5. #5
    Moderator
    Join Date
    Jan 2006
    Location
    Finland
    Posts
    1,422

    Default

    Btw - you can get this lip sync stuff for $35 (if you give credits/stuff to this page).

  6. #6
    Senior Member
    Join Date
    Jul 2006
    Location
    France
    Posts
    1,388

    Default

    Quote Originally Posted by Polycount Productions View Post
    I recorded my own test.wav - and it worked! At first attempt my Mic recording volume was too low... but after adjusting the volume I managed to get the lips sync my voice.
    Yes, I forget to tell you that you could create your own samples for testing.

    JC

  7. #7
    Moderator
    Join Date
    Nov 2004
    Posts
    2,896

    Default

    Sounds like a cool feature to me.. so yeah, I'd go for a license, wether I used it or not

    In fact, it would be cool for a "commander" character I've got a plan for
    www.mindflock.com - social AI-based games

  8. #8
    Moderator
    Join Date
    Nov 2004
    Posts
    2,896

    Default

    Hmm.. so its kind of more like a "lip flapping" than lip sync.. i.e. some filter on the sound to take peeks and fit them into a sequence of frames.

    But still fun

    Does it require fmod?
    www.mindflock.com - social AI-based games

  9. #9
    Moderator
    Join Date
    Nov 2004
    Posts
    2,896

    Default

    Oh, btw.. can it work on a realtime stream?
    www.mindflock.com - social AI-based games

  10. #10
    Senior Member
    Join Date
    May 2005
    Location
    USA
    Posts
    3,664

    Default

    Quote Originally Posted by zoombapup View Post
    Hmm.. so its kind of more like a "lip flapping" than lip sync.. i.e. some filter on the sound to take peeks and fit them into a sequence of frames.
    I don't understand the distinction between the two? This technology is commonly referred to as automatic lip syncing...

  11. #11
    Moderator
    Join Date
    Nov 2004
    Posts
    2,896

    Default

    lip sync to me, means automatically generating the shape of the mouth from phoneme recognition within the audio.

    This is more of a filter that extracts peaks within the audio and changes a variable based on that (which then selects which sprite to show).

    One is basically a hell of a lot harder than the other
    Last edited by zoombapup; 10-21-2006 at 10:25 AM. Reason: mouTH
    www.mindflock.com - social AI-based games

  12. #12
    Senior Member
    Join Date
    May 2005
    Location
    USA
    Posts
    3,664

    Default

    I guess I misunderstood this guy's technology, then. I was under the impression it was doing frequency analysis and adjusting frames accordingly. I didn't realize it was a simple on/off type thing. 'Spose I should have downloaded the demo, huh? Sorry for the misunderstanding!

  13. #13
    Senior Member
    Join Date
    Jul 2006
    Location
    France
    Posts
    1,388

    Default

    > The demo does not run for me... I get an error saying:
    Shame. Try to update your graphic drivers... Might fix it.

    > so its kind of more like a "lip flapping" than lip sync..
    No it is not. The 2D demo is not showing the full potential. It is a cut down version of the 3D. I still have a lot of work to make it work. The output is a percentage of few phonemes. So there is nice blinding and no peaks. The "peaks" are just an artifact of the 2D demo.

    > can it work on a real-time stream?
    I know some similar stuff performing in real time but the lips sync is a little bit dodgy (well it is c..p in comparison ). In my implementation I make the choice of slow but more preferment analysis. The demo is in debug, the progress bar is showing the time it takes to perform the extraction of the data. The final data is about a couple of Ks for small samples. In the final product, the lips data extraction will be an offline process. I might add some parameters for you to tweak but I would like to avoid it. So 100% automatic.

    > Does it require fmod?
    Nope. The playback of the audio is completely independent from the lips sync libs. The only things you need is to tell me how far you are in your audio sample. For example with FMOD you call FMOD_Channel_GetPosition. You have something similar with Direct Sound (GetCurrentPosition I think). Then I have a function that return some values for each phonemes. Then you do as you wish with these values: (select peaks for 2D), or control some animations or perform a morph target on your 3D models.

    JC

  14. #14
    Moderator
    Join Date
    Nov 2004
    Posts
    2,896

    Default

    Ah gotcha, yes, the 2d demo doesnt quite do it justice then. You can get a very similar effect with peak meter filtering for a similar effect.

    Having phoneme blends would be FAAAAR more useful and well worth the cost.
    www.mindflock.com - social AI-based games

  15. #15
    Senior Member
    Join Date
    Jul 2004
    Posts
    380

    Default

    Neat. I wish I could use it, but at the moment I don't have the budget for voice acting.
    Rainer Deyke - Eldwood

  16. #16
    Member
    Join Date
    Feb 2006
    Location
    Darwin, NT Australia
    Posts
    37

    Default

    If you're looking for lip syncing using speech recognition, download 3D Outside the Box, http://www.mxac.com.au/m3d. It comes with a wave editor that will use SR to identify phonemes, which can then be used to lip sync.

    It works okay, but it does a one-to-one mapping of phoneme to mouth shape. (It even has a talking 2D mouth option.) This isn't entirely correct. For a good description of how lip syncing should be done, there's a book I have in my collection... "Stop smiling" or something like that. I can look up the name and author if anyone wants it.

    The wave editor saves the phonemes into a RIFF chunk in the wave. I've never heard of any standards, so I made up my own; a sequence of structures with a float time-stamp and 4 WCHARs for the phoneme. If there is a standard, please tell me, and I'll impliment it.

    The 3D editor has a head model, and with a little bit of work you can create an animation, piping the wave file with lip-sync information to the head. The mouth of the head will lip sync from the wave file.

    -- Edit

    I remembered the name: Stop staring: Facial modelling and animation done right, by Jason Opia.

    Also, the wave editor has a nifty voice disguise utility that's useful for stretching the budget.
    Last edited by MikeRozak; 10-21-2006 at 02:46 PM. Reason: Remembered the name

  17. #17
    Senior Member
    Join Date
    Jul 2006
    Location
    France
    Posts
    1,388

    Default

    mine doesn't requieres any editing and is not discreet throught time: meaning you can ask for phonemes values at any time and you will have a nice blending values. Basically it is smooth and doesn't snap from one phonems to an other. You will need to wait for the 3d version to see demo.

    JC

  18. #18
    Moderator
    Join Date
    Nov 2004
    Posts
    2,896

    Default

    Cool, more options are always worth having.. definitely will get both of your products when available.

    Not that I particularly need lip sync, I just like having an arsenal of tools for all eventualities.
    www.mindflock.com - social AI-based games

  19. #19
    Senior Member
    Join Date
    Jul 2006
    Location
    France
    Posts
    1,388

    Default

    I am please to announce that TalkToMe SDK is now released.
    You can read the news on my blog (http://www.ovogame.com/Blog/?p=8).

    This is the link to the TalkToMe website:
    www.ovogame.com/TalkToMe.

    Feel free to test it and give me feedback. I hope this will be usefull for the indie dev community.

    JC

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •