Lips Syncing anyone?

Discussion in 'Game Development (Technical)' started by jcottier, Oct 20, 2006.

  1. jcottier

    jcottier
    Expand Collapse
    New Member

    Joined:
    Jul 12, 2006
    Messages:
    1,386
    Likes Received:
    0
    Hello,
    I have some cool technology that I would like to share. It is about lips syncing. I am writing about it in my blog http://www.ovogame.com/Blog/?p=8. I need to know if it is something that might interest people in the indie devs community.

    thanks

    JC
     
    #1 jcottier, Oct 20, 2006
    Last edited: Nov 5, 2006
  2. Anthony Flack

    Anthony Flack
    Expand Collapse
    Indie Author

    Joined:
    Jul 27, 2004
    Messages:
    2,178
    Likes Received:
    0
    Neat! I assume it's using some variant of voice recognition to analyse the sound waves and assign the proper lip shapes.

    I have full lip syncing in my game. In my case all the lip-sync data was done by hand (several hundred lines worth), and for anyone who wants to add lip-sync to their games (and why not, it looks great) - I reckon $150 is totally worth it, to avoid the several days of pain I went through. Even if the results were a little rough and you had to go over it afterwards and tweak it in places, it would still be worth it.
     
  3. TimS

    TimS
    Expand Collapse
    Original Member

    Joined:
    Feb 9, 2005
    Messages:
    684
    Likes Received:
    0
    The demo does not run for me... I get an error saying:

    I'm quite interested to see how it stacks up against Papagayo and similar stuff. If it's really doing all the work for you it'd be quite worth the license fee to me!

    -Tim
     
  4. Game Producer

    Game Producer
    Expand Collapse
    Moderator Original Member

    Joined:
    Jan 13, 2006
    Messages:
    1,418
    Likes Received:
    0
    Hah, this was really fun. I recorded my own test.wav - and it worked! At first attempt my Mic recording volume was too low... but after adjusting the volume I managed to get the lips sync my voice. I'm impressed, nice job :)
     
  5. Game Producer

    Game Producer
    Expand Collapse
    Moderator Original Member

    Joined:
    Jan 13, 2006
    Messages:
    1,418
    Likes Received:
    0
    Btw - you can get this lip sync stuff for $35 (if you give credits/stuff to this page).
     
  6. jcottier

    jcottier
    Expand Collapse
    New Member

    Joined:
    Jul 12, 2006
    Messages:
    1,386
    Likes Received:
    0
    Yes, I forget to tell you that you could create your own samples for testing.

    JC
     
  7. zoombapup

    zoombapup
    Expand Collapse
    Moderator Original Member

    Joined:
    Nov 25, 2004
    Messages:
    2,890
    Likes Received:
    0
    Sounds like a cool feature to me.. so yeah, I'd go for a license, wether I used it or not :)

    In fact, it would be cool for a "commander" character I've got a plan for :)
     
  8. zoombapup

    zoombapup
    Expand Collapse
    Moderator Original Member

    Joined:
    Nov 25, 2004
    Messages:
    2,890
    Likes Received:
    0
    Hmm.. so its kind of more like a "lip flapping" than lip sync.. i.e. some filter on the sound to take peeks and fit them into a sequence of frames.

    But still fun :)

    Does it require fmod?
     
  9. zoombapup

    zoombapup
    Expand Collapse
    Moderator Original Member

    Joined:
    Nov 25, 2004
    Messages:
    2,890
    Likes Received:
    0
    Oh, btw.. can it work on a realtime stream?
     
  10. soniCron

    soniCron
    Expand Collapse
    Indie Author

    Joined:
    May 4, 2005
    Messages:
    3,664
    Likes Received:
    0
    I don't understand the distinction between the two? This technology is commonly referred to as automatic lip syncing...
     
  11. zoombapup

    zoombapup
    Expand Collapse
    Moderator Original Member

    Joined:
    Nov 25, 2004
    Messages:
    2,890
    Likes Received:
    0
    lip sync to me, means automatically generating the shape of the mouth from phoneme recognition within the audio.

    This is more of a filter that extracts peaks within the audio and changes a variable based on that (which then selects which sprite to show).

    One is basically a hell of a lot harder than the other :)
     
    #11 zoombapup, Oct 21, 2006
    Last edited: Oct 21, 2006
  12. soniCron

    soniCron
    Expand Collapse
    Indie Author

    Joined:
    May 4, 2005
    Messages:
    3,664
    Likes Received:
    0
    I guess I misunderstood this guy's technology, then. :eek: I was under the impression it was doing frequency analysis and adjusting frames accordingly. I didn't realize it was a simple on/off type thing. 'Spose I should have downloaded the demo, huh? ;) Sorry for the misunderstanding! :)
     
  13. jcottier

    jcottier
    Expand Collapse
    New Member

    Joined:
    Jul 12, 2006
    Messages:
    1,386
    Likes Received:
    0
    > The demo does not run for me... I get an error saying:
    Shame. Try to update your graphic drivers... Might fix it.

    > so its kind of more like a "lip flapping" than lip sync..
    No it is not. The 2D demo is not showing the full potential. It is a cut down version of the 3D. I still have a lot of work to make it work. The output is a percentage of few phonemes. So there is nice blinding and no peaks. The "peaks" are just an artifact of the 2D demo.

    > can it work on a real-time stream?
    I know some similar stuff performing in real time but the lips sync is a little bit dodgy (well it is c..p in comparison ). In my implementation I make the choice of slow but more preferment analysis. The demo is in debug, the progress bar is showing the time it takes to perform the extraction of the data. The final data is about a couple of Ks for small samples. In the final product, the lips data extraction will be an offline process. I might add some parameters for you to tweak but I would like to avoid it. So 100% automatic.

    > Does it require fmod?
    Nope. The playback of the audio is completely independent from the lips sync libs. The only things you need is to tell me how far you are in your audio sample. For example with FMOD you call FMOD_Channel_GetPosition. You have something similar with Direct Sound (GetCurrentPosition I think). Then I have a function that return some values for each phonemes. Then you do as you wish with these values: (select peaks for 2D), or control some animations or perform a morph target on your 3D models.

    JC
     
  14. zoombapup

    zoombapup
    Expand Collapse
    Moderator Original Member

    Joined:
    Nov 25, 2004
    Messages:
    2,890
    Likes Received:
    0
    Ah gotcha, yes, the 2d demo doesnt quite do it justice then. You can get a very similar effect with peak meter filtering for a similar effect.

    Having phoneme blends would be FAAAAR more useful and well worth the cost.
     
  15. Rainer Deyke

    Rainer Deyke
    Expand Collapse
    Indie Author

    Joined:
    Jul 28, 2004
    Messages:
    380
    Likes Received:
    0
    Neat. I wish I could use it, but at the moment I don't have the budget for voice acting.
     
  16. MikeRozak

    MikeRozak
    Expand Collapse
    Original Member

    Joined:
    Feb 4, 2006
    Messages:
    37
    Likes Received:
    0
    If you're looking for lip syncing using speech recognition, download 3D Outside the Box, http://www.mxac.com.au/m3d. It comes with a wave editor that will use SR to identify phonemes, which can then be used to lip sync.

    It works okay, but it does a one-to-one mapping of phoneme to mouth shape. (It even has a talking 2D mouth option.) This isn't entirely correct. For a good description of how lip syncing should be done, there's a book I have in my collection... "Stop smiling" or something like that. I can look up the name and author if anyone wants it.

    The wave editor saves the phonemes into a RIFF chunk in the wave. I've never heard of any standards, so I made up my own; a sequence of structures with a float time-stamp and 4 WCHARs for the phoneme. If there is a standard, please tell me, and I'll impliment it.

    The 3D editor has a head model, and with a little bit of work you can create an animation, piping the wave file with lip-sync information to the head. The mouth of the head will lip sync from the wave file.

    -- Edit

    I remembered the name: Stop staring: Facial modelling and animation done right, by Jason Opia.

    Also, the wave editor has a nifty voice disguise utility that's useful for stretching the budget.
     
    #16 MikeRozak, Oct 21, 2006
    Last edited: Oct 21, 2006
  17. jcottier

    jcottier
    Expand Collapse
    New Member

    Joined:
    Jul 12, 2006
    Messages:
    1,386
    Likes Received:
    0
    mine doesn't requieres any editing and is not discreet throught time: meaning you can ask for phonemes values at any time and you will have a nice blending values. Basically it is smooth and doesn't snap from one phonems to an other. You will need to wait for the 3d version to see demo.

    JC
     
  18. zoombapup

    zoombapup
    Expand Collapse
    Moderator Original Member

    Joined:
    Nov 25, 2004
    Messages:
    2,890
    Likes Received:
    0
    Cool, more options are always worth having.. definitely will get both of your products when available.

    Not that I particularly need lip sync, I just like having an arsenal of tools for all eventualities.
     
  19. jcottier

    jcottier
    Expand Collapse
    New Member

    Joined:
    Jul 12, 2006
    Messages:
    1,386
    Likes Received:
    0
    I am please to announce that TalkToMe SDK is now released.
    You can read the news on my blog (http://www.ovogame.com/Blog/?p=8).

    This is the link to the TalkToMe website:
    www.ovogame.com/TalkToMe.

    Feel free to test it and give me feedback. I hope this will be usefull for the indie dev community.

    JC
     

Share This Page