AS3 based dynamic lip sync: MobileWish LipSyncAS3 soon to be out.

I have created a poc component (named it as “MobileWish LipSyncAS3”) for dynamic  lip sync using ActionScript 3 in Flash CS3. This beta version matches the lip sync from any of the supplied MP3 file containing pure speech. This can be handy tool for adding lip sync for flash characters in any dynamic interactive.  I had been using third party tools like Flash Amp for lip sync in some of my previous projects. But that involved pre processing of audio mp3 files for creating amplitude value. Now with AS3 there is no need to preprocess the mp3 files, rather the lip sync will be instantly done on the run time which will be an added advantage.

Given below is the sample video of my working prototype at Youtube.

I will finetune it and will soon be uploading it to MobileWish website (www.mobilewish.com)

If interested please add your names by replying to this post.

Advertisements

37 thoughts on “AS3 based dynamic lip sync: MobileWish LipSyncAS3 soon to be out.

  1. Arrrggghh!!! – That was to be MY next project!

    I have an old Central application called e2animate that I was going to port to AIR. (I mentioned it recently in my blog http://e2easy.wordpress.com ). I was going to turn it into an easy fun animation app (aimed at non technical users), with the ability to customise your own mouth shapes, and drive them automatically using sound files, or perhaps input from the microphone.

    I was going to look at using prediction gain over a set of fixed filters, or normalised multiplying frequency bin values with a set of patterns to determine the closest match to a particular sound (A,I/E/U/L/W,Q/M, fricatives, etc.) It was going to take some experimentation, but from the nearest sound, I’d be able to dynamically choose mouth shapes.

    Like you, originally, I was going to extract the data from MP3 files. MP3 is just a sub-band coder after all, which uses frequency bin values. And like you, I discovered that AS3 had all I needed. But I was also going to look into Flash 10 player features, and I think someone said that pixelbender could be used for sound processing – I was going to investigate.

    Obviously, amplitude determines the size of the mouth.

    If you did give away your product for free – you would save me a lot of work – but I would be a little disappointed too, at not having done it myself.

  2. Pingback: Automatic lip synch. « e2easy AIR applications

  3. Pingback: Fredericton Designers » Blog Archive » AS3 based dynamic lip sync: MobileWish LipSyncAS3 soon to be out.

  4. I worked for a company called “LIPSinc” whose product “Animeter” output phonemes based on audio files you sent to a server-side program. The result was a text string which I split into an array in Flash. The details are fuzzy now, but I recall a Flash library item that consisted of mouth positions with frame labels that corresponded to the text string’s phonemes. To make a long story short, we were able to make a character talk using this technology. The guy who owned the technology is Donovan Moxy last I heard. Anyway this may be worht checking out.

  5. Hi Daniel,

    Its good to hear from you. Its not that the way I am trying to build is the better way. There may be several other ways to build the same thing…and the idea is we all should try to add value to a concept by continously trying to improve it so that users will get a list of options to choose from.

    regards
    Samir

  6. I hadn’t started working on this yet, so it’s good to have your off-the-shelf solution. If yours is free with no licensing complications – I’ll just use yours. Lip synch is just one aspect of the AIR animation app I want to build. If I don’t have to spend time on the lip synch – I can concentrate more on other features.

    Can you describe your algorithm?

    Yes, there are several ways. I described prediction gain and frequency bin matching on my blog. Even a simple zero-crossing algorithm might yield something useful.

    Mike: Text to speech, and concatenated speech are a whole different ball game.

  7. Hy,

    I’m very interested to see the how LipSyncAS3 works.

    I’m also trying to achive it width the computer spectrum analysis … and love to share or help in developing

    thanks, ciao
    filippo

  8. Pingback: LipSyncAS3 Demo « SAMIR

  9. Thank you, Samir.

    Lip sync is time consuming, tedious work. It has prevented many amateur animators (myself included) from turning thoughts into moving pictures.

    The ability to automate lip sync in Flash would be a godsend. Would you please send me a demo (or a URL if the demo has already been uploaded to the MobileWish website)?

    On behalf of amateur animators everywhere, thank you!

    Best Regards,

    Steve.

  10. Thank you, Samir.

    Lip sync is time consuming, tedious work. It has prevented many amateur animators (myself included) from turning thoughts into moving pictures.

    The ability to automate lip sync in Flash would be a godsend. Would you please send me a demo (or a URL if the demo has already been uploaded to the MobileWish website)?

    On behalf of amateur animators everywhere, thank you!

    Best Regards,

    Steve.

  11. Hy Samir,

    Thanks for the demo I recived shortly.

    As I sad before, I’m very interested to see it in action, width a much “real” mounth/lips and realistic behaviour, that is the proyect I’m in too right now.

    I have most of the “fonemas”, diferents mouth expersion, like “a”, “u” … ecc, simplified in 10 gestures.

    And of course I’d like to try your system and know how is working, if is comparing Spectrum Wave with different Spectrum patterns to recognize them …

    In breafh. When do you publish the component? Is there a way to contribute right now to it to generate the most accurate code, and testing.

    thanks, filippo

  12. Thanks for the feedback filipo,

    I was working on the basis of real gestures, but i found certain problems such as preprocessing and the related even in AS3. As you might have noticed the current demo manages to render nearly close lip type.

    I am not sure of the date of its release. Most of the stuffs I release at mobilewish are not funded projects. I do them whenever I get free time, apart from my priority/office works just to pass time.

    regards
    Samir

  13. Hi Samir,

    Outstanding demo. I used FlashAmp in the past with amplitude data and I’m very interested in the demo.

    Very nice.

    Regards.

Comments are closed.