AS3 based dynamic lip sync: MobileWish LipSyncAS3 soon to be out.

September 28, 2008 in AS3, Flash Lite & AS2, Technology News, Web2.0, e-Learning | Tags: actionscript, AS3, dynamic lipsync, lipsync, mobilewish, mp3

I have created a poc component (named it as “MobileWish LipSyncAS3″) for dynamic lip sync using ActionScript 3 in Flash CS3. This beta version matches the lip sync from any of the supplied MP3 file containing pure speech. This can be handy tool for adding lip sync for flash characters in any dynamic interactive. I had been using third party tools like Flash Amp for lip sync in some of my previous projects. But that involved pre processing of audio mp3 files for creating amplitude value. Now with AS3 there is no need to preprocess the mp3 files, rather the lip sync will be instantly done on the run time which will be an added advantage.

Given below is the sample video of my working prototype at Youtube.

I will finetune it and will soon be uploading it to MobileWish website (www.mobilewish.com)

If interested please add your names by replying to this post.

37 comments

September 28, 2008 at 12:17 pm

Evi Skitsanos

hi there, would it be something commercial or you will release it open source under MIT or BSD or etc flexible license by any chance?

September 28, 2008 at 12:55 pm

matt

I’m interested.

thanks.

MaTT

September 28, 2008 at 1:14 pm

Samir

Hi Evi,

Most probably it will be a Free product just like other stuffs at mobilewish.com . But it is not the final word yet.

regards
Samir

September 28, 2008 at 1:15 pm

Samir

Thanks Matt.

September 28, 2008 at 4:30 pm

Daniel Freeman

Arrrggghh!!! – That was to be MY next project!

I have an old Central application called e2animate that I was going to port to AIR. (I mentioned it recently in my blog http://e2easy.wordpress.com ). I was going to turn it into an easy fun animation app (aimed at non technical users), with the ability to customise your own mouth shapes, and drive them automatically using sound files, or perhaps input from the microphone.

I was going to look at using prediction gain over a set of fixed filters, or normalised multiplying frequency bin values with a set of patterns to determine the closest match to a particular sound (A,I/E/U/L/W,Q/M, fricatives, etc.) It was going to take some experimentation, but from the nearest sound, I’d be able to dynamically choose mouth shapes.

Like you, originally, I was going to extract the data from MP3 files. MP3 is just a sub-band coder after all, which uses frequency bin values. And like you, I discovered that AS3 had all I needed. But I was also going to look into Flash 10 player features, and I think someone said that pixelbender could be used for sound processing – I was going to investigate.

Obviously, amplitude determines the size of the mouth.

If you did give away your product for free – you would save me a lot of work – but I would be a little disappointed too, at not having done it myself.

September 28, 2008 at 5:33 pm

Automatic lip synch. « e2easy AIR applications

[...] someone has beat me to it with an AS3 lip-synch implementation. Samir has created a Flash component that he’s talking about licensing for [...]

September 28, 2008 at 6:27 pm

Fredericton Designers » Blog Archive » AS3 based dynamic lip sync: MobileWish LipSyncAS3 soon to be out.

[...] Link to the original site [...]

September 29, 2008 at 4:59 am

Mike Britton

I worked for a company called “LIPSinc” whose product “Animeter” output phonemes based on audio files you sent to a server-side program. The result was a text string which I split into an array in Flash. The details are fuzzy now, but I recall a Flash library item that consisted of mouth positions with frame labels that corresponded to the text string’s phonemes. To make a long story short, we were able to make a character talk using this technology. The guy who owned the technology is Donovan Moxy last I heard. Anyway this may be worht checking out.

September 29, 2008 at 7:14 am

Samir

Hi Daniel,

Its good to hear from you. Its not that the way I am trying to build is the better way. There may be several other ways to build the same thing…and the idea is we all should try to add value to a concept by continously trying to improve it so that users will get a list of options to choose from.

regards
Samir

September 29, 2008 at 10:38 am

Daniel Freeman

I hadn’t started working on this yet, so it’s good to have your off-the-shelf solution. If yours is free with no licensing complications – I’ll just use yours. Lip synch is just one aspect of the AIR animation app I want to build. If I don’t have to spend time on the lip synch – I can concentrate more on other features.

Can you describe your algorithm?

Yes, there are several ways. I described prediction gain and frequency bin matching on my blog. Even a simple zero-crossing algorithm might yield something useful.

Mike: Text to speech, and concatenated speech are a whole different ball game.

September 29, 2008 at 12:20 pm

najier

I am also interested.

September 30, 2008 at 7:41 am

Ravi

Thanks Samir. Last even’s discussion was nice. I will be looking forward to test ur compomponent.

September 30, 2008 at 12:05 pm

filippo

Hy,

I’m very interested to see the how LipSyncAS3 works.

I’m also trying to achive it width the computer spectrum analysis … and love to share or help in developing

thanks, ciao
filippo

October 1, 2008 at 7:35 am

LipSyncAS3 Demo « SAMIR

[...] who had added there replies to my previous post AS3 based dynamic lip sync: MobileWish LipSyncAS3 soon to be out will receive a demo file of the lip sync component I am working on. This will allow them to [...]

October 1, 2008 at 11:58 am

Jason

Love to see how it works.

October 1, 2008 at 12:04 pm

Samir

Hi jason,
just sent you the demo by email. thanks
samir

October 1, 2008 at 12:14 pm

Ruben Rojas

I’m really interested on this, it’s great and awesome.

October 1, 2008 at 12:56 pm

Samir

Hi Ruben,
just sent you the demo by email.
thanks
samir

October 4, 2008 at 10:45 am

Gani

Hey Samir

Could you send me a demo too please.. thanks

October 9, 2008 at 9:14 am

Steve

Thank you, Samir.

Lip sync is time consuming, tedious work. It has prevented many amateur animators (myself included) from turning thoughts into moving pictures.

The ability to automate lip sync in Flash would be a godsend. Would you please send me a demo (or a URL if the demo has already been uploaded to the MobileWish website)?

On behalf of amateur animators everywhere, thank you!

Best Regards,

Steve.

October 9, 2008 at 9:16 am

Steve

Thank you, Samir.

Lip sync is time consuming, tedious work. It has prevented many amateur animators (myself included) from turning thoughts into moving pictures.

The ability to automate lip sync in Flash would be a godsend. Would you please send me a demo (or a URL if the demo has already been uploaded to the MobileWish website)?

On behalf of amateur animators everywhere, thank you!

Best Regards,

Steve.

October 9, 2008 at 10:16 am

Samir

Gani and Steve,
Sent you a copy to your emails.
Thanks.
Samir

October 9, 2008 at 5:56 pm

Michael

Hi,

Can you send me a demo of your component? I am using CS3 and am looking for a solution like yours.

Michael

October 13, 2008 at 10:10 am

kaidew

amazing:)
i struggling with a 5 second animation manually

October 13, 2008 at 10:13 am

kaidew

..sorry for doble post,i was too excited ,can i have a copy please?

October 13, 2008 at 11:17 am

Samir

Sent u one kaidew.

October 15, 2008 at 9:28 am

filippo

Hy Samir,

Thanks for the demo I recived shortly.

As I sad before, I’m very interested to see it in action, width a much “real” mounth/lips and realistic behaviour, that is the proyect I’m in too right now.

I have most of the “fonemas”, diferents mouth expersion, like “a”, “u” … ecc, simplified in 10 gestures.

And of course I’d like to try your system and know how is working, if is comparing Spectrum Wave with different Spectrum patterns to recognize them …

In breafh. When do you publish the component? Is there a way to contribute right now to it to generate the most accurate code, and testing.

thanks, filippo

October 15, 2008 at 10:31 am

Samir

Thanks for the feedback filipo,

I was working on the basis of real gestures, but i found certain problems such as preprocessing and the related even in AS3. As you might have noticed the current demo manages to render nearly close lip type.

I am not sure of the date of its release. Most of the stuffs I release at mobilewish are not funded projects. I do them whenever I get free time, apart from my priority/office works just to pass time.

regards
Samir

October 16, 2008 at 4:23 pm

dave