Lip Syncing Stop motion with Animatronics

The whole Billy Whiskers project really stemmed from me not knowing anything about lip syncing. I couldn’t get my head around the effort that went into creating individual mouth shapes then replacing them frame by frame, making sure it blended in, all synced up whilst also animating everything else in shot. At the same time i didn’t like the look that 3d printed mouth replacement has in stop motion. So I wanted to create something that would allow me to animate the mouth in advance, but also let me tweak it live if I wanted to.

I started off by taking some servos and seeing if I could make a mouth that I could control by just a handful of servos. I think about a year of experimenting got me to here:

The small circuit board you can see in the corner of that gif is a Pololu Maestro. Its a small usb servo controller that was my first thought for controlling the servos for animation. The software I used to create this movement was a piece of software used to program lighting and firework displays. It wasn’t really cut out for animation.

A lot of research followed and I looked at various animatronic control methods and bits of software but couldn’t find anything that would do the job. Around the same time I started fiddling with arduinos and read about ways to interface with the computer via the serial port. Several pieces of software will communicate with arduinos through the serial port. Usually through additional codes and scripts written by very clever helpful internet people. Blender is one, but I opted for Flash (or adobe animator) as I have a bot of experience with AS3.

Using Flash I made a small program that would allow me to control servos individually. The flash timeline also allowed me to import audio and animate the mouth lip sync using key frames and tweens. Then I could connect the actual servos via an arduino to the computer and simply move them to position 1 frame at a time to create the animation. I could also tweak the mouth movement using the sliders in the program.

 

It won’t win any awards for the GUI, but it did the job. You can see the sliders that represent the 5 servos and a blue line to represent Billy’s lips.

After I’d animated the lip movement in flash, i put the files on a small computer that is mounted to a piece of wood along with a monitor and power supply. The power supply (harvested from an old PC) powers the computer, monitor and servos. The servos are connected via a long cable to a simple circuit board that connects all of the power and grounds as well as the signals to the arduino which is connected to the computer by USB.

You can see most of the setup in this frame before it was removed in post.  Although it kind of just looks like a mess of wires, which it was.

Servo’s aren’t the most space friendly motors around, however they are easy to get cheaply, have handy built in gear boxes and easily controlled. Another advantage of this setup was that I was able to play the lip sync animation back real time, as seen in the first gif below. This made it easy to see if the lip sync matched up correctly.

To speed up the animation process, I set up a bunch of key mouth shapes and phonemes. This allowed me to quickly go through the audio and copy and paste frames to create the animation. Unlike traditional replacement animation however, flash could automatically tween the motion between frames.

These early tests were done at 24 frames per second, I eventually shot the whole film at 12fps, I preferred the jerky movement, also it cut down on a lot of work and deep down i’m just lazy. The second Gif is the first stop motion test with the first billy puppet. Much of the body of the puppet stayed the same.

 

James is the writer/director of Billy Whiskers, scribbling haphazardly on this website to document and share the experiences of making a stop motion animation.

2 Comments

  1. Geoff 3 months ago

    Wow! What a great technique!

    • Author
      James 3 months ago

      Thanks Geoff!

Leave a reply

Your email address will not be published. Required fields are marked *

*