It's been a great year for Joe Letteri, Weta Digital's senior visual effects supervisor, with "Rise of the Planet of the Apes" as the front runner for the VFX Oscar and "The Adventures of Tintin" opening today and competing in the animation category. Both are riding the crest of a new virtual production revolution and powered by the same innovative technology leveraged by "Avatar."
Yet whereas "Apes" is hailed by the industry for transforming Andy Serkis into a CG simian sensation worthy of Oscar recognition in some quarters, "Tintin," which co-stars Serkis as the scene-stealing Captain Haddock, has been criticized by traditionalists for entering the Uncanny Valley of dead-eyed zombies with its performance-captured animation.
But Letteri, who was in LA last week touting "Tintin," counters that critics are being short-sighted: the Uncanny Valley has nothing to do with performance capture: it has to do with how well you translate that performance. "And a lot of that just comes down to understanding the details of what makes faces work, especially around the eyes," Letteri tells me. "Because when you create a unique character, you have different proportions. By definition, you're doing something that's not human. But with the muscles in the face and the eyes, you can't rely on something that's simply biomechanical. You really have to focus on an understanding of how those muscles [function] and work together to create an expression. And then you have to allow for the artistic interpretation, which is the final layer."
Indeed, it's instructive to compare Serkis' Caesar with Haddock. The difference lies in the performance, of course, which is soulful no matter what he does, and yet there is also tremendous animation skill beneath the digital masks.
"If you look at it, Captain Haddock is not human," Letteri asserts. "His nose is sitting up between his eyes and everything is actually different from a human. But humans are incredibly forgiving for what we'll accept as being a human face. We wanted to make the world of 'Tintin' look real and feel real and come alive. But we wanted it to look like the characters that Hergé drew. It's all the same techniques that we would use for Caesar or any other character that would have to live side-by-side with real actors or real photography. The difference is that that there were no live-action elements in 'Tintin': everything was created as a virtual world."
"Tintin" and "Apes"also benefitted from a new fur system that allowed the wizards of Weta to edit and groom the hair individually instead of procedurally. On the rendering side, they used a technique called dual scattering that allowed them to calculate how light bounces inside the hair volume. They applied that to all the apes as well as the characters in "Tintin." And it was especially useful for Tintin's ginger hair and the terrier's curly white fur, which was tough to groom without turning to mush. In fact, Snowy proved the most technically challenging because Hergé drew him so cartoony. Unlike a real dog, his eyes are too close together, his lips are too shallow, his nose rides higher than his muzzle, and his ears are constantly making a box shape.
"So we were constantly experimenting with how far we could push and pull this so that it behaves like a real dog when he barks, but still looks like Snowy when you see him in one of his classic poses next to Tintin," Letteri observes.
But it was the lighting improvements that really made "Tintin" shine, according to the four-time Oscar winner for "Avatar," "King Kong," and the final two "Lord of the Rings" movies.
"'Avatar' was mostly exteriors, but 'Tintin' not only has a lot more interiors, but when you're shooting in these cramped spaces like the ship's corridor, you've got practical lights everywhere," suggests Letteri. "Mostly when you're working with CG [in live action], you're not photographing the digital lights, you're just getting the influence of the lights on the characters. Lights, if they do appear in camera, are on the set anyway. Now, we have to actually shoot the lights — the lighting instruments were there, so we had to design them and make sure they actually did the right thing on camera. And that starts to not only affect lighting design but set design because the art department builds sets and you have to move them around.
"Now in the real world, that can be done organically. As you're framing a set, you can be figuring out where the lights go and the two can go together. But for us it was more of an iterative process. And Steven [Spielberg] was directly involved because he had his own ideas about how he wanted to light this. We started off thinking: Hergé draws everything in very simple and bright-looking, broad colors, so we're going to light everything kind of brightly, but then Steven had the idea that we're going to so this more film noir with deep shadows, so that was a big exploration for us to see what worked and didn't work, especially with 3-D. You want to avoid the whole screen turning black because you lose your depth perception. But, of course, that's what film noir is all about."
They looked at a lot of classic Hitchcock films as well as Spielberg's classical canon (such as "Raiders" and "A.I.") to determine what the lighting style would be, but only as a starting point because Spielberg didn't want to retrace his stylistic footsteps.
But Letteri says that when playing with the lighting in context of the performances, it altered the dramatic emphasis. "For example, the way you might perceive a smile, and the way the light falls on the cheeks and the folds of the skin, just by moving the light a little bit, you change the perception of what that smile looks like because you get a different sense of the lines that you're creating with the shadows. But at the same time, you've got to make sure you've got the light doing what you want to say with the eyes. So it's kind of a puzzle."
For Letteri, it's no puzzle why you'd want to bring actors more fully into the animation process with their complete performances. "They're better at that than a lot of other people because that's what they do. And what we want is that magic you get from a really great performance."