Minecraft Live 2024: The Monster in the Woods

Client: Element Animation, Mojang
Role: Lighting
Link to the video: https://www.youtube.com/watch?v=nDLmBv-2wb4

Minecraft Live is Mojang’s virtual event all about Minecraft, a video game that they’ve made. A recurring part are the animations provided by Element Animation. For Minecraft Live 2024, this was an animation based on Element Animation’s animated series “Villager News”, featuring a new mob recently added to the game. I was in charge of doing the lighting for the animation. This animation was also different from previous Minecraft Live animations, since we were using Unreal Engine to light and render it out, rather than using Autodesk Maya with Pixar’s Renderman, and this is my first time using Unreal Engine.

Writing

It all starts with coming up with ideas and writing a script. Making animations is a lot of work and it takes a long time. Writing for the Minecraft Live 2024 animation started all the way back in June, about five months before the show.

We all come up with a bunch of ideas for what the animation could be. The idea that we ended up going with was an idea incorporating a new feature that Mojang were working hard on: A Villager News episode about a villager whose group was all taken by a monster, but no-one else believes it. So, Villager #9 and a bunch of other villagers decide to investigate to prove that there is no monster. Only it turns out that this monster was the new mob announced in this Minecraft Live, a Creaking.

Sets, Layout, And Animating

The animation plays out in a Dark Forest biome. The set was created in Minecraft and exported out using MiEx. Element Animation uses Autodesk Maya for doing layout and animation, which we extended with our own custom pipeline.

Layout for an entire scene is generally done in one file, where the different shots are set up in a custom sequencer tool. This file is then later split up into individual shot files, so that each shot can be assigned to an animator. Here’s a video showing the layout of a scene all in one file:

Getting it into Unreal Engine

Once animation is done, it’s time to get it into Unreal Engine for effects, lighting, and rendering. Each shot gets exported out individually. The rigs get exported out as an alembic, and the camera and rig locators get exported out as FBXs. In Unreal Engine, we then have a few custom tools that import these files in and setup a Level Sequence for an entire scene with sub-sequences for each shot. Generally, we export out the layout so that lighting can already begin and once animation is finished, we export out the final animation.

Effects, Lighting, and Rendering

Once the shots are in Unreal Engine, it’s time to add in the effects and lighting. This animation didn’t have many effects. The torches have flame and smoke particles, XP are particles, and at the end when a block got broken, there were particles for that. The particle effects were done using Niagara as assets that I could then drag into a shot.

For this project, my contribution was the lighting and rendering. This was the first time that I used Unreal Engine, so I had to learn how to use it while also figuring out the look for the animation.

One of the things that was important to the art style for the animation, were edge highlights on blocks. Previously, we achieved this via an OSL shading node during rendering that changed the shading normals to round off the edges, and later during compositing in Fusion by projecting the render onto a plane that was displaced by the world position, relit by lights exported from Maya, and masked by a Fuse node to detect the edges. But since we were using Unreal Engine now, we wanted to do everything in Unreal.

I ended up creating a custom Actor which set up a Scene Capture Component tied to the render camera, which renders out a depth map to a render target. This render target was then used in the materials for the world, where it would use it to detect edges. For the edges, it would brighten up the diffuse colour to highlight the edge and perturb the normal using the distance field gradient, so that it would be able to catch edge lights better. This could then be adjusted for different kinds of blocks. Foliage, like short grass and flowers had the strength of the edge highlight effect reduced, for example.

Because the video is meant to be spooky, I wanted the forest to be very dark, with only a few pockets of moonlight. To still make sure that there was enough depth in the image and to add to the atmosphere, I added in a lot of fog. There was a constant exponential height fog, but also a VDB volume to provide local pockets of additional fog. I used an additional directional light for the moonlight interacting with the fog, rather than enabling volumetric lighting on the existing directional light for more control over the intensity and colour of the fog.

Since the dark forest is very dark, all of the light available is moonlight and torch light from the torches that the villagers hold. The moonlight is done via multiple directional lights at differing source angles and intensities to provide a nicer falloff, with one directional light having a very high source angle to provide a little bit of additional fill so that the ambient lighting has more variation. Next to the directional lights, there are also local spot lights to add moonlight where needed.

In most cases, we were inside of the forest and could barely see the sky, but there were a few shots where we did see the sky. Typically in Minecraft, you don’t have that many clouds, but I wanted it to feel more ominous and to have an in-world reason for why there are only pockets of moonlight outside of the forest as well (that way I could use the pockets of moonlight to help direct the eye to what is important). In other words, I wanted more cloud coverage. Minecraft clouds are essentially 2D pixelated clouds extruded to provide thickness, but they all exist on a single plane. Increasing the cloud coverage by having more of the pixels in the cloud texture covered by clouds didn’t look so great. Instead, I ended up creating four layers of clouds, each layer up would also scale up the size of the clouds. The large clouds at the top would provide most of the cloud coverage, while the normal sized clouds on the bottom layer would provide the detail. The clouds made use of subsurface scattering to interact with the moonlight in a naturalistic way and they faded into the sky in the distance. To achieve the fade, rather than changing the opacity which would have impacted the shading due to how Unreal handles transparent objects and would also have clouds show the clouds behind them, the fade was achieved by lowering the diffuse colour to black and increasing the emission based on distance. The sky shader was then plugged into the emission colour. Essentially, in the distance you would only have emission, which was the sky shader and so it would just look like the sky. The sky shader during the night was pretty much only stars. At the very end, when we finally see a Creaking, it’s just about to become dawn and so there is a little bit of a gradient in the sky.

The materials for the world are pretty basic, apart from the edge highlight. They have a diffuse texture and a little bit of rough specular. Vegetation, like short grass and flowers, are set to two sided illumination to give a translucent effect. Leaves have subsurface scattering on them to provide some nicer shading and show more shape, especially since their textures are solid colours in Mojang’s art style for animations. Lumen was used to provide global illumination.

A big part of Element Animation’s art style is per-character lighting. Each character gets their own lighting setup that only affects them. Typically we have a key light, a back light, and a face light. The key light provides the main illumination, often tinted slightly orange as if they are illuminated by the torches or a very pale blue for moonlight. Too strong of a blue key light and you loose the natural colours, but if the key light would be white, then they wouldn’t fit into the background anymore. The back light provides a subsurface scattering effect on the edges of the villagers, helping them stand out from the set. The face light puts extra attention on their faces, since they often talk and their facial expressions are important to the animation. Most of the times it’s all pretty much the same setup, slightly orange or very pale blue key light, blue back light, and neutral face light. But for some shots I was able to use it to help sell an idea, like at the start where the witch is saying that the forest is very dangerous. I had the lights turn to red to convey this idea and I was able to animate the lights on the villagers for this as well. The key and face lights turned red, while the back light became stronger, to create more dramatic lighting.

Another technique to help character stand out from the set that Element Animation uses is to have characters be more saturated than the set. Previously in Fusion we used cryptomattes to apply a colour grade on just the characters, bumping up their saturation and lowering the saturation on the rest of the image. For this animation, I created a custom Actor that added in a post process material. This material used the CustomDepth Stencil to figure out where the characters are. The world would render with a stencil value of 0, and the characters would render out with a value between 1 and 4. In the post process material, it would use this stencil value to apply one of five colour grades. One for the set and four for the characters. Most of the times all of the characters were grouped together and got the same grade, but sometimes a character needed a slightly different grade and so got given a different stencil value.

I also added a Post Process Volume with bloom set up to look like halation. This provided a nice natural glow on bright objects and the reddish tint makes it look more interesting, especially with the strong blues from the moonlight. Emissive surfaces like torches, flames, and lanterns were set up by taking the diffuse texture and applying the inverse view transform to them. That’s then saved out into an EXR, imported into Unreal, and put into the emission colour. This makes these surfaces glow realistically, while still looking the same as in the game.

Once it was all lit, it was time to render it out. We used the Movie Render Queue to render out the frames, which then got directly loaded into the Adobe Premiere Pro timeline. We use a small amount of motion blur, only a 90 degree shutter angle rather than the typical 180 degree shutter angle, since we find a 180 degree shutter angle too much motion blur for this style of Minecraft animation.

Mojang also wanted a 9×16 version next to the standard 16×9 version. Luckily, since we were using Unreal Engine, we could easily re-render the animation to be in 9×16. Often we could just switch the aspect ratio and re-render. For some shots, we needed to adjust the camera and for a few shots, the animation needed to be adjusted as well.

Being able to light and render in real-time using Unreal Engine has been very nice and has allowed us to do things that otherwise we couldn’t. But due to it being a real-time game engine, it is missing some stuff that offline render engines, typically used in animation, do provide. This does make it more challenging to use Unreal Engine for lighting and rendering animations. It seems less robust, due to it needing to use approximations to keep things real-time, which can create artifacts when settings are pushed too far. But the fast feedback provided by it being real-time makes lighting more fun and increases productivity.

In the end, it was really fun working on this video, I’ve learned a lot, and it has been great seeing the community’s response to it.