This year at SIGGRAPH 2022, Corey Butler, Brent Burley, Wei-Feng Wayne Huang, Benjamin Huang, and I have a talk that presents the technical and artistic challenges and solutions that went into creating the holographic look for Bruno’s visions in Encanto. In Encanto, Bruno is a character who has a magical gift of being able to see into the future, and the visions he sees of the future get crystalized into a sort of glassy emerald tablet with the vision embedded in the glassy surface with a holographic effect. Coming up with this unique look and an efficient and robust authoring workflow required a tight collaboration between visual development, lookdev, lighting, and the Hyperion rendering team to develop a custom solution in Disney’s Hyperion Renderer [Burley et al. 2018]. On the artist side, Corey was the main lighter and Benjamin was the main lookdev artist for this project, while on the rendering team side, Wayne and I worked closely together to develop a series of prototype shaders that were instrumental in defining how the effect should look and then Brent came up with the implementation approach for the final production version of the shader. This project was a lot of fun to be a part of and in my opinion really demonstrates the benefits of having an in-house rendering team that works closely with and embedded within a production context.

Here is the paper abstract:

In Walt Disney Animation Studios’ “Encanto”, Mirabel discovers the remnants of her Uncle Bruno’s mysterious visions of the future. Developing the look and lighting for the emerald shards required close collaboration between our Visual Development, Look Development, Lighting, and Technology departments to create a holographic effect. With an innovative new teleporting holographic shader, we were able to bring a unique and unusual effect to the screen.

The paper and related materials can be found at:

• Project Page (Author’s Version and Supplementary Material)
• Official Print Version (ACM Library)

When Corey first came to the rendering team with the request for a more efficient way to create the hologram effect that lighting had prototyped using camera mapping, our initial instinct actually wasn’t to develop a new shader at all. Hyperion has an existing “hologram” shader that was developed for use on Big Hero 6 [Joseph et al. 2014], and our initial instinct was to tell Corey that they should use the hologram shader. The way the Big Hero 6 era hologram shader works is: upon hitting a surface that has the hologram shader applied, the ray is moved into a virtual space containing a bunch of imaginary parallel planes, with each plane textured with a 2D slice of a 3D interior. In some ways the hologram shader can be thought of as raymarching through a sparse volumetric representation of a 3D interior, but the sparse volumetric interior really is just a stack of 2D slices. This technique works really well for things like building interiors seen through glass windows. However, our artists… really dislike using the hologram shader, to put things lightly. The problem with the hologram shader is that setting up the 2D slices that are inputs to the shader is an incredibly annoying and difficult process, and since the 2D slice baker has to be run as an offline process before the shader can be authored and rendered, making changes and iterating on the contents of the hologram shader is a slow process. Furthermore, if the inside of the hologram shader has to be animated, the slice baker needs to be run for every frame. We were told in no uncertain terms that the hologram shader was likely more work to set up and iterate on than the already painful manual camera mapping approach that the artists had prototyped the effect with. This request also came to us fairly late in Encanto’s production schedule, so easy setup and fast iteration times along with an extremely accelerated development timeline were hard requirements for whatever approach we took.

Upon receiving this feedback, Wayne and I set out to prototype a version of the teleportation shader that Pixar came up with for the portals in Incredibles 2 [Coleman et al. 2014]. This process was a lot of fun; Wayne and I spent a few days rapidly iterating on several different ideas for both how to implement ray teleportation in Hyperion and on how the artist workflow and interface for this new teleportation system should work. At the same time that we were prototyping, we started giving test builds of our latest prototypes to Corey to try out, which produced a feedback loop where Corey would use our prototypes to further iterate on how the final effect would look and go back and forth with the movie’s production designer and we would use Corey’s feedback to further improve the prototype. One example of where our prototype directly informed the final look was in how the prophecies fade away towards the edges of the emerald tablet- Wayne and I threw in a feature where artists could use a map to paint in the ratio of teleportation effect versus normal surface BSDF that would be applied at each surface point, and this feature wound up driving the faded edges.

The key thing that made our new approach work better than the old hologram shader was in simplicity of setup. Instead of having to run a pre-bake process and then wire up a whole bunch of texture slices into the renderer, our new approach was designed so that all an artist had to do was set up the 3D geometry that they wanted to put inside of the hologram in a target space hidden somewhere in the overall scene (typically below the ground plane in a black box or something), and then select the geometry in the main scene that they wanted to act as the “entrance” portal, select the geometry in the target space that they wanted to act as the “exit” portal, and link the two using the teleportation shader. The renderer then did all of the rest of the work of figuring out how each point on the entrance portal corresponded to the surface of the exit portal, how transforms needed to be calculated, and so on and so forth. Multiple portal pairs could be set up in a single scene too, and the contents of a world seen through a portal could contain more portals, all of which was important because in the movie, Mirabel initially finds Bruno’s prophecy broken into shards, which had to be set up as a separate entrance portal per shard all into the same interior world. Since all of this just piggy-backed off of the normal way artists set up scenes, things like animation just worked out-of-the-box with no additional code or effort.

The last piece of the puzzle fell into place when Wayne and I discussed our progress with Brent. One of the big remaining challenges for us was that tracking correspondences between entrance and exit geometry and transforms was prone to easy breakage if input geometry wasn’t set up exactly the way we expected. At the time Brent was working on a new fracture-aware tessellation system for subdivision surfaces in Hyperion [Burley and Rodriguez 2022], and Brent quickly realized that the approach we were using for figuring out the transform from the entrance to the exit portal could be replaced with something he had already developed for the fracture-aware tessellation system. Specifically, the fracture-aware tessellation system has to be able to calculate correspondences between undeformed unfractured reference points and corresponding points in a deformed fractured fragment space; this is done using a best-fit process to find orthonormal transforms [Horn et al. 1988]. Brent realized that the problem we were trying to solve was actually the same problem he that he had already solved in the fracture system, so he took our latest prototype and reworked the internals to use the same best-fit orthonormal transform solution as in the fracturing system. With Brent’s improvements, we arrived at the final production version of the teleportation shader used on Encanto.

Going from the start of brainstorming and prototyping to delivering the final production version of the shader took us a little over a week, which anyone who has worked in an animation/VFX production setting before will know is very fast for a large new rendering feature. Working tightly with Corey and Benjamin to simultaneously iterate on the art and the software and inform each other was key to this project’s fast development time and key to achieving an amazing looking effect in the film. At Disney Animation, we have a mantra that goes “art challenges technology and technology inspires the art”- this project was a case that exemplifies how we carry out that mantra in real-world filmmaking and demonstrates the amazing results that come out of such a process. Bruno’s visions in Encanto are every bit a case where the artistic vision challenged us to develop new technology, and the process of iterating on the new technology between engineers and artists in turn informed the final artwork that made it into the movie; for me, projects like these are one of the things that makes Disney Animation such a fun and amazing place to be.

References

Brent Burley, David Adler, Matt Jen-Yuan Chiang, Hank Driskill, Ralf Habel, Patrick Kelly, Peter Kutz, Yining Karl Li, and Daniel Teece. 2018. The Design and Evolution of Disney’s Hyperion Renderer. ACM Transactions on Graphics 37, 3 (Jul. 2018), Article 33.

Brent Burley and Francisco Rodriguez. 2022. Fracture-Aware Tessellation of Subdivision Surfaces. In ACM SIGGRAPH 2022 Talks. Article 10.

Patrick Coleman, Darwyn Peachey, Tom Nettleship, Ryusuke Villemin, and Tobin Jones. 2018. Into the Voyd: Teleportation of Light Transport in Incredibles 2. In Proc. of Digital Production Symposium (DigiPro 2018). Article 12.

Berthold K. P. Horn, Hugh M. Hilden, and Shahriar Negahdaripour. 1988. Close-Form Solution of Absolute Orientation using Orthonormal Matrices. Journal of the Optical Society of America A 5, 7 (Jul. 1988), 1127–1135.

Norman Moses Joseph, Brett Achorn, Sean D. Jenkins, and Hank Driskill. Visualizing Building Interiors Using Virtual Windows. In ACM SIGGRAPH Asia 2014 Technical Briefs. Article 18.

Disney Animation has two short-form television series lined up for release in 2022 on Disney+, and the first of the two series is Baymax!, which is comprised of six episodes set after the events of Big Hero 6 and follows Baymax around San Fransokyo as he helps various people. Baymax! notably is the first time Disney Animation has ever produced a television series completely in-house; because Baymax! was made in-house, it was made using the same tech and pipeline, by the same artists, and to the same standards that we make our feature-length films using. Big Hero 6 was the first project at Disney Animation ever rendered using Disney’s Hyperion Renderer (and to a large extent the first version of Hyperion was specifically developed for Big Hero 6), so getting to revisit the world of San Fransokyo in many ways represented a sort of homecoming for Hyperion. However, I joined the Hyperion development team shortly after the completion of Big Hero 6, so for me personally, getting to work a bit for the first time on a project tied to the show where Hyperion began was a very cool experience.

Baymax! was made using a slightly evolved version of the same pipeline and toolset that was used for Raya and the Last Dragon. Since production overlapped with Encanto, some improvements developed for Encanto made their way over to Baymax! as well, which in the case of Hyperion mostly meant bugfixes and improvements to our state-of-the-art in-house machine-learning denoiser [Vogels et al. 2018]. From a computer graphics technology perspective, there is not really much new to tell about the work that went into Baymax!, but I think this in itself is actually an interesting thing to discuss. Production on Baymax! went very smoothly and required very little dedicated renderer development work, which I think is a testament to both how mature Hyperion has become since Big Hero 6 and how proficient our artists have become at using Hyperion. Of course, another factor to consider is that Baymax! has more or less exactly the same challenges that Big Hero 6 had, and all of those problems were solved already (and arguably Hyperion was custom built specifically to solve many of those problems), so of course having a second go at exactly the same kinds of production challenges should be easier than it was the first time, especially when using more advanced, evolved versions of the original solutions.

As an example, the Baymax character’s white translucent look comes from a ton of high-order scattering inside of his inflatable balloon shell, but the effect is different than what one gets from subsurface scattering. Instead of having some kind of solid medium inside, Baymax contains air inside, so the type of scattering occurring inside of Baymax is more akin to something like total internal reflection with no energy loss between surface events, as opposed to subsurface scattering which has extinction due to volumetric effects. The net result is that Baymax requires a ton of surface bounces in path tracing; often many more than our usual maximum path length. On Big Hero 6 the solution to this problem was to allow artists to specify materials and objects that would be permitted to raise the maximum path length for paths that interact with them [Driskill et al. 2015]. On Baymax!, all we had to do was to remember to set this setting for Baymax, and of course everything just worked because this problem had already been solved before.

Previously we’ve ported characters and assets from older shows forward to whatever the latest modern pipeline is many times, but Baymax! was a case where we had to port characters that originated from a previous Hyperion-based show forward. I’ve written before about how porting pre-Hyperion characters and assets to Hyperion is made a lot easier by a lot of our foundational shading technologies spanning between the pre-Hyperion and modern Hyperion pipeline versions, but porting from Big Hero 6 was made even easier by the fact that this was just porting between an older and newer version of the same renderer. Big Hero 6 was the first show to use our modern Disney BSDF [Burley et al. 2015], which expanded upon the older Disney BRDF [Burley et al. 2012] to add things such as refraction and subsurface scattering, so pretty much all solid surfaces ported over with essentially zero effort required. However, Big Hero 6 was also the last and only Hyperion-based show to use our older Tangled-era fur/hair shading model [Sadeghi et al. 2010], with all shows starting with Zootopia using our modern state-of-the-art (and now de-facto industry standard) fur/hair model [Chiang et al. 2016a], which meant that all characters with hair or fur had to have their look updated to use the modern fur/hair shading model. Similarly, for characters, shows starting with Frozen 2 moved off of normalized diffusion subsurface scattering [Christensen and Burley 2015] and onto path traced subsurface scattering [Chiang et al. 2016b], so this switch had to be made for Baymax! as well. On Frozen 2 we found that generally textures and settings meant for normalized diffusion translated to path traced subsurface scattering pretty well, although areas with thinner surfaces sometimes required some additional manual adjustment; the experience on Baymax! was similar. Additionally, additive features on top of the Disney BSDF, such as the softened shadow terminator handling we introduced on Frozen 2 [Chiang et al. 2019], just automatically made everything ported from Big Hero 6 look a bit nicer.

Aside from shading improvements, seeing how other technology and techniques developed in the years since Big Hero 6 fed back into improving the world of San Fransokyo on Baymax! was also very fun and cool. Here are three examples that I thought were really neat: First, for the swimming pool in the second episode of Baymax!, the photon mapping system originally developed for Moana [Burley et al. 2018] and expanded on Olaf’s Frozen Adventure proved useful for water effects once again, providing the caustics on the bottom of the pool. Second, in the fourth episode, a character runs a food truck that only serves fish soup, and to make shots of cooking the soup look really appealing and convincing, our effects artists borrowed from soup simulation and shading techniques originally developed for Raya and the Last Dragon. Third, the studio’s crowds and characters workflows and pipelines have improved by leaps and bounds since Big Hero 6, to the point where many of the new main characters in Baymax! are actually promoted versions of background crowd characters from Big Hero 6 [Hamed et al. 2015], chosen by the directors and upgraded by our artists to serve as new main characters that already fit into the show’s world.

At the end of the day though, I think one of the coolest technical things about Baymax! is simply that it looks every bit as good as the original Big Hero 6, if not even better in some places. Of course one would think that of course it would look as good as the original Big Hero 6 since it’s using improved versions of the same assets and an evolved version of the same pipeline and tooling and renderer, but the thing to note here is that Big Hero 6 was an enormous, high-risk, all-hands-on-deck heavy lift endeavour for the studio, whereas Baymax! is a television series made by a much smaller crew with much tighter resources. Why Baymax! is able to meet the same bar as Big Hero 6 is partially down to smart planning and decisions on the part of the directors and show supervisors, but another large part is due to how much the studio has improved and grown technically and artistically in the 8 years since Big Hero 6.

Baymax! is available for streaming on Disney+; I recommend projecting or casting Disney+ to the largest screen you can to best see all of the amazing work that went into making this television series look every bit as good as our feature films. Here is a selection of stills from Disney+, presented in no particular order:

Here is the credits frame for the Hyperion team, interspersed within the larger credits block for the entire production technology team at Disney Animation:

All images in this post are courtesy of and the property of Walt Disney Animation Studios.

References

Brent Burley. 2012. Physically Based Shading at Disney. In ACM SIGGRAPH 2012 Course Notes: Practical Physically-Based Shading in Film and Game Production.

Brent Burley. 2015. Extending the Disney BRDF to a BSDF with Integrated Subsurface Scattering. In ACM SIGGRAPH 2015 Course Notes: Physically Based Shading in Theory and Practice.

Brent Burley, David Adler, Matt Jen-Yuan Chiang, Hank Driskill, Ralf Habel, Patrick Kelly, Peter Kutz, Yining Karl Li, and Daniel Teece. 2018. The Design and Evolution of Disney’s Hyperion Renderer. ACM Transactions on Graphics 37, 3 (Jul. 2018), Article 33.

Matt Jen-Yuan Chiang, Benedikt Bitterli, Chuck Tappan, and Brent Burley. 2016. A Practical and Controllable Hair and Fur Model for Production Path Tracing. Computer Graphics Forum (Proc. of Eurographics) 35, 2 (May 2016), 275-283.

Matt Jen-Yuan Chiang, Peter Kutz, and Brent Burley. 2016. Practical and Controllable Subsurface Scattering for Production Path Tracing. In ACM SIGGRAPH 2016 Talks. Article 49.

Matt Jen-Yuan Chiang, Yining Karl Li, and Brent Burley. 2019. Taming the Shadow Terminator. In ACM SIGGRAPH 2019 Talks. Article 71.

Per H. Christensen and Brent Burley. 2015. Approximate Reflectance Profiles for Efficient Subsurface Scattering. In ACM SIGGRAPH 2015 Talks. Article 25.

Hank Driskill, Larry Wu, Adolph Lusinsky, and Sean D. Jenkins. 2015. Building San Fransokyo: Creating the World of Disney’s “Big Hero 6”. In ACM SIGGRAPH 2015 Production Sessions. 169.

Yasser Hamed, John Kahwaty, Andy Lin, Evan Goldberg, and Lawrence Chai. 2015. Crowd Character Complexity on Big Hero 6. In ACM SIGGRAPH 2015 Talks. Article 77.

Iman Sadeghi, Heather Pritchett, Henrik Wann Jensen, and Rasmus Tamstorf. 2010. An Artist Friendly Hair Shading System. ACM Transactions on Graphics (Proc. of SIGGRAPH) 29, 4 (Jul. 2010), Article 56.

Thijs Vogels, Fabrice Rousselle, Brian McWilliams, Gerhard Röthlin, Alex Harvill, David Adler, Mark Meyer, and Jan Novák. 2018. Denoising with Kernel Prediction and Asymmetric Loss Functions. ACM Transactions on Graphics (Proc. of SIGGRAPH) 37, 4 (Aug. 2018), Article 124.

Along with Encanto’s release last fall, Disney Animation also released Far from the Tree, which is the studio’s newest short film. Far from the Tree was made in parallel with our last short, Us Again, but the two shorts have completely different visual styles. Both shorts were rendered using Disney’s Hyperion Renderer, but visually they stand as almost polar opposites. From a computer graphics perspective, Us Again is a showcase of the studio’s cutting-edge modern physically based rendering capabilities, while Far from the Tree is a showcase of the studio’s hand-drawn¹ inspired stylized rendering capabilities. The two shorts weren’t set up this way intentionally; in both cases, the visual style was chosen based entirely off of what was right for the story, but as a rendering engineer, these two shorts going into production at the same time was also a serendipitous opportunity to see how far we could push our visual filmmaking capabilities in two very different directions.

Normally in these posts, I write that the project was rendered entirely using Disney’s Hyperion Renderer, but that’s not the whole story here, and writing that would be a disservice to how this short was actually made. There is quite a lot of 3D CG in this short; all of the character animation was made using our normal 3D animation process, and all of the 3D stuff is in fact rendered entirely using Hyperion. However, the final look of the short involves extensive additional 2D work done on top of the base 3D renders. Far from the Tree’s look and production process is an extension of the hand-drawn inspired hybrid 2D-3D approach that Disney Animation previously used on Paperman , Feast, and several Short Circuit shorts. The base 3D renders are essentially unlit flat surface-color passes; in fact, if you look at just the raw beauty passes out of the renderer, they’re completely black! The real look of the film comes from extensive work done on top of the flat surface-color passes and a ton of AOVs output from the renderer. Much like on Paperman [Kahrs et al. 2012, Whited et al. 2012], Far from the Tree uses extensive linework drawn by hand using Meander on top of the rendered layers, and much like on Feast [Osborne and Staub 2014], final 2D lighting was created entirely in the composite in Nuke. The backgrounds are a combination of similar 2D-3D hybrid work to the foreground along with a lot of pure matte paintings, and much of the textural detail across the entire frame is similarly painted and projected in 2D, all of which were evolved at the studio on the Short Circuit experimental shorts program [Newfield and Staub 2020]. To further help enhance the 2D-3D hybrid look, much of the short is animated on twos and threes, to help match the motion of traditional hand-drawn animation.

On the Hyperion front, even though there’s no meaningful path tracing taking place, the renderer was still doing quite a lot on Far from the Tree! By the very nature of where they exist in production pipelines, production renderers tend to function as the final “source of truth” for what all 3D data in the production pipeline actually looks like, and this is definitely true of Hyperion. In our pipeline, Hyperion doesn’t just serve as the final frame renderer; it also acts as a powerful data visualization tool that is used in all departments upstream of lighting to generate authoritative visualizations of what our 3D data actually looks like. To help serve this function, Hyperion has incredibly extensive custom AOV capabilities. As far as I’m aware, I think Hyperion’s custom AOV capabilities are a fair bit more extensive than in even a lot of commercial renderers. Commercial renderers usually break down light transport into a bunch of individual components and expose these as AOVs (so, things like specular vs diffuse, direct vs indirect, individual BXDF lobes, etc), and also expose some basic geometric information as AOVs (things like position, normals, object IDs, cryptomattes, etc). Hyperion provides all of these as AOVs as one would expect, but beyond that, Hyperion allows essentially any signal or snippet of shader code from inside of the pattern generation part of the shading system to be routed into a custom AOV. Hyperion also goes even further and allows for custom AOVs to be driven directly using SeExpr [Disney Animation 2011]; on the first hit from the camera, user-authored SeExpr programs can be run and the result directly splatted to specified custom AOVs. For stylized rendering projects such as Far from the Tree’s hybrid 2D-3D, this capability is really powerful since it allows for artists to drive custom signals from 3D animated geometry, funnel those signals into 2D layers, and then use the result to drive any kind of effect they want in compositing. This system means that even without needing to run full light transport, on highly stylized projects such as Far from the Tree, Hyperion still plays a big role. Actually, on Far from the Tree, Hyperion is doing almost no light transport, but not exactly zero. There is one specific small detail that did make use of light transport: the full eye shader [Chiang & Burley 2018] was used to get enough input to drive the final flatter look of the stylized eyes.

The final result in Far from the Tree is a wonderful combination of Disney Animation’s modern state-of-the-art 3D CG capabilities and rich hand-drawn 2D legacy. Any randomly chosen frame from Far from the Tree basically looks exactly like the concept art used to art-direct the short, and that is both a very cool and technically astonishing feat; it’s a really beautiful film. I also just love the character design and character animation in this short; raccoons are an endless source of interesting animation, and the derpy birds are a fantastically fun piece of cartoon design.

Here are some frames from Far from the Tree from the Blu-ray, presented in no particular order. You can get Far from the Tree with a copy of Encanto on Blu-ray or digital, or watch it on Disney+; as always I recommend watching it on the biggest screen you can!

All images in this post are courtesy of and the property of Walt Disney Animation Studios.

References

Matt Jen-Yuan Chiang and Brent Burley. 2018. Plausible Iris Caustics and Limbal Arc Rendering. In ACM SIGGRAPH 2018 Talks. Article 15.

John Kahrs, Patrick Osborne, Amol Sathe, Jeff Turley, Brian Whited, and Darrin Butters. 2012. The Art and Science Behind Walt Disney Animation Studios’ “Paperman”. In ACM SIGGRAPH 2012 Production Sessions.

Jennifer Newfield and Josh Staub. 2020. How Short Circuit Experiments: Experimental Filmmaking at Walt Disney Animation Studios. In ACM SIGGRAPH 2020 Talks. Article 72.

Patrick Osborne and Josh Staub. 2014. Feast – A Look at Walt Disney Animation Studios’ Newest Short. In ACM SIGGRAPH 2014 Production Sessions.

Brian Whited, Eric Daniels, Michael Kaschalk, Patrick Osborne, and Kyle Odermatt. 2012. Computer-Assisted Animation of Line and Paint in Disney’s Paperman. In ACM SIGGRAPH 2012 Talks. Article 19.

Walt Disney Animation Studios. 2011. SeExpr.

Footnotes

¹ A lot of people use “2D animation” to describe Disney Animation’s work before the CG era, but at Disney Animation we prefer the term “hand-drawn animation”. I think the distinction is really important; a lot of modern 2D animation is made entirely digitally using rigged digital models/puppets similar to what we do for 3D animation. This is totally fine! However, Disney Animation’s previous traditional animation work was distinguished not just by being visually 2D, but really by the fact that everything was drawn by hand, either using pencil on paper or using a stylus and tablet when digital. keyboard_return

For the first time since 2016, Walt Disney Animation Studios is releasing not just one animated feature in a year, but two! The second Disney Animation release of 2021 is Encanto, which marks a major milestone as Disney Animation’s 60th animated feature film. Encanto is a musical set in Colombia about a girl named Mirabel and her family: the amazing, fantastical, magical Madrigals. I’m proud of every Disney Animation project that I’ve had the privilege to work on, but I have to admit that this year was something different and something very special to me, because this year we completed both Raya and the Last Dragon and Encanto, which are together two of my favorite Disney Animation projects so far. Earlier this year, I wrote about the amazing work that went into Raya and the Last Dragon and why I loved working on that project; with Encanto now in theaters, I now get to share why I’ve loved working on Encanto so much as well!

Disney Animation feature films take many years and hundreds of people to make, and often the film’s story can remain in a state of flux for much of the film’s production. All of the above isn’t unusual; large-scale creative endeavors like filmmaking often entail an extremely complex and challenging process. More often than not, a film requires time and many iterations to really find its voice and gain that spark that makes it a great film. Encanto, however, is a film that a lot of my coworkers and I realized was going to be really special very early on in production. Now obviously, that hunch didn’t mean that making Encanto was easy by any means; every film requires tons of hard work from the most amazing, inspiring, talented artists and engineers that I know. But, I think in the end, that initial hunch about Encanto was proven correct: the finished Encanto has a story that is bursting with warmth and meaning, has one of Disney Animation’s best main characters to date with a huge cast of charming supporting characters, has the most beautiful, magical animation and visuals we’ve ever done, and sets all of the above to a wonderful soundtrack with a bunch of catchy, really cleverly written new songs. Both the production process and final film for Encanto were a strong reminder for me of why I love working on Disney Animation films in the first place.

From a technical perspective, Encanto also represents something very special in the history of Disney Animation’s continual advancements in animation technology. To understand why this is, a very brief history review about Disney Animation’s modern production pipeline and toolset is helpful. In retrospect, Disney Animation’s 50th animated feature film, Tangled, was probably one of the most important films the studio has ever made from a technical perspective, because the production of Tangled required a near-total ground-up rebuild of the studio’s production pipeline and tools that wound up laying the technical foundations for Disney Animation’s modern era. While every film we’ve made since Tangled has seen us make enormous technical strides in a variety of eras, the starting point of the production pipeline we’ve used and evolved for every CG film up until Encanto were put into place during Tangled. The fact that Encanto is Disney Animation’s 60th animated feature film is therefore fitting; Encanto is the first film made using the USD-based successor [Miller et al. 2022] to the production pipeline that was first built for Tangled, and just like how Tangled laid the technical foundations for the subsequent ten films that followed, Encanto lays the technical foundations for many more future films to come! As presented in the USD Birds of a Feather session at SIGGRAPH 2021, this new production pipeline is built on the open-source Universal Scene Description project and brings massive upgrades to almost every piece of software and every custom tool that our artists use. Already Encanto’s upgraded production pipeline has enabled cool new tools that would have been much harder to create previously, such as a new command center tool that gives a bird’s eye unified overview of stats and data across the entire movie [Tennant et al. 2022]. An absolutely monumental amount of work was put into building a new USD-based world at Disney Animation, but I think the effort was extremely worthwhile: thanks to the work done on Encanto, Disney Animation is now well set up for another decade of technical innovation and another decade of pushing animation as a medium forward.

Moving to a new production pipeline meant also moving Disney’s Hyperion Renderer to work in the new production pipeline. To me, one of the biggest advantages of an in-house production renderer is the ability for the renderer development team to work extremely closely with other teams in the studio in an integrated fashion, and moving Hyperion to work well in the new USD-based world exemplifies just how important this collaboration is. We couldn’t have pulled off this effort without the huge amount of amazing work that engineers and TDs and artists from many other departments pitched in. However, having to move an existing renderer to a new pipeline isn’t the only impact on rendering that the new USD-based world has had. One of the most exciting things about the new pipeline is all of the new possibilities and capabilities that USD and Hydra unlocks; one of the biggest projects our rendering team worked on during Encanto’s production was a new, very exciting next-generation rendering project. I can’t talk too much about this project yet; all I can say is that we see it as a major step towards the future of rendering at Disney Animation, and that even in its initial deployment on Encanto, we’ve already seen huge fundamental improvements to how our lighters work every day. Hopefully we’ll be able to reveal more soon!

Of course, just because Encanto saw huge foundational changes to how we make movies doesn’t mean that there weren’t the usual fun and interesting show-specific challenges as well. Encanto presented many new, weird, fun problems for the rendering team to think about. Geometry fracturing was a major effect used extensively throughout Encanto, and in order to author and render fractured geometry as efficiently as possible, the rendering team had to devise some really clever new geometry-processing features in Hyperion [Burley and Rodriguez 2021]. Encanto’s cinematography direction called for a beautiful, really colorful look [Robinson 2022] that required pushing artistic controllability in our lighting capabilities even further, and to that end our team developed a bunch of cool new artistic control enhancements in Hyperion’s volume rendering and light shaping systems. One of my favorite show-specific challenges that I got to work on for Encanto was for the holographic effect in Bruno’s emerald crystal prophecies [Butler et al. 2021]. For a variety of reasons, the artists wanted this effect done completely in-render; coming up with an in-render solution required many iterations and prototypes and experiments carried out over several months through a close collaboration between a number of artists and TDs and the rendering team.

Encanto also saw continued advancements to Hyperion’s state-of-the-art deep-learning denoiser and stereo rendering solutions and saw continued advancements in Hyperion’s shading models and traversal system. A particularly notable advancement in our shading model is the addition of a new physically accurate practical multiple-scattering sheen lobe [Zeltner et al. 2022] to the Disney BSDF [Burley 2015]; I think this new sheen model is going to catch on widely in industry due to its combination of accuracy, ease of implementation, and performance, all of which improves greatly over previously existing sheen models [Conty and Kulla 2017]. These advancements helped us tackle many of the interesting complexity and scaling challenges that Encanto presented; effects like Isabella’s flowers and the glowing magical particles associated with the Madrigal family’s miracle pushed instancing counts to incredible new record levels [Finley et al. 2022], and for the first time ever on a Disney Animation film, we actually rendered some of the gorgeous costumes in the movie not as displaced triangle meshes with fuzz on top, but as actual woven curves at the thread-level [Velasquez et al. 2022]. The latter proved crucial to creating the chiffon and tulle in Isabella’s outfit and was a huge part in creating the look of Mirabel’s characteristic custom-embroidered skirt. My mind was thoroughly blown when I saw those renders for the first time; on every film, I’m constantly amazed and impressed by what our artists can do with the tools we provide them with.

Encanto also saw rendering features that we first developed for previous films pushed even further and used in interesting new ways. We first deployed a path guiding implementation [Müller et al. 2017] in Hyperion back on Frozen 2 [Müller 2019], but path guiding wound up not seeing too much use on Raya and the Last Dragon since Raya’s setting was mostly outdoors, and path guiding doesn’t help as much in direct-lighting dominant scenarios such as outdoor scenes. However, since a huge part of Encanto takes place inside of the magical Madrigal casita, indoor indirect illumination was a huge component of Encanto’s lighting. We found that path guiding provided enormous benefits to render times in many indoor scenes, and especially in settings like the Madrigal family’s kitchen at night, where lighting was almost entirely provided by outdoor light sources coming in through windows and from candles and stuff. I think this case was a great example of how we benefit from how closely our lighting artists and our rendering engineers work together on many shows over time; because we had all worked together on similar problems before, we all had shared experiences with past solutions that we were able to draw on together to quickly arrive at a common understanding of the new challenges on Encanto. Another good example of how this collaboration continues to pay dividends over time is in the choices of lens and bokeh effects that were used on Encanto. For Raya and the Last Dragon, we learned a lot about creating non-uniform bokeh and interesting lensing effects, and what we learned on Raya in turn helped further inform early cinematography and lensing experiments on Encanto. One more great example can be found in how eyes are shaded in Encanto- over our last few shows, we’ve been steadily moving our eye shading approach over to a next-generation shading model with advanced, physically accurate iris caustics [Chiang and Burley 2018] sampled using manifold next event estimation, and Encanto is the first show to use this new eye shading model on 100% of characters. The way we push technology further and further on each film isn’t limited to just rendering either; I mostly write about only lighting/shading/rendering topics here because that’s my home domain, but there are countless other examples in things like rigging, animation, simulation, procedural authoring, interactive visualization, and more about how we each film tech advances on top of the previous film. A great example published at SIGGRAPH 2022 is the new hair simulation technique that was developed for Mirabel’s bouncy curly hair [Liu 2022]; ever since Tangled, Disney Animation has been great at hair, but with each movie we still keep advancing what we can do!

In addition to all of the cool renderer development work that I usually do, I also got to take part in something a little bit different on Encanto. Every year, the lighting department brings on a handful of trainees, who are put through several months of in-studio “lighting school” to learn our tools and pipeline and approach to lighting before lighting real shots on the film itself. This year, I got to join in with the lighting trainees while they were going through lighting training; this experience wound up being one of my favorites from the past year. I think that having to sit down and actually learn and use software the same way that the users have to is an extraordinarily valuable experience for any software engineer that is building tools for users. Even though I’ve been working at Disney Animation for six years now, and even though I know the internals of how our renderer works extensively, I still learned a ton from having to actually use Hyperion to light shots and address notes from lighting supervisors and stuff! Encanto’s lighting style required really leaning on the tools that we have for art-directing and pushing and modifying fully physical lighting, which really changed my perspective on some of these tools. For most rendering engineers and researchers, features that allow for breaking purely physical light transport are often seen as annoying and difficult to implement but necessary concessions to the artists. Having now used these features in order to hit artistic notes on short time frames though, I now have a better understanding of just how critical a component these features can be in an artist’s toolbox. I owe a huge amount of thanks to Disney Animation’s technology department leadership and to the lighting department for having made this experience possible and for having strongly supported this entire “exchange program”; I’d strongly recommend that every rendering engineer should go try lighting some shots sometime!

Finally, here are some stills from the movie pulled from the Blu-ray, 100% created using Disney’s Hyperion Renderer by our amazing artists. I’ve ordered the frames randomly, to try to prevent spoiling anything important. These frames showcase just how gorgeous Encanto looks, but they only represent a small fraction of how breathtakingly beautiful and colorful the total film is. I highly recommend seeing Encanto on the biggest screen you can; if you are a computer graphics enthusiast, go see it twice: the first time for the wonderful, magical story and the second time for the incredible artistry that went into every single shot and every single frame! I love working on Disney Animation films because Disney Animation is a place where some of the most amazing artists and engineers in the world work together to simultaneously advance animation as a storytelling medium, as a visual medium, and as a technology goal. Art being inspired by technology and technology being challenged by art is a legacy that is deeply baked into the very DNA of Disney Animation, and that approach is exemplified by every single frame in Encanto:

Here is the credits frame for Disney Animation’s rendering and visualization teams! These two teams collectively are responsible for generating all of the pixels at Disney Animation, be it final frames from Hyperion, or interactive viewports using our internal realtime rasterizer:

All images in this post are courtesy of and the property of Walt Disney Animation Studios.

Also, be sure to catch our new short, Far from the Tree, which is accompanying Encanto in theaters. Far from the Tree deserves its own discussion later; all I’ll write here is that I’m sure it’s going to be fascinating for rendering and computer graphics enthusiasts to see! Far from the Tree tells the story of a parent and child raccoon as they explore a beach; the short has a beautiful hand-drawn watercolor look that is actually CG rendered out of Disney’s Hyperion Renderer and extensively augmented with hand-crafted elements. Be sure to see Far from the Tree in theaters with Encanto!

References

Brent Burley. 2015. Extending the Disney BRDF to a BSDF with Integrated Subsurface Scattering. In ACM SIGGRAPH 2015 Course Notes: Physically Based Shading in Theory and Practice.

Brent Burley and Francisco Rodriguez. 2022. Fracture-Aware Tessellation of Subdivision Surfaces. In ACM SIGGRAPH 2022 Talks. Article 10.

Corey Butler, Brent Burley, Wei-Feng Wayne Huang, Yining Karl Li, and Benjamin Huang. 2022. “Encanto” - Let’s Talk About Bruno’s Visions. In ACM SIGGRAPH 2022 Talks. Article 8.

Matt Jen-Yuan Chiang and Brent Burley. 2018. Plausible Iris Caustics and Limbal Arc Rendering. In ACM SIGGRAPH 2018 Talks. Article 15.

Alejandro Conty and Christopher Kulla. 2017. Production Friendly Microfacet Sheen BRDF. In ACM SIGGRAPH 2017 Course Notes: Physically Based Shading in Theory and Practice.

Henrik Dahlberg, David Adler, and Jeremy Newlin. 2019. Machine-Learning Denoising in Feature Film Production. In ACM SIGGRAPH 2019 Talks. Article 21.

Andrew Finley, Jesse Erickson, Peter De Mund, and Ying Liu. 2022. Modeling Animated Jumbo Floral Display on Disney’s “Encanto”. In ACM SIGGRAPH 2022 Talks. Article 43.

Haixiang Liu. 2022. Gravity Preloading for Maintaining Hair Shape Using the Simulator as a Closed-box Function. In ACM SIGGRAPH 2022 Talks. Article 40.

Tad Miller, Harmony M. Li, Neelima Karanam, Nadim Sinno, and Todd Scopio. 2022. Making Encanto with USD: Rebuilding a Production Pipeline Working from Home. In ACM SIGGRAPH 2022 Talks. Article 12.

Thomas Müller. 2019. Practical Path Guiding in Production. In ACM SIGGRAPH 2019 Course Notes: Path Guiding in Production. 37-50.

Thomas Müller, Markus Gross, and Jan Novák. 2017. Practical Path Guiding for Efficient Light-Transport Simulation. Computer Graphics Forum (Proc. of Eurographics Symposium on Rendering) 36, 4 (Jun. 2017), 91-100.

Michelle Robinson, Michael Woodside, Daniel Rice, Tad Miller, Scott Kersavage, and Tyler Kupferer. 2022. We Don’t Talk About Bruno - An Encanto Musical Sequence Unveiled. In ACM SIGGRAPH 2022 Production Sessions. Article 2.

Justin Tennant, Mitch Counsell, Far Jangtrakool, Salina Ortega, Rajesh Sharma, Tad Miller, and Scott Kersavage. 2022. Visualizing the Production Process of “Encanto” with the Command Center. In ACM SIGGRAPH 2022 Talks. Article 11.

Jose Velasquez, Alexander Alvarado, Ying Liu, and Maryann Simmons. 2022. Embroidery and Cloth Fiber Workflows on Disney’s “Encanto”. In ACM SIGGRAPH 2022 Talks. Article 22.

Tizian Zeltner, Brent Burley, and Matt Jen-Yuan Chiang. 2022. Practical Multiple-Scattering Sheen Using Linearly Transformed Cosines. In ACM SIGGRAPH 2022 Talks. Article 7.