Table of Contents

Introduction

A few years ago I ported Takua Renderer to build and run on arm64 systems. Porting to arm64 proved to be a major effort (see Parts 1, 2, 3, and 4) which wound up paying off in spades; I learned a lot, found and fixed various longstanding platform-specific bugs in the renderer, and wound up being perfectly timed for Apple transitioning the Mac to arm64-based Apple Silicon. As a result, for the past few years I have been routinely building and running Takua Renderer on arm64 Linux and macOS, in addition to building and runninng on x86-64 Linux/Mac/Windows. Even though I take somewhat of a Mac-first approach for personal projects since I daily drive macOS, I make a point of maintaining robust cross-platform support for Takua Renderer for reasons I wrote about in the first part of this series.

Up until recently though, my supported platforms list for Takua Renderer notably did not include Windows on Arm. There are two main reasons why I never ported Takua Renderer to build and run on Windows on Arm. The first reason is that Microsoft’s own support for Windows on Arm has up until recently been in a fairly nascent state. Windows RT added Arm support in 2012 but only for 32-bit processors, and Windows 10 added arm64 support in 2016 but lacked a lot of native applications and developer support; notably, Visual Studio didn’t gain native arm64 support until late in 2022. The second reason I never got around to adding Windows on Arm support is simply that I don’t have any Windows on Arm hardware sitting around and generally there just have not been many good Windows on Arm devices available in the market. However, with the advent of Qualcomm’s Oryon-based Snapdragon X SoCs and Microsoft’s push for a new generation of arm64 PCs using the Snapdragon X SoCs, all of the above finally seems to be changing. Microsoft also authorized arm64 editions of Windows 11 for use in virtual machines on Apple Silicon Macs at the beginning of this year. With Windows on Arm now clearly signaled as a major part of the future of Windows and clearly signaled as here to stay, and now that spinning up a Windows 11 on Arm VM is both formally supported and easy to do, a few weeks ago I finally got around to getting Takua Renderer up and running on native arm64 Windows 11.

Overall this process was very easy compared with my previous efforts to add support for arm64 Mac and Linux. This was not because porting architectures is easier on Windows but rather is a consequence of the fact that I had already solved all of the major architecture-related porting problems for Mac and Linux; the Windows 11 on Arm port just piggy-backed on those efforts. Because of how relatively straightforward this process was, this will be a shorter post, but there were a few interesting gotchas and details that I think are worth noting in case they’re useful to anyone else porting graphics stuff to Windows on Arm.

Note that everything in this post uses arm64 Windows 11 Pro 23H2 and Visual Studio 2022 17.10.x. Noting the specific versions used here is important since Microsoft is still actively fleshing out arm64 support in Windows 11 and Visual Studio 2022; later versions will likely see improvements to problems discussed in this post.

OpenGL on arm64 Windows 11

Takua has two user interface systems: a macOS-specific UI written using a combination of Dear Imgui, Metal, and AppKit, and a cross-platform UI written using a combination of Dear Imgui, OpenGL, and GLFW. On macOS, OpenGL is provided by the operating system itself as part of the standard system frameworks. On most desktop Linux distributions, OpenGL can be provided by several different sources: one option is entirely through the operating system’s provided Mesa graphics stack, another option is through a combination of Mesa for the graphics API and a proprietary driver for the backend hardware support, and the last option is entirely through a proprietary driver (such as with Nvidia’s official drivers). On Windows, however, the operating system does not provide modern OpenGL (“modern” meaning OpenGL 3.3 or newer), support whatsoever and the OpenGL 1.1 support that is available is a wrapper around Direct3D; modern OpenGL support on Windows has to be provided entirely by the graphics driver.

I don’t actually have any native arm64 Windows 11 hardware, so for this porting project, I ran arm64 Windows 11 as a virtual machine on two of my Apple Silicon Macs. I used the excellent UTM app (which under the hood uses QEMU) as the hypervisor. However, UTM does not provide any kind of GPU emulation/virtualization to Windows virtual machines, so the first problem I ran into was that my arm64 Windows 11 environment did not have any kind of modern OpenGL support due to the lack of a GPU driver with OpenGL. Therefore, I had no way to build and run Takua’s UI system.

Fortunately, because OpenGL is so widespread in commonly used applications and games, this is a problem that Microsoft has already anticipated and come up with a solution for. A few years ago, Microsoft developed and released an OpenGL/OpenCL Compatability Pack for Windows on Arm, and they’ve since also added Vulkan support to the compatability pack as well. The compatability pack is available for free on the Windows Store. Under the hood, the compatability pack uses a combination of Microsoft-developed client drivers and a bunch of components from Mesa to translate from OpenGL/OpenCL/Vulkan to Direct3D [Jiang 2020]. This system was originally developed to provide support for specifically Photoshop on arm64 Windows, but has since been expanded to provide general OpenGL 3.3, OpenCL 3.0, and Vulkan 1.2 support to all applications on arm64 Windows. Installing the compatability pack allowed me to get GLFW building and to get GLFW’s example demos working.

Takua’s cross-platform UI is capable of running either using OpenGL 4.5 on systems with support for the latest fanciest OpenGL API version, or using OpenGL 3.3 on systems that only have older OpenGL support (examples include macOS when not using the native Metal-based UI and include many SBC Linux devices such as Raspberry Pi). Since the arm64 Windows compatability pack only fully supports up to OpenGL 3.3, I set up Takua’s arm64 Windows build to fall back to only use the OpenGL 3.3 code path, which was enough to get things up and running. However, I immediately noticed that everything in the UI looked wrong; specifically, everything was clearly not in the correct color space.

The problem turned out to be that the Windows OpenGL/OpenCL/Vulkan compatability pack doesn’t seem to correctly implement GL_FRAMEBUFFER_SRGB; calling glEnable(GL_FRAMEBUFFER_SRGB) did not have any impact on the actual color space that the framebuffer rendered with. To work around this problem, I simply added software sRGB emulation to the output fragment shader and added some code to detect if GL_FRAMEBUFFER_SRGB was working or not and if not, fall back to the fragment shader’s implementation. Implementing the sRGB transform is extremely easy and is something that every graphics programmer inevitably ends up doing a bunch of times throughout one’s career:

float sRGB(float x) {
    if (x <= 0.00031308)
        return 12.92 * x;
    else
        return 1.055*pow(x,(1.0 / 2.4) ) - 0.055;
}

With this fix, Takua’s UI now fully works on arm64 Windows 11 and displays renders correctly:

Building Embree on arm64 Windows 11

Takua has a moderately sized dependency base, and getting all of the dependency base compiled during my ports to arm64 Linux and arm64 macOS was a very large part of the overall effort since arm64 support across the board was still in an early stage in the graphics field three years ago. However, now that libraries such as Embree and OpenEXR and even TBB have been building and running on arm64 for years now, I was expecting that getting Takua’s full dependency base brought up on Windows on Arm would be straightforward. Indeed this was the case for everything except Embree, which proved to be somewhat tricky to get working. I was surprised that Embree proved to be difficult, since Embree for a few years now has had excellent arm64 support on macOS and Linux. Thanks to a contribution from Apple’s Developer Ecosystem Engineer team, arm64 Embree now even has a neat double-pumped NEON option for emulating AVX2 instructions.

As of the time of writing this post, compiling Embree 4.3.1 for arm64 using MSVC 19.x (which ships with Visual Studio 2022) simply does not work. Initially just to get the renderer up and running in some form at all, I disabled Embree in the build. Takua has both an Embree-based traversal system and a standalone traversal system that uses my own custom BVH implementation; I keep both systems at parity with each other because Takua at the end of the day is a hobby renderer that I work on for fun, and writing BVH code is fun! However, a secondary reason for keeping both traversal systems around is because in the past having a non-Embree code path has been useful for getting the renderer bootstrapped on platforms that Embree doesn’t fully support yet, and this was another case of that.

Right off the bat, building Embree with MSVC runs into a bunch of problems with detecting the platform as being a 64-bit platform and also runs into all kinds of problems with including immintrin.h, which is where vector data types and other x86-64 intrinsics stuff is defined. After hacking my way through solving those problems, the next issue I ran into is that MSVC really does not like how Embree carries out static initialisation of NEON datatypes; this is a known problem in MSVC. Supposedly this issue was fixed in MSVC some time ago, but I haven’t been able to get it to work at all. Fixing this issue requires some extensive reworking of how Embree does static initialisation of vector datatypes, which is not a very trivial task; Anthony Roberts previously attempted to actually make these changes in support of getting Embree on Windows on Arm working for use in Blender, but eventually gave up since making these changes while also making sure Embree still passes all of its internal tests proved to be challenging.

In the end, I found a much easier solution to be to just compile Embree using Visual Studio’s version of clang instead of MSVC. This has to be done from the command line; I wasn’t able to get this to work from within Visual Studio’s regular GUI. From within a Developer PowerShell for Visual Studio session, the following worked for me:

cmake -G "Ninja" ../../ -DCMAKE_C_COMPILER="clang-cl" `
                        -DCMAKE_CXX_COMPILER="clang-cl" ` 
                        -DCMAKE_C_FLAGS_INIT="--target=arm64-pc-windows-msvc" `
                        -DCMAKE_CXX_FLAGS_INIT="--target=arm64-pc-windows-msvc" `
                        -DCMAKE_BUILD_TYPE=Release `
                        -DTBB_ROOT="[TBB LOCATION HERE]" `
                        -DCMAKE_INSTALL_PREFIX="[INSTALL PREFIX HERE]"

To do the above, of course you will need both CMake and Ninja installed; fortunately both come with pre-built arm64 Windows binaries on their respective websites. You will also need to install the “C++ Clang Compiler for Windows” component in the Visual Studio Installer application if you haven’t already.

Just building with clang is also the solution that Blender eventually settled on for Windows on Arm, although Blender’s version of this solution is a bit more complex since Blender builds Embree using its own internal clang and LLVM build instead of just using the clang that ships with Visual Studio.

An additional limitation in compiling Embree 4.3.1 for arm64 on Windows right now is that ISPC support seems to be broken. On arm64 macOS and Linux this works just fine; the ISPC project provides prebuilt arm64 binaries on both platforms, and even without a prebuilt arm64 binary, I found that running the x86-64 build of ISPC on arm64 macOS via Rosetta 2 worked without a problem when building Embree. However, on arm64 Windows 11, even though the x86-64 emulation system ran the x86-64 build of ISPC just fine standalone, trying to run it as part of the Embree build didn’t work for me despite me trying a variety of ways to get it to work. I’m not sure if this works with a native arm64 build of ISPC; building ISPC is a sufficiently involved process that I decided it was out of scope for this project.

Running x86-64 code on arm64 Windows 11

Much like how Apple provides Rosetta 2 for running x86-64 applications on arm64 macOS, Microsoft provides a translation layer for running x86 and x86-64 applications on arm64 Windows 11. In my post on porting to arm64 macOS, I included a lengthy section discussing and performance testing Rosetta 2. This time around, I haven’t looked as deeply into x86-64 emulation on arm64 Windows, but I did do some basic testing. Part of why I didn’t go as deeply into this area on Windows is because I’m running arm64 Windows 11 in a virtual machine instead of on native hardware- the comparison won’t be super fair anyway. Another part of why I didn’t go in as deeply is because x86-64 emulation is something that continues to be in an active state of development on Windows; Windows 11 24H2 is supposed to introduce a new x86-64 emulation system called Prism that Microsoft promises to be much faster than the current system in 23H2 [Mehdi 2024]. As of writing though, little to no information is available yet on how Prism works and how it improves on the current system.

The current system for emulating x86 and x86-64 on arm64 Windows is a fairly complex system that differs greatly from Rosetta 2 in a lot of ways. First, arm64 Windows 11 supports emulating both 32-bit x86 and 64-bit x86-64, whereas macOS dropped any kind of 32-bit support long ago and only needs to support 64-bit x86-64 on 64-bit arm64. Windows actually handles 32-bit x86 and 64-bit x86-64 through two basically completely different systems. 32-bit x86 is handled through an extension of the WoW64 (Windows 32-bit on Windows 64-bit) system, while 64-bit x86-64 uses a different system. The 32-bit system uses a JIT compiler called xtajit.dll [Radich et al. 2020, Beneš 2018] to translate blocks of x86 assembly to arm64 assembly and has a caching mechanism for JITed code blocks similar to Rosetta 2 to speed up execution of x86 code that has already been run through the emulation system before [Cylance Research Team 2019]. In the 32-bit system, overall support for providing system calls and whatnot are handled as part of the larger WoW64 system.

The 64-bit system relies on a newer mechanism. The core binary translation system is similar to the 32-bit system, but providing system calls and support for the rest of the surrounding operatin system doesn’t happen through WoW64 at all and instead relies on something that is in some ways similar to Rosetta 2, but is in other crucial ways radically different from Rosetta 2 or the 32-bit WoW64 approach. In Rosetta 2, arm64 code that comes from translation uses a completely different ABI from native arm64 code; the translated arm64 ABI contains a direct mapping between x86-64 and arm64 registers. Microsoft similarly uses a different ABI for translated arm64 code compared with native arm64 code; in Windows, translated arm64 code uses the arm64EC (EC for “Emulation Compatible”) ABI. Here though we find the first major difference between the macOS and Windows 11 approaches. In Rosetta 2, the translated arm64 ABI is an internal implementation detail that is not exposed to users or developers whatsoever; by default there is no way to compile source code against the translated arm64 ABI in Xcode. In the Windows 11 system though, the arm64EC ABI is directly available to developers; Visual Studio 2022 supports compiling source code against either the native arm64 or the translation-focused arm64EC ABI. Code built as arm64EC is capable of interoperating with emulated x86-64 code within the same process, the idea being that this approach allows developers to incrementally port applications to arm64 piece-by-piece while leaving other pieces as x86-64 [Sweetgall et al. 2023]. This… is actually kind of wild if you think about it!

The second major difference between the macOS and Windows 11 approaches is even bigger than the first. On macOS, application binaries can be fat binaries (Apple calls these universal binaries), which contain both full arm64 and x86-64 versions of an application and share non-code resources within a single universal binary file. The entirety of macOS’s core system and frameworks ship as universal binaries, such that at runtime Rosetta 2 can simply translate both the entirety of the user application and all system libraries that the application calls out to into arm64. Windows 11 takes a different approach- on arm64, Windows 11 extends the standard Windows portable executable format (aka .exe files) to be a hybrid binary format called arm64X (X for eXtension). The arm64X format allows for arm64 code compiled against the arm64EC ABI and emulated x86-64 code to interoperate within the same binary; x86-64 code in the binary is translated to arm64EC as needed. Pretty much every 64-bit system component of Windows 11 on Arm ships as arm64X binaries [Niehaus 2021]. Darek Mihocka has a fantastic article that goes into extensive depth about how arm64EC and arm64X work, and Koh Nakagawa has done an extensive analysis of this system as well.

One thing that Windows 11’s emulation system does not seem to be able to do is make special accomodations for TSO memory ordering. As I explored previously, Rosetta 2 gains a very significant performance boost from Apple Silicon’s hardware-level support for emulating x86-64’s strong memory ordering. However, since Microsoft cannot control and custom tailor the hardware that Windows 11 will be running on, arm64 Windows 11 can’t make any guarantees about hardware-level TSO memory ordering support. I don’t know if this situation is any different with the new Prism emulator running on the Snapdragon X Pro/Elite, but in the case of the current emulation framework, the lack of hardware TSO support is likely a huge problem for performance. In my testing of Rosetta 2, I found that Takua typically ran about 10-15% slower as x86-64 under Rosetta 2 with TSO mode enabled (the default) compared with native arm64, but ran 40-50% slower as x86-64 under Rosetta 2 with TSO mode disabled compared with native arm64.

Below are some numbers comparing running Takua on arm64 Windows 11 as a native arm64 application versus as an emulated x86-64 application. The tests used are the same as the ones I used in my Rosetta 2 tests, with the same settings as before. In this case though, because this was all running in a virtual machine (with 6 allocated cores) instead of directly on hardware, the absolute numbers are not as important as the relative difference between native and emulated modes:

The emulated results are… not great; for compute-heavy workloads like path tracing, x86-64 emulation on arm64 Windows 11 seems to to be around 1.7x to 3x slower than native arm64 code. These results are much slower compared with how Rosetta 2 performs, which generally sees only a 10-15% performance penalty over native arm64 when running Takua Renderer. However, a critical caveat has to be pointed out here: reportedly Windows 11’s x86-64 emulation works worse in a VM on Apple Silicon than it does on native hardware because Arm RCpc instructions on Apple Silicon are relatively slow. For Rosetta 2 this behavior doesn’t matter because Rosetta 2 uses TSO mode instead of RCpc instructions for emulating strong memory ordering, but since Windows on Arm does rely on RCpc for emulating strong memory ordering, this means that the results above are likely not fully representative of emulation performance on native Windows on Arm hardware. Nonetheless though, having any form of x86-64 emulation at all is an important part of making Windows on Arm viable for mainstream adoption, and I’m looking forward to see how much of an improvement the new Prism emulation system in Windows 11 24H2 brings. I’ll update these results with the Prism emulator once 24H2 is released, and I’ll also update these results to show comparisons on real Windows on Arm hardware whenever I actually get some real hardware to try out.

Conclusion

I don’t think that x86-64 is going away any time soon, but at the same time, the era of mainstream desktop arm64 adoption is here to stay. Apple’s transition to arm64-based Apple Silicon already made the viability of desktop arm64 unquestionable, and now that Windows on Arm is finally ready for the mainstream as well, I think we will now be living in a multi-architecture world in the desktop computing space for a long time. Having more competitors driving innovation ultimately is a good thing, and as new interesting Windows on Arm devices enter the market alongside Apple Silicon Macs, Takua Renderer is ready to go!

References

ARM Holdings. 2022. Load-Acquire and Store-Release instructions. Retrieved June 7, 2024.

Petr Beneš. 2018. Wow64 Internals: Re-Discovering Heaven’s Gate on ARM. Retrieved June 5, 2024.

Cylance Research Team. 2019. Teardown: Windows 10 on ARM - x86 Emulation. In BlackBerry Blog. Retrieved June 5, 2024.

Angela Jiang. 2020. Announcing the OpenCL™ and OpenGL® Compatibility Pack for Windows 10 on ARM. In DirectX Developer Blog. Retrieved June 5, 2024.

Yusuf Mehdi. 2024. Introducing Copilot+ PCs. In Official Microsoft Blog. Retrieved June 5, 2024.

Derek Mihocka. 2024. ARM64 Boot Camp. Retrieved June 5, 2024.

Koh M. Nakagawa. 2021. Discovering a new relocation entry of ARM64X in recent Windows 10 on Arm. In Project Chameleon. Retrieved June 5, 2024.

Koh M. Nakagawa. 2021. Relock 3.0: Relocation-based obfuscation revisited in Windows 11 on Arm. In Project Chameleon. Retrieved June 5, 2024.

Michael Niehaus. 2021. Running x64 on Windows 10 ARM64: How the heck does that work?. In Out of Office Hours. Retrieved June 5, 2024.

Quinn Radich, Karl Bridge, David Coulter, and Michael Satran. 2020. WOW64 Implementation Details. In Programming Guide for 64-bit Windows. Retrieved June 5, 2024.

Marc Sweetgall, Drew Batchelor, Scott Jones, and Matt Wojciakowski. 2023. Arm64EC - Build and port apps for native performance on ARM. Retrieved June 5, 2024.

Wikipedia. 2024. WoW64. Retrieved June 5, 2024.

Once Upon A Studio is Disney Animation’s newest short, which accompanied Wish’s theatrical release last fall. Together, Wish and Once Upon A Studio celebrate 100 years of Disney Animation; while Wish tells a new story inspired by themes from across the past century of Disney animated films, Once Upon A Studio brings together 543 characters from past Disney Animation features and shorts into a single short, set in the Roy E. Disney Animation Building in Burbank that serves as the studio’s modern headquarters and production facility. I only played a small role on making Once Upon A Studio, providing the usual technical assistance and support for the artists that used Disney’s Hyperion Renderer to make the short. Even only being a small part of this short was still enormously exciting though! This post is a small collection of my thoughts on some of the more interesting rendering challenges that we encountered while making Once Upon A Studio, and also I’ll include some thoughts on what the short and 100 years of Disney Animation means to me.

Once Upon A Studio features an ambitious blend of live action plates, traditional hand-drawn animation, and CG animation in order to depict hundreds of Disney Animation characters from across the studio’s entire history all gathering together. All traditional hand-drawn animation was done the old-school way using paper and pencil for the actual animation and with digital ink and paint, while all CG animation and backgrounds were rendered entirely using Disney’s Hyperion Renderer. The real challenge in making this short came in figuring out how to seamlessly integrate all of these components in a believable fashion; despite having completely different visual styles and completely different animation techniques, everything had to sit together and interact in the same world.

Before even getting into the characters, one interesting challenge was in depicting the Disney Animation building. The short is set in a somewhat-fictionalized version of our building; some of the interior locations seen in the short are real locations in the building, others are completely made up areas that look like real parts of the building but don’t actually exist, and there are even some shots that transition from a real location to a fictional one in a single continuous camera move. Most of the real locations in the building are photographic plates, while all fictional locations are completely CG and rendered using Hyperion. For the shots where the camera transitions from real to fictional location, this means we had to be able to create renders using Hyperion that were photorealistic enough to be completely visually indistinguishable from the photographic plates. In some cases this meant Hyperion renders had to be realistic enough to be composited into part of the photographic plate (for example, when there was a real photographed hallway looking into a fictional room), while in a few other extraordinary cases, we had to be able to swap from a 100% photographic plate to a 100% exact CG replica within a single frame. I think our artists did an incredible job with this; even if you step through the short frame by frame, I don’t think anyone would be able to tell where these seams are without already knowing beforehand.

From a rendering engineer perspective, the big question that having to match photographic plates raised was simply: can we actually do it? While matching photographic plates is very common in VFX workflows, it is something that Disney Animation hasn’t had to do in decades. Hyperion is a renderer designed for and custom-tailored to Disney Animation’s very specific needs [Burley et al. 2018], and since matching photographic plates isn’t something we usually have to do, we never explicitly considered that use case in the renderer’s design. Core parts of the renderer, such as the shading system, are designed with physically based rendering principles as the foundation [Burley 2012], but our target look has always been animation, so outside of some experiments early in the Hyperion project’s history, we’d never spent much time evaluating the shading system’s ability to achieve exact matches to true photographs. Fortunately, we found that Hyperion’s physically based foundations and shading flexibility were more than enough to achieve seamless matches with photographic plates; in fact, our artists reported that this wasn’t even particularly difficult, which was very cool to see.

The single largest technical challenge on Once Upon A Studio wasn’t matching plates though; the single largest challenge by far was having traditional hand-drawn characters interact with CG characters in a visually cohesive manner while being 100% truthful to their respective native mediums. All of the traditional hand-drawn characters in the short were created by an incredible team of in-house traditional animators exclusively using pencil and paper (this team even included several legendary ex-Disney traditional animators who came back just for the occasion, including heroes of mine such as James Baxter and Tony Bancroft). For shots where hand-drawn and CG characters interact, depending on what was happening in the shot, either the hand-drawn character would be animated first and then the CG animation done afterwards to match, or the CG animation would be completed first and the frames would then be printed out on paper for hand-drawn animation to draw on top of before being scanned back in. In a few shots, both the hand-drawn and CG animation were done by a single animator who was proficient in both mediums; Disney Animation has several animators who are masters in both disciplines, which absolutely amazes and boggles my mind.

Any time a hand-drawn character has to interact with CG elements, the solution we took was to bring the hand-drawn character into the 3D world by projecting the hand-drawn animation with final ink and paint onto cards in 3D space. This way, the hand-drawn and CG characters and environments could be rendered out by Hyperion together in a single pass. More importantly, this meant that hand-drawn characters could participate in global illumination, casting proper indirect illumination and shadows on CG elements and showing up in reflections and refractions correctly and so on. Of course a ton of further adjustments had to be made in compositing, but having a correct basis coming out of the renderer provided an important foundation for making all of this work. One part of the short that I think demonstrates this combination particularly well is when Genie bursts out of an animation desk that Olaf is working at; the desk and chair are a combination of photographic plate and CG elements, Olaf is obviously completely CG, and Genie is hand-drawn. When Genie bursts out of the desk, he shows up properly in reflections in the office’s window, which is a CG replacement for the photographic plate. Genie’s blue glow properly illuminates Olaf, and you can even see Genie’s blue glow diffusing through Olaf’s snowy body via path traced subsurface scattering.

Another major challenge was porting all of Disney Animation’s CG characters forward to work in the latest version of our pipeline and render using the latest version of Hyperion. Depending on how old the character was, the level of effort required ranged anywhere from “works perfectly with a click of a single button” to “rebuild the character completely from scratch”. As with any other modern CG studio, Disney Animation’s pipeline is constantly evolving with each successive show, with the modern pipeline tracing its earliest version back to the production of Tangled. Characters from shows before Tangled had to be completely rebuilt from scratch, using the originals as references, while for characters from after Tangled, things varied based on how far back the character was from and whether or not the character had already been ported forward before. While the current pipeline traces its origins to Tangled, essentially every part of the pipeline has since undergone a complete Ship of Theseus styled transformation. From Tangled to the modern day, the studio’s production renderer switched from RenderMan to Hyperion, the underlying data formats across the entire studio moved from proprietary formats to exclusively USD, we moved from Maya to Presto as our core animation DCC and from an internal tool to Houdini’s Solaris system as our lighting DCC, and on top of those big changes there are an uncountable number of other changes.

On the rendering front, pretty much the only thing that is the same since Tangled is that we use Ptex [Burley and Lacewell 2008] and SeExpr [Disney Animation 2011] as the foundation of our shading system; literally everything else has changed, with the largest breaking points being the move from legacy shaders to physically based shading and the Disney BSDF between Tangled and Wreck-it Ralph, and the move to Hyperion between Frozen and Big Hero 6. Fortunately, because every show since Wreck-it Ralph has made use of the Disney BSDF, at least as far as rendering and lookdev is concerned, porting forward Disney BSDF based characters is relatively easy. The Disney BSDF has continuously evolved over many shows, so sometimes characters need to be tuned up a bit to use the latest Disney BSDF features, but usually this is pretty straightforward. The trickiest things to tune up are porting characters from using older normalized diffusion based subsurface scattering [Burley 2015] to our modern path-traced subsurface scattering [Chiang et al. 2016b], and porting eyes from using our older eye shader to our modern manifold next-event estimation based eye shader [Chiang and Burley 2018]. Even more fortuitously, older RenderMan based characters from Frozen and Wreck-it Ralph had already been ported to Hyperion for Frozen 2 and Ralph Breaks the Internet, which in the past was made a lot easier by the fact that the Disney BSDF spans both renderers. Shows before Zootopia also used an older ad-hoc hair shader instead of the modern industry-standard Chiang hair model [Chiang et al. 2016a], but fortunately most Disney BSDF based shows before Zootopia already had their main characters ported forward, while the remaining characters mostly were from shows before Tangled and needed to be rebuilt from scratch anyway. For the case of Asha from Wish, we actually had to backport new renderer features from Wish’s pipeline into Once Upon A Studio’s pipeline, which used a slightly earlier version of the renderer from the end of Moana 2’s production.

Getting to be around for and occasionally help out with Once Upon A Studio was an incredibly cool experience; as a deep animation nerd, being at Disney Animation for the studio’s 100th anniversary is a once-in-a-lifetime sort of thing, and Once Upon A Studio is the ultimate animation nerd’s project. In general I think this project speaks to why I love being at Disney Animation in particular; Disney Animation uniquely combines a rich 100 year old legacy of animation tradition built by the greatest to ever do it with a 100 year old charter to constantly look forwards and push animation technology into the future; this combination was the philosophy that Walt Disney founded the studio with and that the studio still operates under today. One way this combination of simultaneously valuing tradition and looking to the future manifests itself in concrete reality is that Disney Animation is one of the very few places on Earth where you can find some of the greatest traditional hand-drawn animators to ever live, and legendary visdev artists, and people with math and physics and computer science PhDs, and people who are experts in building and operating supercomputers, and everyone in between, all under the same roof working on the same projects. You rarely find beautiful hand-drawn storyboards and character sketches pinned up on boards and partial differential equations and code architecture diagrams scrawled on whiteboards in offices right next to each other (or sometimes even in the same room), and yet, that’s just every day life at Disney Animation. Once Upon A Studio celebrates 100 years of Disney Animation, but more than that it’s really a celebration of everything that comes together to make Disney Animation possible.

Once Upon A Studio is available on the Wish Blu-ray, on Disney+, and for free on Disney Animation’s Youtube channel. Here are some of my favorite frames from the short, present in no particular order. Go watch the short on the biggest screen you can! Also, if you want to learn more about how the short was made, the studio released a longer production brief on the topic with a lot of interesting details I haven’t covered here.

All images in this post are courtesy of and the property of Walt Disney Animation Studios.

References

Brent Burley. 2012. Physically Based Shading at Disney. In ACM SIGGRAPH 2012 Course Notes: Practical Physically-Based Shading in Film and Game Production.

Brent Burley. 2015. Extending the Disney BRDF to a BSDF with Integrated Subsurface Scattering. In ACM SIGGRAPH 2015 Course Notes: Physically Based Shading in Theory and Practice.

Brent Burley, David Adler, Matt Jen-Yuan Chiang, Hank Driskill, Ralf Habel, Patrick Kelly, Peter Kutz, Yining Karl Li, and Daniel Teece. 2018. The Design and Evolution of Disney’s Hyperion Renderer. ACM Transactions on Graphics 37, 3 (Jul. 2018), Article 33.

Brent Burley and Dylan Lacewell. 2008. Ptex: Per-face Texture Mapping for Production Rendering. Computer Graphics Forum (Proc. of Eurographics Symposium on Rendering) 27, 4 (Jun. 2008), 1155-1164.

Matt Jen-Yuan Chiang, Benedikt Bitterli, Chuck Tappan, and Brent Burley. 2016. A Practical and Controllable Hair and Fur Model for Production Path Tracing. Computer Graphics Forum (Proc. of Eurographics) 35, 2 (May 2016), 275-283.

Matt Jen-Yuan Chiang, Peter Kutz, and Brent Burley. 2016. Practical and Controllable Subsurface Scattering for Production Path Tracing. In ACM SIGGRAPH 2016 Talks. Article 49.

Matt Jen-Yuan Chiang and Brent Burley. 2018. Plausible Iris Caustics and Limbal Arc Rendering. In ACM SIGGRAPH 2018 Talks. Article 15.

Walt Disney Animation Studios. 2011. SeExpr.

Disney Animation’s fall 2023 release is Wish, which is the studio’s 62nd animated feature and also the studio’s film celebrating the 100th anniversary of Disney Animation and The Walt Disney Company. Wish is a brand new story but is also steeped in the past century of Disney storytelling; while Asha and Star and Valentino’s adventure is a new musical story, the themes and setting Wish draws upon are timeless and classic. As part of this theme of modern Disney with throwback elements, Wish also has a unique beautiful visual style that combines the latest of our computer graphics animation with a classic watercolor style. Creating this style presented an interesting set of new challenges for our artists, TDs, engineers, and for Disney’s Hyperion Renderer.

At Disney Animation, every one of our films is a new opportunity for us to push our filmmaking art and technology forward. On most films this advancement takes place across many different aspects of the film, but on Wish, there is one obvious challenge that stood out above everything else: the film’s visual style. Of course we still made large improvements in other areas, such as major pipeline optimizations [Li et al. 2024a], but on Wish a large proportion of technology development was focused on achieving the target visual style. One could be forgiven for thinking that stylization is mostly a rendering problem, but on Wish it really was a challenge that reached into every department and every corner or our production process. Stylization on Wish meant stylization in modeling, lookdev, animation, effects, lighting, everything else in between, and even new pipeline challenges!

The decision to give Wish a unique style came from pretty much the very beginning of the project; the studio wanted to do something special for the 100th anniversary film to tie together our modern way of making animated films with the studio’s rich hand-drawn heritage. The look of Wish is especially influenced by early 20th century Disney traditional watercolor animation, with Snow White and the Seven Dwarfs (1937) and Sleeping Beauty (1959) being the largest guideposts. This influence extends all the way to the very shape of the film, so to speak- Wish is the first CG film that Disney Animation has made in the ultrawide 2.55:1 Cinemascope-style aspect ratio, matching the aspect ratio that was used on Sleeping Beauty and Lady and the Tramp (1955). This aspect ratio choice meant that stylization on Wish even impacted layout, since they had to think about how to frame for such an ultrawide image!

Disney Animation has a long history of stylizing 3D CG to resemble and fit in with hand-drawn animation, going all the way back to the studio’s traditional hand-drawn era [Meier 1996, Daniels 1999, Tamstorf et al. 2001, Odermatt and Springfield 2002, Teece 2003]. In the 3D CG era, Disney Animation has continually experimented with stylizing CG as well with a number of different approaches. Paperman focused on integrating 2D linework with 3D rendered characters [Kahrs et al. 2012 ,Whited et al. 2012], while Feast experimented with driving 2D lighting entirely in compositing on top of flat unlit/unshaded 3D renders [Osborne and Staub 2014]. The studio’s recent Short Circuit experimental shorts program had many shorts that experimented with a variety of different stylized looks, ranging from Chinese ink brush watercolors to graphic 2D illustrations to stop motion and wood carved looks [Newfield and Staub 2020]. Disney Research has also worked closely with Disney Animation in the past decade plus to develop various experimental stylization techniques [Schmid et al. 2011, Sýkora et al. 2014]. Both Strange World and Raya and the Last Dragon had small amounts of stylized sequences as well, with Strange World doing a 1950s pulp scifi comic book look and Raya and the Last Dragon doing a more graphic digital mixed media sort of look. Most recently, the short Far From the Tree utilized a look with cel-shaded characters on watercolor backgrounds. All of these were animated using our standard 3D pipeline, with much of the look being built using a combination of render passes from the renderer (Hyperion for everything except Paperman, which preceded Hyperion’s existence by a few years), various tricks in lighting, and a lot of work in compositing; how much of each was used varied widely per show and per target style.

Wish builds upon all of these predecessors. Wish’s stylization system is a vastly expanded version of what was used on Far From the Tree, which in turn was built on top of everything that was developed for Short Circuit, which in turn drew upon lessons from both Feast and Paperman. One of the biggest challenges came from having to scale up a stylization pipeline from a short film to a full feature length project, while also trying to hit a new target style. Early tests on the show were able to reproduce in CG the target visdev paintings essentially exactly, but through entirely ad-hoc and mostly manual approaches, which we then had to systematically take apart and figure out how to apply to the whole movie. My wife, Harmony Li, was an Associate Technical Supervisor on the show and (among a ton of other things) oversaw the development of the entire technical backend that was built out to support stylization on Wish [Li et al. 2024b]; as a member of the rendering team, I got to work with her on this, which was great fun! Meanwhile, much of the development for the actual techniques used was led primarily by lighting and lookdev artists.

An early breakthrough in achieving Wish’s style was finding that combining Kuwahara filters¹ [Kuwahara et al. 1976] with linework generated from the renderer created a convincing starting point for a line-on-watercolor look that could use the renderer’s physical lighting as a starting point, instead of needing to construct stylized lighting entirely from scratch in comp on top of flat-shaded unlit renders. To help really tie together the watercolor look, early tests put the entire image on a watercolor paper texture background, but once we tested the watercolor paper texture background in motion, some issues became apparent. With just a static watercolor paper texture background, the illusion of motion broke as animation looked like it was “swimming” through the texture, but simply texturing everything with watercolor textures in 3D space looked downright bizarre since it looked less like the frame was a watercolor painting and more like all of the characters and the environment were made out of paper. To solve this problem, the Hyperion team invented a new dynamic screen space texture technique [Burley et al. 2024] where the renderer would project screen space textures onto 3D surfaces while tracking motion vectors. The result is that Wish’s watercolor backgrounds look like just a flat sheet of watercolor paper when still, but under motion convincingly move with the characters while neither looking like they’re actually in 3D space nor looking disconnected from motion.

One interesting question I worked on for Wish was making Hyperion robustly handle shading normals that are really dramatically disconnected from the underlying “physically correct” geometric normal. Extreme bending of normals was used extensively on Wish to simplify shapes and art-direct lighting detail and shadows. In a normal physically based path traced render, shading normals coming from bump mapping and normal mapping usually have at least some relationship with the underlying true geometric normal, meaning that the ways shading normals modify light transport are relatively constrained to a plausible range. However, on Wish, extreme shading normals were used for things like simplifying the lighting on an entire complex tree canopy to match what the lighting would be on a simple sphere. Making Hyperion handle these cases both from an authoring perspective and making Hyperion’s light transport robust against these cases took some work!

There were actually also some more traditional physically based rendering problems to solve on Wish too, which one might not necessarily expect for such a stylized film. For some of Magnifico’s magic, the art-direction called for a sort of prismatic look where white light would get split into different colors. We decided to try to achieve this effect through physically based shading, since the starting point for Wish’s entire stylization pipeline was renders with physical lighting in order to provide consistency. To achieve this effect through physically based shading, I extended the Disney BSDF with spectral dispersion support (retrofitting a spectral effect into a non-spectral renderer was a fun challenge worth discussing on its own someday). Once our lookdev artists had access to dispersion within the Disney BSDF, it was fun seeing all of the other places where they started sprinkling the effect in, such as in various glass objects.

Stylization on Wish didn’t just mean new renderer effects and lighting and compositing work; in order to make characters read correctly in a watercolor look, stylization had to be incorporated into all of the characters at a geometric and design level as well, and had to be incorporated into animation and simulation. As an example: a core story device in Wish is the collective wishes of Rosas, which take the form of orbs containing entire small worlds set inside of swirling volumetrics. Creating these wishes required clever pipeline solutions to embed entire stylized animated scenes inside of the orbs in 3D space, which was used instead of a usual compositing-based insert-shot workflow or the teleport-based solution [Butler et al. 2022] used on Encanto; this approach was taken in order to provide animators with the ability to sync and see fully combined shots interactively and to simplify the rendering setup needed for stylization in lighting and compositing [Karanam et al. 2024]. On top of creating the individual wishes, huge numbers of wishes then had to be choreographed into tight, closely synchronized formations to meet the art-direction and shape language of the songs they were a part of, which required developing new crowd rigs and animation controls. The rendering aspect of the wishes was in a lot of ways actually the easy part! Each wish was also an internally emissive object, so when thousands of wishes are massed together in key sequences in the movie, we initially had some concerns about efficiently rendering all of the wishes, but our long-standing cache points many-light selection strategy [Li et al. 2024c] proved to be more than capable for the task.

Another example of stylization far upstream of lighting is in the project’s entire approach to character stylization. Character hair and fur grooms required a different approach from our usual process; normally in more photoreal Disney Animation films, hair and fur grooms are built to be highly detailed to support the rich detailed look of the film, but Wish’s watercolor style meant using a more simplified and graphic shape language across the board, where detail is traded off for a stronger focus on silhouette and overall massing. Hair and fur grooms had to be adjusted to match, and hair and fur simulation had to be adjusted to keep art-directed shapes intact instead of operating on a more individual strand-based level [Kaur and Stratton 2024]. Asha’s braids, with their North African inspired long box braids, required additional attention to create and simulate [Kaur et al. 2024]. The braids themselves were already a major technical challenge; even using our state-of-the-art in-house grooming system Tonic [Simmons and Whited 2014], the braids still required a final groom two orders of magnitude more complex than our average groom. Once Asha’s groom was figured out, her hair then had to also be put through the same stylized simulation setup mentioned earlier, with extensive 2D drawovers being used to art-direct simulations. Character animation then also had to take into account the fact that Wish does not have motion blur and how that impacts how viewers perceive character performances.

Speaking of 2D drawovers, one particularly interesting use of 2D drawings to art-direct stylization on Wish is in Magnifico’s magic and in various effects like flames and torches. Normal volumetric effects created from simulations tend to be highly physical and detailed, but Wish’s style called for these effects to harken back to the much more graphic shape language of magic effects from Disney Animation’s hand-drawn era. To do this, our effects artists built on top of the neural volume style transfer work from Raya and the Last Dragon [Navarro and Rice 2021] and Strange World [Navarro 2023] to develop a new system where effects animation would begin with hand-drawn 2D elements, which were then projected and extruded into the 3D space to provide a guide for neural volume style transfer on top of volume simulations [Tollec and Navarro 2024]. The result is that Wish’s volumetric effects combine the movement and interactions of physical simulations while retaining the shape and style of traditional hand-drawn effects.

Everything I’ve written about here is just what I was familiar with on this film; vastly more work went into every single frame of Wish than even I know. The final look of Wish is something that I think really is unique and beautiful. Wish’s 3D watercolor look speaks to the entire history of Disney Animation and simultaneously roots itself in the studio’s rich traditional hand-drawn legacy while also exemplifying the studio’s long history of innovating and driving filmmaking craft forward. Walt Disney never stopped seeking to innovate in animation, and 100 years after he founded the studio, the animation studio that carries his name today continues to embody that same bold spirit on every new film. As someone who’s a lifelong fan and student of animation, I feel incredibly humbled and fortunate to get to contribute towards that legacy every day. Wish is also accompanied by the short Once Upon A Studio, which is a wonderful tribute to 100 years of Disney Animation characters.

Below are some frames from Wish, pulled from the Blu-ray and presented in no particular order. As always, I’d highly recommend seeing Wish on the biggest screen you can find!

Here are two credits frames from Wish; the first is the fancy hero-credit card for my wife Harmony Li and her fellow Associate Technical Supervisors, and the second is for the Hyperion team, along with several of the Hyperion’s sister technology teams that all support lighting and lookdev. Also, Wish has a lovely post-credits scene that I’d encourage sticking around for!

All images in this post are courtesy of and the property of Walt Disney Animation Studios.

References

Brent Burley, Brian Green, and Daniel Teece. 2024. Dynamic Screen Space Textures for Coherent Stylization. In ACM SIGGRAPH 2024 Talks. Article 50.

Corey Butler, Brent Burley, Wei-Feng Wayne Huang, Yining Karl Li, and Benjamin Huang. 2022. “Encanto” - Let’s Talk About Bruno’s Visions. In ACM SIGGRAPH 2022 Talks. Article 8.

Eric Daniels. 1999. Deep Canvas in Disney’s Tarzan. In ACM SIGGRAPH 1999 Sketches & Applications. 200.

Avneet Kaur, Jennifer Stratton, David Hutchins, and Nikki Mull. 2024. Art-Directing Asha’s Braids in Disney’s Wish. In ACM SIGGRAPH 2024 Talks. Article 4.

Avneet Kaur and Jennifer Stratton. 2024. Character Stylization in Disney’s Wish. In ACM SIGGRAPH 2024 Talks. Article 5.

John Kahrs, Patrick Osborne, Amol Sathe, Jeff Turley, Brian Whited, and Darrin Butters. 2012. The Art and Science Behind Walt Disney Animation Studios’ “Paperman”. In ACM SIGGRAPH 2012 Production Sessions.

Neelima Karanam, Joel Einhorn, Emily Vo, and Harmony M. Li. 2024. Creating the Wishes of Rosas. In ACM SIGGRAPH 2024 Talks. Article 6.

Michiyoshi Kuwahara, Kozaburo Hachimura, Shigeru Eiho, and Masato Kinoshita. 1976. Processing of RI-Angiocardiographic Images. In Digital Processing of Biomedical Images. 187-202.

Harmony M. Li, George Rieckenberg, Neelima Karanam, Emily Vo, and Kelsey Hurley. 2024a. Optimizing Assets for Authoring and Consumption in USD. In ACM SIGGRAPH 2024 Talks. Article 30.

Harmony M. Li, Angela McBride, Sari Rodrig, and Gregory Culp. 2024b. A Pipeline for Effective and Extensible Stylization. In ACM SIGGRAPH 2024 Talks. Article 51.

Yining Karl Li, Charlotte Zhu, Gregory Nichols, Peter Kutz, Wei-Feng Wayne Huang, David Adler, Brent Burley, and Daniel Teece. 2024c. Cache Points for Production-Scale Occlusion-Aware Many-Lights Sampling and Volumetric Scattering. In Proc. of Digital Production Symposium (DigiPro 2024). Article 6.

Barbara J. Meier. 1996. Painterly Rendering for Animation. In SIGGRAPH 1996: Proceedings of the 23rd Annual Conference on Computer Graphics and Interactive Techniques. 477-484.

Mike Navarro and Jacob Rice. 2021. Stylizing Volumes with Neural Networks. In ACM SIGGRAPH 2021 Talks. Article 54.

Mike Navarro. 2023. Diving Deeper Into Volume Style Transfer. In ACM SIGGRAPH 2023 Talks. Article 39.

Jennifer Newfield and Josh Staub. 2020. How Short Circuit Experiments: Experimental Filmmaking at Walt Disney Animation Studios. In ACM SIGGRAPH 2020 Talks. Article 72.

Kyle Odermatt and Chris Springfield. 2002. Creating 3D Painterly Environments for Disney’s “Treasure Planet”. In ACM SIGGRAPH 2002 Sketches & Applications. 160.

Patrick Osborne and Josh Staub. 2014. Feast – A Look at Walt Disney Animation Studios’ Newest Short. In ACM SIGGRAPH 2014 Production Sessions.

Johannes Schmid, Martin Sebastian Senn, Markus Gross, and Robert W. Sumner. 2011. OverCoat: An Implicit Canvas for 3D Painting. ACM Transactions on Graphics (Proc. of SIGGRAPH) 30, 4 (Jul. 2011), Article 28.

Maryann Simmons and Brian Whited. 2014. Disney’s Hair Pipeline: Crafting Hair Styles From Design to Motion. In Eurographics 2014 Industrial Presentation.

Daniel Sýkora, Ladislav Kavan, Martin Čadik, Ondrej Jamriška, Alec Jacobson, Brian Whited, Maryann Simmons, and Olga Sorkine-Hornung. 2014. Ink-and-Ray: Bas-relief Meshes for Adding Global Illumination Effects to Hand-Drawn Characters. ACM Transactions on Graphics 33, 2 (Apr. 2016), Article 16.

Rasmus Tamstorf, Ramón Montoya-Vozmediano, Daniel Teece, and Patrick Dalton. 2001. Hybrid Ink-Line Rendering in a Production Environment. In ACM SIGGRAPH 2001 Sketches & Applications. 201.

Daniel Teece. 2003. Sable - a Painterly Renderer for Film Animation. In ACM SIGGRAPH 2003 Sketches & Applications.

Marie Tollec and Mike Navarro. 2024. Making Magic with 3D Volume Style Transfer. In ACM SIGGRAPH 2024 Talks. Article 48.

Brian Whited, Eric Daniels, Michael Kaschalk, Patrick Osborne, and Kyle Odermatt. 2012. Computer-Assisted Animation of Line and Paint in Disney’s Paperman. In ACM SIGGRAPH 2012 Talks. Article 19.

Footnotes

¹ I recently learned that the Kuwahara filter originated from completely outside of graphics; it was originally invented at Kyoto University’s medical school and at Shiga University of Medical Science for medical imaging purposes. Specifically, it was invented for reducing noise in radioisotopic heart scans without blurring sharp features, and much later graphics people realized it made for a great edge-preserving blur for painting-like effects. I love when graphics intersects with other fields to produce interesting results! keyboard_return

This year at SIGGRAPH 2023, we have a conference-track technical paper in collaboration with Zackary Misso and Wojciech Jarosz from Dartmouth College! The paper is titled “Progressive Null-tracking for Volumetric Rendering” and is the result of work that Zackary did while he was a summer intern with the Hyperion development team last summer. On the Disney Animation side, Brent Burley, Dan Teece, and I oversaw Zack’s internship work, while on the the Dartmouth side, Wojciech was involved in the project as both Zack’s PhD advisor and as a consultant to Disney Animation.

Here is the paper abstract:

Null-collision approaches for estimating transmittance and sampling free-flight distances are the current state-of-the-art for unbiased rendering of general heterogeneous participating media. However, null-collision approaches have a strict requirement for specifying a tightly bounding total extinction in order to remain both robust and performant; in practice this requirement restricts the use of null-collision techniques to only participating media where the density of the medium at every possible point in space is known a-priori. In production rendering, a common case is a medium in which density is defined by a black-box procedural function for which a bounding extinction cannot be determined beforehand. Typically in this case, a bounding extinction must be approximated by using an overly loose and therefore computation- ally inefficient conservative estimate. We present an analysis of how null-collision techniques degrade when a more aggressive initial guess for a bounding extinction underestimates the true maximum density and turns out to be non-bounding. We then build upon this analysis to arrive at two new techniques: first, a practical, efficient, consistent progressive algorithm that allows us to robustly adapt null-collision techniques for use with procedural media with unknown bounding extinctions, and second, a new importance sampling technique that improves ratio-tracking based on zero-variance sampling.

The paper and related materials can be found at:

• Project Page (Author’s Version and Supplementary Material)
• Wojciech Jarosz’s Project Page (Author’s Version and Supplementary Material)
• Official Print Version (ACM Library)

One cool thing about this project is that this project both served as a direct extension of Zack’s PhD research area and served as a direct extension of the approach we’ve been taking to volume rendering in Disney’s Hyperion Renderer over the past 6 years. Hyperion has always used unbiased transmittance estimators for volume rendering (as opposed to biased ray marching) [Fong et al. 2017], and Hyperion’s modern volume rendering system is heavily based on null-collision theory [Woodcock et al. 1965]. We’ve put significant effort into making a null-collision based volume rendering system robust and practical in production, which led to projects such as residual ratio tracking [Novák et al. 2014], spectral and decomposition tracking [Kutz et al. 2017] and approaches for unbiased emission and scattering importance sampling in heterogeneous volumes [Huang et al. 2021]. Over the past decade, many other production renderers [Christensen et al. 2018, Gamito 2018, Novák et al. 2018] have similarly made the shift to null-collision based volume rendering because of the many benefits that the null-collision framework brings, such as unbiased volume rendering and efficient handling of volumes with lots of high-order scattering due to the null-collision framework’s ability to cheaply perform distance sampling. Vanilla null-collision volume rendering does have shortcomings, such as difficulty in efficiently sampling optically thin volumes due to the fact that null-collision tracking techniques produce a binary transmittance estimate that is super noisy. A lot of progress has been made in improving null-collision volume rendering’s efficiency and robustness in these thin volumes cases [Villemin and Hery 2013, Villemin et al. 2018, Herholz et al. 2019, Miller et al. 2019]; the intro to the paper goes into much more extensive detail about these advancements.

However, one major limitation of null-collision volume rendering that remained unsolved until this paper is that the null-collision framework requires knowing the maximum density, or bounding majorant of a heterogeneous volume beforehand. This is a fundamental requirement of null-collision volume rendering that makes using procedurally defined volumes difficult, since the maximum possible density value of a procedurally defined volume cannot be known a-priori without either putting into place a hard clamp or densely evaluating the procedural function. As a result, renderers that use null-collision volume rendering typically only support procedurally defined volumes by pre-rasterizing the procedural function onto a fixed voxel grid, à la the volume pre-shading in Manuka [Fascione et al. 2018]. The need to pre-rasterize procedural volumes negates a lot of the workflow and artistic benefits of using procedural volumes; this is one of several reasons why other renderers continue to use ray-marching based integrators for volumes despite the bias and loss of efficiency at handling high-order scattering. Inspired by ongoing challenges we were facing with rendering huge volume-scapes on Strange World at the time, we gave Zack a very open-ended challenge for his internship: brainstorm and experiment with ways to lift this limitation in null-collision volume rendering.

Zack’s PhD research coming into this internship revolved around deeply investigating the math behind modern volume rendering theory, and from these investigations, Zack had previously found deep new insights into how to formulate volumetric transmittance [Georgiev et al. 2019] and cool new ways to de-bias previously biased techniques such as ray marching [Misso et al. 2022]. Zack’s solution to the procedural volumes in null-collision volume rendering problem very much follows in the same trend as his previous papers; after initially attempting to find ways to adapt de-biased ray marching to fit into a null-collision system, Zack went back to first principles and had the insight that a better solution was to find a way to de-bias the result that one gets from clamping the majorant of a procedural function. This idea really surprised me when he first proposed it; I had never thought about the problem from this perspective before. Dan, Brent, and I were highly impressed!

In addition to the acknowledgements in the paper, I wanted to acknowledge here Henrik Falt and Jesse Erickson from Disney Animation, who spoke with Zack and us early in the project to help us better understand how better procedural volumes support in Hyperion could benefit FX artist workflows. We are also very grateful to Disney Animation’s CTO, Nick Cannon, for granting us permission to include example code implemented in Mitsuba as part of the paper’s supplemental materials.

A bit of a postscript: during the Q&A session after Zack’s paper presentation at SIGGRAPH, Zack and I had a chat with Wenzel Jakob, Merlin Nimier-David, Delio Vicini, and Sébastien Speierer from EPFL’s Realistic Graphics Lab. Wenzel’s group brought up a potential use case for this paper that we hadn’t originally thought of. Neural radiance fields (NeRFs) [Mildenhall et al. 2020, Takikawa et al. 2023] are typically rendered using ray marching, but this is often inefficient. Rendering NeRFs using null tracking instead of ray marching is an interesting idea, but the neural networks that underpin NeRFs are essentially similar to procedural functions as far as null-collision tracking is concerned because there’s no way to know a tight bounding majorant for a neural network a-priori without densely evaluating the neural network. Progressive null tracking solves this problem and potentially opens the door to more efficient and interesting new ways to render NeRFs! If you happen to be interested in this problem, please feel free to reach out to Zack, Wojciech, and myself.

Getting to work with Zack and Wojciech on this project was an honor and a blast; I count myself as very lucky that working at Disney Animation continues to allow me to meet and work with rendering folks from across our field!

References

Brent Burley, David Adler, Matt Jen-Yuan Chiang, Hank Driskill, Ralf Habel, Patrick Kelly, Peter Kutz, Yining Karl Li, and Daniel Teece. 2018. The Design and Evolution of Disney’s Hyperion Renderer. ACM Transactions on Graphics 37, 3 (Jul. 2018), Article 33.

Per Christensen, Julian Fong, Jonathan Shade, Wayne Wooten, Brenden Schubert, Andrew Kensler, Stephen Friedman, Charlie Kilpatrick, Cliff Ramshaw, Marc Bannister, Brenton Rayner, Jonathan Brouillat, and Max Liani. 2018. RenderMan: An Advanced Path-Tracing Architeture for Movie Rendering. ACM Transactions on Graphics 37, 3 (Jul. 2018), Article 30.

Luca Fascione, Johannes Hanika, Mark Leone, Marc Droske, Jorge Schwarzhaupt, Tomáš Davidovič, Andrea Weidlich and Johannes Meng. 2018. Manuka: A Batch-Shading Architecture for Spectral Path Tracing in Movie Production. ACM Transactions on Graphics 37, 3 (Jul. 2018), Article 31.

Julian Fong, Magnus Wrenninge, Christopher Kulla, and Ralf Habel. 2017. Production Volume Rendering. In ACM SIGGRAPH 2017 Courses. Article 2.

Manuel Gamito. 2018. Path Tracing the Framestorian Way. In ACM SIGGRAPH 2018 Course Notes: Path Tracing in Production. 52-61.

Sebastian Herholz, Yangyang Zhao, Oskar Elek, Derek Nowrouzezahrai, Hendrik P A Lensch, and Jaroslav Křivánek. 2019. Volume Path Guiding Based on Zero-Variance Random Walk Theory. ACM Transactions on Graphics 38, 3 (Jun. 2019), Article 25.

Wei-Feng Wayne Huang, Peter Kutz, Yining Karl Li, and Matt Jen-Yuan Chiang. 2021. Unbiased Emission and Scattering Importance Sampling For Heterogeneous Volumes. In ACM SIGGRAPH 2021 Talks. Article 3.

Peter Kutz, Ralf Habel, Yining Karl Li, and Jan Novák. 2017. Spectral and Decomposition Tracking for Rendering Heterogeneous Volumes. ACM Transactions on Graphics (Proc. of SIGGRAPH) 36, 4 (Aug. 2017), Article 111.

Ben Mildenhall, Pratul P. Srinivasan, Matthew Tancik, Jonathan T. Barron, Ravi Ramamoorthi, and Ren Ng. 2020. NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis. In Proc. of European Conference on Computer Vision (ECCV 2020). 405-421.

Bailey Miller, Iliyan Georgiev, and Wojciech Jarosz. 2019. A Null-Scattering Path Integral Formulation of Light Transport. ACM Transactions on Graphics (Proc. of SIGGRAPH) 38, 4 (Jul. 2019), Article 44.

Jan Novák, Iliyan Georgiev, Johannes Hanika, and Wojciech Jarosz. 2018. Monte Carlo Methods for Volumetric Light Transport Simulation. Computer Graphics Forum (Proc. of Eurographics) 37, 2 (May 2018), 551-576.

Jan Novák, Andrew Selle and Wojciech Jarosz. 2014. Residual Ratio Tracking for Estimating Attenuation in Participating Media. ACM Transactions on Graphics (Proc. of SIGGRAPH Asia) 33, 6 (Nov. 2014), Article 179.

Towaki Takikawa, Shunsuke Saito, James Tompkin, Vincent Sitzmann, Srinath Sridhar, Or Litany, and Alex Yu. 2023. Neural Fields for Visual Computing. In ACM SIGGRAPH 2023 Courses. Article 10.

Ryusuke Villemin and Christophe Hery. 2013. Practical Illumination from Flames. Journal of Computer Graphics Techniques 2, 2 (Dec. 2013), 142-155.

Ryusuke Villemin, Magnus Wrenninge, and Julian Fong. 2018. Efficient Unbiased Rendering of Thin Participating Media. Journal of Computer Graphics Techniques 7, 3 (Sep. 2018), 50-65.

E. R. Woodcock, T. Murphy, P. J. Hemmings, and T. C. Longworth. 1965. Techniques used in the GEM code for Monte Carlo neutronics calculations in reactors and other systems of complex geometry. In Applications of Computing Methods to Reactor Problems. Argonne National Laboratory.