Hello, yuzu fans! Tired of broken cutscenes and having to mash your controller buttons in hopes of skipping them? Well, look no further! Thanks to the efforts of epicboy, yuzu can now play (most of) your favorite in-game cutscene videos. Jump right in to find out more!

NVDEC - What is it?

For a long time, one of the most noticeable missing features of yuzu was support for NVDEC.
What is NVDEC? NVDEC stands for Nvidia Video Decoder and is a feature in Nvidia GPUs that performs video decoding. And since the Switch has an Nvidia Tegra X1 SoC, it too makes use of its NVDEC module to offload all CPU intensive video decoding tasks to the GPU.

NVDEC supports video codecs like MPEG-2, VC-1, H.264 (AVC), H.265 (HEVC), VP8, and VP9, but Nintendo only exposes H.264, H.265, VP8 and VP9. Most games seem to use H.264 because of its wide support and efficiency ratio. But many first party games are known to use VP9 too - eg. Super Smash Bros. Ultimate, Pokémon: Let’s Go, Pikachu/Eevee!

With NVDEC support, users no longer have to rely on old save files or crazy button mashing to bypass broken cutscenes. Grab the latest yuzu Early Access build to try it out now!

Development

The Switch uses NVDEC along with VIC (Video Image Composer) in the GPU for video decoding. Games are exposed to higher level APIs by Nintendo which abstract the configuration of these modules.

NVDEC and VIC work together to decode incoming videos and the process usually goes like this:

- Allocate memory
- Place encoded video frame data in known memory location
- Decode and place frame data into GPU memory
- Clear intermediary allocated memory

Although this seems fairly simple, implementing it was quite the task. epicboy began working on this by implementing the service/command calls to NVDEC and VIC. It was decided to use the FFmpeg library to decode the frame data once we knew its codec (H.264 or VP9).

VP9

VP9 was the biggest challenge as its decoding required knowledge on the different attributes of a specific frame. Each frame carries two headers and raw frame data with it. The headers contain the information as to which previous frame the current frame refers to or depends on. Frames usually refer to previously decoded frames, along with transformations to be done on various parts of those, to compose new frames. This is done in an effort to reduce the memory footprint of each frame.

Here is where things got complicated. On the Switch, these headers are parsed by the Nvidia library in the games; therefore, NVDEC never receives this data. But for us to decode the frames using FFmpeg, we need to provide it with the full frame - Two headers and raw frame data from NVDEC. And since NVDEC hardware doesn’t expose all of the data of the two VP9 headers, epicboy had to manually compose these headers from the provided information.

epicboy took an interesting approach to this problem - he buffered two frames in advance. As the frame data from NVDEC holds some data on previous frames, knowing two frames in advance was useful. But this wasn’t sufficient as a few games resulted in issues with this approach. We are currently researching these edge cases to properly fix it for all games.

H.264

H.264 was relatively easier compared to VP9. Each H.264 frame contains two headers and raw frame data with it. Unlike VP9, H.264 doesn’t have different headers for each frame but has same headers for entire video.

ogniK had, long ago, already implemented H.264 support in his experimental branch. epicboy based his work off of ogniK’s and fixed a bug in it which caused distorted videos. It turned out that ogniK was using incorrect dimensions when writing the pixel location.

Currently NVDEC operations are synchronous, but are asynchronous compatible. This means that in the future, NVDEC operations can be made asynchronous thus yielding even better performance. Also, thanks to the FFmpeg library, we have access to hardware acceleration for faster decoding. And in the future, this will allow yuzu to offload video decoding to the host GPU (user’s GPU).

VP8 & H.265

The Switch officially supports VP8 and H.265 too, along with VP9 and H.264. However, we are yet to see games make use of these codecs for in-game videos and hence support for these codecs remains unimplemented for now.

Testing

As development work progressed, there were a lot of challenges and issues with games. Link’s Awakening, for example, wasn’t providing the proper memory locations to write frame data to and the NVDEC/VIC service/command calls were inaccurate as well. This led to data corruption when we wrote frame data to wrong memory locations.

epicboy and our testers spent tens of hours testing various games in yuzu to make sure any minor issues were ironed out. This rigorous testing also brought to our attention more games that exhibit weird edge cases. Over the next couple of weeks, we plan to fix these bugs and make our decoding even more accurate.

Huge shoutout to our testers for testing and compiling these lists.

Fin

As always, we would like to remind users that the features released on Early Access are still being worked on. Hence not all games might behave in the way we want them to. If you come across more games (other than the ones mentioned above) that encounter bugs or issues, feel free to reach out to us on our Discord server and share your findings.

See you next time,
- yuzu development team!

Please consider supporting us on Patreon!
If you would like to contribute to this project, check out our GitHub!