Take a fresh look at your lifestyle.

How Fox Sports uses virtual fans in Unreal Engine to fill empty stadiums in MLB broadcasts

Baseball is back. But the MLB will look a little different in the COVID-19 world, with empty stadiums full of silent cardboard cutouts instead of thousands of screaming fans trying not to spread the virus. Unless you check out Fox Sports. Starting this weekend, the network will not only piping into fake fan audio, but it will also fill the stands with digitally rendered fans for its MLB broadcasts, starting with Saturday’s Yankees-Nationals game.

All 30 MLB teams will be rely on fake fan sound in the actual stadiums (from Sony’s MLB: The Show games), along with the usual stadium announcers, walk-up music and other aspects of pomp and circumstance that accompany a Major League Baseball game. But Fox Sports is the only broadcaster to announce plans for virtual fans.

“We believe that the crowd and seeing people in seats is part of a broadcast, part of broadcasting sports at a high level in the major leagues. So we wanted to come up with a solution for that, ”explains Brad Zager, executive producer and head of production and operations at Fox Sports.

“We don’t want to fool anyone, it’s still about the game. But shot-to-shot, when you watch a broadcast, you no longer notice what is normally a broadcast with fans in an empty stadium and it feels weird – we want to give people a sense of normality ‘, Zager says. “And we felt that we would go down this road and try to use a virtual crowd, hopefully make it part of the game, and focus more on the game because you don’t think about the void from a Major League Baseball stadium during the game. “

The effect is a combination of technologies that you may have seen before. The augmented reality software used to insert the crowds is called Pixotope, who has worked on AR graphics for things like the Super Bowl and The Weather Channel’s terrifying storm warning demonstrations.

It works by using images (created by a creative agency Silver spoon animation) built in Epic’s Unreal Engine. Unreal Engine is used here for the same reason that it is popular for creating video games or creating virtual wallpapers on set for shows like The Mandalorian. Unlike most movie images, which must be displayed in post-production afterwards, Unreal can render in real time, making it much more suitable for live television.

Finally, SportsMedia Technology (SMT) – the same company that handles most sports-oriented computer overlays that you are familiar with, such as the yellow first line in football, the on-screen clock and scoreboard, and more – ensures the camera follows to insert those images into the live camera feeds. Four cameras – high home, centerfield, high first and high third cameras – show the virtual fans.

Fox Sports producers will be able to arrange things like how full the virtual ‘crowds’ are for a particular game, what weather fans are dressed for, and what percentage of the crowd will be home fans or not, although the company is still figuring it out is how it will make some of those decisions. “We are now figuring this out and we will continue to evolve that process as we continue this weekend and from game-to-game, week by week,” said Zager.

The Fox Sports broadcasts will also integrate the feigned fan audio played in the stadium into the background of the player’s microphone clips (such as during pitches) and other in-game audio, but it will amplify that live audio on its own, too, similar to how it is done for MLS games.

And while the interactions between the virtual crowds and the audio will be basic for the time being (like standard cheers, anger, and the wave), the company hopes to better integrate the two as it gets more practice throughout the season. “It’s something we hope will work even more together and become more on the same page as we do this more and more, because we can combine the audio and the audience together.”

For the time being, the virtual fans will only be broadcast on MLB games on Fox, but it is possible that the digitally generated fans may also appear elsewhere. Zager says the company is partnering with all of its sports sponsors, and that “as we get closer to the fall for football and other sports, if we feel like it’s going in the right direction, we’ll launch it and roll it out to other sports.” also.