How off-the-shelf cameras are powering baseball’s next-gen replays

As crowds fill the stands in AT&T Park this year, they'll be focused on the usual ballpark festivities: eating hot dogs, drinking beer and, of course, watching the San Francisco Giants play baseball. But if they were to venture onto the park's club level and look upward, they'd notice something odd hanging from the deck: around 140 Panasonic GH5 cameras pointed at the field. The purpose of these cameras? To capture seamless slow-motion video from multiple angles in near real time.

The company behind the tech is Korean outfit 4DReplay Sports, which has been around for a few years in Asian sports circles, especially the Korean Baseball League. 4DReplay has also dabbled in concert videos, action movies, event photo booths and documentaries. It's been trying to expand to international markets like the US, and even set up shop in Silicon Valley about a year ago to court US sports teams.

The first US team to step up to the plate was the San Francisco Giants. After a trial run last year in a game against the Kansas City Royals, the Giants' interest was piqued. "Everyone loved it, even the broadcasters," said the Giants' CIO Bill Schlough. "We thought, hey, we might have something here." So this year, the Giants invited 4DReplay to set up shop permanently, making AT&T Park the first sports venue in the US to do so.

2017_MLB_4DReplay from 4DReplay on Vimeo.

360-degree replay is not a new thing in US sports; you've likely seen something similar in NBA and NFL games via Intel's True View (originally called FreeD before Intel acquired it). At the Super Bowl in Minnesota, for example, Intel had 38 360-degree cameras in the stadium which it used to provide slow-motion rotational shots of various plays. It's also been used in baseball, like at the 2016 All-Star game in San Diego's Petco Park.

The Giants considered True View too, but ultimately passed for one reason: speed. It typically takes Intel's system several minutes to render the images into a 360-degree shot. "That's way too long," said Schlough. "You can't have that in a live game. If there's a great play on the field, you'd be able to show it maybe at the end of the inning, but fans want to see it quickly. We need a technology that can show it in seconds." Plus, Intel's tech needs a tremendous amount of compute power, often requiring hundreds of servers -- and a lot of ballpark real estate to house said servers -- in order to process terabytes of data.

4DReplay's system, on the other hand, can output that shot in just five to ten seconds. And it only requires a single Windows 10 PC. The system installed at AT&T Park uses one or two extra PCs, as well as a few servers in the back room for faster processing and redundancy, but that still takes up far less space than Intel's system.

4DReplay uses way more cameras -- anywhere between 40 to 150 -- placed all around the park. But those cameras aren't the expensive ones that Intel uses. Instead, they're regular off-the-shelf mirrorless models like the aforementioned Panasonic GH5s. They're small, lightweight, and relatively easy to install. All of them capture 4K, 60fps shots. They're hooked up to the PC via either fiber or LAN (AT&T Park uses LAN). 4DReplay actually worked with Panasonic to come up with custom firmware for the cameras so that they would work with its tech.

The other, perhaps most important, part of the equation, is that while Intel's True View uses rendering to compose the slow-mo video, 4DReplay doesn't. Thanks to all those cameras and a bit of proprietary software magic, a 4DReplay system operator is able to just select a bunch of frames, apply them to a template, and voila, that awesome bullet-time clip of SF Giants' Brandon Belt hitting a homerun is stitched together in five seconds flat.

"It's so well-calibrated and synchronized, that it doesn't need rendering," said 4DReplay COO Henry Chon. "Visually it looks rendered, but it isn't. If you're really visually keen, you might be able to spot the difference, but most people can't tell."

Beyond just having a cool clip to show on the scoreboard or in live broadcasts, this idea of a near-instant 360-degree recap done without any rendering has big ramifications in sports, especially in the era of replay reviews. When there's a question on whether a player is safe or out at the plate, umpires need that video replay fast, in multiple angles, and they need to know that the video is untouched. A computer-rendered video is no good, because a machine fills in the gap between each shot with what it thinks is right. With 4DReplay's system, the replays capture what the cameras actually saw, with no manipulation.

[System] Instant Replay Review X Baseball from 4DReplay on Vimeo.

"It'll help us improve the accuracy of replays," said Schlough. Historically, he says, a lot of plays are ruled inconclusive, where it's just too close to call either way, so the call on the field stands. "But with 140 cameras, there must be at least one angle. I think you're gonna catch it." Currently, the 4DReplay cameras at AT&T Park are focused in on first, second, and third base, home plate, and the pitcher's mound, because that's where most of the action is. But if there's a contested play in the outfield, then it might not catch that.

In the future though, Chon hopes to have better custom-made cameras that might help. The company is working with a Japanese manufacturer to make a version of the mirrorless cameras without the battery and storage compartments (because it doesn't need them) that can be mounted on a movable pivot which can be controlled remotely. That way, it won't need quite as many cameras, and it can catch those occasional shots of a flyball in center field.

But while 4Dreplay has already been used to great success in replay reviews in Korea and Japan, it'll take a while longer before it's implemented in the US. Not only do both teams need to have access to the same video angles to be fair, the remote umpires in New York need to be able to view them too. In order for it to be implemented league-wide, it needs to be in every stadium, and video replay booths need to be set up for each team.

There are other potential uses for these replays, too. They can act as an instructional tool, where players and coaches compare bat swings up close or analyze pitcher throws from every angle. Sports commentators could liven up the broadcast by tapping into the replay video and rotating the video from side to side, using a pen tool to circle over an elbow or a hand to point out a player's technique.

4DLive_Capture_(time_Remapping)_2 from 4DReplay on Vimeo.

One intriguing avenue for replays is to send them to a mobile or tablet app so fans can explore the videos themselves. For example, they could scrub a pitcher throw from all angles by swiping up or down, and take a closer look at how he holds and lets go of the ball by zooming in. Chon said these interactive highlights are already in use in Korea, mostly because fans in Korea tend to watch baseball on their phones, not on TV. Chon said they could very well replicate the same thing over here in the US, perhaps to something like the MLB app.

4DReplay has been making strides outside the US, too. It recently partnered with the Olympics Broadcasting Service to provide 360-degree recaps of a few Winter Olympics events like figure skating and half pipe and is hoping to get a similar deal for the upcoming Summer Olympics in Tokyo. It's also working to bring 4DReplay to a boxing event in the UK in June.

As for the US, it's not just baseball. Last year, 4DReplay had a trial run with CBS Sports at a couple of NCAA games between Ohio State and North Carolina, as well as Kentucky and UCLA. As a result, the NBA is reportedly exploring possibilities as well, and Chon dropped hints that you might see 4DReplay in use at an NBA game in the near future.

"My phone has been ringing off the hook," joked Schlough, as he's been forwarding requests on to Chon and his team. Schlough says he's had at least 10 other baseball teams reach out to him since the Giants installed it, who are interested in installing it in their own parks. According to Chon, Major League Baseball has also expressed interest in taking a closer look at the tech, especially for replays.

"They want to see more of us in other properties before they make a decision," said Chon. "We're currently in talks with about three different teams."