Virtual Production and the importance of having the right tools for the job
"For decades we have been shooting talent or products in green screen studios and adding (compositing) photo-realistic 3D graphics in post-production to finalize the shot. This is the foundation of a visual effects shot (VFX). It was and still is very much a two-stage process." Michael Geissler, CEO of Mo-Sys, talks to InBroadcast about the evolution of Virtual Production.
Virtual production in most cases makes this a one-stage process, by enabling the VFX shot to be captured on set, in camera, in real-time. The virtual production process saves time, money, and is much more efficient, but requires a greater level of pre-production preparation, virtual production knowledge and experience, together with the right tools for the job.
The concept of Virtual Production is not new. Tracked virtual graphics arrived during the mid-1990’s where they were used predominantly for broadcast, albeit with limited tracking, no lens calibration and would typically produce poor virtual image quality.
Thankfully, the story doesn’t end there. In what Mo-Sys have named Virtual Production 1.0 (VP 1.0), circa. 2016 we saw the introduction of games engines capable of producing photo-realistic onset pre-viz. The next logical step would have been real-time VFX in green/blue screen studios, but we jumped straight to VP 2.0, where production of The Mandalorian illustrated the exciting new capabilities of real-time VFX using LED volumes.
Hype, coupled with a surge in demand from the locked-down live events industry, who cleverly pivoted into virtual production for survival, meant that many early LED VP stages were using lower quality live event LED tiles, live event-oriented media servers and mo-cap tracking systems, rather than cinematic LED tiles, dedicated cinematic content servers, and precision camera tracking systems such as Mo-Sys StarTracker.
Right now, virtual production is in its third stage of evolution – VP 3.0. It has matured over the last five years and in 2020, the global market was reported to be worth in excess of £2 billion and is expected to grow beyond £5 billion by 2028. Mo-Sys’ experience gained from 25 years within the broadcast and film industry, together with bleeding edge innovation is driving VP from what it was in early 2020, to what it needs to be in order to deliver on many of the benefits originally promised.
Within broadcast and film, there is now a greater understanding surrounding the limitations of live event LED tiles and events media servers which were developed for entertainment applications such as projection mapping and timeline lighting shows.
Cinematographers and VFX Supervisors are demanding a dedicated, focused toolset that meets their needs and enables a closer content quality match to traditional post-production compositing quality.
Mo-Sys is delivering exactly that with its award-winning VP Pro XR. Designed by in-house cinematic innovators for XR creatives, Mo-Sys VP Pro XR is an award-winning, dedicated XR server solution that takes a radical new approach to delivering cinematic standards to on-set, real-time virtual production using LED volumes. Designed for LED stages with or without set extensions, it can also be used with blue/green screens and enables traditional shooting techniques within an LED volume, with a focus on composite image quality.
Typically, directors and cinematographers must make continuous calculations to make the real and virtual elements match in post-production. This is often a costly and repetitive process. A pioneer in providing a platform for graphics rendering as well as camera and lens tracking to create higher quality virtual productions, Mo-Sys has made it even easier to produce seamless, high-end productions with VP Pro XR. Designed to enable the use of traditional shooting techniques within virtual productions, it removes the limitations on the ability to tell stories that are imposed by current XR stage designs.
Another major hurdle in VP and the use of LED volume comes when switching between multiple cameras pointing at an LED volume. Whilst the camera outputs can be switched in 1 frame, the virtual scene displayed on the LED volume typically take 5-6 frames to update. This means on every camera switch there will be 5-6 frames of the previous camera’s virtual scene displayed on the LED volume, before it updates with the correct perspective view for the new camera. As a result, you get a background flash on every camera switch, which is unusable in production.
The latest iteration of VP Pro XR addresses this and orchestrates the delay between the operator switching cameras, and the LED volume updating with the correct perspective view of the live camera. More importantly, it will do this at up to the full UHD4K resolution of the LED processor input, whereas previous workarounds would reduce the resolution of the content to HD to achieve the same outcome.
Along with full resolution multi-cam switching, Mo-Sys Cinematic XR Focus enables seamless interaction between virtual and real worlds. This feature ensures that an LED wall can be used as more than just a backdrop, allowing it to integrate with the real stage. This gives cinematographers the means to seamlessly rack focus deep into the virtual world and create immersive shots which enhance their visual storytelling.
NearTime® is Mo-Sys’ HPA Engineering award-winning solution for solving real-time VFX virtual production image quality. The solution comprises a cloud-based auto-re-rendering system utilizing Smart Green, a method of separating talent from the LED background without introducing copious green spill, when combined deliver higher quality real-time VFX content.
