Creating digital characters with KéexFrame
KéexFrame utilised StarTracker and Mo-Sys VP for an augmented digital character proof of concept for a major broadcaster. Using Epic Games’ Unreal Engine, KéexFrame collaborated with Mo-Sys, Dynamixyz (real-time facial capture), Vicon (motion capture) and Nansense (additional motion capture) to deliver a real-time performance-driven motion capture demonstration for live daily broadcast.
Not too long ago the idea of interacting with a digital human would have conjured up one or two nightmarish thoughts. While these organic-digital interactions have previously been found only in fiction, recent developments in performance-driven capture technology have enabled digital humans to become one of the hottest areas of virtual production. Along with media creators relentlessly seeking new ways to engage with audiences, immersive mixed-reality products and experiences are an increasingly popular feature in our current entertainment landscape.
New York-based creative studio, KéexFrame is one of the latest visual disruptors to really push the boundaries of virtual production with a real-time digital character proof of concept. Bringing together both performance-driven capture and camera tracking technology, KéexFrame showcased the conceptualisation and utilisation of a digital character for a live daily broadcaster, illustrating how it can enrich their entertainment programs and enhance marketing opportunities.
Arturo Brena, CEO and Founder of KéexFrame said: “The broadcaster requested the character to be a CGI soccer influencer that could connect with the audience, not only as a soccer specialist but on a personal level. They also wanted to steer away from the typical “superhero” character and create something more “real” and relatable. After a few brainstorming sessions we came up with the idea of ‘Félix’; an older grumpy but lovable grandpa that is obsessed with his soccer team.”
What technology was used?
Utilising the power of Epic Games’ Unreal Engine, KéexFrame combined facial capture from Dynamixyz, full-body motion capture from both Vicon and Nansense as well as camera tracking from Mo-Sys. This blend of software and hardware enabled an off-screen performer to transfer their live full unique acting performance to the expressions of the digital character in real-time, bringing realism and personality to the character.
The addition of StarTracker, allowed the main camera to be moved around the whole studio without the augmented digital character drifting from its position, which would break the illusion. StarTracker was set-up on a jib for smooth floating shots around the talent creating an immersive and dynamic production for the audience. Using retro-reflective stickers the StarTracker sensor and processing unit captured the absolute position of the camera. The tracking data was then sent into to the Unreal Engine using Mo-Sys VP, the full virtual production plugin for the Unreal Engine. This provided virtual camera movement, lens distortion compensation, colour correction, particle effects and augmented reality shadows to lock the digital character in the scene.
Arturo Brena added: “The Mo-Sys StarTracker solution was the perfect solution for this, the system was not only easy to calibrate but it allowed all sorts of flexibility in the studio. The cameras could be moved in and out of the tracking space and the StarTracker would automatically start tracking again as soon as it entered the calibrated space. This flexibility allowed for a seamless transition for the production team when having a CGI personality on set. There were no complications at all with the camera tracking system it simply worked perfectly, it was very refreshing not to think or worry about it during production.”.
About KéexFrame
KéexFrame is a creative studio based in NYC, dedicated to the conceptualisation, design and execution of compelling stories through visual digital mediums. From traditional 2D and 3D animation to the most advanced technical implementations in the media industry.
They have worked on some of the most ambitious projects in TV, primarily in animation, design, data implementation and real-time graphics. They have collaborated with some of the biggest networks in the industry (NBC, CBS, Televisa, TNT, CCTV China, among others). Click here to visit the KéexFrame website.
Arturo Brena - Creative direction / 3D lead / Unreal Engine implementation / Project Manager
Matthew Cornelius - Inertial Mocap / Optical Mocap
Omar Hernandez - Rigging / 3D technical direction
Robert Lester - Facial mocap
