Strictly Augmented Reality

It is one of the greatest global television hits of the last 15 years. First seen as Strictly Come Dancing, produced by BBC, it is now licensed around the world, as Dancing with the Stars or its equivalent in local languages, and is massively popular.

In the UK, Strictly is the most popular entertainment program on television. For the 2020 season, the producers wanted to add even more glitz and glamour, and were considering upping the graphics game even before the full extent of the Covid-19 pandemic meant the production plan had to be completely rethought.

Graphics specialist company Potion Pictures was already a part of the Strictly production team, building excitement through the screens which are part of the set, and through the dramatic floor projections. Without the usual enthusiastic studio audience, it was clear that the graphics were going to play an even bigger part than usual in building the atmosphere.

Because the programme is broadcast live, there is a very short period between each dance in which to put specific furniture and props in place. Social distancing requirements meant a smaller crew on the studio floor, so physical props had to be cut to an absolute minimum. The producers turned to augmented reality (AR) to make the scenes sparkle.

From steam trains to elephants, the 2020 series of Strictly Come Dancing went all out to provide the visual wow factor and make up for the lack of a live audience. In this case study session for MPTS, Potion Pictures explained how they delivered their most exciting and extravagant graphics to date to bring the series to life using Mo-Sys StarTracker and VP Pro.

Mo-Sys had previously worked with Potion Pictures on a POC (proof-of-concept) in another studio at BBC Elstree. This provided the opportunity to show to the creative and technical team exactly what could be achieved with live real-time AR virtual graphics - and it made them see what really could be possible.

There are layers of complexity in adding AR to Strictly. First, it is primarily a dance competition, so Mo-Sys could not get in the way. The digital objects had to be moved to match the choreography. More important, nothing could obscure the dancers: if the voting audience could not see what the partners were doing, it could have huge consequences for the fairness of the judging.

Second, the show was already full of shine and sparkle, and nothing could affect that. If the video processing to composite real and virtual elements together degraded the live pictures, that too would be unacceptable.

Third, whatever Mo-Sys did had to be in absolute real time, with the minimum of latency. The show depends on music, and the audience would not tolerate pictures out of time with the sound.

The system built for Strictly used the Mo-Sys VP Pro software, designed to integrate directly into Epic Games’ Unreal Engine editor interface. This system provided the real-time compositing and 3D graphics rendering of Potion Pictures’ creative vision.

Camera tracking was provided using Mo-Sys' StarTracker 6-axis tracking system. The augmented reality images were linked to two of the live cameras: one on a crane, the other on a Steadicam. These are the most challenging camera mounts, which added to the excitement.

The way that the production decided to use AR was to introduce elements at the start and end of the dance. A whole range of virtual graphics was created for the series, from an ice castle and a Barbie house to a train and – bizarrely – an elephant.

For augmented reality to be completely convincing there has to be no visible join between the real and visible worlds. The graphics have to be rooted firmly in position. Strictly was a very testing environment for the Mo-Sys technology; the combination of a dark studio ceiling (where the camera tracking markers are located), rapidly changing lighting, glitterball reflections, meant that this was probably the sternest test of tracking accuracy and integrity possible. It was agreed that there were three key requirements if the virtual graphics illusion was going work and be maintained:

• ultra-precise camera tracking on all 6-axes of movement
• extremely accurate lens calibration, so the virtual graphics are distorted to exactly match the camera lens attributes, particularly during zoom shots
• excellent synchronisation – the camera frequently moves quickly to keep the dancers perfectly framed, and the virtual graphics had to keep up

Mo-Sys delivered on all three. Indeed, the Mo-Sys synchronisation above all else realised this ambition – proving that without this technology the results could not have been achieved.

Mo-Sys Technical Director, James Uren, supervised the system over the three studio days for every episode. That gave the company a remarkable insight into how real, creative production people want to use Mo-Sys equipment, pushing it to the limit. It meant continually refining what could be done, particularly in association with the Unreal engine.

The ice castle saw some incredible refractions enabling the development of ever-better translucency on particle effects. The models had dynamic shadows, again to make them sit naturally into the live studio. This and other continuing improvements were because of the ongoing understanding and recognition of exactly what Strictly was trying to achieve, and interfacing with Epic Games to make even better use of the very latest Unreal Engine features.

Besides the AR graphics, Potion Pictures also used a second type of tracked real-time graphics enhancement. In previous series of Strictly, viewers would have seen the floor displaying a ‘drop-away' 3D graphic of the top of a building, on which the dancers would appear to perform their routines. However, the camera would never move, and the 3D visual impression was more symbolic than photo-realistic. This series however used an off-axis tracked camera, again utilising the Mo-Sys StarTracker system, meaning that this time the camera could move, and as a result it gave the very realistic impression that the dancers were performing on top of a building.

This series of Strictly produced several key technological ‘firsts’ for virtual production use in a live broadcast:

• The use of Epic Games’s Unreal Engine natively to produce the real-time graphics
• The use of Mo-Sys' VP Pro plug-in software to deliver the real-time compositing and synchronisation
• The use of Unreal Engine’s image distortion features to show the real studio distorted through ice AR graphics

It is fair to say that AR on Strictly was a learning experience for everyone. But there is no doubt that it added yet more glamour and visual interest to this iconic programme, at a surprisingly low cost. With minimal addition to the programme budget, AR added a huge amount to the visual richness.