The creators of Mario Kart Live predict the future of gaming

Nintendo’s Mario Kart Live: Home Circuit is a fascinating little toy. Not the first of its kind, but for its performance and price it’s one of the best. In a way it’s a robot, or or an advanced connected peripheral, that can interact with invisible video game obstacles. It can make me feel like I’m a tiny person shrunk down into another space, exploring it in detail I wouldn’t otherwise pick up. It blends augmented reality and the real world at high speeds.

Robotic toys were a big thing a few years ago, and then they faded a bit. Anki Cozmo, all the WowWee creatures, all the Sphero Star Wars droids and cars. Some died, some remain. But those were mostly RC vehicles. What Mario Kart makes me think of is telepresence.

One field trip I regret not going on, years ago, was a drone race deep in the Vegas desert organized in part by headset maker Avegant. I had commitments in massive halls full of tech at CES, and couldn’t make a journey out to fly drones and see them through my headset display, piloting them like little fighter ships in a living video game. It was my loss, but years later, racing a Mario Kart through my dining room and looking at it through a Nintendo Switch display, I felt like things had come full circle. A little drone race was here in my home in 2020.

Karthik Bala and Dan Doptis, the CEO and Game Director for New York-based Velan Studios, the company that created the Mario Kart Live AR-enabled kart, along with Nintendo’s Yosuke Tamori, Hiroki Ikuta, and Yuji Ichijo, spoke to me over email about the process of making Mario Kart Live: Home Circuit. 

“The project started mid-2017, which was initially inspired by drone racing and the fun of RC cars,” Bala said. “We built a prototype RC car with a camera and sensors. The feeling of high-speed driving in the real world was fun, but we are game developers. We wanted to build a game around it — not just a fun toy. That led us to the idea of a “mixed reality” experience. As the prototype took form, it became clear to us that we had to show it to Nintendo.”

Bala explains that a lot of the AR work is done in tandem with the physical kart and the Switch, with the kart doing the camera and mapping work that is normally done on phones, tablets, and headsets in AR. “The kart itself is a ‘mini-console’ on wheels. It has an HD camera and sensors that connect wirelessly and directly to the Nintendo Switch system,” Bala says. “It’s a custom-built solution that delivers low-latency HD video, computer vision, and SLAM (real-world positioning of the kart) to deliver a convincing mixed reality experience for players. The work is split between the kart and the Nintendo Switch system, which does a lot of the heavy lifting.”

“When it came to the camera on the karts, we worked hard on synchronizing video from the camera with the sensor data and the controls to make the AR appear natural,” Nintendo’s Ikuta adds. “On top of that, we optimized the system to minimize any lag that would have a major effect on gameplay. After the footage goes through all that processing, it is sent to the Nintendo Switch system via wireless communication. After receiving the camera footage, the system processes the graphics to bring the AR experience to life.” Ikuta also added that the kart behavior had to keep getting tweaked to account for different home lighting, and for driving over carpeting at low speeds.

Robots not as things we control, but things we embody

Telepresence robots have been around for a number of years, too: Remember those big rolling iPads on wheels that became a tech industry joke? And yet, at a CES show years ago, I remember being moved by a virtual tour where dozens of those roving screens-on-wheels were being driven by real people somewhere else, unable to travel, maybe even unable to walk. These devices were a way to be somewhere else.

William Gibson’s recent books, The Peripheral and Agency, explore a lot of ideas surrounding people embodying other robots and drones through advanced wearable sensory systems. I’ve wondered about a world where thousands of robotic devices could exist, mapping the real world through depth-sensing sensors just like the ones on AR headsets and the iPhone 12 Pro, available for me to jump into at any time. It’s classic sci-fi stuff, and yet, when I think about the potential for home robot toys and gadgets, I feel like that’s exactly where a new wave of products could head next.

Velan Studios’ Bala seems to agree. “The breakthrough technology is that we’ve fundamentally decoupled the camera and sensors from the host device (the Nintendo Switch system), which unlocks all sorts of new possibilities. It’s a rich new paradigm to explore in the future!”

A way to explore impossible places?

As I raced Mario Kart around my home in miniature, I started to think about the possibilities for building miniature theme parks you could explore through tiny devices. I can see a little animated Mario Kart world sprouting up around my furniture on my Nintendo Switch screen, seen from a doll-sized point of view. What if I could shrink down with a VR headset on and go into a tiny Disney ride that existed somewhere else? What if I was a tiny person running from big real-sized people? What if I was riding through the secret insides of the in-park Millennium Falcon via a droid I was controlling through my headset? 

Trapped at home this year, I’ve been escaping virtually in all sorts of ways. But embodying another device, or robot, or drone, isn’t something I’ve done until Mario Kart Live Home Circuit. And I’m surprised more toys haven’t explored this yet. I’ve used AR to hold a virtual light saber, but that didn’t put me in another point of view… it augmented my own.

“I believe that in Mario Kart Live: Home Circuit, it’s possible to see the Nintendo philosophy of developing new, unique integrated hardware-software entertainment,” Nintendo’s Tamori added. This isn’t Nintendo’s first hardware experiment, in other words, and it won’t be the last.

But if Nintendo and Velan Studios could get this to work well on the three-year-old Nintendo Switch, what could it mean for more advanced phones and tablets? AR already exists on phones, and it’ll live on headsets. But it could very well live inside robots, too.