This is the final article in the XR Tech Stack Overview for VR Development. We’ve covered both the VR and AR tech stacks. This piece is about Mixed Reality, which is some kind of mix of AR and VR but is neither. 

‍

The table below shows how VR and AR features both overlap with MR features.

Mixed Reality is an emergent form of immersive media that takes the best of both worlds to create an even richer experience. Thus far there are 5 main categories of MR devices but for this article, we’ll only be focusing on building for HMDs and mobile phones.

‍

Types of MR Devices

1. CAVE (Cave Automatic Virtual Environment) - A CAVE is an immersive VR system that projects 3D images onto the walls of a cube-shaped room. Users wear special glasses and, sometimes, have controllers to view and interact with the projections. The projected images are also designed to adjust based on user movements which are tracked by sensors in the room. Computers generate the images and 3D audio. CAVEs are used in various fields like engineering for prototyping and simulation. They provide a lifelike experience, allowing users to interact with virtual objects before committing resources.

2. HUD (Head-up Display) - A HUD projects information directly in the user's field of view using a projector, combiner, and computer. Originally used in military aviation, HUDs are now common in commercial planes and cars. Automotive HUDs, like Pioneer's Heads-up system, replace the sun visor and project navigation data onto the road ahead. The display adjusts based on the driver's movements and the vehicle's position to provide relevant, real-time information without obstructing the view.
   
   

2. HMD (Head-mounted displays) - HMDs use one or two optics to project images directly in front of the user's eyes, providing an immersive experience across various fields. Consumer HMDs, like the Meta Quest 3, are popular for entertainment. In contrast, enterprise-focused augmented reality HMDs overlay digital information onto the physical world, such as the Microsoft HoloLens and HoloLens 2, and Apple Vision Pro. These devices have medical applications, giving doctors real-time insights, and engineering. The Meta Quest Pro is another notable HMD that blends virtual and augmented reality capabilities.
   ‍
   
   
3. Mobile Phones - Modern smartphones and tablets have become increasingly powerful, portable, and capable of running augmented reality applications. These devices come with toolkits that allow developers to overlay digital content onto the physical world. Pokémon GO, released in 2016, was the first widely successful AR mobile game. Beyond entertainment, productivity apps like Google Maps and Google Translate now include AR features for navigation and real-time translation. The iPhone 15 takes this further by enabling users to capture spatial (3D) videos that can be viewed on devices like the Apple Vision Pro, enhancing the immersive experience. The constant presence of mobile devices in our daily lives makes them unique platforms for delivering augmented reality experiences to users anytime, anywhere

‍

So, how do you build for MR? It’s a similar process to building for VR and AR seeing as it incorporates aspects of both. As such, the usual suspects are involved. We cover that in the next section.

‍

‍

How to build for MR 

‍

Like with VR and AR projects building an MR app starts with figuring out a few things, which are:

‍

1. The first consideration is platform? Is it cross-platform? I.e. is it compatible with multiple devices? Is it an enhanced environment app? E.g. Google maps? Or a blended environment app, e.g. Pokemon Go? Deciding this helps you decide what devices you can build for.
   ‍
2. Who is your audience and what device/devices will they experience your MR app with? What platforms are they on? Which mobile operating system do they use? What types of network connections are they on? You need as much data as you can gather on your customers and the type of devices they use. Have a baseline device that serves as a minimum requirement.
   ‍
3. How will you trigger the app's MR functions? Is it with a marker like a QR code or a special type of graphic? Or will it work with the device’s sensors to kickstart in real-time, thus being markerless? Or will it be for a headset and have its triggers built in? 
   ‍
4. The next thing to consider would be what SDK you intend to use for the app and this depends on the previous two points, your audience, and their devices, and how the MR functions in the app will be triggered. You’d need an SDK that supports the platform of your choice and supports a game engine, e.g., Unity.

‍

Let’s expand a bit more on each option:

‍

1. Platform 

Most current XR experiences are carried out either via Smartphones or HMDs such as the Meta Quest 3, Apple Vision Pro, or Microsoft HoloLens 2. There are MR apps/experiences that are cross-platform either courtesy of SDK or courtesy of the manufacturers. Some manufacturers such as Meta Inc. have made their MR operating system, Meta Horizon OS, open source and free for 3rd party hardware makers to use and are also working on a spatial framework for mobile devices. Different operating systems or SDKs are compatible with different platforms. Go with the ones that support your audience’s devices.

‍

2. Audience & Device

There are many types of MR devices, each with its tradeoffs. Smartphones allow for enhanced and blended environment apps, but the variety of blended environment apps you can create on a smartphone is limited compared to an HMD and smartphones have zero capacity for immersive environment MR. That said, smartphones are more accessible than HMDs given the number of people using a smartphone compared to an HMD.
‍

‍

3. Feature Triggering

Depending on the device MR features can be triggered in a variety of ways. If your target device is a smartphone, your triggers will be AR-based, i.e., either via markers like QR codes, or via GPS (location-based), or via SLAM technology (Markerless). A headset can use all those triggers and can kick start its own MR experiences as well, courtesy of the amount of complexity that is possible with headsets.

‍

4. SDKs and Game Engines

We defined SDKs in our previous article on VR tech stacks and as mentioned they’re toolkits that come with libraries, guides, elements, and blueprints that developers can use to create XR experiences for devices. Your choice of SDK depends heavily on the platform and audience you’re building for, among others. Below is a list of factors to consider when choosing an SDK:

• Free or paid: Many SDKs are free because most manufacturers provide SDKs for developers to use; Meta has Meta Open XR, Microsoft has Mixed Reality Toolkit, Apple supports ARKit and Vision OS, the list is long. Things are different with web-based XR, however, many of the popular and reliable development tools, e.g., 8th Wall, Spark AR, etc. all require payment.   
  ‍
• Platform: Are you developing for iOS, Android, or the web? 
  ‍
• Device: Is it for PCs, mobile phones, Mixed Reality headsets, or Smart Glasses?
  ‍
• Cloud support: Does the SDK offer cloud integration? Cloud storage reduces the memory and performance demands running an MR app places on your device. SDKs like ARCore and ARit offer cloud support via content delivery networks (CDNs) like AWS and Google Cloud platform. 
  ‍
• Game engine support: Not all SDKs work with all game engines, you want to ensure you pick an SDK that’s compatible with the game engine of your choice. 
  ‍
• GeoLocation support: Necessary if you want to build a location-based AR app
  ‍
• SLAM (Simultaneous Localization and Mapping): SLAM technology is what allows self-driving cars and automated robots to map an environment and detect objects in real-time. Your SDK should support this technology if you’re working with a markerless AR app
  ‍
• 3D tracking: Does the SDK have functionality to track 3D objects?
  ‍

Conclusion

Once you have all these questions answered you’re set to start development. Stay tuned for more detailed articles on SDKs and game engines.

‍

What is your favorite MR app at the moment? Answer with a comment.

‍