The idea of a spectrum
Mixed reality (MR) describes experiences in which digital content and the physical world appear together and, importantly, respond to each other. The terms around it overlap, so it helps to picture a spectrum.
- At one end is the ordinary physical world.
- Augmented reality adds information on top of it, such as labels or arrows on a phone screen. The digital layer often does not know much about the room.
- Mixed reality goes further: a virtual object can sit on a real table, be hidden when a real person walks in front of it, or bounce off a real wall.
- At the far end is fully virtual reality, where the real surroundings are replaced.
Different companies and researchers draw the lines in slightly different places, so the labels are not perfectly standardized. What matters in practice is how much the system understands about the space around the user.
What spatial computing means
Spatial computing is a broader phrase for computing that treats physical space as part of the interface. Instead of a flat screen with windows, applications can be placed in the room, and users interact by looking, pointing, speaking or using their hands. The device needs to know where it is, what surrounds it and where the user is looking.
The term is used loosely. Sometimes it refers to headsets, sometimes to phones and tablets that map rooms, and sometimes to the wider set of techniques, such as mapping and tracking, that make this possible. When reading about a product, it is worth checking what the writer means by it.
How it works
Several technical pieces are typically combined.
Sensors and cameras. Outward-facing cameras and depth sensors observe the room. Motion sensors in the device track how it is moving.
Mapping and tracking. Software builds a rough 3D model of surfaces such as floors, walls and tables, while continuously estimating the device’s position within it. This is often called simultaneous localization and mapping. It is why a virtual object can appear to stay in place as the user walks around it.
Anchors. A digital object is attached to a point in the mapped space. If tracking is accurate, the object stays put. If it is poor, content may drift or jitter.
Display method. Some devices use transparent lenses and project light onto them (“see-through” displays). Others use cameras to capture the real world and show it on screens inside the headset, a method called passthrough. Each has trade-offs in brightness, field of view, image quality and delay.
Input. Hand tracking, eye tracking, controllers and voice commands let people interact. Computer vision helps recognize surfaces, objects and hands, and some systems use machine learning to improve that recognition.
Examples of use
- Training and maintenance. A worker might see step-by-step instructions anchored to a machine, or practice a procedure with virtual parts on a real workbench.
- Design and planning. A team can place a virtual piece of furniture or equipment in a real room to judge size and fit, which is quicker than a full-scale mock-up.
- Collaboration. Remote participants can appear as avatars or share 3D models in the same physical space as colleagues.
- Entertainment. Games can use the actual living room as a play area.
- Accessibility. Some applications may help with navigation or reading, although the quality of such tools varies and they should not be assumed to replace established aids.
Mixed reality is also sometimes tied to digital twins, where a virtual model of a physical asset is shown in the real setting, and to edge computing, because processing close to the device can reduce delay.
Limits and concerns
Hardware. Devices can be costly, heavy or short on battery life. Fields of view may be narrower than human vision, and virtual objects can look less solid in bright light on see-through displays.
Accuracy. Mapping can fail on shiny, dark or featureless surfaces, or in changing rooms.
Comfort. Like VR, extended use can cause fatigue or eye strain for some people. Manufacturer guidance and medical advice should be followed.
Privacy. Because the device views and maps the environment, it may capture images of homes, workplaces and bystanders. People nearby usually have not agreed to be recorded. Questions about what personal data is stored, whether processing happens on the device or elsewhere, and how long it is kept are worth asking. Organizations may need to consider rules about recording in workplaces, which differ by place.
Social acceptance. Wearing a camera-equipped device in public or in meetings can make others uncomfortable.
What to take from this
Mixed reality is best understood as a way to attach digital information to real places and objects, not as a single product. It is most useful where the physical context matters, such as a machine, a room or a build site, and least useful when a flat screen already does the job. Which of these uses will become routine is uncertain, so claims about rapid adoption are best treated with caution.