What is merged reality and how will it affect your future?

Two new examples of “merged reality” technology have been making waves, but what does it actually mean?

Sign Up

Get the New Statesman's Morning Call email.

This is your future:

Windows

At least according to Intel and Microsoft, who, at the Intel Developer Forum in San Francisco on Tuesday, discussed two new developments. The first is Project Alloy, a wireless headset that allows you to bring real objects into the virtual world using 3D cameras. The second is Windows Holographic Shell, and it does the reverse – allowing virtual computer screens, objects, and people into the real world.

Both are examples of merged (or mixed) reality. Nice. Very good. Article over. Except, well, what does that mean?

What is mixed/merged reality?

In short, mixed reality is when the real and virtual world are merged and objects from both can interact. So, for example, if you were playing a virtual video game, you could pick up your nearby real-life water bottle and hit Dr Eggman over the head with it.

Specifically, Intel’s Project Alloy uses mixed reality to allow people to use their real hands in the virtual world. Before this point, users had to hold controllers that represented their hands. Now, the headset’s RealSense cameras will detect your finger movements and allow you to manipulate virtual objects. It is quite an exciting development, but it also looks like this:

The headset will also allow you to see your friends in the virtual world, and keep the confines of the virtual world in the real world space around you. In short, this means less walking-into-walls than in a traditional virtual world. 

The other development, Windows Holographic, uses the HoloLens headset to let you interact with holographic screens that display things like your emails and calendars. The HoloLens is also one single operating system for virtual reality, mixed reality, and augmented reality. According to Windows' promotional video, you can also visit the Pantheon.


Windows via YouTube

How is it different from virtual and augmented reality?

Virtual reality is the umbrella term for all these magical artificial worlds that are popping up around us. Most commonly, users plonk on a headset and get a 360° view of an immersive fake world. Naturally, it’s thus far predominantly been used for video games and porn.

Augmented reality – of which the most famous example is Pokemon Go – is when virtual stuff is added to your real world environment. The virtual reality element, ie. Pikachu, is overlayed onto the real world. This has been used for sporting events and tourism, as information about the game or place can be overlayed as you look around. Although this is similar to merged reality, Microsoft and Intel's new projects go further as they do not require sensors and are wireless, so more seemlessly merge the real and virtual worlds. 

When is mixed reality coming to destroy life as we know it forever?

Windows Holographic was unveiled last year, but the latest announcement revealed that it will be soaring into any Windows 10 user’s life next year. Project Alloy’s consumer release date is set for 2018, and for the time being, there is no word on the price. Although these are big developments, it could take years for the technology to become commonplace. In practice, although it sounds cool, there might not be much demand or it may be considered socially unacceptable (remember Google Glass, anyone?).

What will it change about your life?

Theoretically in a few years, you could attend a sporting event or concert without leaving your home. Ok, sure, TV has let you do this for a fair few decades, but Intel's CEO Brian Krzanich explains the difference in a Medium post:

“Walk onto the field and right into the goalie box, or join the huddle, looking over the quarterback’s shoulder to see the calls before they’re played. Stand on the stage with your favorite band and feel the love of the fans from their point of view. Choose your experience and navigate it. It’s all up to you. The virtual world doesn’t need to be virtual; it can be real.”

Krzanich also theorises that you could practise tennis without visiting a court, go on a virtual holiday, and – for some reason, if you were so inclined  play the piano and cello at the same time.

Intel's announcement trailer also features a man enjoying a virtual party under the premise: What if virtual reality felt less virtual and more real? The idea is that people could come together to party while physically being in different parts of the world, and that you could physically walk around the party without the need for cords and controls. Sounds great, right? Except they then demonstrate that you can pause the party and throw a waiter in a pool. The fact the comment section is disabled says more about this then we ever could:

For gamers, mixed reality will also potentially mean a better experience, whereby users can be completely immersed in fictional words by using their own hands and body instead of controllers. 

In the world of work, Windows Holographic could let you have meetings with people where you're physically remote but virtually in the same place. 

Finally, according to Windows’ video demonstration, you’ll also have a fake puppy.


Windows via YouTube

No word yet on whether it leaves fake mess to clean up too.

Amelia Tait is a freelance journalist, and was previously the New Statesman's tech and digital culture writer. She tweets at @ameliargh