The console camera of the future could track your movements through walls

MIT researchers have developed a camera that can build 3D models of users throughout a house - a potential game controller of the future.

The newest generation of consoles is going to get us used to being constantly watched by cameras in our living rooms. This might not be how Microsoft would put it, but that’s what the effect of an always-on Kinect will be. People have been penalised for swearing, at themselves, in their own living rooms:

To clarify, that’s a man getting an in-game penalty in NBA 2k14 because he swore outside of the game. Other games have been found to have similar features. Players can toggle the game’s listening-in on or off, but the important point here is that it’s incredibly creepy, and it’s the default, by choice of the developers. The console offers that function, so games will use it.

Let’s look forward, then, to what we might see in the generation after this one. We’re probably going to have cameras that track us through walls. Researchers at the MIT Computer Science and Artificial Intelligence Lab (CSAIL) have developed a system called “WiTrack” that uses radio waves to build 3D images of game players as they move throughout their home, even when they’re in other rooms.

It builds on an earlier system called “WiVi”, developed by some of the same researchers, that was a rough system for tracking people using the Wi-Fi on a typical smartphone. It works based on quite a simple, but clever, insight - the radio waves that a Wi-Fi device emits, like a smartphone or router, are deflected and reflected by objects they come across in just the same way that the radio waves used in radar systems do.

The resolution of Wi-Vi was pretty unimpressive - it could pretty much only tell you if something was moving closer towards you or further away - but it was an exciting idea. At the time, it was suggested that it could be used in hostage situations to figure out where people were inside buildings, or used by rescue teams to find those trapped beneath rubble in the aftermath of an earthquake.

WiTrack, conversely, uses its own radio waves instead of Wi-Fi, meaning it can build up full 3D models of its immediate environment. Here’s more info from MIT:

WiTrack operates by tracking specialized radio signals reflected off a person’s body to pinpoint location and movement. The system uses multiple antennas: one for transmitting signals and three for receiving. The system then builds a geometric model of the user’s location by transmitting signals between the antennas and using the reflections off a person’s body to estimate the distance between the antennas and the user. WiTrack is able to locate motion with significantly increased accuracy, as opposed to tracking devices that rely on wireless signals, according to Adib.

“Because of the limited bandwidth, you cannot get very high location accuracy using WiFi signals,” Adib says. “WiTrack transmits a very low-power radio signal, 100 times smaller than WiFi and 1,000 times smaller than what your cell phone can transmit. But the signal is structured in a particular way to measure the time from when the signal was transmitted until the reflections come back. WiTrack has a geometric model that maps reflection delays to the exact location of the person. The model can also eliminate reflections off walls and furniture to allow us to focus on tracking human motion.”

If that's not enough, there's a video that walks through how it works:

Gamers can look forward, in theory, to a future Kinect-like system with WiTrack capability that can track them throughout their homes. You could lead a squad of soldiers into your kitchen, make a sandwich, and head back to the action in the living room. Or, more usefully, it could be deployed in care homes to detect when elderly patients fall.

It is also, as a bit of technology that can literally see through walls, quite creepy. Imagine the privacy headache that could arise if everyone in a block of flats could see through into their neighbours’ homes, and imagine the paranoia that people would have that the NSA was keeping watch on them (especially considering that we now know that governments have infiltrated games like World of Warcraft).

It's still hard to see quite how this technology could be rolled out in a consumer setting, but give it time and there are probably hints of what's yet to come here.

A researcher using the WiTrack system. (Image: MIT)

Ian Steadman is a staff science and technology writer at the New Statesman. He is on Twitter as @iansteadman.

Hulton Archive/Stringer
Show Hide image

3D cinema without the glasses: a potential new technology could change how we watch films

Early-stage research success hints at a visionary future in which an immersive glass-free 3D experience could be possible at the cinema. 

The rise of film-on-demand streaming sites such as Netflix and MUBI threatens to make visits to the cinema a redundant pastime; why head out to watch a film when you can just watch one from the comfort of your own home?

A deterrent for many has been the influx of 3D blockbuster films released in theatres. An all-too-familiar routine has developed that causes audiences to let out a big sigh at the thought of 3D films: get excited about the latest Marvel flick, travel to your local cinema, sit through previews of future releases and then as the film is about to start...stick on a pair of flimsy plastic 3D glasses.

It’s an experience that has come to feel lacklustre for people who hope to experience more from 3D technology than just a gimmick. However, recent news that researchers at MIT have developed a prototype screen which can show 3D films without glasses may be just the development needed for the medium to attract fans back to the cinema.

A team of scientists from MIT’s Computer Science and Artificial Intelligence Lab paired up with the Weizmann Institute of Science from Israel to create “Cinema 3D” – a model cinema screen which could potentially allow cinema-goers to have the full, immersive 3D experience sans glasses, no matter where they are sitting in the theatre.

Detailing their research in a paper, the scientists outlined the technology used, which includes “automultiscopic displays” – a 3D enabler that presents “multiple angular images of the same scene” and doesn’t require glasses. The research has had to build upon conventional automultiscopic displays that alone aren’t sufficient for a cinema setting; they don’t accommodate for the varying angles at which people view a film in a generally widely-spaced theatre

Wojciech Matusik, an MIT professor who worked on the research said: “Existing approaches to glasses-free 3D require screens whose resolution requirements are so enormous that they are completely impractical. This is the first technical approach that allows for glasses-free 3D on a larger scale.”

Cinema 3D aims to optimise the experience by making use of the cinema setting: the fixed seat positions, the sloped rows, the width of the screen. 3D televisions work as a result of parallax barriers – essentially a set of slits in front of a screen that filter pixels to create the illusion of depth. Traditional parallax barriers tend to fail with anything larger than a television, as they don’t recreate the same image when viewed from different distances and angles.

The researchers have combated this by using multiple parallax barriers in conjunction with slanted horizontal mirrors and vertical lenslets – a small but crucial change which now allows viewers to see the same 3D images play out, whether they’re in the middle row, the back row, or far off in the periphery. According the paper, the design “only displays the narrow angular range observed within the limited width of a single seat.” This can then be replicated for every seat in the theatre.

Cinema 3D will require a lot more work if it is to become practical. As it stands, the prototype is about a pad of paper in size and needs 50 sets of mirrors and lenses. For the researchers though, there is reason to remain optimistic as the technology works in theory at a cinema-scale.

It’s important to note that 3d technology without glasses isn’t new; it has been used in a limited way with televisions. What is new with this research is its potential application to the film industry along with improvements in picture quality. Matusik has stressed that “it remains to be seen whether the approach is financially feasible enough to scale up to a full-blown theatre”, but went on to say “we are optimistic that this is an important next step in developing glasses-free 3D for large spaces like movie theatres and auditoriums.”

It could take a while for the technology to get to a stage where it can be used in multiplexes, and the market may need convincing to adopt something which is expected to cost a lot of money. It could prove to be attractive to the advertising industry who may want to use it for billboards, allowing the technology to be introduced at incrementally larger stages.

The thought of seeing James Cameron’s next Avatar instalment or the latest high-octane thriller played out in 3D without glasses could push the technology forward and get people to return in droves to the silver screen.