ʴǰéDz GO craze shows that augmented reality is hitting its stride, U of T experts say
Published: July 13, 2016
It's not even available in Canada yet, but ʴǰéDz GO, the new augmented reality (AR) app from Niantic Labs, is dominating watercooler conversations, social media and headlines both here and abroad.
The GPS-enabled game for Android and Apple smartphones is based on the Nintendo ʴǰéDz characters. Players wander around searching for ʴǰéDz, which the game triggers at semi-random physical locations. When they find one, it activates the phone’s camera and shows them the ʴǰéDz.
While ʴǰéDz Go is a new phenomenon, U of T researchers have been working with augmented reality for years. U of T News spoke to computer science professor Eugene Fiume and electrical and computer engineering professor Steve Mann about ʴǰéDz GO and the future of augmented reality.
Eugene Fiume
What is augmented reality (AR)?
As the name would suggest, augmented reality (AR) is a computer-aided way to augment the physical world by inserting computer generated information into the view of a real scene. Normally, this is done by streaming one channel or more of video of the world into a device and then inserting computer synthesized imagery into those channels just in time for interactive viewing. For the most part, augmented reality is about continuous visual phenomena, though one could imagine augmenting sound, images, touch, and one day perhaps even taste. Common uses for AR include museum exhibits, automatic sign translation, interactive games, mapping, defect analysis, artistic tools, see-through displays on cars, etc. AR should be distinguished from virtual reality (VR), whose purpose is to provide an immersive synthetic approximation of reality whose use generally is isolated from "real" reality. AR blends the real and the virtual, while VR replaces the real by the virtual. These distinctions will in time blur.
How does it work?
The main technology needed for AR is the ability to recognise some aspects of how a user is moving and the position and direction of the camera. If I know where a person is and how he or she is holding the camera, I can then approximate (with a little modelling) apparent features in the real scene, such as walls, roads, and posts. This kind of information comes from accelerometers and position sensors: the same things that our fitness applications use to track our distance and speed. As well, some image processing, computer vision and AR is often used to recognise objects or just basic geometry such as a large flat region in the incoming video stream. Then computer graphics jumps in to generate synthetic images (signs, ʴǰéDz characters, narratives, animation) and blends them into the video stream just before it appears on the viewer's screen.
Augmented reality apps have been around for several years now. Why has ʴǰéDz GO suddenly become so popular?
AR (and its cousin, virtual reality) have been around for a long time. Steve Feiner from Columbia University did AR walking around with a huge backpack, doing a lot of the stuff that is becoming commonplace on a handset today. But AR is only now hitting its stride. Generally as a technology this has to do with Moore's Law combining with devices such as mobile handsets that integrate sensors and functions such as cameras, GPS, accelerometers, image processing, graphics, etc. So now we're starting to see AR become commonplace. But the problem is that the fundamental functionality that AR needs such as registration of live video to a synthetic character or sign is quite hard to accomplish for austere applications that require accuracy. And now we come to ʴǰéDz GO. Here we have a perfect combination of an application that combines nostalgic characters, easy graphics, a relatively low need for accuracy, mobility, and community.
Will its popularity be good for augmented reality apps, or will it lead developers to focus on apps with a narrow commercial potential?
I think its popularity will be good for AR overall, as long as people understand that AR has a very broad scope, and that there are many unsolved problems with this technology (as well as with Virtual Reality).
What other AR apps would you recommend to people who’ve become interested because of ʴǰéDz GO?
There are some terrific little applications that do things like sign translation (Google Translate), Crayola Colo[u]r, Wikitude. There are some beautiful star-chart semi-AR applications that allow you to explore the night sky.
ʴǰéDz GO is an entertainment app. Can you give examples of serious uses for augmented reality?
AR is already part of the landscape in high-tech situations, for example, BMW's Mini augmented glasses, which you can see . I particularly like "austere" AR applications that are being developed now to do defect analysis in cars or boats, or to do 3D/AR training. See, for example, .
What is happening at U of T in terms of AR research?
Karan Singh, in our , is doing work on VR and has a startup called JanusVR. Steve Mann, in Electrical and Computer Engineering, has been doing for many years.
A lot of the media coverage of ʴǰéDz GO is about dangerous consequences – players hurting themselves because they’re not paying attention; players being lured to remote locations where they’ve been robbed, etc. But does Augmented Reality pose any dangers?
Looking at the world through a narrow portal is potentially dangerous, but certainly AR is not the only possible concern. The problem here is one of attentional focus as well as "change blindness". The former reflects ongoing research that humans are in fact terrible multi-taskers. The latter points to the fact that in many situations we do not detect changes in our visual field. There are some very graphic examples of this, but the nicest one I've seen is a commercial (!) for . In fact, AR systems could well be better "augmented" to detect environmental changes.
Steve Mann
You are considered one of the pioneers of augmented reality. Tell us about your involvement with AR.
I've been doing mobile, portable, and wearable AR for 42 years now, . Games often accelerate technologies that have been around for a long time, but technologies of the present day make it easier to do. In my original system, a lot of heavy radio and wireless communications equipment was necessary, but today these features are built into smartphones. Some of the tracking systems I developed many years ago that required a large computer can now be implemented in a small device like a smartphone.
Computer games were responsible for accelerated computer development, such as GPGPU (our lab was the first to use parallel GPGPU systems for general-purpose computation, even though they were designed mainly for computer graphics processing as needed in video games).
Will the success of ʴǰéDz GO be good for AR?
Games like ʴǰéDz GO help to make more people aware of the concepts of Augmented and Mediated ("Augmediated") Reality, and the fact that these realities not only augment our vision but modify (mediate) it.
Is ʴǰéDz GO a good example of the current state of AR development?
The current state of AR development goes way beyond what's implemented in these computer games. Take for example the Metavision glasses that implement a Natural User Interface (NUI). I coined the term "NUI" back in the 1990s and built a natural computational system at founding the MIT Wearable Computing project as its first member.
ʴǰéDz GO is a smartphone app. Are smartphones the future of AR? What about other platforms, such as eyewear or other wearables?
Smartphones are not the future of AR, but merely a transitional technology on the way to something I called "Digital Eye Glass" originally. DEG (Digital Eye Glass) like EyeTap or Metavision represents the future of AR.
Topics
