Grand Challenges for Augmented Reality Research

What research needs to be done so that Augmented Reality reaches its full potential?

Overview

The first working Augmented Reality (AR) system was developed over 50 years ago, but todays AR technology is still far removed from the ultimate potential of the technology. So there are a number of research Grand Challenges that need to be addressed.

Researcher Ron Azuma [1] provided one of the best definitions of the AR, writing that AR technology has three key elements; (1) it combines real and virtual images, (2) it is interactive in real time, and (3) the virtual content is registered in three dimensions. Each of these three areas provide important areas for research; (1) display technology, (2) interaction techniques, and (3) tracking methods.

Display Technology

One Grand Challenge with AR display technology is to create a wide field of view, high resolution, see-through display in a socially acceptable form factor. Azuma [2] lists a number of research problems in optics and displays that need to be addressed, such as providing sufficient brightness and contrast, having a high resolution and wide field of view, addressing eyestrain, and being in a sunglass like form factor. Other important topics include addressing the AR vergence accommodation problem, showing photorealistic content, and new form factors such as contact lenses.

Interaction Techniques

Another Grand Challenge is to enable people to manipulate AR content as easily as they do with objects in the real world. One method that has been explored is using real objects to interact with AR in a technique called Tangible AR [3]. Modern AR systems support free-hand gestures, but this could be improved further by adding speech recognition, and supporting combined speech and gesture input in multimodal interfaces. Eye-tracking, full-body input, and other non-verbal cues provide other possibilities for research. Finally, research also needs to be conducted into interaction techniques not possible in the real world, such as brain computer input.

Tracking Methods

In order to fix AR content in the real world, the user’s viewpoint needs to be known, so a fundamental challenge is tracking techniques. There has been lots of significant research on using computer vision for AR tracking. However new hybrid methods combine computer vision with other sensors for improved tracking, such as [4]. There is a need to extend this further by researching wide area tracking techniques using sensor fusion from a dynamic combination of mobile and stationary tracking. This could be combined with machine learning methods for scene understanding and AR cloud techniques to provide a ubiquitous tracking service.

In addition to these Grand Challenges in fundamental technologies for AR, there are a number of other important research opportunities. Three that I think are important are (1) Perception and Neuroscience, (2) Collaboration, and (3) Social and Ethical Issues.

Perception and Neuroscience

The overall goal of AR systems is to create a perceptual illusion that convinces the brain that virtual content actually exists in the real world. Considerable research has been conducted on how to make AR content perceptually the same as real objects, including the use of virtual lighting and shadows, and occlusion of and by real objects, and similar methods. Recent research has shown that brain activity can be detected to provide an objective measure of how real AR content seems [5]. So neuroscience could provide valuable insights into AR experiences.

Collaboration

One of the most important areas where AR could significantly improve quality of life is through enabling new ways of face to face and remote collaboration. AR can provide the rich spatial cues often missing from video conferencing, and create the illusion that remote people are standing in the real world with a local user. It can also be used to enable people in the same location to view and interact with AR content in a natural way. However, there is still very little research conducted on collaborative AR. A survey of ten years of AR user studies until 2015, found that only 4% of them involved collaboration [6]. There are many interesting directions for collaborative work. For example, the emerging field of Empathic Computing [7] explores how physiological cues can be combined with AR in collaborative interfaces to enable remote people to share what they are seeing, hearing and feeling.

Social and Ethical Issues

Even if AR technical challenges are addressed, significant social and ethical issues will need to be overcome. For example, do people feel awkward usign AR displays in public, who should be able to place AR content in the view of a person, and what are the privacy implications of seeing personal information in public spaces? Pase [7] lists a number of questionable ethical uses of pervasive AR, such as for deception, surveillance, behaviour modification, and punishment. Widespread AR technology use may depend more on social than technical issues.

In conclusion, AR could fundamentally change how people interact with digital content. However, as we have show, there are many Grand Challenges that must be addressed before the full potential of the technology can be realised.

Note: This is a shortened version of the article “Grand Challenges for Augmented Reality” published in the Frontiers in Virtual Reality journal.

References

[1] Azuma, R. T. (1997). A Survey of Augmented Reality. Presence: Teleoperators & Virtual Environments, 6(4), 355–385.

[2] Azuma, R. T. (2017). Making Augmented Reality a Reality. In Applied Industrial Optics: Spectroscopy, Imaging and Metrology (pp. JTu1F-1). Optical Society of America.

[3] Billinghurst, M., Kato, H., and Poupyrev, I. (2008). Tangible Augmented Reality. ACM SIGGRAPH ASIA, 7(2), 1–10.

[4] Liu, H., Zhang, G., and Bao, H. (2016). Robust Keyframe-based Monocular SLAM for Augmented Reality. In 2016 IEEE International Symposium on Mixed and Augmented Reality (ISMAR), 1–10. IEEE.

[5] Bauman, B., and Seeling, P. (2018). Evaluation of EEG-Based Predictions of Image QoE in Augmented Reality Scenarios. In 2018 IEEE 88th Vehicular Technology Conference (VTC-Fall) (pp. 1–5). IEEE.

[6] Dey, A., Billinghurst, M., Lindeman, R. W., and Swan, J. (2018). A systematic review of 10 years of augmented reality usability studies: 2005 to 2014. Frontiers in Robotics and AI, 5, 37.

[7] Piumsomboon, T., Lee, Y., Lee, G. A., Dey, A., and Billinghurst, M. (2017). Empathic mixed reality: Sharing what you feel and interacting with what you see. In 2017 International Symposium on Ubiquitous Virtual Reality (ISUVR) (pp. 38–41). IEEE.

[8] Pase, S. (2012). Ethical considerations in augmented reality applications. In Proceedings of the International Conference on e-Learning, e-Business, Enterprise Information Systems, and e-Government (EEE) (p. 1). The Steering Committee of The World Congress in Computer Science, Computer Engineering and Applied Computing (WorldComp).