augmented reality research papers

Thank you for visiting nature.com. You are using a browser version with limited support for CSS. To obtain the best experience, we recommend you use a more up to date browser (or turn off compatibility mode in Internet Explorer). In the meantime, to ensure continued support, we are displaying the site without styles and JavaScript.

• View all journals
• My Account Login
• Explore content
• About the journal
• Publish with us
• Sign up for alerts
• Review Article
• Open access
• Published: 25 October 2021

Augmented reality and virtual reality displays: emerging technologies and future perspectives

• Jianghao Xiong 1 ,
• En-Lin Hsiang 1 ,
• Ziqian He 1 ,
• Tao Zhan   ORCID: orcid.org/0000-0001-5511-6666 1 &
• Shin-Tson Wu   ORCID: orcid.org/0000-0002-0943-0440 1  

Light: Science & Applications volume  10 , Article number:  216 ( 2021 ) Cite this article

118k Accesses

474 Citations

40 Altmetric

Metrics details

• Liquid crystals

With rapid advances in high-speed communication and computation, augmented reality (AR) and virtual reality (VR) are emerging as next-generation display platforms for deeper human-digital interactions. Nonetheless, to simultaneously match the exceptional performance of human vision and keep the near-eye display module compact and lightweight imposes unprecedented challenges on optical engineering. Fortunately, recent progress in holographic optical elements (HOEs) and lithography-enabled devices provide innovative ways to tackle these obstacles in AR and VR that are otherwise difficult with traditional optics. In this review, we begin with introducing the basic structures of AR and VR headsets, and then describing the operation principles of various HOEs and lithography-enabled devices. Their properties are analyzed in detail, including strong selectivity on wavelength and incident angle, and multiplexing ability of volume HOEs, polarization dependency and active switching of liquid crystal HOEs, device fabrication, and properties of micro-LEDs (light-emitting diodes), and large design freedoms of metasurfaces. Afterwards, we discuss how these devices help enhance the AR and VR performance, with detailed description and analysis of some state-of-the-art architectures. Finally, we cast a perspective on potential developments and research directions of these photonic devices for future AR and VR displays.

Similar content being viewed by others

Advanced liquid crystal devices for augmented reality and virtual reality displays: principles and applications

Achromatic diffractive liquid-crystal optics for virtual reality displays

Full-colour 3D holographic augmented-reality displays with metasurface waveguides

Introduction.

Recent advances in high-speed communication and miniature mobile computing platforms have escalated a strong demand for deeper human-digital interactions beyond traditional flat panel displays. Augmented reality (AR) and virtual reality (VR) headsets 1 , 2 are emerging as next-generation interactive displays with the ability to provide vivid three-dimensional (3D) visual experiences. Their useful applications include education, healthcare, engineering, and gaming, just to name a few 3 , 4 , 5 . VR embraces a total immersive experience, while AR promotes the interaction between user, digital contents, and real world, therefore displaying virtual images while remaining see-through capability. In terms of display performance, AR and VR face several common challenges to satisfy demanding human vision requirements, including field of view (FoV), eyebox, angular resolution, dynamic range, and correct depth cue, etc. Another pressing demand, although not directly related to optical performance, is ergonomics. To provide a user-friendly wearing experience, AR and VR should be lightweight and ideally have a compact, glasses-like form factor. The above-mentioned requirements, nonetheless, often entail several tradeoff relations with one another, which makes the design of high-performance AR/VR glasses/headsets particularly challenging.

In the 1990s, AR/VR experienced the first boom, which quickly subsided due to the lack of eligible hardware and digital content 6 . Over the past decade, the concept of immersive displays was revisited and received a new round of excitement. Emerging technologies like holography and lithography have greatly reshaped the AR/VR display systems. In this article, we firstly review the basic requirements of AR/VR displays and their associated challenges. Then, we briefly describe the properties of two emerging technologies: holographic optical elements (HOEs) and lithography-based devices (Fig. 1 ). Next, we separately introduce VR and AR systems because of their different device structures and requirements. For the immersive VR system, the major challenges and how these emerging technologies help mitigate the problems will be discussed. For the see-through AR system, we firstly review the present status of light engines and introduce some architectures for the optical combiners. Performance summaries on microdisplay light engines and optical combiners will be provided, that serve as a comprehensive overview of the current AR display systems.

The left side illustrates HOEs and lithography-based devices. The right side shows the challenges in VR and architectures in AR, and how the emerging technologies can be applied

Key parameters of AR and VR displays

AR and VR displays face several common challenges to satisfy the demanding human vision requirements, such as FoV, eyebox, angular resolution, dynamic range, and correct depth cue, etc. These requirements often exhibit tradeoffs with one another. Before diving into detailed relations, it is beneficial to review the basic definitions of the above-mentioned display parameters.

Definition of parameters

Taking a VR system (Fig. 2a ) as an example. The light emitting from the display module is projected to a FoV, which can be translated to the size of the image perceived by the viewer. For reference, human vision’s horizontal FoV can be as large as 160° for monocular vision and 120° for overlapped binocular vision 6 . The intersection area of ray bundles forms the exit pupil, which is usually correlated with another parameter called eyebox. The eyebox defines the region within which the whole image FoV can be viewed without vignetting. It therefore generally manifests a 3D geometry 7 , whose volume is strongly dependent on the exit pupil size. A larger eyebox offers more tolerance to accommodate the user’s diversified interpupillary distance (IPD) and wiggling of headset when in use. Angular resolution is defined by dividing the total resolution of the display panel by FoV, which measures the sharpness of a perceived image. For reference, a human visual acuity of 20/20 amounts to 1 arcmin angular resolution, or 60 pixels per degree (PPD), which is considered as a common goal for AR and VR displays. Another important feature of a 3D display is depth cue. Depth cue can be induced by displaying two separate images to the left eye and the right eye, which forms the vergence cue. But the fixed depth of the displayed image often mismatches with the actual depth of the intended 3D image, which leads to incorrect accommodation cues. This mismatch causes the so-called vergence-accommodation conflict (VAC), which will be discussed in detail later. One important observation is that the VAC issue may be more serious in AR than VR, because the image in an AR display is directly superimposed onto the real-world with correct depth cues. The image contrast is dependent on the display panel and stray light. To achieve a high dynamic range, the display panel should exhibit high brightness, low dark level, and more than 10-bits of gray levels. Nowadays, the display brightness of a typical VR headset is about 150–200 cd/m 2 (or nits).

a Schematic of a VR display defining FoV, exit pupil, eyebox, angular resolution, and accommodation cue mismatch. b Sketch of an AR display illustrating ACR

Figure 2b depicts a generic structure of an AR display. The definition of above parameters remains the same. One major difference is the influence of ambient light on the image contrast. For a see-through AR display, ambient contrast ratio (ACR) 8 is commonly used to quantify the image contrast:

where L on ( L off ) represents the on (off)-state luminance (unit: nit), L am is the ambient luminance, and T is the see-through transmittance. In general, ambient light is measured in illuminance (lux). For the convenience of comparison, we convert illuminance to luminance by dividing a factor of π, assuming the emission profile is Lambertian. In a normal living room, the illuminance is about 100 lux (i.e., L am  ≈ 30 nits), while in a typical office lighting condition, L am  ≈ 150 nits. For outdoors, on an overcast day, L am  ≈ 300 nits, and L am  ≈ 3000 nits on a sunny day. For AR displays, a minimum ACR should be 3:1 for recognizable images, 5:1 for adequate readability, and ≥10:1 for outstanding readability. To make a simple estimate without considering all the optical losses, to achieve ACR = 10:1 in a sunny day (~3000 nits), the display needs to deliver a brightness of at least 30,000 nits. This imposes big challenges in finding a high brightness microdisplay and designing a low loss optical combiner.

Tradeoffs and potential solutions

Next, let us briefly review the tradeoff relations mentioned earlier. To begin with, a larger FoV leads to a lower angular resolution for a given display resolution. In theory, to overcome this tradeoff only requires a high-resolution-display source, along with high-quality optics to support the corresponding modulation transfer function (MTF). To attain 60 PPD across 100° FoV requires a 6K resolution for each eye. This may be realizable in VR headsets because a large display panel, say 2–3 inches, can still accommodate a high resolution with acceptable manufacture cost. However, for a glasses-like wearable AR display, the conflict between small display size and the high solution becomes obvious as further shrinking the pixel size of a microdisplay is challenging.

To circumvent this issue, the concept of the foveated display is proposed 9 , 10 , 11 , 12 , 13 . The idea is based on that the human eye only has high visual acuity in the central fovea region, which accounts for about 10° FoV. If the high-resolution image is only projected to fovea while the peripheral image remains low resolution, then a microdisplay with 2K resolution can satisfy the need. Regarding the implementation method of foveated display, a straightforward way is to optically combine two display sources 9 , 10 , 11 : one for foveal and one for peripheral FoV. This approach can be regarded as spatial multiplexing of displays. Alternatively, time-multiplexing can also be adopted, by temporally changing the optical path to produce different magnification factors for the corresponding FoV 12 . Finally, another approach without multiplexing is to use a specially designed lens with intended distortion to achieve non-uniform resolution density 13 . Aside from the implementation of foveation, another great challenge is to dynamically steer the foveated region as the viewer’s eye moves. This task is strongly related to pupil steering, which will be discussed in detail later.

A larger eyebox or FoV usually decreases the image brightness, which often lowers the ACR. This is exactly the case for a waveguide AR system with exit pupil expansion (EPE) while operating under a strong ambient light. To improve ACR, one approach is to dynamically adjust the transmittance with a tunable dimmer 14 , 15 . Another solution is to directly boost the image brightness with a high luminance microdisplay and an efficient combiner optics. Details of this topic will be discussed in the light engine section.

Another tradeoff of FoV and eyebox in geometric optical systems results from the conservation of etendue (or optical invariant). To increase the system etendue requires a larger optics, which in turn compromises the form factor. Finally, to address the VAC issue, the display system needs to generate a proper accommodation cue, which often requires the modulation of image depth or wavefront, neither of which can be easily achieved in a traditional geometric optical system. While remarkable progresses have been made to adopt freeform surfaces 16 , 17 , 18 , to further advance AR and VR systems requires additional novel optics with a higher degree of freedom in structure design and light modulation. Moreover, the employed optics should be thin and lightweight. To mitigate the above-mentioned challenges, diffractive optics is a strong contender. Unlike geometric optics relying on curved surfaces to refract or reflect light, diffractive optics only requires a thin layer of several micrometers to establish efficient light diffractions. Two major types of diffractive optics are HOEs based on wavefront recording and manually written devices like surface relief gratings (SRGs) based on lithography. While SRGs have large design freedoms of local grating geometry, a recent publication 19 indicates the combination of HOE and freeform optics can also offer a great potential for arbitrary wavefront generation. Furthermore, the advances in lithography have also enabled optical metasurfaces beyond diffractive and refractive optics, and miniature display panels like micro-LED (light-emitting diode). These devices hold the potential to boost the performance of current AR/VR displays, while keeping a lightweight and compact form factor.

Formation and properties of HOEs

HOE generally refers to a recorded hologram that reproduces the original light wavefront. The concept of holography is proposed by Dennis Gabor 20 , which refers to the process of recording a wavefront in a medium (hologram) and later reconstructing it with a reference beam. Early holography uses intensity-sensitive recording materials like silver halide emulsion, dichromated gelatin, and photopolymer 21 . Among them, photopolymer stands out due to its easy fabrication and ability to capture high-fidelity patterns 22 , 23 . It has therefore found extensive applications like holographic data storage 23 and display 24 , 25 . Photopolymer HOEs (PPHOEs) have a relatively small refractive index modulation and therefore exhibits a strong selectivity on the wavelength and incident angle. Another feature of PPHOE is that several holograms can be recorded into a photopolymer film by consecutive exposures. Later, liquid-crystal holographic optical elements (LCHOEs) based on photoalignment polarization holography have also been developed 25 , 26 . Due to the inherent anisotropic property of liquid crystals, LCHOEs are extremely sensitive to the polarization state of the input light. This feature, combined with the polarization modulation ability of liquid crystal devices, offers a new possibility for dynamic wavefront modulation in display systems.

The formation of PPHOE is illustrated in Fig. 3a . When exposed to an interfering field with high-and-low intensity fringes, monomers tend to move toward bright fringes due to the higher local monomer-consumption rate. As a result, the density and refractive index is slightly larger in bright regions. Note the index modulation δ n here is defined as the difference between the maximum and minimum refractive indices, which may be twice the value in other definitions 27 . The index modulation δ n is typically in the range of 0–0.06. To understand the optical properties of PPHOE, we simulate a transmissive grating and a reflective grating using rigorous coupled-wave analysis (RCWA) 28 , 29 and plot the results in Fig. 3b . Details of grating configuration can be found in Table S1 . Here, the reason for only simulating gratings is that for a general HOE, the local region can be treated as a grating. The observation of gratings can therefore offer a general insight of HOEs. For a transmissive grating, its angular bandwidth (efficiency > 80%) is around 5° ( λ  = 550 nm), while the spectral band is relatively broad, with bandwidth around 175 nm (7° incidence). For a reflective grating, its spectral band is narrow, with bandwidth around 10 nm. The angular bandwidth varies with the wavelength, ranging from 2° to 20°. The strong selectivity of PPHOE on wavelength and incident angle is directly related to its small δ n , which can be adjusted by controlling the exposure dosage.

a Schematic of the formation of PPHOE. Simulated efficiency plots for b1 transmissive and b2 reflective PPHOEs. c Working principle of multiplexed PPHOE. d Formation and molecular configurations of LCHOEs. Simulated efficiency plots for e1 transmissive and e2 reflective LCHOEs. f Illustration of polarization dependency of LCHOEs

A distinctive feature of PPHOE is the ability to multiplex several holograms into one film sample. If the exposure dosage of a recording process is controlled so that the monomers are not completely depleted in the first exposure, the remaining monomers can continue to form another hologram in the following recording process. Because the total amount of monomer is fixed, there is usually an efficiency tradeoff between multiplexed holograms. The final film sample would exhibit the wavefront modulation functions of multiple holograms (Fig. 3c ).

Liquid crystals have also been used to form HOEs. LCHOEs can generally be categorized into volume-recording type and surface-alignment type. Volume-recording type LCHOEs are either based on early polarization holography recordings with azo-polymer 30 , 31 , or holographic polymer-dispersed liquid crystals (HPDLCs) 32 , 33 formed by liquid-crystal-doped photopolymer. Surface-alignment type LCHOEs are based on photoalignment polarization holography (PAPH) 34 . The first step is to record the desired polarization pattern in a thin photoalignment layer, and the second step is to use it to align the bulk liquid crystal 25 , 35 . Due to the simple fabrication process, high efficiency, and low scattering from liquid crystal’s self-assembly nature, surface-alignment type LCHOEs based on PAPH have recently attracted increasing interest in applications like near-eye displays. Here, we shall focus on this type of surface-alignment LCHOE and refer to it as LCHOE thereafter for simplicity.

The formation of LCHOEs is illustrated in Fig. 3d . The information of the wavefront and the local diffraction pattern is recorded in a thin photoalignment layer. The volume liquid crystal deposited on the photoalignment layer, depending on whether it is nematic liquid crystal or cholesteric liquid crystal (CLC), forms a transmissive or a reflective LCHOE. In a transmissive LCHOE, the bulk nematic liquid crystal molecules generally follow the pattern of the bottom alignment layer. The smallest allowable pattern period is governed by the liquid crystal distortion-free energy model, which predicts the pattern period should generally be larger than sample thickness 36 , 37 . This results in a maximum diffraction angle under 20°. On the other hand, in a reflective LCHOE 38 , 39 , the bulk CLC molecules form a stable helical structure, which is tilted to match the k -vector of the bottom pattern. The structure exhibits a very low distorted free energy 40 , 41 and can accommodate a pattern period that is small enough to diffract light into the total internal reflection (TIR) of a glass substrate.

The diffraction property of LCHOEs is shown in Fig. 3e . The maximum refractive index modulation of LCHOE is equal to the liquid crystal birefringence (Δ n ), which may vary from 0.04 to 0.5, depending on the molecular conjugation 42 , 43 . The birefringence used in our simulation is Δ n  = 0.15. Compared to PPHOEs, the angular and spectral bandwidths are significantly larger for both transmissive and reflective LCHOEs. For a transmissive LCHOE, its angular bandwidth is around 20° ( λ  = 550 nm), while the spectral bandwidth is around 300 nm (7° incidence). For a reflective LCHOE, its spectral bandwidth is around 80 nm and angular bandwidth could vary from 15° to 50°, depending on the wavelength.

The anisotropic nature of liquid crystal leads to LCHOE’s unique polarization-dependent response to an incident light. As depicted in Fig. 3f , for a transmissive LCHOE the accumulated phase is opposite for the conjugated left-handed circular polarization (LCP) and right-handed circular polarization (RCP) states, leading to reversed diffraction directions. For a reflective LCHOE, the polarization dependency is similar to that of a normal CLC. For the circular polarization with the same handedness as the helical structure of CLC, the diffraction is strong. For the opposite circular polarization, the diffraction is negligible.

Another distinctive property of liquid crystal is its dynamic response to an external voltage. The LC reorientation can be controlled with a relatively low voltage (<10 V rms ) and the response time is on the order of milliseconds, depending mainly on the LC viscosity and layer thickness. Methods to dynamically control LCHOEs can be categorized as active addressing and passive addressing, which can be achieved by either directly switching the LCHOE or modulating the polarization state with an active waveplate. Detailed addressing methods will be described in the VAC section.

Lithography-enabled devices

Lithography technologies are used to create arbitrary patterns on wafers, which lays the foundation of the modern integrated circuit industry 44 . Photolithography is suitable for mass production while electron/ion beam lithography is usually used to create photomask for photolithography or to write structures with nanometer-scale feature size. Recent advances in lithography have enabled engineered structures like optical metasurfaces 45 , SRGs 46 , as well as micro-LED displays 47 . Metasurfaces exhibit a remarkable design freedom by varying the shape of meta-atoms, which can be utilized to achieve novel functions like achromatic focus 48 and beam steering 49 . Similarly, SRGs also offer a large design freedom by manipulating the geometry of local grating regions to realize desired optical properties. On the other hand, micro-LED exhibits several unique features, such as ultrahigh peak brightness, small aperture ratio, excellent stability, and nanosecond response time, etc. As a result, micro-LED is a promising candidate for AR and VR systems for achieving high ACR and high frame rate for suppressing motion image blurs. In the following section, we will briefly review the fabrication and properties of micro-LEDs and optical modulators like metasurfaces and SRGs.

Fabrication and properties of micro-LEDs

LEDs with a chip size larger than 300 μm have been widely used in solid-state lighting and public information displays. Recently, micro-LEDs with chip sizes <5 μm have been demonstrated 50 . The first micro-LED disc with a diameter of about 12 µm was demonstrated in 2000 51 . After that, a single color (blue or green) LED microdisplay was demonstrated in 2012 52 . The high peak brightness, fast response time, true dark state, and long lifetime of micro-LEDs are attractive for display applications. Therefore, many companies have since released their micro-LED prototypes or products, ranging from large-size TVs to small-size microdisplays for AR/VR applications 53 , 54 . Here, we focus on micro-LEDs for near-eye display applications. Regarding the fabrication of micro-LEDs, through the metal-organic chemical vapor deposition (MOCVD) method, the AlGaInP epitaxial layer is grown on GaAs substrate for red LEDs, and GaN epitaxial layers on sapphire substrate for green and blue LEDs. Next, a photolithography process is applied to define the mesa and deposit electrodes. To drive the LED array, the fabricated micro-LEDs are transferred to a CMOS (complementary metal oxide semiconductor) driver board. For a small size (<2 inches) microdisplay used in AR or VR, the precision of the pick-and-place transfer process is hard to meet the high-resolution-density (>1000 pixel per inch) requirement. Thus, the main approach to assemble LED chips with driving circuits is flip-chip bonding 50 , 55 , 56 , 57 , as Fig. 4a depicts. In flip-chip bonding, the mesa and electrode pads should be defined and deposited before the transfer process, while metal bonding balls should be preprocessed on the CMOS substrate. After that, thermal-compression method is used to bond the two wafers together. However, due to the thermal mismatch of LED chip and driving board, as the pixel size decreases, the misalignment between the LED chip and the metal bonding ball on the CMOS substrate becomes serious. In addition, the common n-GaN layer may cause optical crosstalk between pixels, which degrades the image quality. To overcome these issues, the LED epitaxial layer can be firstly metal-bonded with the silicon driver board, followed by the photolithography process to define the LED mesas and electrodes. Without the need for an alignment process, the pixel size can be reduced to <5 µm 50 .

a Illustration of flip-chip bonding technology. b Simulated IQE-LED size relations for red and blue LEDs based on ABC model. c Comparison of EQE of different LED sizes with and without KOH and ALD side wall treatment. d Angular emission profiles of LEDs with different sizes. Metasurfaces based on e resonance-tuning, f non-resonance tuning and g combination of both. h Replication master and i replicated SRG based on nanoimprint lithography. Reproduced from a ref. 55 with permission from AIP Publishing, b ref. 61 with permission from PNAS, c ref. 66 with permission from IOP Publishing, d ref. 67 with permission from AIP Publishing, e ref. 69 with permission from OSA Publishing f ref. 48 with permission from AAAS g ref. 70 with permission from AAAS and h , i ref. 85 with permission from OSA Publishing

In addition to manufacturing process, the electrical and optical characteristics of LED also depend on the chip size. Generally, due to Shockley-Read-Hall (SRH) non-radiative recombination on the sidewall of active area, a smaller LED chip size results in a lower internal quantum efficiency (IQE), so that the peak IQE driving point will move toward a higher current density due to increased ratio of sidewall surface to active volume 58 , 59 , 60 . In addition, compared to the GaN-based green and blue LEDs, the AlGaInP-based red LEDs with a larger surface recombination and carrier diffusion length suffer a more severe efficiency drop 61 , 62 . Figure 4b shows the simulated result of IQE drop in relation with the LED chip size of blue and red LEDs based on ABC model 63 . To alleviate the efficiency drop caused by sidewall defects, depositing passivation materials by atomic layer deposition (ALD) or plasma enhanced chemical vapor deposition (PECVD) is proven to be helpful for both GaN and AlGaInP based LEDs 64 , 65 . In addition, applying KOH (Potassium hydroxide) treatment after ALD can further reduce the EQE drop of micro-LEDs 66 (Fig. 4c ). Small-size LEDs also exhibit some advantages, such as higher light extraction efficiency (LEE). Compared to an 100-µm LED, the LEE of a 2-µm LED increases from 12.2 to 25.1% 67 . Moreover, the radiation pattern of micro-LED is more directional than that of a large-size LED (Fig. 4d ). This helps to improve the lens collection efficiency in AR/VR display systems.

Metasurfaces and SGs

Thanks to the advances in lithography technology, low-loss dielectric metasurfaces working in the visible band have recently emerged as a platform for wavefront shaping 45 , 48 , 68 . They consist of an array of subwavelength-spaced structures with individually engineered wavelength-dependent polarization/phase/ amplitude response. In general, the light modulation mechanisms can be classified into resonant tuning 69 (Fig. 4e ), non-resonant tuning 48 (Fig. 4f ), and combination of both 70 (Fig. 4g ). In comparison with non-resonant tuning (based on geometric phase and/or dynamic propagation phase), the resonant tuning (such as Fabry–Pérot resonance, Mie resonance, etc.) is usually associated with a narrower operating bandwidth and a smaller out-of-plane aspect ratio (height/width) of nanostructures. As a result, they are easier to fabricate but more sensitive to fabrication tolerances. For both types, materials with a higher refractive index and lower absorption loss are beneficial to reduce the aspect ratio of nanostructure and improve the device efficiency. To this end, titanium dioxide (TiO 2 ) and gallium nitride (GaN) are the major choices for operating in the entire visible band 68 , 71 . While small-sized metasurfaces (diameter <1 mm) are usually fabricated via electron-beam lithography or focused ion beam milling in the labs, the ability of mass production is the key to their practical adoption. The deep ultraviolet (UV) photolithography has proven its feasibility for reproducing centimeter-size metalenses with decent imaging performance, while it requires multiple steps of etching 72 . Interestingly, the recently developed UV nanoimprint lithography based on a high-index nanocomposite only takes a single step and can obtain an aspect ratio larger than 10, which shows great promise for high-volume production 73 .

The arbitrary wavefront shaping capability and the thinness of the metasurfaces have aroused strong research interests in the development of novel AR/VR prototypes with improved performance. Lee et al. employed nanoimprint lithography to fabricate a centimeter-size, geometric-phase metalens eyepiece for full-color AR displays 74 . Through tailoring its polarization conversion efficiency and stacking with a circular polarizer, the virtual image can be superimposed with the surrounding scene. The large numerical aperture (NA~0.5) of the metalens eyepiece enables a wide FoV (>76°) that conventional optics are difficult to obtain. However, the geometric phase metalens is intrinsically a diffractive lens that also suffers from strong chromatic aberrations. To overcome this issue, an achromatic lens can be designed via simultaneously engineering the group delay and the group delay dispersion 75 , 76 , which will be described in detail later. Other novel and/or improved near-eye display architectures include metasurface-based contact lens-type AR 77 , achromatic metalens array enabled integral-imaging light field displays 78 , wide FoV lightguide AR with polarization-dependent metagratings 79 , and off-axis projection-type AR with an aberration-corrected metasurface combiner 80 , 81 , 82 . Nevertheless, from the existing AR/VR prototypes, metasurfaces still face a strong tradeoff between numerical aperture (for metalenses), chromatic aberration, monochromatic aberration, efficiency, aperture size, and fabrication complexity.

On the other hand, SRGs are diffractive gratings that have been researched for decades as input/output couplers of waveguides 83 , 84 . Their surface is composed of corrugated microstructures, and different shapes including binary, blazed, slanted, and even analogue can be designed. The parameters of the corrugated microstructures are determined by the target diffraction order, operation spectral bandwidth, and angular bandwidth. Compared to metasurfaces, SRGs have a much larger feature size and thus can be fabricated via UV photolithography and subsequent etching. They are usually replicated by nanoimprint lithography with appropriate heating and surface treatment. According to a report published a decade ago, SRGs with a height of 300 nm and a slant angle of up to 50° can be faithfully replicated with high yield and reproducibility 85 (Fig. 4g, h ).

Challenges and solutions of VR displays

The fully immersive nature of VR headset leads to a relatively fixed configuration where the display panel is placed in front of the viewer’s eye and an imaging optics is placed in-between. Regarding the system performance, although inadequate angular resolution still exists in some current VR headsets, the improvement of display panel resolution with advanced fabrication process is expected to solve this issue progressively. Therefore, in the following discussion, we will mainly focus on two major challenges: form factor and 3D cue generation.

Form factor

Compact and lightweight near-eye displays are essential for a comfortable user experience and therefore highly desirable in VR headsets. Current mainstream VR headsets usually have a considerably larger volume than eyeglasses, and most of the volume is just empty. This is because a certain distance is required between the display panel and the viewing optics, which is usually close to the focal length of the lens system as illustrated in Fig. 5a . Conventional VR headsets employ a transmissive lens with ~4 cm focal length to offer a large FoV and eyebox. Fresnel lenses are thinner than conventional ones, but the distance required between the lens and the panel does not change significantly. In addition, the diffraction artifacts and stray light caused by the Fresnel grooves can degrade the image quality, or MTF. Although the resolution density, quantified as pixel per inch (PPI), of current VR headsets is still limited, eventually Fresnel lens will not be an ideal solution when a high PPI display is available. The strong chromatic aberration of Fresnel singlet should also be compensated if a high-quality imaging system is preferred.

a Schematic of a basic VR optical configuration. b Achromatic metalens used as VR eyepiece. c VR based on curved display and lenslet array. d Basic working principle of a VR display based on pancake optics. e VR with pancake optics and Fresnel lens array. f VR with pancake optics based on purely HOEs. Reprinted from b ref. 87 under the Creative Commons Attribution 4.0 License. Adapted from c ref. 88 with permission from IEEE, e ref. 91 and f ref. 92 under the Creative Commons Attribution 4.0 License

It is tempting to replace the refractive elements with a single thin diffractive lens like a transmissive LCHOE. However, the diffractive nature of such a lens will result in serious color aberrations. Interestingly, metalenses can fulfil this objective without color issues. To understand how metalenses achieve achromatic focus, let us first take a glance at the general lens phase profile \(\Phi (\omega ,r)\) expanded as a Taylor series 75 :

where \(\varphi _0(\omega )\) is the phase at the lens center, \(F\left( \omega \right)\) is the focal length as a function of frequency ω , r is the radial coordinate, and \(\omega _0\) is the central operation frequency. To realize achromatic focus, \(\partial F{{{\mathrm{/}}}}\partial \omega\) should be zero. With a designed focal length, the group delay \(\partial \Phi (\omega ,r){{{\mathrm{/}}}}\partial \omega\) and the group delay dispersion \(\partial ^2\Phi (\omega ,r){{{\mathrm{/}}}}\partial \omega ^2\) can be determined, and \(\varphi _0(\omega )\) is an auxiliary degree of freedom of the phase profile design. In the design of an achromatic metalens, the group delay is a function of the radial coordinate and monotonically increases with the metalens radius. Many designs have proven that the group delay has a limited variation range 75 , 76 , 78 , 86 . According to Shrestha et al. 86 , there is an inevitable tradeoff between the maximum radius of the metalens, NA, and operation bandwidth. Thus, the reported achromatic metalenses at visible usually have limited lens aperture (e.g., diameter < 250 μm) and NA (e.g., <0.2). Such a tradeoff is undesirable in VR displays, as the eyepiece favors a large clear aperture (inch size) and a reasonably high NA (>0.3) to maintain a wide FoV and a reasonable eye relief 74 .

To overcome this limitation, Li et al. 87 proposed a novel zone lens method. Unlike the traditional phase Fresnel lens where the zones are determined by the phase reset, the new approach divides the zones by the group delay reset. In this way, the lens aperture and NA can be much enlarged, and the group delay limit is bypassed. A notable side effect of this design is the phase discontinuity at zone boundaries that will contribute to higher-order focusing. Therefore, significant efforts have been conducted to find the optimal zone transition locations and to minimize the phase discontinuities. Using this method, they have demonstrated an impressive 2-mm-diameter metalens with NA = 0.7 and nearly diffraction-limited focusing for the designed wavelengths (488, 532, 658 nm) (Fig. 5b ). Such a metalens consists of 681 zones and works for the visible band ranging from 470 to 670 nm, though the focusing efficiency is in the order of 10%. This is a great starting point for the achromatic metalens to be employed as a compact, chromatic-aberration-free eyepiece in near-eye displays. Future challenges are how to further increase the aperture size, correct the off-axis aberrations, and improve the optical efficiency.

Besides replacing the refractive lens with an achromatic metalens, another way to reduce system focal length without decreasing NA is to use a lenslet array 88 . As depicted in Fig. 5c , both the lenslet array and display panel adopt a curved structure. With the latest flexible OLED panel, the display can be easily curved in one dimension. The system exhibits a large diagonal FoV of 180° with an eyebox of 19 by 12 mm. The geometry of each lenslet is optimized separately to achieve an overall performance with high image quality and reduced distortions.

Aside from trying to shorten the system focal length, another way to reduce total track is to fold optical path. Recently, polarization-based folded lenses, also known as pancake optics, are under active development for VR applications 89 , 90 . Figure 5d depicts the structure of an exemplary singlet pancake VR lens system. The pancake lenses can offer better imaging performance with a compact form factor since there are more degrees of freedom in the design and the actual light path is folded thrice. By using a reflective surface with a positive power, the field curvature of positive refractive lenses can be compensated. Also, the reflective surface has no chromatic aberrations and it contributes considerable optical power to the system. Therefore, the optical power of refractive lenses can be smaller, resulting in an even weaker chromatic aberration. Compared to Fresnel lenses, the pancake lenses have smooth surfaces and much fewer diffraction artifacts and stray light. However, such a pancake lens design is not perfect either, whose major shortcoming is low light efficiency. With two incidences of light on the half mirror, the maximum system efficiency is limited to 25% for a polarized input and 12.5% for an unpolarized input light. Moreover, due to the existence of multiple surfaces in the system, stray light caused by surface reflections and polarization leakage may lead to apparent ghost images. As a result, the catadioptric pancake VR headset usually manifests a darker imagery and lower contrast than the corresponding dioptric VR.

Interestingly, the lenslet and pancake optics can be combined to further reduce the system form. Bang et al. 91 demonstrated a compact VR system with a pancake optics and a Fresnel lenslet array. The pancake optics serves to fold the optical path between the display panel and the lenslet array (Fig. 5e ). Another Fresnel lens is used to collect the light from the lenslet array. The system has a decent horizontal FoV of 102° and an eyebox of 8 mm. However, a certain degree of image discontinuity and crosstalk are still present, which can be improved with further optimizations on the Fresnel lens and the lenslet array.

One step further, replacing all conventional optics in catadioptric VR headset with holographic optics can make the whole system even thinner. Maimone and Wang demonstrated such a lightweight, high-resolution, and ultra-compact VR optical system using purely HOEs 92 . This holographic VR optics was made possible by combining several innovative optical components, including a reflective PPHOE, a reflective LCHOE, and a PPHOE-based directional backlight with laser illumination, as shown in Fig. 5f . Since all the optical power is provided by the HOEs with negligible weight and volume, the total physical thickness can be reduced to <10 mm. Also, unlike conventional bulk optics, the optical power of a HOE is independent of its thickness, only subject to the recording process. Another advantage of using holographic optical devices is that they can be engineered to offer distinct phase profiles for different wavelengths and angles of incidence, adding extra degrees of freedom in optical designs for better imaging performance. Although only a single-color backlight has been demonstrated, such a PPHOE has the potential to achieve full-color laser backlight with multiplexing ability. The PPHOE and LCHOE in the pancake optics can also be optimized at different wavelengths for achieving high-quality full-color images.

Vergence-accommodation conflict

Conventional VR displays suffer from VAC, which is a common issue for stereoscopic 3D displays 93 . In current VR display modules, the distance between the display panel and the viewing optics is fixed, which means the VR imagery is displayed at a single depth. However, the image contents are generated by parallax rendering in three dimensions, offering distinct images for two eyes. This approach offers a proper stimulus to vergence but completely ignores the accommodation cue, which leads to the well-known VAC that can cause an uncomfortable user experience. Since the beginning of this century, numerous methods have been proposed to solve this critical issue. Methods to produce accommodation cue include multifocal/varifocal display 94 , holographic display 95 , and integral imaging display 96 . Alternatively, elimination of accommodation cue using a Maxwellian-view display 93 also helps to mitigate the VAC. However, holographic displays and Maxwellian-view displays generally require a totally different optical architecture than current VR systems. They are therefore more suitable for AR displays, which will be discussed later. Integral imaging, on the other hand, has an inherent tradeoff between view number and resolution. For current VR headsets pursuing high resolution to match human visual acuity, it may not be an appealing solution. Therefore, multifocal/varifocal displays that rely on depth modulation is a relatively practical and effective solution for VR headsets. Regarding the working mechanism, multifocal displays present multiple images with different depths to imitate the original 3D scene. Varifocal displays, in contrast, only show one image at each time frame. The image depth matches the viewer’s vergence depth. Nonetheless, the pre-knowledge of the viewer’s vergence depth requires an additional eye-tracking module. Despite different operation principles, a varifocal display can often be converted to a multifocal display as long as the varifocal module has enough modulation bandwidth to support multiple depths in a time frame.

To achieve depth modulation in a VR system, traditional liquid lens 97 , 98 with tunable focus suffers from the small aperture and large aberrations. Alvarez lens 99 is another tunable-focus solution but it requires mechanical adjustment, which adds to system volume and complexity. In comparison, transmissive LCHOEs with polarization dependency can achieve focus adjustment with electronic driving. Its ultra-thinness also satisfies the requirement of small form factors in VR headsets. The diffractive behavior of transmissive LCHOEs is often interpreted by the mechanism of Pancharatnam-Berry phase (also known as geometric phase) 100 . They are therefore often called Pancharatnam-Berry optical elements (PBOEs). The corresponding lens component is referred as Pancharatnam-Berry lens (PBL).

Two main approaches are used to switch the focus of a PBL, active addressing and passive addressing. In active addressing, the PBL itself (made of LC) can be switched by an applied voltage (Fig. 6a ). The optical power of the liquid crystal PBLs can be turned-on and -off by controlling the voltage. Stacking multiple active PBLs can produce 2 N depths, where N is the number of PBLs. The drawback of using active PBLs, however, is the limited spectral bandwidth since their diffraction efficiency is usually optimized at a single wavelength. In passive addressing, the depth modulation is achieved through changing the polarization state of input light by a switchable half-wave plate (HWP) (Fig. 6b ). The focal length can therefore be switched thanks to the polarization sensitivity of PBLs. Although this approach has a slightly more complicated structure, the overall performance can be better than the active one, because the PBLs made of liquid crystal polymer can be designed to manifest high efficiency within the entire visible spectrum 101 , 102 .

Working principles of a depth switching PBL module based on a active addressing and b passive addressing. c A four-depth multifocal display based on time multiplexing. d A two-depth multifocal display based on polarization multiplexing. Reproduced from c ref. 103 with permission from OSA Publishing and d ref. 104 with permission from OSA Publishing

With the PBL module, multifocal displays can be built using time-multiplexing technique. Zhan et al. 103 demonstrated a four-depth multifocal display using two actively switchable liquid crystal PBLs (Fig. 6c ). The display is synchronized with the PBL module, which lowers the frame rate by the number of depths. Alternatively, multifocal displays can also be achieved by polarization-multiplexing, as demonstrated by Tan et al. 104 . The basic principle is to adjust the polarization state of local pixels so the image content on two focal planes of a PBL can be arbitrarily controlled (Fig. 6d ). The advantage of polarization multiplexing is that it does not sacrifice the frame rate, but it can only support two planes because only two orthogonal polarization states are available. Still, it can be combined with time-multiplexing to reduce the frame rate sacrifice by half. Naturally, varifocal displays can also be built with a PBL module. A fast-response 64-depth varifocal module with six PBLs has been demonstrated 105 .

The compact structure of PBL module leads to a natural solution of integrating it with above-mentioned pancake optics. A compact VR headset with dynamic depth modulation to solve VAC is therefore possible in practice. Still, due to the inherent diffractive nature of PBL, the PBL module face the issue of chromatic dispersion of focal length. To compensate for different focal depths for RGB colors may require additional digital corrections in image-rendering.

Architectures of AR displays

Unlike VR displays with a relatively fixed optical configuration, there exist a vast number of architectures in AR displays. Therefore, instead of following the narrative of tackling different challenges, a more appropriate way to review AR displays is to separately introduce each architecture and discuss its associated engineering challenges. An AR display usually consists of a light engine and an optical combiner. The light engine serves as display image source, while the combiner delivers the displayed images to viewer’s eye and in the meantime transmits the environment light. Some performance parameters like frame rate and power consumption are mainly determined by the light engine. Parameters like FoV, eyebox and MTF are primarily dependent on the combiner optics. Moreover, attributes like image brightness, overall efficiency, and form factor are influenced by both light engine and combiner. In this section, we will firstly discuss the light engine, where the latest advances in micro-LED on chip are reviewed and compared with existing microdisplay systems. Then, we will introduce two main types of combiners: free-space combiner and waveguide combiner.

Light engine

The light engine determines several essential properties of the AR system like image brightness, power consumption, frame rate, and basic etendue. Several types of microdisplays have been used in AR, including micro-LED, micro-organic-light-emitting-diodes (micro-OLED), liquid-crystal-on-silicon (LCoS), digital micromirror device (DMD), and laser beam scanning (LBS) based on micro-electromechanical system (MEMS). We will firstly describe the working principles of these devices and then analyze their performance. For those who are more interested in final performance parameters than details, Table 1 provides a comprehensive summary.

Working principles

Micro-LED and micro-OLED are self-emissive display devices. They are usually more compact than LCoS and DMD because no illumination optics is required. The fundamentally different material systems of LED and OLED lead to different approaches to achieve full-color displays. Due to the “green gap” in LEDs, red LEDs are manufactured on a different semiconductor material from green and blue LEDs. Therefore, how to achieve full-color display in high-resolution density microdisplays is quite a challenge for micro-LEDs. Among several solutions under research are two main approaches. The first is to combine three separate red, green and blue (RGB) micro-LED microdisplay panels 106 . Three single-color micro-LED microdisplays are manufactured separately through flip-chip transfer technology. Then, the projected images from three microdisplay panels are integrated by a trichroic prism (Fig. 7a ).

a RGB micro-LED microdisplays combined by a trichroic prism. b QD-based micro-LED microdisplay. c Micro-OLED display with 4032 PPI. Working principles of d LCoS, e DMD, and f MEMS-LBS display modules. Reprinted from a ref. 106 with permission from IEEE, b ref. 108 with permission from Chinese Laser Press, c ref. 121 with permission from Jon Wiley and Sons, d ref. 124 with permission from Spring Nature, e ref. 126 with permission from Springer and f ref. 128 under the Creative Commons Attribution 4.0 License

Another solution is to assemble color-conversion materials like quantum dot (QD) on top of blue or ultraviolet (UV) micro-LEDs 107 , 108 , 109 (Fig. 7b ). The quantum dot color filter (QDCF) on top of the micro-LED array is mainly fabricated by inkjet printing or photolithography 110 , 111 . However, the display performance of color-conversion micro-LED displays is restricted by the low color-conversion efficiency, blue light leakage, and color crosstalk. Extensive efforts have been conducted to improve the QD-micro-LED performance. To boost QD conversion efficiency, structure designs like nanoring 112 and nanohole 113 , 114 have been proposed, which utilize the Förster resonance energy transfer mechanism to transfer excessive excitons in the LED active region to QD. To prevent blue light leakage, methods using color filters or reflectors like distributed Bragg reflector (DBR) 115 and CLC film 116 on top of QDCF are proposed. Compared to color filters that absorb blue light, DBR and CLC film help recycle the leaked blue light to further excite QDs. Other methods to achieve full-color micro-LED display like vertically stacked RGB micro-LED array 61 , 117 , 118 and monolithic wavelength tunable nanowire LED 119 are also under investigation.

Micro-OLED displays can be generally categorized into RGB OLED and white OLED (WOLED). RGB OLED displays have separate sub-pixel structures and optical cavities, which resonate at the desirable wavelength in RGB channels, respectively. To deposit organic materials onto the separated RGB sub-pixels, a fine metal mask (FMM) that defines the deposition area is required. However, high-resolution RGB OLED microdisplays still face challenges due to the shadow effect during the deposition process through FMM. In order to break the limitation, a silicon nitride film with small shadow has been proposed as a mask for high-resolution deposition above 2000 PPI (9.3 µm) 120 .

WOLED displays use color filters to generate color images. Without the process of depositing patterned organic materials, a high-resolution density up to 4000 PPI has been achieved 121 (Fig. 7c ). However, compared to RGB OLED, the color filters in WOLED absorb about 70% of the emitted light, which limits the maximum brightness of the microdisplay. To improve the efficiency and peak brightness of WOLED microdisplays, in 2019 Sony proposed to apply newly designed cathodes (InZnO) and microlens arrays on OLED microdisplays, which increased the peak brightness from 1600 nits to 5000 nits 120 . In addition, OLEDWORKs has proposed a multi-stacked OLED 122 with optimized microcavities whose emission spectra match the transmission bands of the color filters. The multi-stacked OLED shows a higher luminous efficiency (cd/A), but also requires a higher driving voltage. Recently, by using meta-mirrors as bottom reflective anodes, patterned microcavities with more than 10,000 PPI have been obtained 123 . The high-resolution meta-mirrors generate different reflection phases in the RGB sub-pixels to achieve desirable resonant wavelengths. The narrow emission spectra from the microcavity help to reduce the loss from color filters or even eliminate the need of color filters.

LCoS and DMD are light-modulating displays that generate images by controlling the reflection of each pixel. For LCoS, the light modulation is achieved by manipulating the polarization state of output light through independently controlling the liquid crystal reorientation in each pixel 124 , 125 (Fig. 7d ). Both phase-only and amplitude modulators have been employed. DMD is an amplitude modulation device. The modulation is achieved through controlling the tilt angle of bi-stable micromirrors 126 (Fig. 7e ). To generate an image, both LCoS and DMD rely on the light illumination systems, with LED or laser as light source. For LCoS, the generation of color image can be realized either by RGB color filters on LCoS (with white LEDs) or color-sequential addressing (with RGB LEDs or lasers). However, LCoS requires a linearly polarized light source. For an unpolarized LED light source, usually, a polarization recycling system 127 is implemented to improve the optical efficiency. For a single-panel DMD, the color image is mainly obtained through color-sequential addressing. In addition, DMD does not require a polarized light so that it generally exhibits a higher efficiency than LCoS if an unpolarized light source is employed.

MEMS-based LBS 128 , 129 utilizes micromirrors to directly scan RGB laser beams to form two-dimensional (2D) images (Fig. 7f ). Different gray levels are achieved by pulse width modulation (PWM) of the employed laser diodes. In practice, 2D scanning can be achieved either through a 2D scanning mirror or two 1D scanning mirrors with an additional focusing lens after the first mirror. The small size of MEMS mirror offers a very attractive form factor. At the same time, the output image has a large depth-of-focus (DoF), which is ideal for projection displays. One shortcoming, though, is that the small system etendue often hinders its applications in some traditional display systems.

Comparison of light engine performance

There are several important parameters for a light engine, including image resolution, brightness, frame rate, contrast ratio, and form factor. The resolution requirement (>2K) is similar for all types of light engines. The improvement of resolution is usually accomplished through the manufacturing process. Thus, here we shall focus on other three parameters.

Image brightness usually refers to the measured luminance of a light-emitting object. This measurement, however, may not be accurate for a light engine as the light from engine only forms an intermediate image, which is not directly viewed by the user. On the other hand, to solely focus on the brightness of a light engine could be misleading for a wearable display system like AR. Nowadays, data projectors with thousands of lumens are available. But the power consumption is too high for a battery-powered wearable AR display. Therefore, a more appropriate way to evaluate a light engine’s brightness is to use luminous efficacy (lm/W) measured by dividing the final output luminous flux (lm) by the input electric power (W). For a self-emissive device like micro-LED or micro-OLED, the luminous efficacy is directly determined by the device itself. However, for LCoS and DMD, the overall luminous efficacy should take into consideration the light source luminous efficacy, the efficiency of illumination optics, and the efficiency of the employed spatial light modulator (SLM). For a MEMS LBS engine, the efficiency of MEMS mirror can be considered as unity so that the luminous efficacy basically equals to that of the employed laser sources.

As mentioned earlier, each light engine has a different scheme for generating color images. Therefore, we separately list luminous efficacy of each scheme for a more inclusive comparison. For micro-LEDs, the situation is more complicated because the EQE depends on the chip size. Based on previous studies 130 , 131 , 132 , 133 , we separately calculate the luminous efficacy for RGB micro-LEDs with chip size ≈ 20 µm. For the scheme of direct combination of RGB micro-LEDs, the luminous efficacy is around 5 lm/W. For QD-conversion with blue micro-LEDs, the luminous efficacy is around 10 lm/W with the assumption of 100% color conversion efficiency, which has been demonstrated using structure engineering 114 . For micro-OLEDs, the calculated luminous efficacy is about 4–8 lm/W 120 , 122 . However, the lifetime and EQE of blue OLED materials depend on the driving current. To continuously display an image with brightness higher than 10,000 nits may dramatically shorten the device lifetime. The reason we compare the light engine at 10,000 nits is that it is highly desirable to obtain 1000 nits for the displayed image in order to keep ACR>3:1 with a typical AR combiner whose optical efficiency is lower than 10%.

For an LCoS engine using a white LED as light source, the typical optical efficiency of the whole engine is around 10% 127 , 134 . Then the engine luminous efficacy is estimated to be 12 lm/W with a 120 lm/W white LED source. For a color sequential LCoS using RGB LEDs, the absorption loss from color filters is eliminated, but the luminous efficacy of RGB LED source is also decreased to about 30 lm/W due to lower efficiency of red and green LEDs and higher driving current 135 . Therefore, the final luminous efficacy of the color sequential LCoS engine is also around 10 lm/W. If RGB linearly polarized lasers are employed instead of LEDs, then the LCoS engine efficiency can be quite high due to the high degree of collimation. The luminous efficacy of RGB laser source is around 40 lm/W 136 . Therefore, the laser-based LCoS engine is estimated to have a luminous efficacy of 32 lm/W, assuming the engine optical efficiency is 80%. For a DMD engine with RGB LEDs as light source, the optical efficiency is around 50% 137 , 138 , which leads to a luminous efficacy of 15 lm/W. By switching to laser light sources, the situation is similar to LCoS, with the luminous efficacy of about 32 lm/W. Finally, for MEMS-based LBS engine, there is basically no loss from the optics so that the final luminous efficacy is 40 lm/W. Detailed calculations of luminous efficacy can be found in Supplementary Information .

Another aspect of a light engine is the frame rate, which determines the volume of information it can deliver in a unit time. A high volume of information is vital for the construction of a 3D light field to solve the VAC issue. For micro-LEDs, the device response time is around several nanoseconds, which allows for visible light communication with bandwidth up to 1.5 Gbit/s 139 . For an OLED microdisplay, a fast OLED with ~200 MHz bandwidth has been demonstrated 140 . Therefore, the limitation of frame rate is on the driving circuits for both micro-LED and OLED. Another fact concerning driving circuit is the tradeoff between resolution and frame rate as a higher resolution panel means more scanning lines in each frame. So far, an OLED display with 480 Hz frame rate has been demonstrated 141 . For an LCoS, the frame rate is mainly limited by the LC response time. Depending on the LC material used, the response time is around 1 ms for nematic LC or 200 µs for ferroelectric LC (FLC) 125 . Nematic LC allows analog driving, which accommodates gray levels, typically with 8-bit depth. FLC is bistable so that PWM is used to generate gray levels. DMD is also a binary device. The frame rate can reach 30 kHz, which is mainly constrained by the response time of micromirrors. For MEMS-based LBS, the frame rate is limited by the scanning frequency of MEMS mirrors. A frame rate of 60 Hz with around 1 K resolution already requires a resonance frequency of around 50 kHz, with a Q-factor up to 145,000 128 . A higher frame rate or resolution requires a higher Q-factor and larger laser modulation bandwidth, which may be challenging.

Form factor is another crucial aspect for the light engines of near-eye displays. For self-emissive displays, both micro-OLEDs and QD-based micro-LEDs can achieve full color with a single panel. Thus, they are quite compact. A micro-LED display with separate RGB panels naturally have a larger form factor. In applications requiring direct-view full-color panel, the extra combining optics may also increase the volume. It needs to be pointed out, however, that the combing optics may not be necessary for some applications like waveguide displays, because the EPE process results in system’s insensitivity to the spatial positions of input RGB images. Therefore, the form factor of using three RGB micro-LED panels is medium. For LCoS and DMD with RGB LEDs as light source, the form factor would be larger due to the illumination optics. Still, if a lower luminous efficacy can be accepted, then a smaller form factor can be achieved by using a simpler optics 142 . If RGB lasers are used, the collimation optics can be eliminated, which greatly reduces the form factor 143 . For MEMS-LBS, the form factor can be extremely compact due to the tiny size of MEMS mirror and laser module.

Finally, contrast ratio (CR) also plays an important role affecting the observed images 8 . Micro-LEDs and micro-OLEDs are self-emissive so that their CR can be >10 6 :1. For a laser beam scanner, its CR can also achieve 10 6 :1 because the laser can be turned off completely at dark state. On the other hand, LCoS and DMD are reflective displays, and their CR is around 2000:1 to 5000:1 144 , 145 . It is worth pointing out that the CR of a display engine plays a significant role only in the dark ambient. As the ambient brightness increases, the ACR is mainly governed by the display’s peak brightness, as previously discussed.

The performance parameters of different light engines are summarized in Table 1 . Micro-LEDs and micro-OLEDs have similar levels of luminous efficacy. But micro-OLEDs still face the burn-in and lifetime issue when driving at a high current, which hinders its use for a high-brightness image source to some extent. Micro-LEDs are still under active development and the improvement on luminous efficacy from maturing fabrication process could be expected. Both devices have nanosecond response time and can potentially achieve a high frame rate with a well-designed integrated circuit. The frame rate of the driving circuit ultimately determines the motion picture response time 146 . Their self-emissive feature also leads to a small form factor and high contrast ratio. LCoS and DMD engines have similar performance of luminous efficacy, form factor, and contrast ratio. In terms of light modulation, DMD can provide a higher 1-bit frame rate, while LCoS can offer both phase and amplitude modulations. MEMS-based LBS exhibits the highest luminous efficacy so far. It also exhibits an excellent form factor and contrast ratio, but the presently demonstrated 60-Hz frame rate (limited by the MEMS mirrors) could cause image flickering.

Free-space combiners

The term ‘free-space’ generally refers to the case when light is freely propagating in space, as opposed to a waveguide that traps light into TIRs. Regarding the combiner, it can be a partial mirror, as commonly used in AR systems based on traditional geometric optics. Alternatively, the combiner can also be a reflective HOE. The strong chromatic dispersion of HOE necessitates the use of a laser source, which usually leads to a Maxwellian-type system.

Traditional geometric designs

Several systems based on geometric optics are illustrated in Fig. 8 . The simplest design uses a single freeform half-mirror 6 , 147 to directly collimate the displayed images to the viewer’s eye (Fig. 8a ). This design can achieve a large FoV (up to 90°) 147 , but the limited design freedom with a single freeform surface leads to image distortions, also called pupil swim 6 . The placement of half-mirror also results in a relatively bulky form factor. Another design using so-called birdbath optics 6 , 148 is shown in Fig. 8b . Compared to the single-combiner design, birdbath design has an extra optics on the display side, which provides space for aberration correction. The integration of beam splitter provides a folded optical path, which reduces the form factor to some extent. Another way to fold optical path is to use a TIR-prism. Cheng et al. 149 designed a freeform TIR-prism combiner (Fig. 8c ) offering a diagonal FoV of 54° and exit pupil diameter of 8 mm. All the surfaces are freeform, which offer an excellent image quality. To cancel the optical power for the transmitted environmental light, a compensator is added to the TIR prism. The whole system has a well-balanced performance between FoV, eyebox, and form factor. To release the space in front of viewer’s eye, relay optics can be used to form an intermediate image near the combiner 150 , 151 , as illustrated in Fig. 8d . Although the design offers more optical surfaces for aberration correction, the extra lenses also add to system weight and form factor.

a Single freeform surface as the combiner. b Birdbath optics with a beam splitter and a half mirror. c Freeform TIR prism with a compensator. d Relay optics with a half mirror. Adapted from c ref. 149 with permission from OSA Publishing and d ref. 151 with permission from OSA Publishing

Regarding the approaches to solve the VAC issue, the most straightforward way is to integrate a tunable lens into the optical path, like a liquid lens 152 or Alvarez lens 99 , to form a varifocal system. Alternatively, integral imaging 153 , 154 can also be used, by replacing the original display panel with the central depth plane of an integral imaging module. The integral imaging can also be combined with varifocal approach to overcome the tradeoff between resolution and depth of field (DoF) 155 , 156 , 157 . However, the inherent tradeoff between resolution and view number still exists in this case.

Overall, AR displays based on traditional geometric optics have a relatively simple design with a decent FoV (~60°) and eyebox (8 mm) 158 . They also exhibit a reasonable efficiency. To measure the efficiency of an AR combiner, an appropriate measure is to divide the output luminance (unit: nit) by the input luminous flux (unit: lm), which we note as combiner efficiency. For a fixed input luminous flux, the output luminance, or image brightness, is related to the FoV and exit pupil of the combiner system. If we assume no light waste of the combiner system, then the maximum combiner efficiency for a typical diagonal FoV of 60° and exit pupil (10 mm square) is around 17,000 nit/lm (Eq. S2 ). To estimate the combiner efficiency of geometric combiners, we assume 50% of half-mirror transmittance and the efficiency of other optics to be 50%. Then the final combiner efficiency is about 4200 nit/lm, which is a high value in comparison with waveguide combiners. Nonetheless, to further shrink the system size or improve system performance ultimately encounters the etendue conservation issue. In addition, AR systems with traditional geometric optics is hard to achieve a configuration resembling normal flat glasses because the half-mirror has to be tilted to some extent.

Maxwellian-type systems

The Maxwellian view, proposed by James Clerk Maxwell (1860), refers to imaging a point light source in the eye pupil 159 . If the light beam is modulated in the imaging process, a corresponding image can be formed on the retina (Fig. 9a ). Because the point source is much smaller than the eye pupil, the image is always-in-focus on the retina irrespective of the eye lens’ focus. For applications in AR display, the point source is usually a laser with narrow angular and spectral bandwidths. LED light sources can also build a Maxwellian system, by adding an angular filtering module 160 . Regarding the combiner, although in theory a half-mirror can also be used, HOEs are generally preferred because they offer the off-axis configuration that places combiner in a similar position like eyeglasses. In addition, HOEs have a lower reflection of environment light, which provides a more natural appearance of the user behind the display.

a Schematic of the working principle of Maxwellian displays. Maxwellian displays based on b SLM and laser diode light source and c MEMS-LBS with a steering mirror as additional modulation method. Generation of depth cues by d computational digital holography and e scanning of steering mirror to produce multiple views. Adapted from b, d ref. 143 and c, e ref. 167 under the Creative Commons Attribution 4.0 License

To modulate the light, a SLM like LCoS or DMD can be placed in the light path, as shown in Fig. 9b . Alternatively, LBS system can also be used (Fig. 9c ), where the intensity modulation occurs in the laser diode itself. Besides the operation in a normal Maxwellian-view, both implementations offer additional degrees of freedom for light modulation.

For a SLM-based system, there are several options to arrange the SLM pixels 143 , 161 . Maimone et al. 143 demonstrated a Maxwellian AR display with two modes to offer a large-DoF Maxwellian-view, or a holographic view (Fig. 9d ), which is often referred as computer-generated holography (CGH) 162 . To show an always-in-focus image with a large DoF, the image can be directly displayed on an amplitude SLM, or using amplitude encoding for a phase-only SLM 163 . Alternatively, if a 3D scene with correct depth cues is to be presented, then optimization algorithms for CGH can be used to generate a hologram for the SLM. The generated holographic image exhibits the natural focus-and-blur effect like a real 3D object (Fig. 9d ). To better understand this feature, we need to again exploit the concept of etendue. The laser light source can be considered to have a very small etendue due to its excellent collimation. Therefore, the system etendue is provided by the SLM. The micron-sized pixel-pitch of SLM offers a certain maximum diffraction angle, which, multiplied by the SLM size, equals system etendue. By varying the display content on SLM, the final exit pupil size can be changed accordingly. In the case of a large-DoF Maxwellian view, the exit pupil size is small, accompanied by a large FoV. For the holographic display mode, the reduced DoF requires a larger exit pupil with dimension close to the eye pupil. But the FoV is reduced accordingly due to etendue conservation. Another commonly concerned issue with CGH is the computation time. To achieve a real-time CGH rendering flow with an excellent image quality is quite a challenge. Fortunately, with recent advances in algorithm 164 and the introduction of convolutional neural network (CNN) 165 , 166 , this issue is gradually solved with an encouraging pace. Lately, Liang et al. 166 demonstrated a real-time CGH synthesis pipeline with a high image quality. The pipeline comprises an efficient CNN model to generate a complex hologram from a 3D scene and an improved encoding algorithm to convert the complex hologram to a phase-only one. An impressive frame rate of 60 Hz has been achieved on a desktop computing unit.

For LBS-based system, the additional modulation can be achieved by integrating a steering module, as demonstrated by Jang et al. 167 . The steering mirror can shift the focal point (viewpoint) within the eye pupil, therefore effectively expanding the system etendue. When the steering process is fast and the image content is updated simultaneously, correct 3D cues can be generated, as shown in Fig. 9e . However, there exists a tradeoff between the number of viewpoint and the final image frame rate, because the total frames are equally divided into each viewpoint. To boost the frame rate of MEMS-LBS systems by the number of views (e.g., 3 by 3) may be challenging.

Maxwellian-type systems offer several advantages. The system efficiency is usually very high because nearly all the light is delivered into viewer’s eye. The system FoV is determined by the f /# of combiner and a large FoV (~80° in horizontal) can be achieved 143 . The issue of VAC can be mitigated with an infinite-DoF image that deprives accommodation cue, or completely solved by generating a true-3D scene as discussed above. Despite these advantages, one major weakness of Maxwellian-type system is the tiny exit pupil, or eyebox. A small deviation of eye pupil location from the viewpoint results in the complete disappearance of the image. Therefore, to expand eyebox is considered as one of the most important challenges in Maxwellian-type systems.

Pupil duplication and steering

Methods to expand eyebox can be generally categorized into pupil duplication 168 , 169 , 170 , 171 , 172 and pupil steering 9 , 13 , 167 , 173 . Pupil duplication simply generates multiple viewpoints to cover a large area. In contrast, pupil steering dynamically shifts the viewpoint position, depending on the pupil location. Before reviewing detailed implementations of these two methods, it is worth discussing some of their general features. The multiple viewpoints in pupil duplication usually mean to equally divide the total light intensity. In each time frame, however, it is preferable that only one viewpoint enters the user’s eye pupil to avoid ghost image. This requirement, therefore, results in a reduced total light efficiency, while also conditioning the viewpoint separation to be larger than the pupil diameter. In addition, the separation should not be too large to avoid gap between viewpoints. Considering that human pupil diameter changes in response to environment illuminance, the design of viewpoint separation needs special attention. Pupil steering, on the other hand, only produces one viewpoint at each time frame. It is therefore more light-efficient and free from ghost images. But to determine the viewpoint position requires the information of eye pupil location, which demands a real-time eye-tracking module 9 . Another observation is that pupil steering can accommodate multiple viewpoints by its nature. Therefore, a pupil steering system can often be easily converted to a pupil duplication system by simultaneously generating available viewpoints.

To generate multiple viewpoints, one can focus on modulating the incident light or the combiner. Recall that viewpoint is the image of light source. To duplicate or shift light source can achieve pupil duplication or steering accordingly, as illustrated in Fig. 10a . Several schemes of light modulation are depicted in Fig. 10b–e . An array of light sources can be generated with multiple laser diodes (Fig. 10b ). To turn on all or one of the sources achieves pupil duplication or steering. A light source array can also be produced by projecting light on an array-type PPHOE 168 (Fig. 10c ). Apart from direct adjustment of light sources, modulating light on the path can also effectively steer/duplicate the light sources. Using a mechanical steering mirror, the beam can be deflected 167 (Fig. 10d ), which equals to shifting the light source position. Other devices like a grating or beam splitter can also serve as ray deflector/splitter 170 , 171 (Fig. 10e ).

a Schematic of duplicating (or shift) viewpoint by modulation of incident light. Light modulation by b multiple laser diodes, c HOE lens array, d steering mirror and e grating or beam splitters. f Pupil duplication with multiplexed PPHOE. g Pupil steering with LCHOE. Reproduced from c ref. 168 under the Creative Commons Attribution 4.0 License, e ref. 169 with permission from OSA Publishing, f ref. 171 with permission from OSA Publishing and g ref. 173 with permission from OSA Publishing

Nonetheless, one problem of the light source duplication/shifting methods for pupil duplication/steering is that the aberrations in peripheral viewpoints are often serious 168 , 173 . The HOE combiner is usually recorded at one incident angle. For other incident angles with large deviations, considerable aberrations will occur, especially in the scenario of off-axis configuration. To solve this problem, the modulation can be focused on the combiner instead. While the mechanical shifting of combiner 9 can achieve continuous pupil steering, its integration into AR display with a small factor remains a challenge. Alternatively, the versatile functions of HOE offer possible solutions for combiner modulation. Kim and Park 169 demonstrated a pupil duplication system with multiplexed PPHOE (Fig. 10f ). Wavefronts of several viewpoints can be recorded into one PPHOE sample. Three viewpoints with a separation of 3 mm were achieved. However, a slight degree of ghost image and gap can be observed in the viewpoint transition. For a PPHOE to achieve pupil steering, the multiplexed PPHOE needs to record different focal points with different incident angles. If each hologram has no angular crosstalk, then with an additional device to change the light incident angle, the viewpoint can be steered. Alternatively, Xiong et al. 173 demonstrated a pupil steering system with LCHOEs in a simpler configuration (Fig. 10g ). The polarization-sensitive nature of LCHOE enables the controlling of which LCHOE to function with a polarization converter (PC). When the PC is off, the incident RCP light is focused by the right-handed LCHOE. When the PC is turned on, the RCP light is firstly converted to LCP light and passes through the right-handed LCHOE. Then it is focused by the left-handed LCHOE into another viewpoint. To add more viewpoints requires stacking more pairs of PC and LCHOE, which can be achieved in a compact manner with thin glass substrates. In addition, to realize pupil duplication only requires the stacking of multiple low-efficiency LCHOEs. For both PPHOEs and LCHOEs, because the hologram for each viewpoint is recorded independently, the aberrations can be eliminated.

Regarding the system performance, in theory the FoV is not limited and can reach a large value, such as 80° in horizontal direction 143 . The definition of eyebox is different from traditional imaging systems. For a single viewpoint, it has the same size as the eye pupil diameter. But due to the viewpoint steering/duplication capability, the total system eyebox can be expanded accordingly. The combiner efficiency for pupil steering systems can reach 47,000 nit/lm for a FoV of 80° by 80° and pupil diameter of 4 mm (Eq. S2 ). At such a high brightness level, eye safety could be a concern 174 . For a pupil duplication system, the combiner efficiency is decreased by the number of viewpoints. With a 4-by-4 viewpoint array, it can still reach 3000 nit/lm. Despite the potential gain of pupil duplication/steering, when considering the rotation of eyeball, the situation becomes much more complicated 175 . A perfect pupil steering system requires a 5D steering, which proposes a challenge for practical implementation.

Pin-light systems

Recently, another type of display in close relation with Maxwellian view called pin-light display 148 , 176 has been proposed. The general working principle of pin-light display is illustrated in Fig. 11a . Each pin-light source is a Maxwellian view with a large DoF. When the eye pupil is no longer placed near the source point as in Maxwellian view, each image source can only form an elemental view with a small FoV on retina. However, if the image source array is arranged in a proper form, the elemental views can be integrated together to form a large FoV. According to the specific optical architectures, pin-light display can take different forms of implementation. In the initial feasibility demonstration, Maimone et al. 176 used a side-lit waveguide plate as the point light source (Fig. 11b ). The light inside the waveguide plate is extracted by the etched divots, forming a pin-light source array. A transmissive SLM (LCD) is placed behind the waveguide plate to modulate the light intensity and form the image. The display has an impressive FoV of 110° thanks to the large scattering angle range. However, the direct placement of LCD before the eye brings issues of insufficient resolution density and diffraction of background light.

a Schematic drawing of the working principle of pin-light display. b Pin-light display utilizing a pin-light source and a transmissive SLM. c An example of pin-mirror display with a birdbath optics. d SWD system with LBS image source and off-axis lens array. Reprinted from b ref. 176 under the Creative Commons Attribution 4.0 License and d ref. 180 with permission from OSA Publishing

To avoid these issues, architectures using pin-mirrors 177 , 178 , 179 are proposed. In these systems, the final combiner is an array of tiny mirrors 178 , 179 or gratings 177 , in contrast to their counterparts using large-area combiners. An exemplary system with birdbath design is depicted in Fig. 11c . In this case, the pin-mirrors replace the original beam-splitter in the birdbath and can thus shrink the system volume, while at the same time providing large DoF pin-light images. Nonetheless, such a system may still face the etendue conservation issue. Meanwhile, the size of pin-mirror cannot be too small in order to prevent degradation of resolution density due to diffraction. Therefore, its influence on the see-through background should also be considered in the system design.

To overcome the etendue conservation and improve see-through quality, Xiong et al. 180 proposed another type of pin-light system exploiting the etendue expansion property of waveguide, which is also referred as scanning waveguide display (SWD). As illustrated in Fig. 11d , the system uses an LBS as the image source. The collimated scanned laser rays are trapped in the waveguide and encounter an array of off-axis lenses. Upon each encounter, the lens out-couples the laser rays and forms a pin-light source. SWD has the merits of good see-through quality and large etendue. A large FoV of 100° was demonstrated with the help of an ultra-low f /# lens array based on LCHOE. However, some issues like insufficient image resolution density and image non-uniformity remain to be overcome. To further improve the system may require optimization of Gaussian beam profile and additional EPE module 180 .

Overall, pin-light systems inherit the large DoF from Maxwellian view. With adequate number of pin-light sources, the FoV and eyebox can be expanded accordingly. Nonetheless, despite different forms of implementation, a common issue of pin-light system is the image uniformity. The overlapped region of elemental views has a higher light intensity than the non-overlapped region, which becomes even more complicated considering the dynamic change of pupil size. In theory, the displayed image can be pre-processed to compensate for the optical non-uniformity. But that would require knowledge of precise pupil location (and possibly size) and therefore an accurate eye-tracking module 176 . Regarding the system performance, pin-mirror systems modified from other free-space systems generally shares similar FoV and eyebox with original systems. The combiner efficiency may be lower due to the small size of pin-mirrors. SWD, on the other hand, shares the large FoV and DoF with Maxwellian view, and large eyebox with waveguide combiners. The combiner efficiency may also be lower due to the EPE process.

Waveguide combiner

Besides free-space combiners, another common architecture in AR displays is waveguide combiner. The term ‘waveguide’ indicates the light is trapped in a substrate by the TIR process. One distinctive feature of a waveguide combiner is the EPE process that effectively enlarges the system etendue. In the EPE process, a portion of the trapped light is repeatedly coupled out of the waveguide in each TIR. The effective eyebox is therefore enlarged. According to the features of couplers, we divide the waveguide combiners into two types: diffractive and achromatic, as described in the followings.

Diffractive waveguides

As the name implies, diffractive-type waveguides use diffractive elements as couplers. The in-coupler is usually a diffractive grating and the out-coupler in most cases is also a grating with the same period as the in-coupler, but it can also be an off-axis lens with a small curvature to generate image with finite depth. Three major diffractive couplers have been developed: SRGs, photopolymer gratings (PPGs), and liquid crystal gratings (grating-type LCHOE; also known as polarization volume gratings (PVGs)). Some general protocols for coupler design are that the in-coupler should have a relatively high efficiency and the out-coupler should have a uniform light output. A uniform light output usually requires a low-efficiency coupler, with extra degrees of freedom for local modulation of coupling efficiency. Both in-coupler and out-coupler should have an adequate angular bandwidth to accommodate a reasonable FoV. In addition, the out-coupler should also be optimized to avoid undesired diffractions, including the outward diffraction of TIR light and diffraction of environment light into user’s eyes, which are referred as light leakage and rainbow. Suppression of these unwanted diffractions should also be considered in the optimization process of waveguide design, along with performance parameters like efficiency and uniformity.

The basic working principles of diffractive waveguide-based AR systems are illustrated in Fig. 12 . For the SRG-based waveguides 6 , 8 (Fig. 12a ), the in-coupler can be a transmissive-type or a reflective-type 181 , 182 . The grating geometry can be optimized for coupling efficiency with a large degree of freedom 183 . For the out-coupler, a reflective SRG with a large slant angle to suppress the transmission orders is preferred 184 . In addition, a uniform light output usually requires a gradient efficiency distribution in order to compensate for the decreased light intensity in the out-coupling process. This can be achieved by varying the local grating configurations like height and duty cycle 6 . For the PPG-based waveguides 185 (Fig. 12b ), the small angular bandwidth of a high-efficiency transmissive PPG prohibits its use as in-coupler. Therefore, both in-coupler and out-coupler are usually reflective types. The gradient efficiency can be achieved by space-variant exposure to control the local index modulation 186 or local Bragg slant angle variation through freeform exposure 19 . Due to the relatively small angular bandwidth of PPG, to achieve a decent FoV usually requires stacking two 187 or three 188 PPGs together for a single color. The PVG-based waveguides 189 (Fig. 12c ) also prefer reflective PVGs as in-couplers because the transmissive PVGs are much more difficult to fabricate due to the LC alignment issue. In addition, the angular bandwidth of transmissive PVGs in Bragg regime is also not large enough to support a decent FoV 29 . For the out-coupler, the angular bandwidth of a single reflective PVG can usually support a reasonable FoV. To obtain a uniform light output, a polarization management layer 190 consisting of a LC layer with spatially variant orientations can be utilized. It offers an additional degree of freedom to control the polarization state of the TIR light. The diffraction efficiency can therefore be locally controlled due to the strong polarization sensitivity of PVG.

Schematics of waveguide combiners based on a SRGs, b PPGs and c PVGs. Reprinted from a ref. 85 with permission from OSA Publishing, b ref. 185 with permission from John Wiley and Sons and c ref. 189 with permission from OSA Publishing

The above discussion describes the basic working principle of 1D EPE. Nonetheless, for the 1D EPE to produce a large eyebox, the exit pupil in the unexpanded direction of the original image should be large. This proposes design challenges in light engines. Therefore, a 2D EPE is favored for practical applications. To extend EPE in two dimensions, two consecutive 1D EPEs can be used 191 , as depicted in Fig. 13a . The first 1D EPE occurs in the turning grating, where the light is duplicated in y direction and then turned into x direction. Then the light rays encounter the out-coupler and are expanded in x direction. To better understand the 2D EPE process, the k -vector diagram (Fig. 13b ) can be used. For the light propagating in air with wavenumber k 0 , its possible k -values in x and y directions ( k x and k y ) fall within the circle with radius k 0 . When the light is trapped into TIR, k x and k y are outside the circle with radius k 0 and inside the circle with radius nk 0 , where n is the refractive index of the substrate. k x and k y stay unchanged in the TIR process and are only changed in each diffraction process. The central red box in Fig. 13b indicates the possible k values within the system FoV. After the in-coupler, the k values are added by the grating k -vector, shifting the k values into TIR region. The turning grating then applies another k -vector and shifts the k values to near x -axis. Finally, the k values are shifted by the out-coupler and return to the free propagation region in air. One observation is that the size of red box is mostly limited by the width of TIR band. To accommodate a larger FoV, the outer boundary of TIR band needs to be expanded, which amounts to increasing waveguide refractive index. Another important fact is that when k x and k y are near the outer boundary, the uniformity of output light becomes worse. This is because the light propagation angle is near 90° in the waveguide. The spatial distance between two consecutive TIRs becomes so large that the out-coupled beams are spatially separated to an unacceptable degree. The range of possible k values for practical applications is therefore further shrunk due to this fact.

a Schematic of 2D EPE based on two consecutive 1D EPEs. Gray/black arrows indicate light in air/TIR. Black dots denote TIRs. b k-diagram of the two-1D-EPE scheme. c Schematic of 2D EPE with a 2D hexagonal grating d k-diagram of the 2D-grating scheme

Aside from two consecutive 1D EPEs, the 2D EPE can also be directly implemented with a 2D grating 192 . An example using a hexagonal grating is depicted in Fig. 13c . The hexagonal grating can provide k -vectors in six directions. In the k -diagram (Fig. 13d ), after the in-coupling, the k values are distributed into six regions due to multiple diffractions. The out-coupling occurs simultaneously with pupil expansion. Besides a concise out-coupler configuration, the 2D EPE scheme offers more degrees of design freedom than two 1D EPEs because the local grating parameters can be adjusted in a 2D manner. The higher design freedom has the potential to reach a better output light uniformity, but at the cost of a higher computation demand for optimization. Furthermore, the unslanted grating geometry usually leads to a large light leakage and possibly low efficiency. Adding slant to the geometry helps alleviate the issue, but the associated fabrication may be more challenging.

Finally, we discuss the generation of full-color images. One important issue to clarify is that although diffractive gratings are used here, the final image generally has no color dispersion even if we use a broadband light source like LED. This can be easily understood in the 1D EPE scheme. The in-coupler and out-coupler have opposite k -vectors, which cancels the color dispersion for each other. In the 2D EPE schemes, the k -vectors always form a closed loop from in-coupled light to out-coupled light, thus, the color dispersion also vanishes likewise. The issue of using a single waveguide for full-color images actually exists in the consideration of FoV and light uniformity. The breakup of propagation angles for different colors results in varied out-coupling situations for each color. To be more specific, if the red and the blue channels use the same in-coupler, the propagating angle for the red light is larger than that of the blue light. The red light in peripheral FoV is therefore easier to face the mentioned large-angle non-uniformity issue. To acquire a decent FoV and light uniformity, usually two or three layers of waveguides with different grating pitches are adopted.

Regarding the system performance, the eyebox is generally large enough (~10 mm) to accommodate different user’s IPD and alignment shift during operation. A parameter of significant concern for a waveguide combiner is its FoV. From the k -vector analysis, we can conclude the theoretical upper limit is determined by the waveguide refractive index. But the light/color uniformity also influences the effective FoV, over which the degradation of image quality becomes unacceptable. Current diffractive waveguide combiners generally achieve a FoV of about 50°. To further increase FoV, a straightforward method is to use a higher refractive index waveguide. Another is to tile FoV through direct stacking of multiple waveguides or using polarization-sensitive couplers 79 , 193 . As to the optical efficiency, a typical value for the diffractive waveguide combiner is around 50–200 nit/lm 6 , 189 . In addition, waveguide combiners adopting grating out-couplers generate an image with fixed depth at infinity. This leads to the VAC issue. To tackle VAC in waveguide architectures, the most practical way is to generate multiple depths and use the varifocal or multifocal driving scheme, similar to those mentioned in the VR systems. But to add more depths usually means to stack multiple layers of waveguides together 194 . Considering the additional waveguide layers for RGB colors, the final waveguide thickness would undoubtedly increase.

Other parameters special to waveguide includes light leakage, see-through ghost, and rainbow. Light leakage refers to out-coupled light that goes outwards to the environment, as depicted in Fig. 14a . Aside from decreased efficiency, the leakage also brings drawback of unnatural “bright-eye” appearance of the user and privacy issue. Optimization of the grating structure like geometry of SRG may reduce the leakage. See-through ghost is formed by consecutive in-coupling and out-couplings caused by the out-coupler grating, as sketched in Fig. 14b , After the process, a real object with finite depth may produce a ghost image with shift in both FoV and depth. Generally, an out-coupler with higher efficiency suffers more see-through ghost. Rainbow is caused by the diffraction of environment light into user’s eye, as sketched in Fig. 14c . The color dispersion in this case will occur because there is no cancellation of k -vector. Using the k -diagram, we can obtain a deeper insight into the formation of rainbow. Here, we take the EPE structure in Fig. 13a as an example. As depicted in Fig. 14d , after diffractions by the turning grating and the out-coupler grating, the k values are distributed in two circles that shift from the origin by the grating k -vectors. Some diffracted light can enter the see-through FoV and form rainbow. To reduce rainbow, a straightforward way is to use a higher index substrate. With a higher refractive index, the outer boundary of k diagram is expanded, which can accommodate larger grating k -vectors. The enlarged k -vectors would therefore “push” these two circles outwards, leading to a decreased overlapping region with the see-through FoV. Alternatively, an optimized grating structure would also help reduce the rainbow effect by suppressing the unwanted diffraction.

Sketches of formations of a light leakage, b see-through ghost and c rainbow. d Analysis of rainbow formation with k-diagram

Achromatic waveguide

Achromatic waveguide combiners use achromatic elements as couplers. It has the advantage of realizing full-color image with a single waveguide. A typical example of achromatic element is a mirror. The waveguide with partial mirrors as out-coupler is often referred as geometric waveguide 6 , 195 , as depicted in Fig. 15a . The in-coupler in this case is usually a prism to avoid unnecessary color dispersion if using diffractive elements otherwise. The mirrors couple out TIR light consecutively to produce a large eyebox, similarly in a diffractive waveguide. Thanks to the excellent optical property of mirrors, the geometric waveguide usually exhibits a superior image regarding MTF and color uniformity to its diffractive counterparts. Still, the spatially discontinuous configuration of mirrors also results in gaps in eyebox, which may be alleviated by using a dual-layer structure 196 . Wang et al. designed a geometric waveguide display with five partial mirrors (Fig. 15b ). It exhibits a remarkable FoV of 50° by 30° (Fig. 15c ) and an exit pupil of 4 mm with a 1D EPE. To achieve 2D EPE, similar architectures in Fig. 13a can be used by integrating a turning mirror array as the first 1D EPE module 197 . Unfortunately, the k -vector diagrams in Fig. 13b, d cannot be used here because the k values in x-y plane no longer conserve in the in-coupling and out-coupling processes. But some general conclusions remain valid, like a higher refractive index leading to a larger FoV and gradient out-coupling efficiency improving light uniformity.

a Schematic of the system configuration. b Geometric waveguide with five partial mirrors. c Image photos demonstrating system FoV. Adapted from b , c ref. 195 with permission from OSA Publishing

The fabrication process of geometric waveguide involves coating mirrors on cut-apart pieces and integrating them back together, which may result in a high cost, especially for the 2D EPE architecture. Another way to implement an achromatic coupler is to use multiplexed PPHOE 198 , 199 to mimic the behavior of a tilted mirror (Fig. 16a ). To understand the working principle, we can use the diagram in Fig. 16b . The law of reflection states the angle of reflection equals to the angle of incidence. If we translate this behavior to k -vector language, it means the mirror can apply any length of k -vector along its surface normal direction. The k -vector length of the reflected light is always equal to that of the incident light. This puts a condition that the k -vector triangle is isosceles. With a simple geometric deduction, it can be easily observed this leads to the law of reflection. The behavior of a general grating, however, is very different. For simplicity we only consider the main diffraction order. The grating can only apply a k -vector with fixed k x due to the basic diffraction law. For the light with a different incident angle, it needs to apply different k z to produce a diffracted light with equal k -vector length as the incident light. For a grating with a broad angular bandwidth like SRG, the range of k z is wide, forming a lengthy vertical line in Fig. 16b . For a PPG with a narrow angular bandwidth, the line is short and resembles a dot. If multiple of these tiny dots are distributed along the oblique line corresponding to a mirror, then the final multiplexed PPGs can imitate the behavior of a tilted mirror. Such a PPHOE is sometimes referred as a skew-mirror 198 . In theory, to better imitate the mirror, a lot of multiplexed PPGs is preferred, while each PPG has a small index modulation δn . But this proposes a bigger challenge in device fabrication. Recently, Utsugi et al. demonstrated an impressive skew-mirror waveguide based on 54 multiplexed PPGs (Fig. 16c, d ). The display exhibits an effective FoV of 35° by 36°. In the peripheral FoV, there still exists some non-uniformity (Fig. 16e ) due to the out-coupling gap, which is an inherent feature of the flat-type out-couplers.

a System configuration. b Diagram demonstrating how multiplexed PPGs resemble the behavior of a mirror. Photos showing c the system and d image. e Picture demonstrating effective system FoV. Adapted from c – e ref. 199 with permission from ITE

Finally, it is worth mentioning that metasurfaces are also promising to deliver achromatic gratings 200 , 201 for waveguide couplers ascribed to their versatile wavefront shaping capability. The mechanism of the achromatic gratings is similar to that of the achromatic lenses as previously discussed. However, the current development of achromatic metagratings is still in its infancy. Much effort is needed to improve the optical efficiency for in-coupling, control the higher diffraction orders for eliminating ghost images, and enable a large size design for EPE.

Generally, achromatic waveguide combiners exhibit a comparable FoV and eyebox with diffractive combiners, but with a higher efficiency. For a partial-mirror combiner, its combiner efficiency is around 650 nit/lm 197 (2D EPE). For a skew-mirror combiner, although the efficiency of multiplexed PPHOE is relatively low (~1.5%) 199 , the final combiner efficiency of the 1D EPE system is still high (>3000 nit/lm) due to multiple out-couplings.

Table 2 summarizes the performance of different AR combiners. When combing the luminous efficacy in Table 1 and the combiner efficiency in Table 2 , we can have a comprehensive estimate of the total luminance efficiency (nit/W) for different types of systems. Generally, Maxwellian-type combiners with pupil steering have the highest luminance efficiency when partnered with laser-based light engines like laser-backlit LCoS/DMD or MEM-LBS. Geometric optical combiners have well-balanced image performances, but to further shrink the system size remains a challenge. Diffractive waveguides have a relatively low combiner efficiency, which can be remedied by an efficient light engine like MEMS-LBS. Further development of coupler and EPE scheme would also improve the system efficiency and FoV. Achromatic waveguides have a decent combiner efficiency. The single-layer design also enables a smaller form factor. With advances in fabrication process, it may become a strong contender to presently widely used diffractive waveguides.

Conclusions and perspectives

VR and AR are endowed with a high expectation to revolutionize the way we interact with digital world. Accompanied with the expectation are the engineering challenges to squeeze a high-performance display system into a tightly packed module for daily wearing. Although the etendue conservation constitutes a great obstacle on the path, remarkable progresses with innovative optics and photonics continue to take place. Ultra-thin optical elements like PPHOEs and LCHOEs provide alternative solutions to traditional optics. Their unique features of multiplexing capability and polarization dependency further expand the possibility of novel wavefront modulations. At the same time, nanoscale-engineered metasurfaces/SRGs provide large design freedoms to achieve novel functions beyond conventional geometric optical devices. Newly emerged micro-LEDs open an opportunity for compact microdisplays with high peak brightness and good stability. Further advances on device engineering and manufacturing process are expected to boost the performance of metasurfaces/SRGs and micro-LEDs for AR and VR applications.

Data availability

All data needed to evaluate the conclusions in the paper are present in the paper. Additional data related to this paper may be requested from the authors.

Cakmakci, O. & Rolland, J. Head-worn displays: a review. J. Disp. Technol. 2 , 199–216 (2006).

Article   ADS   Google Scholar  

Zhan, T. et al. Augmented reality and virtual reality displays: perspectives and challenges. iScience 23 , 101397 (2020).

Rendon, A. A. et al. The effect of virtual reality gaming on dynamic balance in older adults. Age Ageing 41 , 549–552 (2012).

Article   Google Scholar  

Choi, S., Jung, K. & Noh, S. D. Virtual reality applications in manufacturing industries: past research, present findings, and future directions. Concurrent Eng. 23 , 40–63 (2015).

Li, X. et al. A critical review of virtual and augmented reality (VR/AR) applications in construction safety. Autom. Constr. 86 , 150–162 (2018).

Kress, B. C. Optical Architectures for Augmented-, Virtual-, and Mixed-Reality Headsets (Bellingham: SPIE Press, 2020).

Cholewiak, S. A. et al. A perceptual eyebox for near-eye displays. Opt. Express 28 , 38008–38028 (2020).

Lee, Y. H., Zhan, T. & Wu, S. T. Prospects and challenges in augmented reality displays. Virtual Real. Intell. Hardw. 1 , 10–20 (2019).

Kim, J. et al. Foveated AR: dynamically-foveated augmented reality display. ACM Trans. Graph. 38 , 99 (2019).

Tan, G. J. et al. Foveated imaging for near-eye displays. Opt. Express 26 , 25076–25085 (2018).

Lee, S. et al. Foveated near-eye display for mixed reality using liquid crystal photonics. Sci. Rep. 10 , 16127 (2020).

Yoo, C. et al. Foveated display system based on a doublet geometric phase lens. Opt. Express 28 , 23690–23702 (2020).

Akşit, K. et al. Manufacturing application-driven foveated near-eye displays. IEEE Trans. Vis. Computer Graph. 25 , 1928–1939 (2019).

Zhu, R. D. et al. High-ambient-contrast augmented reality with a tunable transmittance liquid crystal film and a functional reflective polarizer. J. Soc. Inf. Disp. 24 , 229–233 (2016).

Lincoln, P. et al. Scene-adaptive high dynamic range display for low latency augmented reality. In Proc. 21st ACM SIGGRAPH Symposium on Interactive 3D Graphics and Games . (ACM, San Francisco, CA, 2017).

Duerr, F. & Thienpont, H. Freeform imaging systems: fermat’s principle unlocks “first time right” design. Light.: Sci. Appl. 10 , 95 (2021).

Bauer, A., Schiesser, E. M. & Rolland, J. P. Starting geometry creation and design method for freeform optics. Nat. Commun. 9 , 1756 (2018).

Rolland, J. P. et al. Freeform optics for imaging. Optica 8 , 161–176 (2021).

Jang, C. et al. Design and fabrication of freeform holographic optical elements. ACM Trans. Graph. 39 , 184 (2020).

Gabor, D. A new microscopic principle. Nature 161 , 777–778 (1948).

Kostuk, R. K. Holography: Principles and Applications (Boca Raton: CRC Press, 2019).

Lawrence, J. R., O'Neill, F. T. & Sheridan, J. T. Photopolymer holographic recording material. Optik 112 , 449–463 (2001).

Guo, J. X., Gleeson, M. R. & Sheridan, J. T. A review of the optimisation of photopolymer materials for holographic data storage. Phys. Res. Int. 2012 , 803439 (2012).

Jang, C. et al. Recent progress in see-through three-dimensional displays using holographic optical elements [Invited]. Appl. Opt. 55 , A71–A85 (2016).

Xiong, J. H. et al. Holographic optical elements for augmented reality: principles, present status, and future perspectives. Adv. Photonics Res. 2 , 2000049 (2021).

Tabiryan, N. V. et al. Advances in transparent planar optics: enabling large aperture, ultrathin lenses. Adv. Optical Mater. 9 , 2001692 (2021).

Zanutta, A. et al. Photopolymeric films with highly tunable refractive index modulation for high precision diffractive optics. Optical Mater. Express 6 , 252–263 (2016).

Moharam, M. G. & Gaylord, T. K. Rigorous coupled-wave analysis of planar-grating diffraction. J. Optical Soc. Am. 71 , 811–818 (1981).

Xiong, J. H. & Wu, S. T. Rigorous coupled-wave analysis of liquid crystal polarization gratings. Opt. Express 28 , 35960–35971 (2020).

Xie, S., Natansohn, A. & Rochon, P. Recent developments in aromatic azo polymers research. Chem. Mater. 5 , 403–411 (1993).

Shishido, A. Rewritable holograms based on azobenzene-containing liquid-crystalline polymers. Polym. J. 42 , 525–533 (2010).

Bunning, T. J. et al. Holographic polymer-dispersed liquid crystals (H-PDLCs). Annu. Rev. Mater. Sci. 30 , 83–115 (2000).

Liu, Y. J. & Sun, X. W. Holographic polymer-dispersed liquid crystals: materials, formation, and applications. Adv. Optoelectron. 2008 , 684349 (2008).

Xiong, J. H. & Wu, S. T. Planar liquid crystal polarization optics for augmented reality and virtual reality: from fundamentals to applications. eLight 1 , 3 (2021).

Yaroshchuk, O. & Reznikov, Y. Photoalignment of liquid crystals: basics and current trends. J. Mater. Chem. 22 , 286–300 (2012).

Sarkissian, H. et al. Periodically aligned liquid crystal: potential application for projection displays. Mol. Cryst. Liq. Cryst. 451 , 1–19 (2006).

Komanduri, R. K. & Escuti, M. J. Elastic continuum analysis of the liquid crystal polarization grating. Phys. Rev. E 76 , 021701 (2007).

Kobashi, J., Yoshida, H. & Ozaki, M. Planar optics with patterned chiral liquid crystals. Nat. Photonics 10 , 389–392 (2016).

Lee, Y. H., Yin, K. & Wu, S. T. Reflective polarization volume gratings for high efficiency waveguide-coupling augmented reality displays. Opt. Express 25 , 27008–27014 (2017).

Lee, Y. H., He, Z. Q. & Wu, S. T. Optical properties of reflective liquid crystal polarization volume gratings. J. Optical Soc. Am. B 36 , D9–D12 (2019).

Xiong, J. H., Chen, R. & Wu, S. T. Device simulation of liquid crystal polarization gratings. Opt. Express 27 , 18102–18112 (2019).

Czapla, A. et al. Long-period fiber gratings with low-birefringence liquid crystal. Mol. Cryst. Liq. Cryst. 502 , 65–76 (2009).

Dąbrowski, R., Kula, P. & Herman, J. High birefringence liquid crystals. Crystals 3 , 443–482 (2013).

Mack, C. Fundamental Principles of Optical Lithography: The Science of Microfabrication (Chichester: John Wiley & Sons, 2007).

Genevet, P. et al. Recent advances in planar optics: from plasmonic to dielectric metasurfaces. Optica 4 , 139–152 (2017).

Guo, L. J. Nanoimprint lithography: methods and material requirements. Adv. Mater. 19 , 495–513 (2007).

Park, J. et al. Electrically driven mid-submicrometre pixelation of InGaN micro-light-emitting diode displays for augmented-reality glasses. Nat. Photonics 15 , 449–455 (2021).

Khorasaninejad, M. et al. Metalenses at visible wavelengths: diffraction-limited focusing and subwavelength resolution imaging. Science 352 , 1190–1194 (2016).

Li, S. Q. et al. Phase-only transmissive spatial light modulator based on tunable dielectric metasurface. Science 364 , 1087–1090 (2019).

Liang, K. L. et al. Advances in color-converted micro-LED arrays. Jpn. J. Appl. Phys. 60 , SA0802 (2020).

Jin, S. X. et al. GaN microdisk light emitting diodes. Appl. Phys. Lett. 76 , 631–633 (2000).

Day, J. et al. Full-scale self-emissive blue and green microdisplays based on GaN micro-LED arrays. In Proc. SPIE 8268, Quantum Sensing and Nanophotonic Devices IX (SPIE, San Francisco, California, United States, 2012).

Huang, Y. G. et al. Mini-LED, micro-LED and OLED displays: present status and future perspectives. Light.: Sci. Appl. 9 , 105 (2020).

Parbrook, P. J. et al. Micro-light emitting diode: from chips to applications. Laser Photonics Rev. 15 , 2000133 (2021).

Day, J. et al. III-Nitride full-scale high-resolution microdisplays. Appl. Phys. Lett. 99 , 031116 (2011).

Liu, Z. J. et al. 360 PPI flip-chip mounted active matrix addressable light emitting diode on silicon (LEDoS) micro-displays. J. Disp. Technol. 9 , 678–682 (2013).

Zhang, L. et al. Wafer-scale monolithic hybrid integration of Si-based IC and III–V epi-layers—A mass manufacturable approach for active matrix micro-LED micro-displays. J. Soc. Inf. Disp. 26 , 137–145 (2018).

Tian, P. F. et al. Size-dependent efficiency and efficiency droop of blue InGaN micro-light emitting diodes. Appl. Phys. Lett. 101 , 231110 (2012).

Olivier, F. et al. Shockley-Read-Hall and Auger non-radiative recombination in GaN based LEDs: a size effect study. Appl. Phys. Lett. 111 , 022104 (2017).

Konoplev, S. S., Bulashevich, K. A. & Karpov, S. Y. From large-size to micro-LEDs: scaling trends revealed by modeling. Phys. Status Solidi (A) 215 , 1700508 (2018).

Li, L. Z. et al. Transfer-printed, tandem microscale light-emitting diodes for full-color displays. Proc. Natl Acad. Sci. USA 118 , e2023436118 (2021).

Oh, J. T. et al. Light output performance of red AlGaInP-based light emitting diodes with different chip geometries and structures. Opt. Express 26 , 11194–11200 (2018).

Shen, Y. C. et al. Auger recombination in InGaN measured by photoluminescence. Appl. Phys. Lett. 91 , 141101 (2007).

Wong, M. S. et al. High efficiency of III-nitride micro-light-emitting diodes by sidewall passivation using atomic layer deposition. Opt. Express 26 , 21324–21331 (2018).

Han, S. C. et al. AlGaInP-based Micro-LED array with enhanced optoelectrical properties. Optical Mater. 114 , 110860 (2021).

Wong, M. S. et al. Size-independent peak efficiency of III-nitride micro-light-emitting-diodes using chemical treatment and sidewall passivation. Appl. Phys. Express 12 , 097004 (2019).

Ley, R. T. et al. Revealing the importance of light extraction efficiency in InGaN/GaN microLEDs via chemical treatment and dielectric passivation. Appl. Phys. Lett. 116 , 251104 (2020).

Moon, S. W. et al. Recent progress on ultrathin metalenses for flat optics. iScience 23 , 101877 (2020).

Arbabi, A. et al. Efficient dielectric metasurface collimating lenses for mid-infrared quantum cascade lasers. Opt. Express 23 , 33310–33317 (2015).

Yu, N. F. et al. Light propagation with phase discontinuities: generalized laws of reflection and refraction. Science 334 , 333–337 (2011).

Liang, H. W. et al. High performance metalenses: numerical aperture, aberrations, chromaticity, and trade-offs. Optica 6 , 1461–1470 (2019).

Park, J. S. et al. All-glass, large metalens at visible wavelength using deep-ultraviolet projection lithography. Nano Lett. 19 , 8673–8682 (2019).

Yoon, G. et al. Single-step manufacturing of hierarchical dielectric metalens in the visible. Nat. Commun. 11 , 2268 (2020).

Lee, G. Y. et al. Metasurface eyepiece for augmented reality. Nat. Commun. 9 , 4562 (2018).

Chen, W. T. et al. A broadband achromatic metalens for focusing and imaging in the visible. Nat. Nanotechnol. 13 , 220–226 (2018).

Wang, S. M. et al. A broadband achromatic metalens in the visible. Nat. Nanotechnol. 13 , 227–232 (2018).

Lan, S. F. et al. Metasurfaces for near-eye augmented reality. ACS Photonics 6 , 864–870 (2019).

Fan, Z. B. et al. A broadband achromatic metalens array for integral imaging in the visible. Light.: Sci. Appl. 8 , 67 (2019).

Shi, Z. J., Chen, W. T. & Capasso, F. Wide field-of-view waveguide displays enabled by polarization-dependent metagratings. In Proc. SPIE 10676, Digital Optics for Immersive Displays (SPIE, Strasbourg, France, 2018).

Hong, C. C., Colburn, S. & Majumdar, A. Flat metaform near-eye visor. Appl. Opt. 56 , 8822–8827 (2017).

Bayati, E. et al. Design of achromatic augmented reality visors based on composite metasurfaces. Appl. Opt. 60 , 844–850 (2021).

Nikolov, D. K. et al. Metaform optics: bridging nanophotonics and freeform optics. Sci. Adv. 7 , eabe5112 (2021).

Tamir, T. & Peng, S. T. Analysis and design of grating couplers. Appl. Phys. 14 , 235–254 (1977).

Miller, J. M. et al. Design and fabrication of binary slanted surface-relief gratings for a planar optical interconnection. Appl. Opt. 36 , 5717–5727 (1997).

Levola, T. & Laakkonen, P. Replicated slanted gratings with a high refractive index material for in and outcoupling of light. Opt. Express 15 , 2067–2074 (2007).

Shrestha, S. et al. Broadband achromatic dielectric metalenses. Light.: Sci. Appl. 7 , 85 (2018).

Li, Z. Y. et al. Meta-optics achieves RGB-achromatic focusing for virtual reality. Sci. Adv. 7 , eabe4458 (2021).

Ratcliff, J. et al. ThinVR: heterogeneous microlens arrays for compact, 180 degree FOV VR near-eye displays. IEEE Trans. Vis. Computer Graph. 26 , 1981–1990 (2020).

Wong, T. L. et al. Folded optics with birefringent reflective polarizers. In Proc. SPIE 10335, Digital Optical Technologies 2017 (SPIE, Munich, Germany, 2017).

Li, Y. N. Q. et al. Broadband cholesteric liquid crystal lens for chromatic aberration correction in catadioptric virtual reality optics. Opt. Express 29 , 6011–6020 (2021).

Bang, K. et al. Lenslet VR: thin, flat and wide-FOV virtual reality display using fresnel lens and lenslet array. IEEE Trans. Vis. Computer Graph. 27 , 2545–2554 (2021).

Maimone, A. & Wang, J. R. Holographic optics for thin and lightweight virtual reality. ACM Trans. Graph. 39 , 67 (2020).

Kramida, G. Resolving the vergence-accommodation conflict in head-mounted displays. IEEE Trans. Vis. Computer Graph. 22 , 1912–1931 (2016).

Zhan, T. et al. Multifocal displays: review and prospect. PhotoniX 1 , 10 (2020).

Shimobaba, T., Kakue, T. & Ito, T. Review of fast algorithms and hardware implementations on computer holography. IEEE Trans. Ind. Inform. 12 , 1611–1622 (2016).

Xiao, X. et al. Advances in three-dimensional integral imaging: sensing, display, and applications [Invited]. Appl. Opt. 52 , 546–560 (2013).

Kuiper, S. & Hendriks, B. H. W. Variable-focus liquid lens for miniature cameras. Appl. Phys. Lett. 85 , 1128–1130 (2004).

Liu, S. & Hua, H. Time-multiplexed dual-focal plane head-mounted display with a liquid lens. Opt. Lett. 34 , 1642–1644 (2009).

Wilson, A. & Hua, H. Design and demonstration of a vari-focal optical see-through head-mounted display using freeform Alvarez lenses. Opt. Express 27 , 15627–15637 (2019).

Zhan, T. et al. Pancharatnam-Berry optical elements for head-up and near-eye displays [Invited]. J. Optical Soc. Am. B 36 , D52–D65 (2019).

Oh, C. & Escuti, M. J. Achromatic diffraction from polarization gratings with high efficiency. Opt. Lett. 33 , 2287–2289 (2008).

Zou, J. Y. et al. Broadband wide-view Pancharatnam-Berry phase deflector. Opt. Express 28 , 4921–4927 (2020).

Zhan, T., Lee, Y. H. & Wu, S. T. High-resolution additive light field near-eye display by switchable Pancharatnam–Berry phase lenses. Opt. Express 26 , 4863–4872 (2018).

Tan, G. J. et al. Polarization-multiplexed multiplane display. Opt. Lett. 43 , 5651–5654 (2018).

Lanman, D. R. Display systems research at facebook reality labs (conference presentation). In Proc. SPIE 11310, Optical Architectures for Displays and Sensing in Augmented, Virtual, and Mixed Reality (AR, VR, MR) (SPIE, San Francisco, California, United States, 2020).

Liu, Z. J. et al. A novel BLU-free full-color LED projector using LED on silicon micro-displays. IEEE Photonics Technol. Lett. 25 , 2267–2270 (2013).

Han, H. V. et al. Resonant-enhanced full-color emission of quantum-dot-based micro LED display technology. Opt. Express 23 , 32504–32515 (2015).

Lin, H. Y. et al. Optical cross-talk reduction in a quantum-dot-based full-color micro-light-emitting-diode display by a lithographic-fabricated photoresist mold. Photonics Res. 5 , 411–416 (2017).

Liu, Z. J. et al. Micro-light-emitting diodes with quantum dots in display technology. Light.: Sci. Appl. 9 , 83 (2020).

Kim, H. M. et al. Ten micrometer pixel, quantum dots color conversion layer for high resolution and full color active matrix micro-LED display. J. Soc. Inf. Disp. 27 , 347–353 (2019).

Xuan, T. T. et al. Inkjet-printed quantum dot color conversion films for high-resolution and full-color micro light-emitting diode displays. J. Phys. Chem. Lett. 11 , 5184–5191 (2020).

Chen, S. W. H. et al. Full-color monolithic hybrid quantum dot nanoring micro light-emitting diodes with improved efficiency using atomic layer deposition and nonradiative resonant energy transfer. Photonics Res. 7 , 416–422 (2019).

Krishnan, C. et al. Hybrid photonic crystal light-emitting diode renders 123% color conversion effective quantum yield. Optica 3 , 503–509 (2016).

Kang, J. H. et al. RGB arrays for micro-light-emitting diode applications using nanoporous GaN embedded with quantum dots. ACS Applied Mater. Interfaces 12 , 30890–30895 (2020).

Chen, G. S. et al. Monolithic red/green/blue micro-LEDs with HBR and DBR structures. IEEE Photonics Technol. Lett. 30 , 262–265 (2018).

Hsiang, E. L. et al. Enhancing the efficiency of color conversion micro-LED display with a patterned cholesteric liquid crystal polymer film. Nanomaterials 10 , 2430 (2020).

Kang, C. M. et al. Hybrid full-color inorganic light-emitting diodes integrated on a single wafer using selective area growth and adhesive bonding. ACS Photonics 5 , 4413–4422 (2018).

Geum, D. M. et al. Strategy toward the fabrication of ultrahigh-resolution micro-LED displays by bonding-interface-engineered vertical stacking and surface passivation. Nanoscale 11 , 23139–23148 (2019).

Ra, Y. H. et al. Full-color single nanowire pixels for projection displays. Nano Lett. 16 , 4608–4615 (2016).

Motoyama, Y. et al. High-efficiency OLED microdisplay with microlens array. J. Soc. Inf. Disp. 27 , 354–360 (2019).

Fujii, T. et al. 4032 ppi High-resolution OLED microdisplay. J. Soc. Inf. Disp. 26 , 178–186 (2018).

Hamer, J. et al. High-performance OLED microdisplays made with multi-stack OLED formulations on CMOS backplanes. In Proc. SPIE 11473, Organic and Hybrid Light Emitting Materials and Devices XXIV . Online Only (SPIE, 2020).

Joo, W. J. et al. Metasurface-driven OLED displays beyond 10,000 pixels per inch. Science 370 , 459–463 (2020).

Vettese, D. Liquid crystal on silicon. Nat. Photonics 4 , 752–754 (2010).

Zhang, Z. C., You, Z. & Chu, D. P. Fundamentals of phase-only liquid crystal on silicon (LCOS) devices. Light.: Sci. Appl. 3 , e213 (2014).

Hornbeck, L. J. The DMD TM projection display chip: a MEMS-based technology. MRS Bull. 26 , 325–327 (2001).

Zhang, Q. et al. Polarization recycling method for light-pipe-based optical engine. Appl. Opt. 52 , 8827–8833 (2013).

Hofmann, U., Janes, J. & Quenzer, H. J. High-Q MEMS resonators for laser beam scanning displays. Micromachines 3 , 509–528 (2012).

Holmström, S. T. S., Baran, U. & Urey, H. MEMS laser scanners: a review. J. Microelectromechanical Syst. 23 , 259–275 (2014).

Bao, X. Z. et al. Design and fabrication of AlGaInP-based micro-light-emitting-diode array devices. Opt. Laser Technol. 78 , 34–41 (2016).

Olivier, F. et al. Influence of size-reduction on the performances of GaN-based micro-LEDs for display application. J. Lumin. 191 , 112–116 (2017).

Liu, Y. B. et al. High-brightness InGaN/GaN Micro-LEDs with secondary peak effect for displays. IEEE Electron Device Lett. 41 , 1380–1383 (2020).

Qi, L. H. et al. 848 ppi high-brightness active-matrix micro-LED micro-display using GaN-on-Si epi-wafers towards mass production. Opt. Express 29 , 10580–10591 (2021).

Chen, E. G. & Yu, F. H. Design of an elliptic spot illumination system in LED-based color filter-liquid-crystal-on-silicon pico projectors for mobile embedded projection. Appl. Opt. 51 , 3162–3170 (2012).

Darmon, D., McNeil, J. R. & Handschy, M. A. 70.1: LED-illuminated pico projector architectures. Soc. Inf. Disp. Int. Symp . Dig. Tech. Pap. 39 , 1070–1073 (2008).

Essaian, S. & Khaydarov, J. State of the art of compact green lasers for mobile projectors. Optical Rev. 19 , 400–404 (2012).

Sun, W. S. et al. Compact LED projector design with high uniformity and efficiency. Appl. Opt. 53 , H227–H232 (2014).

Sun, W. S., Chiang, Y. C. & Tsuei, C. H. Optical design for the DLP pocket projector using LED light source. Phys. Procedia 19 , 301–307 (2011).

Chen, S. W. H. et al. High-bandwidth green semipolar (20–21) InGaN/GaN micro light-emitting diodes for visible light communication. ACS Photonics 7 , 2228–2235 (2020).

Yoshida, K. et al. 245 MHz bandwidth organic light-emitting diodes used in a gigabit optical wireless data link. Nat. Commun. 11 , 1171 (2020).

Park, D. W. et al. 53.5: High-speed AMOLED pixel circuit and driving scheme. Soc. Inf. Disp. Int. Symp . Dig. Tech. Pap. 41 , 806–809 (2010).

Tan, L., Huang, H. C. & Kwok, H. S. 78.1: Ultra compact polarization recycling system for white light LED based pico-projection system. Soc. Inf. Disp. Int. Symp. Dig. Tech. Pap. 41 , 1159–1161 (2010).

Maimone, A., Georgiou, A. & Kollin, J. S. Holographic near-eye displays for virtual and augmented reality. ACM Trans. Graph. 36 , 85 (2017).

Pan, J. W. et al. Portable digital micromirror device projector using a prism. Appl. Opt. 46 , 5097–5102 (2007).

Huang, Y. et al. Liquid-crystal-on-silicon for augmented reality displays. Appl. Sci. 8 , 2366 (2018).

Peng, F. L. et al. Analytical equation for the motion picture response time of display devices. J. Appl. Phys. 121 , 023108 (2017).

Pulli, K. 11-2: invited paper: meta 2: immersive optical-see-through augmented reality. Soc. Inf. Disp. Int. Symp . Dig. Tech. Pap. 48 , 132–133 (2017).

Lee, B. & Jo, Y. in Advanced Display Technology: Next Generation Self-Emitting Displays (eds Kang, B., Han, C. W. & Jeong, J. K.) 307–328 (Springer, 2021).

Cheng, D. W. et al. Design of an optical see-through head-mounted display with a low f -number and large field of view using a freeform prism. Appl. Opt. 48 , 2655–2668 (2009).

Zheng, Z. R. et al. Design and fabrication of an off-axis see-through head-mounted display with an x–y polynomial surface. Appl. Opt. 49 , 3661–3668 (2010).

Wei, L. D. et al. Design and fabrication of a compact off-axis see-through head-mounted display using a freeform surface. Opt. Express 26 , 8550–8565 (2018).

Liu, S., Hua, H. & Cheng, D. W. A novel prototype for an optical see-through head-mounted display with addressable focus cues. IEEE Trans. Vis. Computer Graph. 16 , 381–393 (2010).

Hua, H. & Javidi, B. A 3D integral imaging optical see-through head-mounted display. Opt. Express 22 , 13484–13491 (2014).

Song, W. T. et al. Design of a light-field near-eye display using random pinholes. Opt. Express 27 , 23763–23774 (2019).

Wang, X. & Hua, H. Depth-enhanced head-mounted light field displays based on integral imaging. Opt. Lett. 46 , 985–988 (2021).

Huang, H. K. & Hua, H. Generalized methods and strategies for modeling and optimizing the optics of 3D head-mounted light field displays. Opt. Express 27 , 25154–25171 (2019).

Huang, H. K. & Hua, H. High-performance integral-imaging-based light field augmented reality display using freeform optics. Opt. Express 26 , 17578–17590 (2018).

Cheng, D. W. et al. Design and manufacture AR head-mounted displays: a review and outlook. Light.: Adv. Manuf. 2 , 24 (2021).

Google Scholar  

Westheimer, G. The Maxwellian view. Vis. Res. 6 , 669–682 (1966).

Do, H., Kim, Y. M. & Min, S. W. Focus-free head-mounted display based on Maxwellian view using retroreflector film. Appl. Opt. 58 , 2882–2889 (2019).

Park, J. H. & Kim, S. B. Optical see-through holographic near-eye-display with eyebox steering and depth of field control. Opt. Express 26 , 27076–27088 (2018).

Chang, C. L. et al. Toward the next-generation VR/AR optics: a review of holographic near-eye displays from a human-centric perspective. Optica 7 , 1563–1578 (2020).

Hsueh, C. K. & Sawchuk, A. A. Computer-generated double-phase holograms. Appl. Opt. 17 , 3874–3883 (1978).

Chakravarthula, P. et al. Wirtinger holography for near-eye displays. ACM Trans. Graph. 38 , 213 (2019).

Peng, Y. F. et al. Neural holography with camera-in-the-loop training. ACM Trans. Graph. 39 , 185 (2020).

Shi, L. et al. Towards real-time photorealistic 3D holography with deep neural networks. Nature 591 , 234–239 (2021).

Jang, C. et al. Retinal 3D: augmented reality near-eye display via pupil-tracked light field projection on retina. ACM Trans. Graph. 36 , 190 (2017).

Jang, C. et al. Holographic near-eye display with expanded eye-box. ACM Trans. Graph. 37 , 195 (2018).

Kim, S. B. & Park, J. H. Optical see-through Maxwellian near-to-eye display with an enlarged eyebox. Opt. Lett. 43 , 767–770 (2018).

Shrestha, P. K. et al. Accommodation-free head mounted display with comfortable 3D perception and an enlarged eye-box. Research 2019 , 9273723 (2019).

Lin, T. G. et al. Maxwellian near-eye display with an expanded eyebox. Opt. Express 28 , 38616–38625 (2020).

Jo, Y. et al. Eye-box extended retinal projection type near-eye display with multiple independent viewpoints [Invited]. Appl. Opt. 60 , A268–A276 (2021).

Xiong, J. H. et al. Aberration-free pupil steerable Maxwellian display for augmented reality with cholesteric liquid crystal holographic lenses. Opt. Lett. 46 , 1760–1763 (2021).

Viirre, E. et al. Laser safety analysis of a retinal scanning display system. J. Laser Appl. 9 , 253–260 (1997).

Ratnam, K. et al. Retinal image quality in near-eye pupil-steered systems. Opt. Express 27 , 38289–38311 (2019).

Maimone, A. et al. Pinlight displays: wide field of view augmented reality eyeglasses using defocused point light sources. In Proc. ACM SIGGRAPH 2014 Emerging Technologies (ACM, Vancouver, Canada, 2014).

Jeong, J. et al. Holographically printed freeform mirror array for augmented reality near-eye display. IEEE Photonics Technol. Lett. 32 , 991–994 (2020).

Ha, J. & Kim, J. Augmented reality optics system with pin mirror. US Patent 10,989,922 (2021).

Park, S. G. Augmented and mixed reality optical see-through combiners based on plastic optics. Inf. Disp. 37 , 6–11 (2021).

Xiong, J. H. et al. Breaking the field-of-view limit in augmented reality with a scanning waveguide display. OSA Contin. 3 , 2730–2740 (2020).

Levola, T. 7.1: invited paper: novel diffractive optical components for near to eye displays. Soc. Inf. Disp. Int. Symp . Dig. Tech. Pap. 37 , 64–67 (2006).

Laakkonen, P. et al. High efficiency diffractive incouplers for light guides. In Proc. SPIE 6896, Integrated Optics: Devices, Materials, and Technologies XII . (SPIE, San Jose, California, United States, 2008).

Bai, B. F. et al. Optimization of nonbinary slanted surface-relief gratings as high-efficiency broadband couplers for light guides. Appl. Opt. 49 , 5454–5464 (2010).

Äyräs, P., Saarikko, P. & Levola, T. Exit pupil expander with a large field of view based on diffractive optics. J. Soc. Inf. Disp. 17 , 659–664 (2009).

Yoshida, T. et al. A plastic holographic waveguide combiner for light-weight and highly-transparent augmented reality glasses. J. Soc. Inf. Disp. 26 , 280–286 (2018).

Yu, C. et al. Highly efficient waveguide display with space-variant volume holographic gratings. Appl. Opt. 56 , 9390–9397 (2017).

Shi, X. L. et al. Design of a compact waveguide eyeglass with high efficiency by joining freeform surfaces and volume holographic gratings. J. Optical Soc. Am. A 38 , A19–A26 (2021).

Han, J. et al. Portable waveguide display system with a large field of view by integrating freeform elements and volume holograms. Opt. Express 23 , 3534–3549 (2015).

Weng, Y. S. et al. Liquid-crystal-based polarization volume grating applied for full-color waveguide displays. Opt. Lett. 43 , 5773–5776 (2018).

Lee, Y. H. et al. Compact see-through near-eye display with depth adaption. J. Soc. Inf. Disp. 26 , 64–70 (2018).

Tekolste, R. D. & Liu, V. K. Outcoupling grating for augmented reality system. US Patent 10,073,267 (2018).

Grey, D. & Talukdar, S. Exit pupil expanding diffractive optical waveguiding device. US Patent 10,073, 267 (2019).

Yoo, C. et al. Extended-viewing-angle waveguide near-eye display with a polarization-dependent steering combiner. Opt. Lett. 45 , 2870–2873 (2020).

Schowengerdt, B. T., Lin, D. & St. Hilaire, P. Multi-layer diffractive eyepiece with wavelength-selective reflector. US Patent 10,725,223 (2020).

Wang, Q. W. et al. Stray light and tolerance analysis of an ultrathin waveguide display. Appl. Opt. 54 , 8354–8362 (2015).

Wang, Q. W. et al. Design of an ultra-thin, wide-angle, stray-light-free near-eye display with a dual-layer geometrical waveguide. Opt. Express 28 , 35376–35394 (2020).

Frommer, A. Lumus: maximus: large FoV near to eye display for consumer AR glasses. In Proc. SPIE 11764, AVR21 Industry Talks II . Online Only (SPIE, 2021).

Ayres, M. R. et al. Skew mirrors, methods of use, and methods of manufacture. US Patent 10,180,520 (2019).

Utsugi, T. et al. Volume holographic waveguide using multiplex recording for head-mounted display. ITE Trans. Media Technol. Appl. 8 , 238–244 (2020).

Aieta, F. et al. Multiwavelength achromatic metasurfaces by dispersive phase compensation. Science 347 , 1342–1345 (2015).

Arbabi, E. et al. Controlling the sign of chromatic dispersion in diffractive optics with dielectric metasurfaces. Optica 4 , 625–632 (2017).

Download references

Acknowledgements

The authors are indebted to Goertek Electronics for the financial support and Guanjun Tan for helpful discussions.

Author information

Authors and affiliations.

College of Optics and Photonics, University of Central Florida, Orlando, FL, 32816, USA

Jianghao Xiong, En-Lin Hsiang, Ziqian He, Tao Zhan & Shin-Tson Wu

You can also search for this author in PubMed   Google Scholar

Contributions

J.X. conceived the idea and initiated the project. J.X. mainly wrote the manuscript and produced the figures. E.-L.H., Z.H., and T.Z. contributed to parts of the manuscript. S.W. supervised the project and edited the manuscript.

Corresponding author

Correspondence to Shin-Tson Wu .

Ethics declarations

Conflict of interest.

The authors declare no competing interests.

Supplementary information

Supplementary information, rights and permissions.

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made. The images or other third party material in this article are included in the article’s Creative Commons license, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons license and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this license, visit http://creativecommons.org/licenses/by/4.0/ .

Reprints and permissions

About this article

Cite this article.

Xiong, J., Hsiang, EL., He, Z. et al. Augmented reality and virtual reality displays: emerging technologies and future perspectives. Light Sci Appl 10 , 216 (2021). https://doi.org/10.1038/s41377-021-00658-8

Download citation

Received : 06 June 2021

Revised : 26 September 2021

Accepted : 04 October 2021

Published : 25 October 2021

DOI : https://doi.org/10.1038/s41377-021-00658-8

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

This article is cited by

Metaverse, virtual reality and augmented reality in total shoulder arthroplasty: a systematic review.

• Umile Giuseppe Longo
• Alberto Lalli
• Ara Nazarian

BMC Musculoskeletal Disorders (2024)

Color liquid crystal grating based color holographic 3D display system with large viewing angle

• Qiong-Hua Wang

Light: Science & Applications (2024)

Mass-produced and uniformly luminescent photochromic fibers toward future interactive wearable displays

• Manu Gopakumar
• Gun-Yeal Lee
• Gordon Wetzstein

Nature (2024)

Enhancing the color gamut of waveguide displays for augmented reality head-mounted displays through spatially modulated diffraction grating

• Jae-Sang Lee
• Seong-Hyeon Cho
• Young-Wan Choi

Scientific Reports (2024)

Quick links

• Explore articles by subject
• Guide to authors
• Editorial policies

Help | Advanced Search

Computer Science > Computer Vision and Pattern Recognition

Title: modern augmented reality: applications, trends, and future directions.

Abstract: Augmented reality (AR) is one of the relatively old, yet trending areas in the intersection of computer vision and computer graphics with numerous applications in several areas, from gaming and entertainment, to education and healthcare. Although it has been around for nearly fifty years, it has seen a lot of interest by the research community in the recent years, mainly because of the huge success of deep learning models for various computer vision and AR applications, which made creating new generations of AR technologies possible. This work tries to provide an overview of modern augmented reality, from both application-level and technical perspective. We first give an overview of main AR applications, grouped into more than ten categories. We then give an overview of around 100 recent promising machine learning based works developed for AR systems, such as deep learning works for AR shopping (clothing, makeup), AR based image filters (such as Snapchat's lenses), AR animations, and more. In the end we discuss about some of the current challenges in AR domain, and the future directions in this area.

Submission history

Access paper:.

• Other Formats

References & Citations

• Google Scholar
• Semantic Scholar

BibTeX formatted citation

Bibliographic and Citation Tools

Code, data and media associated with this article, recommenders and search tools.

• Institution

arXivLabs: experimental projects with community collaborators

arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.

Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.

Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs .

• Open access
• Published: 16 September 2020

Systematic review and meta-analysis of augmented reality in medicine, retail, and games

• Pranav Parekh 1 ,
• Shireen Patel 1 ,
• Nivedita Patel 1 &
• Manan Shah 2  

Visual Computing for Industry, Biomedicine, and Art volume  3 , Article number:  21 ( 2020 ) Cite this article

86 Citations

10 Altmetric

Metrics details

This paper presents a detailed review of the applications of augmented reality (AR) in three important fields where AR use is currently increasing. The objective of this study is to highlight how AR improves and enhances the user experience in entertainment, medicine, and retail. The authors briefly introduce the topic of AR and discuss its differences from virtual reality. They also explain the software and hardware technologies required for implementing an AR system and the different types of displays required for enhancing the user experience. The growth of AR in markets is also briefly discussed. In the three sections of the paper, the applications of AR are discussed. The use of AR in multiplayer gaming, computer games, broadcasting, and multimedia videos, as an aspect of entertainment and gaming is highlighted. AR in medicine involves the use of AR in medical healing, medical training, medical teaching, surgery, and post-medical treatment. AR in retail was discussed in terms of its uses in advertisement, marketing, fashion retail, and online shopping. The authors concluded the paper by detailing the future use of AR and its advantages and disadvantages in the current scenario.

Introduction

Significant advances in technology in contemporary times have made many things possible such as creating virtual worlds or enhancing existing real-world objects and scenarios through multiple sensory modes [ 1 ]. Augmented reality (AR) and virtual reality (VR) have the capability to alter the way entertainment, shopping, health activities, recreation, etc. are perceived [ 2 ]. Although VR and AR are often assumed to be the same, they are considerably different. AR, also termed mixed reality [ 3 , 4 ], is a mapping of virtual objects onto the real world whose elements are augmented using sensory inputs. VR is a complete immersion in an artificial environment created using software [ 5 ]. This environment is presented to and accepted by the user as a real environment. This difference forms the basis of the functioning of virtual technology. Both AR and VR are often combined to attain specific goals [ 6 ].

AR, which was commercialized long ago, has played a major role in reshaping the existing manners of performing activities. However, owing to certain challenges, the technology did not achieve the expected results in the early days [ 7 ]. Investors were hesitant to invest heavily in this field because they believed that the augmented world was yet to be adequately developed to yield the desired outputs [ 8 ]. However, many industries are gradually recognizing the need to invest in AR to remain at the top of the ladder and expand their brand, by attracting more customers with something new and innovative, as mentioned in ref. [ 9 ]. Since the introductory stage of AR, gaming has been its primary application. However, according to the report drafted by Goldman Sachs in 2016, AR is expected to improve retail, healthcare, and real estate markets in the coming years [ 10 ]. AR is used by various industries for product design; according to ref. [ 11 ], immersive service prototyping is in significant demand in the service design sector. AR has also been used in academics [ 12 ], aeronautics [ 13 ], and military [ 14 ]. It has a substantial potential to make every aspect of living enjoyable, easier, and more creative [ 15 ].

AR technologies are broadly classified into hardware, mainly consisting of varied displays and sensors, and the software algorithms required for integrating the augmentations with the real world. These technologies are used in several fields such as tourism and hospitality [ 16 ], education, medicine, retail, and gaming and entertainment. Hardware and software were integrated in the field of AR-based prototyping methods [ 17 ]. Integration is accomplished by accurately mapping a functional hardware prototype onto a virtual display. AR displays include optical projection systems, monitors, handheld devices, head mounted display (HMD) or head-up display (HUD), and eye tap. A handheld AR system was created to track optical markers in real time [ 18 ]. An optical projection system was generated via a mouse [ 19 ], enabling the configuration of input devices along with AR displays. HMD displays are described in ref. [ 20 ] as real-time three-dimensional interactive displays that allow free head motion and full body mobility; according to ref. [ 21 ], they are used widely as modelers. A usage method for HUD was provided by incorporating it into a laminated windshield [ 22 ], and it was patented. Spatial AR, the branch of AR that does not require displays to function, was studied in detail in ref. [ 23 ]. The authors of that study provided examples such as shader lamps, iLamps and mobile projectors, Being There, HoloStations, and smart projectors.

This paper presents a review of the use of AR in three applications: gaming, medicine, and retail. Gaming has been the leading sector in the use of AR, as a result of which gamers have experienced immense creativity, innovation, and unforgettable experiences. Gamers find AR-enhanced games better and more thrilling because of the engaging experience provided by the technology.

The use of AR in the medical industry has grown over the years. It has proven to be helpful to both doctors and patients. Patients can be educated about their diseases through AR, and the technology can also be used for complex surgeries, helping doctors to perform them with high accuracy. AR has also been used in the retail industry, and several companies have started investing in AR to create apps and amazing experiences to promote and sell their products. In-store technology, as well as online AR technology, has changed the way people shop. Different sectors of fashion that have been affected by AR and experienced retail change are discussed in this paper. AR has impacted our lives in previously unimaginable ways. Thus, it could be said that AR is the future of gaming, retail, and medicine. The expansion of the AR technology in these three sectors and its acceptance by the public was analyzed in this study. Surveys were performed and feedback from various customers was scrutinized to understand their perception of the new technology.

AR in entertainment

The future of entertainment is likely to be influenced by advanced technologies such as AR [ 24 , 25 ]. Mobile technological devices have made it possible for the entertainment industry to change the way people interact and engage with games, sports, tours, performances, among other activities. AR combines real and virtual worlds in 3D while being interactive [ 26 , 27 , 28 ].

In addition to redefining traditional gaming, AR is also already being used to increase the effectiveness of multimedia presentations and videos. However, it can be extended to a considerably greater array of entertainment fields, such as the way we listen to music and the way that we travel. Interface and visualization technology, along with some basic enabling technologies, are being incorporated to achieve heterogeneous and tangible interfaces [ 29 ]. AR may also be used collaboratively to display personalized information to each user. Further, it enhances broadcasting in sports events, concerts, and other events by highlighting or inserting information.

Ivan Sutherland was the creator of the first complete AR system with simple graphics [ 30 ] and a very heavy HMD. Subsequently, AR use in the entertainment industry has made tremendous advances, considering the latest well-known hit, which is an example of location-based gaming. AR completely changes users’ interaction, encouraging people to walk outside and read more, by transforming their books into an AR play space, whereas non-AR experiences limit users to a screen.

Most AR entertainment systems have software components that run on the device such as local game control and user tracking; server connection, which is often necessary in cases where there are shared resources, location-driven games, and where constant synchronization is required, is also used [ 31 ]. Although every system has its unique architecture, real-time performance can be achieved using cloud [ 19 ]. This data and workflow flowchart is depicted in Fig.  1 , shown specifically for AR mobile systems. As shown in Fig.  2 , every architecture mainly comprises three parts: layers that allow the integration of diverse hardware, application container, which is also a run-time context that contains application logic, including things such as navigation and assembly, and workflow abstraction layer, which is where all of the computational tasks occur, whether on the device or on cloud. The results of all these tasks are integrated with real contents and presented on the displays, which users interact with.

Data and workflow for mobile AR

Framework for AR in mobile games

There are mainly two types of AR systems used for the purpose of entertainment. The first is marker-based applications, which are based on image recognition. This technology uses black and white markers that are used to detect the augmented object. To illustrate, the camera in a phone is first pointed at any marker’s position; after the marker has been identified, the required or embedded digital content is superimposed on the marker on top of the real object. Here, the images are to be coded into the system beforehand, making them easier to detect. Most AR apps seen in the market are marker-based. One of the most popular marker-based applications is Snapchat, which has attracted almost all of the population, and is very popular among the youth.

The second is location-based applications, which work without markers. The technology makes use of global positioning system (GPS) or some digital compass that helps in detecting the user’s position, following which the real-world physical objects are replaced with or incorporated with the augmented objects. Such applications enable users to find the best restaurants nearby or locate their cars in parking lots. They can also be used in games that require the player’s location (Fig.  3 ).

a Location-based game: Pokémon Go (Source: Forbes.com ); b Marker-based application: Snapchat (Source: Vox.com )

Some popular software used to create AR applications are Unity, Vuforia, ARToolKit, Google ARCore, Android studio, and AR Spark studio. Unity is the most popular game engine used for developing games and AR apps. The aforementioned softwares are generally used by professionals and regular programmers. Figure  4 illustrates how a simple entertainment application that allows the user to project 3D animals into reality was implemented using Android studio.

Implementation of simple application using Android studio, which allows users to project 3D animals into reality

The biggest application in entertainment is gaming. Although AR games may be limited in physical aspects and face-to-face communication, there is increased collaborative gaming and relationship building through remote multiplayer games [ 32 ]. There is significant potential for the emotional and mental aspects of a game, and AR makes it possible to create all types of scenarios and supports highly complex puzzles, models, and virtual opponents.

Despite the popularity of computer games, pervasive gaming, defined as gaming which increases physical movement and social interactions, enhances gamer experience [ 33 ]. This type of gaming focuses on the aspect of bringing virtual gaming back to the real world. One of the main goals of pervasive computing is to develop context-aware applications that analyze and collect information from the environment, as a result of which users alter their conduct accordingly [ 34 ]. This is achieved by using pervasive computing in combination with technologies, such as smart toys, and creating location-aware games that use the architecture that we currently live in as a game board.

A local collaborative environment study, as shown in Fig.  5 , in which multiple users could interact with the environment and communicate with each other at the same time using see-through HMDs and face-snapping, which allows fast and precise direct object manipulation, was conducted [ 35 ]. The gaming space was subdivided into spatial regions, and a layering concept was introduced for individual views and privacy management. It was observed that numerous board games and console games fit into this model, as a result of which they provide additional benefits and protect individualism and privacy.

AR-based gaming

Further, an emerging trend of serious games, which are computer games meant for non-leisure and educational purposes, has also been observed. They are different and better than traditional games because they may be used for simulations in areas, such as medicine, military operations, and education, thereby linking entertainment and work [ 36 ]. As case study, two AR games were used; the AR Puzzle, a puzzle game based on the City University campus in London, and AR Breakout, an old arcade game that was moved to a tangible environment. Based on the results collected, it could be said that compared to video games, AR games are easier to adapt to. It was observed that AR Puzzle turned out to be a very interesting and effective learning tool. In general, it was observed that tangible AR interactions were preferred by people over traditional ways of playing games.

AR games are also gaining momentum as learning guides, considering the younger generations’ immense use of media. They impact motivation and knowledge acquisition. As mentioned in ref. [ 37 ], real-world games, based on real and virtual elements, along with highly augmented computing functionality, create exciting and fun gaming experiences, potentially leading to high learning motivation. The ability of games to promote teamwork, collaboration, social interaction, and cooperation in a learning environment is frequently emphasized [ 38 ]. According to nine studies, the use of AR games in learning boosts learning performance and increases student motivation and enjoyment by 58% and 10%. However, the limitations of these systems, such as lack of interdisciplinary programs and students being distracted by the virtual novelty factors, persist [ 39 ]. AR-based learning has substantial potential, if proper approaches are developed based on study and analysis.

AR broadcasting is divided into two crucial elements: AR tracking and AR display. Although AR display techniques for broadcasting purposes are still nascent, they are used to project content into three-dimensional space. They are of three forms: head-mounted, monitor-based, and projection-based. Current AR tracking approaches are classified into three types: model-based, marker-based, and tracking without prior knowledge. Technologies, such as cameras, infrared sensors, hybrid sensors, and 2D and 3D markers, can all be implemented to identify a pattern and track its position in the real world. Robotic cameraman systems have been proposed to increase the quality of broadcasting systems and replace human operators [ 40 ]. It has been shown that robotic cameramen facilitate more precise and advanced interaction with virtual elements, and through zooming and multiple angle views, improve the performance of AR broadcasting in all sorts of environments.

An enhanced AR system displays statistical players’ information on captured images of a sports game [ 41 ]. It is an image enhancement technique based on an algorithm that implements multi-scale retinex. It was designed to improve the accuracy of player detection during adverse conditions such as intense sunlight. This is followed by face detection performed using adaptive boosting and Haar features for feature extraction and classification. Discriminant analysis and the nearest-neighbor classifier are used for classification. The system can also display player statistics. This model was tested on several images in immensely diverse conditions, and it was concluded that it could be extended to all types of sports where the inputs are images and the desired output is information displayed around recognized players.

A haptically enhanced broadcasting system that uses AR techniques to synthesize videos in real time, a multimedia streaming technology, and haptics were also implemented. The system operation sequence has four different stages: scene capture, haptic editing, data transmission, and display with haptic interaction. It can be used for creating haptic effects for cartoons and in the context of live sports broadcasts. The most noticeable feature of haptics is the sense of being social presence at the location displayed remotely. In live broadcasting, haptic interaction can enable an audience to take part in communication and discussions with those viewing the same program. Haptic interaction refers to the technology that creates an experience of touch through vibrations, motions, and the application of forces [ 42 ].

AR makes spectator sports more entertaining because of the additional information provided to the viewers. An AR-based sports system involves two major steps: homography estimation and automatic player detection, as described in ref. [ 43 ]. A marker-based approach that uses image patterns was designed for homography estimation, and a markerless approach that works for natural images with distinctive local patterns was designed for automatic player detection. For baseball fields, contours must be extracted and geometric primitives, estimated. For the player detection methods, an algorithm based on Adaboost learning, which is both fast and robust, was used. However, it failed to detect players sometimes. This system, which is based on still images captured using mobile phones, was implemented on mobile platforms. It made it possible to accept all the images taken from different angles, with large variations in the size and pose of the players, and different lighting conditions in the playground. Photos were taken with Apple iPhone 3GS, and a PC with an Intel 2.67 GHz Core I7 CPU was used to test the algorithm. In addition, Table  1 also discusses AR games, their advantages, and what technology was used to make them.

AR in medicine

As mentioned previously, the use of AR to enhance natural environments and alter the perceptions of reality is being exploited in various fields such as entertainment, education, retail, and marketing [ 52 , 53 , 54 ]. It is also being applied to the field of medicine. AR has been defined as a real-time indirect or direct view of the surrounding world that has been augmented with computer-generated virtual information [ 55 ]. AR is indeed highly beneficial to the medical field; however, considerable effort and care must be taken to reap its benefits. The use and function of AR in medicine depends on the skill of the technician, as well as that of the doctors and medical teachers involved. AR systems are also extremely costly, compared to the normal medical methodologies. Hence, to reap maximum benefits, the AR systems must be deployed with significant care and accuracy [ 56 , 57 ].

Ref. [ 58 ] discusses the importance of AR and VR in the fields of medical anatomy and health sciences. The purpose of this research was to assess whether medical students who used VR and AR were more effective than those that used other mobile applications. Fifty-nine participants were randomly assigned three learning modes: VR, tablet-based applications, and AR. The senses of the users using VR are fully immersed in a virtual environment that mimics the properties of the real world through HMDs, stereo headphones, high-resolution motion tracking systems. AR, on the other hand, is used to superimpose digital models on the real world. 3D tablet displays are used mainly for user interaction. Using these teaching modes, a lesson on skull anatomy was conducted. The anatomical knowledge of the medical students was assessed through a repetition of experiments with different lessons. It was noted that both AR and VR were more beneficial, as they promoted increased engagement of the medical students.

On the other hand, ref. [ 59 ] conducted a review that evaluated the past, present, and future of the usage of computer-aided AR in surgeries. Computer-aided AR, also known as computer-aided drawing is a drawing tool that allows the user to make accurate data models using AR. The review centered on the different types of surgeries where AR can be used as a display or a model. A systematic review of the effectiveness of AR applications in medical training yields a promising outlook as well [ 60 ]. The training applications were assigned to three different categories: echocardiography training, laparoscopic surgery, and AR and VR training for neurosurgical procedures. This literature suggests that although AR may have gained scientific interest, no recorded evidence suggests that AR can transfer information to the user seamlessly and promisingly.

Medical displays and accurate medical imaging technology are significant because they enable physicians to fully exploit rich sources of heterogeneous intraoperative and preoperative data (as shown in Fig. 6 which depicts intraoperative brain imaging system). Ref. [ 61 ] discussed these advanced medical displays and also established a relation between the subsets of such bodies of work, to give an idea of the challenges that may occur during the application of such displays. They discussed AR technologies, such as HMD-based AR systems, augmented optics, augmented windows, monitors, and endoscopes, and their specific applications in the medical field. In the study, the solutions that can be provided by AR were acknowledged, and its use in the workflow was encouraged. HMD-based AR headsets consist of OLED microdisplays on which AR systems, such as augmented optics and windows, can run.

Brain imaging and brain surgery using AR

Surgeons are often the earliest adopters of technical tools that can enhance the surgical and patient experience. The application and limitations of a digital surgical environment that uses AR and VR were discussed by ref. [ 62 ]. The applications include operative benefits, broadcasting and recording of surgery, anatomical evaluation, telementoring, and provision of medical education. Limited battery life, large devices, and cumbersome cables are the limitations of the technology. However, it has been stated that significant progress will be made in the coming generations with the development of these tools, which may potentially lead to an increase in their usage as surgical loupes.

An ophthalmic AR environment was developed to allow for more accurate laser treatment for ophthalmic diseases, telemedicine, and real-time image analysis, measurement, and comparison [ 63 ]. The system was designed around a standard slit lamp biomicroscope. A camera, interfaced with the biomicroscope, was used to capture the images, which were then sent to a video capture board. The image was processed using a single computer workstation, and fast registration algorithms were applied to it. The output given by the computer was a VGA resolution video display with adjustable contrast and brightness attached to the oculars of the slit lamp microscope.

A medical AR system performs three tasks: camera or instrument tracking, patient registration, and creation of preoperative planning data. A video see-through system for medical AR was described in ref. [ 64 ]. The system was based on a VectorVision image-guided surgery device. They demonstrated that their system could perform all the above mentioned tasks. VectorVision is an optical tracking IGS platform consisting of two infrared cameras, a PC, and a touch screen display. A vector vision link is a TCP/IP-based interface that is integrated and used with the vector vision cranial system. The tests showed that an augmented video stream with an average frame rate of 10 fps was generated by the augmented video stream using a 640 × 480 pixel webcam. Furthermore, they recorded a latency period of approximately 80 ms, and the camera tracking method exhibited good accuracy. Hence, they provided a novel approach for realizing AR applications in the medical field.

A specific technology that is used extensively for visualization is the HMD. Ref. [ 65 ] discussed AR visualization performed through the use of a head-mounted operating binocular required in the field of medicine. The head-mounted operating binocular is a somewhat modified version of the HMD. The radioscope was adopted because it is a miniature and cost-effective system that can be conveniently deployed for visualization. In this study, a basic design of the modified HMD was displayed, and the results of a detailed laboratory study for photogrammetric calibration of the varioscope’s computer display to a real-world scene, was presented. The location of a position measurement probe of an optical tracking system was transformed to the binocular display with an error of less than 1 mm in the real world in 56% of all cases. In other cases, the error was found to be less than 2 mm. Hence, we can conclude that sufficient accuracy was achieved, such that it could be applied for a wide range of CAS applications.

A haptic AR environment was used to design cranial implants, as described in ref. [ 66 ]. A haptic AR environment conveys the sense of touch to the user; ‘haptic’, in general, refers to any technology that provides the experience of touch through motions, vibrations, and forces. The data obtained from the patient CT was used to create virtual 3D cranial models that were superimposed over their hands. Through such an environment, the medical cranial sculptor could feel and view the model. The personal augmented reality immersive system (PARIS), a new prototype display system, was also used alongside the models. The PARIS system creates the illusion of a 3D tool that can be held by the sculptor. Neurosurgeons, paleontologists, and radiologists have expressed interest in utilizing the system.

Ref. [ 67 ] (Fig. 7 ) presented a paper on an AR system for thermal ablation of the liver. This system was first evaluated on an abdominal phantom, and subsequently, on patients in an operating room. The preoperative image of patients and the needle position that a medical practitioner manipulates were registered in a common coordinate system. The feature points were extracted and processed through validated algorithms. The experiment showed that a predefined target with an accuracy of 2 mm could be achieved at an insertion time of less than a minute. The output inspired confidence that the system provided accurate guidance and information during the complete breathing cycle of the patient.

Hepatic surgery using AR

A rehabilitation system for hand and arm movement was implemented through a spatial AR system, as described in ref. [ 68 ]. The system created a virtual audio and visual experience by tracking a subject’s hand during rehabilitation-related tasks that involve the elbow, shoulder, and wrist movements. Real-time data and photos were sent to the clinic for further evaluation and assessment of the system. The study proved that the system was functional through the application of the technology in the laboratory. The system made it possible to incorporate real objects into tasks, as desired. It also controlled for external objects and ensured the safety and comfort of the patients. Another advantage provided by the system was that the tasks could be modified by a therapist based on the needs of the patient. The system they outlined depicted a performance-driven exercise program for stroke rehabilitation.

Apart from medical teaching and anatomy, medical surgery is an essential use case of AR in medicine as well. Table  2 shows such essential examples of AR in surgery.

AR in retail

AR significantly impacts how companies compete with one other in the technologically advancing environment. AR, as a result of its growing acceptance rate over the years, has heavily influenced brand awareness and expansion. The concept of AR in retail is anything but new. According to ref. [ 77 ], some of the largest firms, such as Coca-Cola, McDonald’s, and General Electric, have invested in AR for better retail experience and more innovative ways of marketing their products. The sales department of Coca-Cola collaborated with Augment to deploy AR to build a software that could help visualize the look of coolers in retail stores. This will help B2B customers make better product choices. Trigger developed an AR app for McDonald’s by bringing selected few animated figures and characters to life for an interactive experience for children. The software platform used was Vuforia. The main aim of the app was to feature characters from DreamWorks movies, such as How to Train Your Dragon and Mr. Peabody and Sherman, on an AR platform to make kids experience healthy fun. The surface around the Happy Meal box would come to life, and a garden filled with cherries, apples, tomatoes, and carrots would emerge.

The best way to compete healthily in the market is to build strong relations with customers and gain their loyalty by enhancing their engagement with the products. Ref. [ 77 ] talks about three different types of consumer engagement facilitated by AR. User-brand engagement occurs between a customer and the product that he/she wishes to buy. This type of engagement could be made as immersive as possible, allowing the users to manipulate and interact with the technology. User-user engagement helps customers interact with each other based on the AR content. They can modify each other’s digital data, resulting in the strengthening of their bond, as well as their individual relation with the company. User-bystander engagement enables customers to make artifacts of their experience with AR and share them on a social platform, thus leading to the advertisement of the product, which in turn benefits the company.

As mentioned in ref. [ 78 ], AR has expanded into various forms such as HMDs, mobile applications, contact lenses, and devices. One such smart device is the Memory Mirror, set up by Neiman Markus, which helps customers look at outfits from different angles and compare the various selected outfits simultaneously (Fig.  8 ).

a AR Use by Coca-Cola (source: augment.com ); b AR Applied to McDonald’s Happy Meal box (source: triggerglobal.com )

It has been demonstrated that AR can help build customer relations and boost sales by reducing the risks that customers face while purchasing a product. Ref. [ 79 ] discusses how AR can improve customer insights, make the shopping experience enjoyable, and reduce the customer perceived risks. The perceived risk is the uncertainty faced by the customers or the negative results they might get from the purchase of a product. A research model indicating that AR can indeed help in reducing the various risks was proposed. The entire model consisted of six different dimensions of risks to be eliminated or reduced through the use of AR. These include social, financial, psychological, performance, physical, and time risks. Thus, it has been assumed in this paper that AR can reduce the perceived risks but empirical proof is yet to be given.

According to ref. [ 80 ], retailers have lost sales to online shopping over the years. However, with the introduction of AR, retailers can reinvent the customer experience and make it far more interesting than traditional shopping. Furthermore, this study also centered on price optimization. Loyalty programs help retailers keep track of customer identification and provide the customers with discounts in return for their data. Thus, integrating AR with loyalty program data could help retailers optimize the prices of products, according to a specific customer. Such personalized shopping experiences could improve the customer experience; further, such AR systems could help the customers navigate easily through the products that are affordable to them. Thus, with the ease brought on in the shopping experience, the customers might prefer going to stores to shopping online. However, an increase in use of AR in online shopping apps has been also gaining momentum for example a jewellery app, as shown in Fig.  9 .

AR-based mobile app for online shopping

Using an app launched by a Swedish eye retailer, Synsam, as shown in Fig.  10 [ 81 ]. studied the impact of an AR-based smartphone application on the customer’s product purchase intentions. The app helped the customers try out different eyeglasses without actually putting them on physically. Their survey aimed to investigate whether the digital experience had a positive effect on the decision to purchase and the determinants that led to it. It was reported that many people found the experience very helpful and fun. It was observed that the females enjoyed trying out different pieces of eyewear using the selfie feature, whereas, the males were more fascinated by the technological side of AR. However, there were people who felt that going to a store and trying on the eyewears physically before buying them was better. However, a significant percentage agreed that AR was a useful technology for buying products.

Virtually trying on glasses using synsam AR application

The introduction of virtual fitting rooms (VFRs) has taken AR to new heights [ 82 ]. These VFRs enable a person to try on outfits without actually being present in the store. This concept can also be used for in-store shopping, making the customer experience fun and easy. A combination of a variety of technologies, such as natural interaction (NI), 3D scanning, 3D models, and omnipresent social networking features, have been used to make the idea of VFRs a considerable success. NI enables users to interact with the augmented environment using hand gestures, speech, and body language. 3D sensors are used to scan a user’s body to create a 3D avatar-type model, which is then integrated with other data, such as gender and different retailers. Customers can be granted access to a variety of clothing, creating a real-time shopping experience. As shopping can be-time consuming and exhausting, adopting such innovative ways can make the customer experience more interactive and fun and less tiring. Furthermore, the biggest obstacle to online shopping concerning whether the garment would fit or not, can be eliminated by such future VFRs. Figure  11 depicts the possible look of a VFR.

VFR (source: ref. [ 82 ])

Ref. [ 83 ] presented an AR-based virtual trial room that allowed the user to try on clothes virtually. This study aimed to enhance the online shopping experience and reduce the time spent on in-store shopping by decreasing the queuing time. In this method, a human was detected from the background using light variations. Frame extraction, blurring, Red, Green, Blue (RGB) to Hue, Saturation, value (HSV) conversion, current frame subtraction, thresholding, Binary Large Object detection, gesture estimation, and post-processing were the steps that made up the desired system (Fig.  12 ). Relevant frames and data were extracted from the camera input, following which the Gaussian blur was applied, to remove unnecessary image noise. This was followed by the RGB to HSV conversion, to achieve greater accuracy and image registration, where different sets of data were transformed into one coordinate system. Then, frame subtraction was performed to reduce the background noise and emphasize the foreground details. The gesture estimation step familiarized the system with general gesture functionalities such as “try next cloth”, or ‘dislike’, or ‘like’. The final post-processing step was necessary, to add final touches to the output. This model could be further enhanced by adding social features that enabled users to take pictures and share them with friends or family.

Working of virtual trial room based on steps mentioned in ref. [ 83 ]

Ref. [ 84 ] discussed how AR had impacted the customer experience; with the aim of integrating psychological, technological, and behavioral perspectives, an embodiment-presence-interactivity cube was proposed based on a variety of existing technologies. AR has an important role to play at every stage of the customer experience. In the pre-experience stage, customers obtain detailed information on the products, which enhances their decision-making. In the experience stage, the shopping experience is made more immersive and enjoyable for the users. The post-experience stage enables customers to evaluate their experience, create content, and share with other people. This leads to customer loyalty and brand awareness.

The effects of AR on customer behavior in purchasing a fashion product have been mentioned in ref. [ 85 ]. An experiment involving 162 participants aged between 18 and 35 was conducted. They took the help of an AR app of a makeup retail brand, which enabled the participants to apply different makeup to their faces using a virtual mirror. The entire experiment session included interaction with the app, following which questions were posed to the customers on their experience and purchase intention. It was observed that people who had experienced augmentation shared positive feedbacks on their experience and purchase intention. The customers who were hedonically motivated experienced more positive emotional response. The present model can be extended to include more features and yield more outcomes beyond purchase intention such as customer satisfaction and loyalty.

Ref. [ 86 ] discussed how fashion retail has evolved and how the emergence and growth of technology is expanding the fashion retail market. In the past, retailers had to create large portfolios and have a big store space to gain the attention of the users. Further, they explained how the introduction of online shopping was a revolutionary step that changed the scenario of a shopping experience. At present, with the advancement and acceptance of AR, brands are including AR in their strategy for remaining at the top. They proceeded to discuss omni-channel retailing, which consists of a cross-channel customer experience, through which a user can access multiple retail channels and also use multiple devices. Ref. [ 87 ] also discussed the penetration of the fashion industry by AR, leading to its growth in technological aspects. Further, ref. [ 87 ] mentioned the acceptance scenario of the AR technologies. The technology acceptance model (TAM) was the first model to focus on the insights of why the customers may accept or reject a new technology. The TAM has been used several times for research purposes, as it is a very helpful model in determining user acceptance. Ref. [ 88 ] also reviewed the implementation of AR by retailers, its applications, and consumer acceptance. The TAM model has been used to determine user acceptance and highlight the need for efficient and consumer-friendly devices in the future for retail growth.

Table  3 discusses different AR applications and devices created for retail experience, and surveyed upon by their respective developers along with the kind of response the technology received.

Applying AR in fight against COVID-19 crisis

The COVID-19 virus has spread to the entire world. It has caused a significant number of deaths and significantly changed the lives of the people who have been affected by it. Many countries have gone into lockdown to prevent the spread of the virus, thereby resulting in economic collapse within them. Many businesses have shut down, and schools have been closed. A lot of measures have been taken to reduce the effects of the virus and, expectedly, the scientific community is developing various technological methods that can benefit the society, amidst these trying times. For example, a framework for change was proposed for medical education [ 96 ], and ref. [ 97 ] discussed the monitoring of hospitals and clinics through technological methods. AR can be very useful for navigating life during the crisis. Sodar, an AR application that upholds social distancing by helping individuals maintain a distance of 2 m from other people, was launched by Google (Fig.  13 ). Such an application will prove to be very useful once the lockdown ends and people start to go outside again.

a Launch of sodar application; b 2-m radius being displayed after camera is pointed toward area

Case Western Reserve University, in collaboration with Cleveland Clinic, developed an AR app called HoloAnatomy, which helps medical students to learn about the human body in 3D. HoloAnatomy teaches students anatomy using Microsoft Hololens (Fig.  14 ). The students can learn about the smallest details in the human body without having to dissect cadavers. Such online educational AR apps can be extremely useful in the current COVID-19 crisis.

HoloAnatomy AR system (source: CWRU website)

AR can also be used to develop long-distance healthcare systems for managing the pain and wellbeing of patients suffering from chronic pain and health care issues due to the COVID-19 outbreak. Telemedicine and web-based systems are some existing prevalent approaches. Telemedicine refers to the short message services, video conferencing, and telephone consultation. Web-based systems, such as PAIN OUT in Europe and CHOIR in the United States [ 98 ], make it possible to review patients before appointments. However, they depend on customer inputs and lack functionality. AR involves a projection being mapped onto the physical world to improve perception and give acute vision to doctors to facilitate the speedy recovery of patients [ 99 ].

AR can also be beneficial in the surgical field, as well. Virtual technology is responsible for saving lives and safeguarding surgical practices during the pandemic; the Proxemie platform is one such example [ 100 ]. The Proxemie platform connects surgeons to a live environment through which experts can provide support to their colleagues and supervise procedures. Proxemie’s AR telehealth solution is used for conducting multidisciplinary meetings to assess patients. The platform also provides a surgical library that provides useful information on surgeries. Hence, it is an extremely essential platform and a suitable example of the usefulness of AR in the current pandemic.

However, a more significant challenge awaits the society at the end of the pandemic. The road to recovery from the pandemic will be extremely difficult. AR software and hardware shall be used to mitigate such effects even after the pandemic is long over. AR can be used to impact practical knowledge because the processes of learning and implementation. Using an AR headset, a skilled technician can seamlessly guide fellow workers and teach students. Companies can also train their workforce using AR, thereby improving their workflow and the economy. For example, Microsoft Hololens 2 AR headset can be used by companies to guide their employees (Fig.  15 ). It provides hands-free visual assistance and data, along with robust security and collaboration with other Microsoft apps. Companies that depend on on-site technical maintenance for their cash flow need AR solutions as well. AR-assisted service prevents physical contact and encourages social distancing, thereby satisfying the requirements of the present and the near future.

Dynamic 365 remote assist on Microsoft Hololens 2 AR headset

Another important area where AR can prove necessary is in retail, from the customer’s perspective. Whether online or in-store, people will never buy products without being sure if a particular product fits them. AR-based technologies, such as VFRs and mobile apps, can enable the customers to try out clothes, jewelry, makeup, sunglasses, or shoes, without actually trying any of these products in reality. Such AR-based solutions will help people practice social distancing. Digital and safe shopping experiences are among the current needs of customers. As explained previously regarding various AR systems used in retail, such applications can prove to be really useful in the coming days, thus boosting the AR market.

Challenges and future scope

Before AR can be accepted by everyone on a large scale, it is important to note that AR faces a large number of challenges that must be overcome for it to thrive. Every technology consists of a well-defined business model based on which investments are generated. However, for AR, there is no particularly defined business model that can work long-term. It is also very early to evaluate the profitability of an AR-based business because the technology is still in its development stages. Further, because of the lack of AR development and application design standards, AR technology faces a problem relating to compatibility with the overall scenario. Security and privacy are also major concerns in the AR industry. Poor content quality, in addition to some technical software and hardware limitations in each game design, is an ongoing challenge in AR gaming. For specific surgeries and in the medical field, accuracy is of prime importance because it is essential for surgeons to have tangible information on how and when the technology is used [ 101 ]. For fashion retail, scant research exists on AR, and its impact on the industry has not yet been realized significantly. Hence, many brands still hesitate to invest in AR.

Despite the numerous challenges, AR has an enormous scope in the near future to transform many industries. On overcoming the above mentioned obstacles, AR could have the power to revolutionize the entire market in every aspect. It has tremendous potential in areas, such as education, medicine, military, construction, automobile, travel, retail, art, and architecture [ 102 ]. AR is a futuristic technology that will change and reshape a number of business strategies developed by organizations. With increase in market competition, customers trust only companies who offer good quality products and extraordinary service. This means that many companies will prioritize incorporating AR, as it promises a personalized experience with products, which would attract more customers. It is also conjectured that the mobile AR technology, which will rise in the coming years, would lead to greater social acceptance. As many people are familiar with operating mobile phones, it would be easier for them to adapt to new technology. Further, as mentioned in the previous section, AR can be very helpful in the current COVID-19 crisis, as it would be, in similar situations that may arise in the future. Figure  16 represents the estimation of the projected AR/VR scenario in different sectors in 2025. However, this report was given by Goldman Sachs in 2016. Considering the present COVID-19 situation and the likely post-lockdown scenario, it appears that people will still be hesitant to use the entertainment, retail, or medical facilities freely. This necessitates the use of AR to provide a fully immersive experience to the customers in almost every field. Hence, it would be wise to say that the AR/VR estimation for 2025 could supersede Goldman Sachs’ 2016 prediction.

Estimated scenario of AR/VR in different sectors in 2025 (source: Goldman Sachs Global Investment Research, 2016)

AR provides unique entertainment options that are not available with common types of digital media. With new research, future AR systems are bound to be significantly more advanced, compared to the currently available ones. Owing to AR, interactivity and content quality are noticeably different, and personalization is possible. The technology is new, and despite having been around for a considerable amount of time, it has not been fully and functionally incorporated in day-to-day activities such as retail and medicine owing to concerns such as technology, social acceptance, and usability. However, upon overcoming these challenges, AR has the ability to redefine gaming through enhanced content in real time. The use of AR in medicine may change the way surgeries are performed. Medical training and post-surgical treatments can be performed with ease using AR displays. As consumers desire new innovations that may simplify shopping experiences and make them more comfortable, they are most likely to welcome AR with excitement. We have also studied the existing AR solutions that are being implemented and have discussed its importance to recovery from the pandemic. Hence, AR is playing a very important role in providing users with technology experience like never before in almost all areas.

The most recent inventions are proofs of the growing improvements in AR. AR in gaming can be seen in Pokémon Go, which also makes use of GPS, and is therefore a location-based application. Snapchat, on the other hand, is an example of a marker-based application, which uses image recognition in addition to AR. There are many AR-based software development kits and the factors determining the choice of an appropriate SDK include the cost, platforms, image recognition technology and the possibility of 3D tracking and recognition. Unity and AR toolkit are a few of the engines that can be used to create AR apps. Google and Android have also provided their respective kits, Google AR Core and AR Spark studio. The instances that have been discussed show the growing market base of AR systems and their importance in the market. Hence, the importance of a review that provides insight into three major fields where AR systems are being used cannot be overemphasized.

Availability of data and materials

All relevant data and material are presented in the main paper.

Abbreviations

• Augmented reality

Virtual reality

Head mounted displays

Head-up display

Global positioning system

Enhanced augmented reality

Personal augmented reality immersive system

Virtual fitting rooms

Integral video

Natural Orifice Transluminal Endoscopic Surgery

Arteriovenous malformations

Natural interaction

Red, Green, Blue

Hue, Saturation, value

Technology acceptance model

Shah G, Shah A, Shah M (2019) Panacea of challenges in real-world application of big data analytics in the healthcare sector. J Data Inf Manage 1(3):107–116. https://doi.org/10.1007/s42488-019-00010-1

Article   Google Scholar  

Pandya R, Nadiadwala S, Shah R, Shah M (2020) Buildout of methodology for meticulous diagnosis of K-complex in EEG for aiding the detection of Alzheimer's by artificial intelligence. Augment Hum Res 5(1):3. https://doi.org/10.1007/s41133-019-0021-6

Silva R, Oliveira JC, Giraldi GA (2003) Introduction to augmented reality. National Laboratory for Scientific Computation, Av Getulio Vargas, Petropolis

Kundalia K, Patel Y, Shah M (2020) Multi-label movie genre detection from a movie poster using knowledge transfer learning. Augment Hum Res 5(1):11. https://doi.org/10.1007/s41133-019-0029-y

Chavan SR (2016) Augmented reality vs. virtual reality: differences and similarities. https://www.semanticscholar.org/paper/Augmented-Reality-vs.-Virtual-Reality%3A-Differences-Chavan/7dda32ae482e926941c872990840d654f9e761ba . Accessed 18 Feb 2020.

Google Scholar  

Gandhi M, Kamdar J, Shah M (2020) Preprocessing of non-symmetrical images for edge detection. Augment Hum Res 5(1):10. https://doi.org/10.1007/s41133-019-0030-5

Patel D, Shah Y, Thakkar N, Shah K, Shah M (2020) Implementation of artificial intelligence techniques for cancer detection. Augment Hum Res 5(1):6. https://doi.org/10.1007/s41133-019-0024-3

Ahir K, Govani K, Gajera R, Shah M (2020) Application on virtual reality for enhanced education learning, military training and sports. Augment Hum Res 5(1):7. https://doi.org/10.1007/s41133-019-0025-2

Leena Lakra P, Verma P (2017) Augmented reality in marketing: role and applications. 8(11):74–81. https://www.academia.edu/36282179/Augmented_Reality_in_Marketing_Role_and_Applications . Accessed 18 Feb 2020.

Parekh V, Shah D, Shah M (2020) Fatigue detection using artificial intelligence framework. Augment Hum Res 5(1):5. https://doi.org/10.1007/s41133-019-0023-4

Razek ARA, van Husen C, Pallot M, Richir S (2018) A comparative study on conventional versus immersive service prototyping (VR, AR, MR). Paper presented at the virtual reality international conference - Laval virtual. ACM, Laval, April 2018. https://doi.org/10.1145/3234253.3234296

Lee K (2012) Augmented reality in education and training. TechTrends 56(2):13–21. https://doi.org/10.1007/s11528-012-0559-3

Hincapié M, Caponio A, Rios H, Mendívil EG (2011) An introduction to augmented reality with applications in aeronautical maintenance. Paper presented at the 13th international conference on transparent optical networks. IEEE, Stockholm, 26-30 June 2011. https://doi.org/10.1109/ICTON.2011.5970856

Livingston MA, Rosenblum LJ, Brown DG, Schmidt GS, Julier SJ, Baillot Y et al (2011) Military applications of augmented reality. In: Furht B (ed) Handbook of augmented reality. Springer, New York, pp 671–706. https://doi.org/10.1007/978-1-4614-0064-6_31

Chapter   Google Scholar  

Jani K, Chaudhuri M, Patel H, Shah M (2020) Machine learning in films: an approach towards automation in film censoring. J Data, Inf Manage 2(1):55–64. https://doi.org/10.1007/s42488-019-00016-9

Nayyar A, Mahapatra B, Le DN, Suseendran G (2018) Virtual reality (VR) & augmented reality (AR) technologies for tourism and hospitality industry. Int J Eng Technol 7(2):156–160. https://doi.org/10.14419/ijet.v7i2.21.11858

Nam TJ, Lee W (2003) Integrating hardware and software: augmented reality based prototyping method for digital products. In: Extended abstracts on human factors in computing systems. ACM, Lauderdale, pp 956–957

Wagner D, Schmalstieg D (2003) First steps towards handheld augmented reality. Paper presented at the 7th IEEE international symposium on wearable computers. IEEE, White Plains, 21-23 October 2003. https://doi.org/10.1109/ISWC.2003.1241402

Huang ZP, Li WK, Hui P, Peylo C (2014) CloudRidAR: a cloud-based architecture for mobile augmented reality. Paper presented at 2014 workshop on mobile augmented reality and robotic technology-based systems. ACM, Bretton Woods, June 16 2014. https://doi.org/10.1145/2609829.2609832

Rolland J, Hua H (2005) Head-mounted display systems. In: Johnson RB, Driggers RG (eds) Encyclopedia of optical engineering. Marcel Dekker, New York, pp 1–14

Butterworth J, Davidson A, Hench S, Olano MT (1992) 3DM: a three dimensional modeler using a head-mounted display. Paper presented at the 1992 symposium on interactive 3D graphics. ACM, Cambridge, June 1992. https://doi.org/10.1145/147156.147182

Harada T, Furuya Y (2013) Head-up display device. US Patent 7,528,798, 16 May 2005

Bimber O, Raskar R (2005) Spatial augmented reality. Paper presented at the 3rd IEEE/ACM international symposium on mixed and augmented reality, IEEE, Arlington, 5 November 2004. http://pages.cs.wisc.edu/~dyer/cs534/papers/SAR.pdf

Talaviya T, Shah D, Patel N, Yagnik H, Shah M (2020) Implementation of artificial intelligence in agriculture for optimisation of irrigation and application of pesticides and herbicides. Artif Intell Agric 4:58–73. https://doi.org/10.1016/j.aiia.2020.04.002

Jha K, Doshi A, Patel P, Shah M (2019) A comprehensive review on automation in agriculture using artificial intelligence. Artif Intell Agric 2:1–12. https://doi.org/10.1016/j.aiia.2019.05.004

Von Itzstein GS, Billinghurst M, Smith RT, Thomas BH (2017) Augmented reality entertainment: taking gaming out of the box. In: Lee N (ed) Encyclopedia of computer graphics and games. Springer, Cham, pp 1–9. https://doi.org/10.1007/978-3-319-08234-9_81-1

Kakkad V, Patel M, Shah M (2019) Biometric authentication and image encryption for image security in cloud framework. Multiscale Multidiscip Model Exp Des 2(4):233–248. https://doi.org/10.1007/s41939-019-00049-y

Patel H, Prajapati D, Mahida D, Shah M (2020) Transforming petroleum downstream sector through big data: a holistic review. J Petrol Explor Prod Technol 10(6):2601–2611. https://doi.org/10.1007/s13202-020-00889-2

Azuma R, Baillot Y, Behringer R, Feiner S, Julier S, MacIntyre B (2001) Recent advances in augmented reality. IEEE Comput Graph Appl 21(6):34–47. https://doi.org/10.1109/38.963459

Sutherland IE (1968) A head-mounted three dimensional display. Paper presented at the fall joint computer conference. ACM, San Francisco December 9-11, 1968. https://doi.org/10.1145/1476589.1476686

Herbst I, Braun AK, McCall R, Broll W (2008) TimeWarp: interactive time travel with a mobile mixed reality game. Paper presented at the 10th international conference on human computer interaction with mobile devices and services. ACM, Amsterdam, 2-5 September 2008. https://doi.org/10.1145/1409240.1409266

Nilsen T, Linton S, Looser J (2004) Motivations for AR gaming. In: Proc fuse new Zealand game developers conf, Dunedin, New Zealand, 26-29 June 2004, pp 86–93 https://dl.acm.org/doi/10.1109/ISMAR.2005.22

Magerkurth C, Cheok AD, Mandryk RL, Nilsen T (2005) Pervasive games: bringing computer entertainment back to the real world. Comput Entertain 3(3):4. https://doi.org/10.1145/1077246.1077257

Abowd GD, Mynatt ED (2000) Charting past, present, and future research in ubiquitous computing. ACM Trans Comput-Hum Int 7(1):29–58. https://doi.org/10.1145/344949.344988

Szalavári Z, Schmalstieg D, Fuhrmann A, Gervautz M (1998) “Studierstube”: an environment for collaboration in augmented reality. Virtual Real 3(1):37–48. https://doi.org/10.1007/BF01409796

Liarokapis F, Macan L, Malone G, Rebolledo-Mendez G, de Freitas S (2009) Multimodal augmented reality tangible gaming. Vis Comput 25(12):1109–1120. https://doi.org/10.1007/s00371-009-0388-3

Winkler T, Ide M, Herczeg M (2008) Mobile co-operative game-based learning with moles. Paper presented at SITE 2008, society for information technology & teacher education international conference. AACE, Chesapeak, March 2008

Schmitz B, Klemke R, Specht M (2012) An analysis of the educational potential of augmented reality games for learning. Paper presented at the 11th world conference on mobile and contextual learning. mLearn 2012, Helsinki, October 16-18 2012

Fotaris P, Pellas N, Kazanidis I, Smith P (2017) A systematic review of augmented reality game-based applications in primary education. Paper presented at the 11th European conference on games based learning. The FH JOANNEUM University of Applied Science, Graz, 4-5 October 2017

Yan DT, Hu HS (2017) Application of augmented reality and robotic technology in broadcasting: a survey. Robotics 6(3):18. https://doi.org/10.3390/robotics6030018

Mahmood Z, Ali T, Muhammad N, Bibi N, Shahzad I, Azmat S (2017) EAR: enhanced augmented reality system for sports entertainment applications. KSII Trans Int Inform Syst 11(12):6069–6091. https://doi.org/10.3837/tiis.2017.12.021

Cha J, Ryu J, Kim S, Eom S, Ahn B (2004) Haptic interaction in realistic multimedia broadcasting. Paper presented at the 5th pacific rim conference on advances in multimedia information processing. ACM, Tokyo, November 30-December 3 2004. https://doi.org/10.1007/978-3-540-30543-9_61

Lee SO, Ahn SC, Hwang JI, Kim HG (2011) A vision-based mobile augmented reality system for baseball games. In: Shumaker R (ed) Virtual and mixed reality - new trends. International conference, virtual and mixed reality 2011, held as part of HCI international 2011. Lecture notes in computer science, vol 6773. Springer, Berlin, Heidelberg

Ulbricht C, Schmalstieg D (2003) Tangible augmented reality for computer games. Paper presented at the 3rd IASTED international conference on visualization, imaging and image processing. ACTA, Benalmádena, 8-10 September 2003

Rohs M (2007) Marker-based embodied interaction for handheld augmented reality games. J Virtual Real Broadcast 4(5):618–639

Lyons K, Gandy M, Starner T (2000) Guided by voices: an audio augmented reality system. Paper presented at the International conference on auditory display. Atlanta, April 2-5 2000

Linaza MT, Gutierrez A, García A (2013) Pervasive augmented reality games to experience tourism destinations. In: Xiang Z, Tussyadiah I (eds) Information and communication technologies in tourism 2014. Springer, Cham, pp 497–509. https://doi.org/10.1007/978-3-319-03973-2_36

Chandaria J, Thomas GA, Stricker D (2007) The MATRIS project: real-time markerless camera tracking for augmented reality and broadcast applications. J Real-Time Image Process 2(2–3):69–79. https://doi.org/10.1007/s11554-007-0043-z

Han JG, Farin D, de With PHN (2007) A real-time augmented-reality system for sports broadcast video enhancement. Paper presented at the 15th international conference on multimedia. ACM, Augsburg, September 2007. https://doi.org/10.1145/1291233.1291306

Boyle E, Curran T, Demiris A, Klein K, Garcia C, Malerczyk C et al (2002) The creation of mpeg-4 content and its delivery over DVB infrastructure, pp 1–3

Kim S, Choi B, Jeong Y, Hong J, Chung J (2012) An architecture of augmented broadcasting service for next generation smart TV. Paper presented at the IEEE international symposium on broadband multimedia systems and broadcasting. IEEE, Seoul, June 27-29 2012. https://doi.org/10.1109/BMSB.2012.6264289

Sukhadia A, Upadhyay K, Gundeti M, Shah S, Shah M (2020) Optimization of smart traffic governance system using artificial intelligence. Augment Hum Res 5(1):13. https://doi.org/10.1007/s41133-020-00035-x

Shah K, Patel H, Sanghvi D, Shah M (2020) A comparative analysis of logistic regression, random forest and KNN models for the text classification. Augment Hum Res 5(1):12. https://doi.org/10.1007/s41133-020-00032-0

Shah D, Dixit R, Shah A, Shah P, Shah M (2020) A comprehensive analysis regarding several breakthroughs based on computer intelligence targeting various syndromes. Augment Hum Res 5(1):14. https://doi.org/10.1007/s41133-020-00033-z

Carmigniani J, Furht B (2011) Augmented reality: an overview. In: Furht B (ed) Handbook of augmented reality. Springer, New York. https://doi.org/10.1007/978-1-4614-0064-6_1

Patel D, Shah D, Shah M (2020) The intertwine of brain and body: a quantitative analysis on how big data influences the system of sports. Ann Data Sci 7(1):1–16. https://doi.org/10.1007/s40745-019-00239-y

Panchiwala S, Shah M (2020) Comprehensive study on critical security issues and challenges of the IoT world. J Data, Inf Manage. https://doi.org/10.1007/s42488-020-00030-2

Moro C, Štromberga Z, Raikos A, Stirling A (2017) The effectiveness of virtual and augmented reality in health sciences and medical anatomy. Anat Sci Educ 10(6):549–559. https://doi.org/10.1002/ase.1696

Shuhaiber JH (2004) Augmented reality in surgery. Arch Surg 139(2):170–174. https://doi.org/10.1001/archsurg.139.2.170

Barsom EZ, Graafland M, Schijven MP (2016) Systematic review on the effectiveness of augmented reality applications in medical training. Surg Endosc 30(10):4174–4183. https://doi.org/10.1007/s00464-016-4800-6

Sielhorst T, Feuerstein M, Navab N (2008) Advanced medical displays: a literature review of augmented reality. J Disp Technol 4(4):451–467. https://doi.org/10.1109/JDT.2008.2001575

Khor WS, Baker B, Amin K, Chan A, Patel K, Wong J (2016) Augmented and virtual reality in surgery-the digital surgical environment: applications, limitations and legal pitfalls. Ann Transl Med 4(23):454. https://doi.org/10.21037/atm.2016.12.23

Berger JW, Leventon ME, Kikinis R (1999) Technique for creating an ophthalmic augmented reality environment. US patent 5,912,720, 2 December 1998

Fischer J, Neff M, Freudenstein D, Bartz D (2004) Medical augmented reality based on commercial image guided surgery. Paper presented at the 10th Eurographics conference on virtual environments (EGVE). ACM, Grenoble, June 2004

Birkfellner W, Figl M, Huber K, Watzinger F, Wanschitz F, Hummel J et al (2002) A head-mounted operating binocular for augmented reality visualization in medicine - design and initial evaluation. IEEE Trans Med Imaging 21(8):991–997. https://doi.org/10.1109/TMI.2002.803099

Article   MATH   Google Scholar  

Scharver C, Evenhouse R, Johnson A, Leigh J (2004) Designing cranial implants in a haptic augmented reality environment. Commun ACM 47(8):32–38. https://doi.org/10.1145/1012037.1012059

Nicolau SA, Pennec X, Soler L, Buy X, Gangi A, Ayache N et al (2009) An augmented reality system for liver thermal ablation: design and evaluation on clinical cases. Med Image Anal 13(3):494–506. https://doi.org/10.1016/j.media.2009.02.003

Hondori HM, Khademi M, Dodakian L, Cramer SC, Lopes CV (2013) A spatial augmented reality rehab system for post-stroke hand rehabilitation. Stud Health Technol Inform 184:279–285

Nijmeh AD, Goodger NM, Hawkes D, Edwards PJ, McGurk M (2005) Image-guided navigation in oral and maxillofacial surgery. Br J Oral Maxillofac Surg 43(4):294–302. https://doi.org/10.1016/j.bjoms.2004.11.018

Tran HH, Suenaga H, Kuwana K, Masamune K, Dohi T, Nakajima S et al (2011) Augmented reality system for oral surgery using 3D auto stereoscopic visualization. Paper presented at the 14th international conference on medical image computing and computer-assisted intervention. ACM, Toronto, September 2011. https://doi.org/10.1007/978-3-642-23623-5_11

Kersten-Oertel M, Gerard I, Drouin S, Mok K, Sirhan D, Sinclair DS et al (2015) Augmented reality in neurovascular surgery: feasibility and first uses in the operating room. Int J Comput Assist Radiol Surg 10(11):1823–1836. https://doi.org/10.1007/s11548-015-1163-8

Bornik A, Beichel R, Reitinger B, Gotschuli G, Sorantin E, Leberl FW et al (2003) Computer-aided liver surgery planning: an augmented reality approach. Paper presented at the SPIE 5029, medical imaging 2003: visualization, image-guided procedures, and display. SPIE, San Diego, 15 February 2003. https://doi.org/10.1117/12.479743

Tonet O, Megali G, D'Attanasio S, Dario P, Carrozza MC, Marcacci M et al (2000) An augmented reality navigation system for computer assisted arthroscopic surgery of the knee. In: Delp SL, AM DG, Jaramaz B (eds) Medical image computing and computer-assisted intervention - MICCAI 2000. 3rd international conference, Pittsburgh, PA, USA, October 11–14 2000. Lecture notes in computer science (Lecture notes in computer science), vol 1935. Springer, Berlin. https://doi.org/10.1007/978-3-540-40899-4_121

Vosburgh KG, Estépar RSJ (2007) Natural orifice Transluminal endoscopic surgery (NOTES): an opportunity for augmented reality guidance. Stud Health Technol Inform 125:485–490

Soler L, Nicolau S, Schmid J, Koehl C, Marescaux J, Pennec X et al (2004) Virtual reality and augmented reality in digestive surgery. Paper presented at the 3rd IEEE and ACM international symposium on mixed and augmented reality. IEEE, Arlington, 24 January 2005. https://doi.org/10.1109/ISMAR.2004.64

Cabrilo I, Bijlenga P, Schaller K (2014) Augmented reality in the surgery of cerebral arteriovenous malformations: technique assessment and considerations. Acta Neurochir 156(9):1769–1774. https://doi.org/10.1007/s00701-014-2183-9

Scholz J, Smith AN (2016) Augmented reality: designing immersive experiences that maximize consumer engagement. Bus Horiz 59(2):149–161. https://doi.org/10.1016/j.bushor.2015.10.003

Poushneh A (2018) Augmented reality in retail: a trade-off between user's control of access to personal information and augmentation quality. J Retailing Consum Serv 41:169–176. https://doi.org/10.1016/j.jretconser.2017.12.010

Alimamy S, Deans KR, Gnoth J (2017) Augmented reality: uses and future considerations in marketing. In: Benlamri R, Sparer M (eds) Leadership, innovation and entrepreneurship as driving forces of the global economy. Springer, Cham. https://doi.org/10.1007/978-3-319-43434-6_62

Daiker M, Ariyachandra T, Frolick M (2017) The influence of augmented reality on retail pricing. Iss Inform Syst 18(4):116–123 https://iacis.org/iis/2017/4_iis_2017_116-123.pdf

Wakim RS, Drak Al Sebai L, Miladinovic M, Ozturkcan S (2018) A study of Swedish eyewear retailer's smartphone-based augmented reality application. In: Çebi F, Bozbura FT, Gözlü S (eds) Engineering and technology management summit 2018: engineering and technology management in the smart age. Istanbul Technical University, Istanbul

Pachoulakis I, Kapetanakis K (2012) Augmented reality platforms for virtual fitting rooms. Int J Multimed Appl 4(4):35–46. https://doi.org/10.5121/ijma.2012.4404

Gunjal K, Patil P, Phalle A, Kanade AV (2017) A survey on virtual changing room using augmented reality. Int J Adv Res Comput Eng Technol 6(10):1545–1547

Flavián C, Ibáñez-Sánchez S, Orús C (2019) The impact of virtual, augmented and mixed reality technologies on the customer experience. J Bus Res 100:547–560. https://doi.org/10.1016/j.jbusres.2018.10.050

Watson A, Alexander B, Salavati L (2018) The impact of experiential augmented reality applications on fashion purchase intention. Int J Retail Distrib Manage 48(5):433–451. https://doi.org/10.1108/IJRDM-06-2017-0117

McCormick H, Cartwright J, Perry P, Barnes L, Lynch S, Ball G (2014) Fashion retailing - past, present and future. Textile Prog 46(3):227–321. https://doi.org/10.1080/00405167.2014.973247

Boardman R, Henninger CE, Zhu AL (2019) Augmented reality and virtual reality: new drivers for fashion retail? In: Vignali G, Reid LF, Ryding D, Henninger CE (eds) Technology-driven sustainability. Palgrave Macmillan, Cham, pp 155–172

Bonetti F, Warnaby G, Quinn L (2017) Augmented reality and virtual reality in physical and online retailing: a review, synthesis and research agenda. In: Jung T, Tom Dieck MC (eds) Augmented reality and virtual reality: empowering human, place and business. Springer, Cham, pp 119–132. https://doi.org/10.1007/978-3-319-64027-3_9

Bonetti F, Pantano E, Warnaby G, Quinn L, Perry P (2019) Augmented reality in real stores: empirical evidence from consumers’ interaction with AR in a retail format. In: Tom Dieck MC, Jung T (eds) Augmented reality and virtual reality: the power of AR and VR for business. Springer, Cham, pp 3–16. https://doi.org/10.1007/978-3-030-06246-0_1

Dacko SG (2017) Enabling smart retail settings via mobile augmented reality shopping apps. Technol Forecasting Soc Change 124:243–256. https://doi.org/10.1016/j.techfore.2016.09.032

Richter T, Raška K (2017) Influence of augmented reality on purchase intention: the IKEA case. Jönköping University http://hj.diva-portal.org/smash/record.jsf?pid=diva2%3A1115470&dswid=-5433 . Accessed 18 Feb 2020

Spreer P, Kallweit K (2014) Augmented reality in retail: assessing the acceptance and potential for multimedia product presentation at the PoS. Trans Marketing Res 1(1):21–25.

Presle P. (2012) A virtual dressing room based on depth data. Master's thesis, Institut für Softwaretechnik und Interaktive Systeme

Poushneh A, Vasquez-Parraga AZ (2017) Customer dissatisfaction and satisfaction with augmented reality in shopping and entertainment. J Consum Satisf Dissatisfaction Complain Behav 30:97–118

Chang HT, Li YW, Chen HT, Feng SY, Chien TT (2013) A dynamic fitting room based on microsoft kinect and augmented reality technologies. Paper presented at the 15th international conference on human-computer interaction: interaction modalities and techniques. ACM, Las Vegas, 21-23 July 2013. https://doi.org/10.1007/978-3-642-39330-3_19

Goh P, Sandars J (2020) A vision of the use of technology in medical education after the COVID-19 pandemic. MededPublish, pp 1–8. https://doi.org/10.15694/mep.2020.000049.1

Ting DSW, Carin L, Dzau V, Wong TY (2020) Digital technology and COVID-19. Nat Med 26(4):459–461. https://doi.org/10.1038/s41591-020-0824-5

Meterko M, Restuccia JD, Stolzmann K, Mohr D, Brennan C, Glasgow J et al (2015) Response rates, nonresponse bias, and data quality: results from a national survey of senior healthcare leaders. Public Opin Quart 79(1):130–144. https://doi.org/10.1093/poq/nfu052

Eccleston C, Blyth FM, Dear BF, Fisher EA, Keefe FJ, Lynch ME et al (2020) Managing patients with chronic pain during the COVID-19 outbreak: considerations for the rapid introduction of remotely supported (eHealth) pain management services. PAIN 161(5):889–893. https://doi.org/10.1097/j.pain.0000000000001885

Singh RP, Javaid M, Kataria R, Tyagi M, Haleem A, Suman R (2020) Significant applications of virtual reality for COVID-19 pandemic. Diabetes Metab Syndr 14(4):661–664. https://doi.org/10.1016/j.dsx.2020.05.011

Mekni M, Lemieux A (2014) Augmented reality: applications, challenges and future trends. Appl Comput Sci:205–214

Agarwal C, Thakur N (2014) The evolution and future scope of augmented reality. Int J Comput Sci Iss 11(6):59–66

Download references

Acknowledgements

The authors are grateful to Nirma University and School of Technology, Pandit Deendayal Petroleum University for the permission to publish this research.

Not applicable.

Author information

Authors and affiliations.

Department of Computer Engineering, Nirma University, Ahmedabad, Gujarat, 382481, India

Pranav Parekh, Shireen Patel & Nivedita Patel

Department of Chemical Engineering, School of Technology, Pandit Deendayal Petroleum University, Gandhinagar, Gujarat, India

You can also search for this author in PubMed   Google Scholar

Contributions

All the authors make substantial contribution in this manuscript. NP, SP, PP and MS participated in drafting the manuscript. NP, SP and PP wrote the main manuscript, all the authors discussed the results and implication on the manuscript at all stages. The author(s) read and approved the final manuscript.

Corresponding author

Correspondence to Manan Shah .

Ethics declarations

Competing interests.

The authors declare that they have no competing interests.

Additional information

Publisher’s note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/ .

Reprints and permissions

About this article

Cite this article.

Parekh, P., Patel, S., Patel, N. et al. Systematic review and meta-analysis of augmented reality in medicine, retail, and games. Vis. Comput. Ind. Biomed. Art 3 , 21 (2020). https://doi.org/10.1186/s42492-020-00057-7

Download citation

Received : 19 February 2020

Accepted : 28 August 2020

Published : 16 September 2020

DOI : https://doi.org/10.1186/s42492-020-00057-7

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

REVIEW article

Augmented reality marketing: a systematic literature review and an agenda for future inquiry.

• Business School of Sport, Beijing Sport University, Beijing, China

Augmented reality (AR) is a potentially disruptive technology that enriches the consumer experience and transforms marketing. With the surging popularity of AR in marketing practice, academic efforts to investigate its effects on consumer experience, response, and behavior have increased significantly. To obtain an integrated and comprehensive view of the front-line in AR marketing research and identify the gaps for future research, we analyze the existing AR marketing literature through a systematic literature review. Using 99 journal articles selected from the Web of Science core collections, this research sheds light on the general characteristics such as publication year, publication outlet, research design, and research method. Moreover, this research also gains insight into the AR marketing relevant factors such as application area, application context, AR type, and theoretical lenses. The findings of the analyses reveal the state-of-the-art of scholarly publications on AR marketing research. First, the number of journal articles on AR marketing increased rapidly in the past few years, and the journals that published articles on AR marketing cover a wide range of disciplines. Second, the empirical studies in most literature adopted the quantitative research design and used survey or experiment methods. Third, the studies in more than half of the journal articles used mobile AR applications in various online contexts. Fourth, the Technology Acceptance Model (TAM) and the Stimulus-Organism-Response (S-O-R) framework are the two most widely used theoretical lenses used in the literature. After that, the major application areas of AR in marketing are retail, tourism, and advertising. To identify the focal themes discussed in the three application areas, this research summarizes the studies by the outcome variables. Specifically, the outcome variables have five categories: technology-related, product-related, brand-related, tourist destination-related, and advertisement-related. Finally, this research proposes the agenda for future academic efforts in AR marketing.

Introduction

Augmented reality (AR) is an emerging cutting-edge technology in marketing, It enhances the visual, auditory, tactile, and olfactory perception of users by augmenting or superimposing digital content such as text, geolocation information, graphics, audios, and videos onto a live view of the physical objects and environments in real-time ( Carmigniani et al., 2011 ; Fan et al., 2020 ; Sung, 2021 ). AR establishes a closer relationship between users’ physical space and virtual objects. Therefore, the user experience with AR is more immersive, more vivid, more interactive, and more realistic ( Cipresso et al., 2018 ). With the popularity of mobile devices and the availability of high-speed wireless networks, an increasing number of web-based AR applications and mobile AR apps have emerged to create novel, immersive, enjoyable, informative, and valuable user experiences. Accordingly, AR is becoming a disruptive technology that will transform marketing in the coming years ( Tan et al., 2022 ). An industry report released by PwC claimed that AR brought net economic benefits of $33 billion in 2019. Furthermore, the benefits will reach $338.1 billion by 2025 and $1.0924 trillion by 2030 ( PwC, 2019 ).

The surging popularity of AR in marketing practice has attracted more and more academic efforts to investigate its effects on consumer experience, response, and behavior ( Rauschnabel et al., 2022 ). This growing interest in AR marketing calls for a synthesis of the existing literature to offer guidance for future research. However, as scholarly investigations on AR marketing are still in the infant stage, the extant literature on AR marketing is fragmented. In this regard, we analyze AR marketing literature through a systematic literature review to obtain an integrated and comprehensive view of the state-of-the-art of AR marketing research and identify the gaps for future research. Specifically, this research sheds light on the generic characteristics of the literature, such as publication year, publication outlet, research design, and research method. In addition, this research also gains insight into the factors specific to AR marketing, such as application area, application context, AR type, and theoretical lenses. We also identify the focal themes in each application area according to the outcome variables to illustrate the current status of scholarly investigation. Moreover, we propose the agenda for future research.

This systematic review differs from existing literature reviews in four ways. First, this review conducts an extensive examination of AR marketing literature. We initially identified 442 journal articles from 200 journals for manual evaluation. These journal articles cover a publication period from 2000 to 2021. After assessing the details following the guidelines of a systematic review methodology, we have 99 journal articles in the final set for analysis. Second, this study adopts a systematic review approach, thus allowing better synthesis and integration. It can help AR marketing researchers better understand existing findings and identify potential topics for future research. Third, this research gains insight into the factors specific to AR marketing, such as application areas, application context, AR type, and theoretical lenses, and summarizes the literature in terms of these factors. Finally, this research identifies and categorizes the AR marketing literature by its application areas, which offers a new perspective to gain insight into the state-of-the-art of AR marketing research.

The remainder of this paper is organized as follows. First, we present the concept of AR and discuss its application in marketing. First, we present the concept of AR and discuss its applications in marketing. Second, we explain the methodology used in the literature search and select the journal articles reviewed in this study. Third, we summarize the journal articles for final analysis in terms of the general factors (e.g., publication year, publication journal, research design, and research method) and the AR relevant factors (e.g., application area, AR type, application context, and theoretical lenses). Fourth, we analyze the focal themes in the three application areas of retail, tourism, and advertising. Lastly, we present the contributions and concluding remarks, future research agenda, and limitations of this research.

Augmented Reality and Augmented Reality Marketing

Augmented reality.

Augmented reality originated from Morton Heilig’s bold and innovative idea that cinema needed to draw viewers into the onscreen activities by effectively taking in all senses ( Carmigniani et al., 2011 ). Although we can track the history of AR back to the 1950s, the way of AR from laboratories to the industry has taken more than half a century. The exposure of AR to a mass audience has not realized until the explosive popularity of Pokémon GO in 2016, which provided both the social and fashionable acceptance for the success of AR in the market ( Rauschnabel et al., 2017 ).

Augmented reality is built on computer vision and object recognition technologies. It enhances consumer experiences by augmenting or superimposing digital content (e.g., text, geolocation information, graphics, audio, and videos onto) a live view of the physical objects and environments (e.g., consumers’ faces, bodies, and surroundings) in real-time ( Sung, 2021 ). The discussions of AR and Virtual Reality (VR) usually connect closely. Compared with traditional media, AR and VR aim to provide users with enriched, interactive, and immersive media experiences ( Yim et al., 2017 ). While VR creates a fully computer-generated virtual environment, AR enriches the real environment by integrating context-aware digital information ( Huang and Liao, 2015 ; Yim et al., 2017 ).

A typical AR system consists of three components: a geospatial datum for the virtual object, a surface to project virtual elements, and an image processing unit ( Carmigniani et al., 2011 ). Early AR systems have limited applications in business practices. They need to be built on dedicated devices such as smart glasses (e.g., HoloLens Magic and Google Glass) ( Poushneh, 2018 ), somatosensory devices (e.g., Kinect) ( Huang and Liao, 2017 ; Huang, 2021 ), or fixed devices (e.g., PC and its connected webcam) and smart mirror ( Rese et al., 2017 ; Baek et al., 2018 ). Recently, with the prevalence of personal mobile devices (e.g., smartphones and tablets) and the availability of high-speed wireless networks, the application of AR has proliferated in a variety of fields such as education ( Wu et al., 2013 ), manufacturing ( Nee et al., 2012 ), healthcare ( Ferrari et al., 2019 ), and marketing ( Tan et al., 2022 ).

Augmented Reality Marketing

Augmented reality marketing refers to the application of AR in marketing to enhance consumers’ experiences, increase their satisfaction, shape their behavior, and boost companies’ revenues ( Huang and Liao, 2015 ; Javornik, 2016 ; Poushneh and Vasquez-Parraga, 2017 ; Bell et al., 2018 ). The novel and attractive media of presentation and interaction enabled by AR play a crucial role in achieving the desired effects. Specifically, AR integrates digital information or objects into consumers’ perceptions of the physical objects and environments, thus providing consumers with rich information about products or services and allowing them to experience products and services easily. Specifically, AR not only improves online experiences and engagement but creates novel and fantastic on-site experiences ( Javornik, 2016 ; Yuan et al., 2021 ).

First, AR engages consumers in online settings by providing real-time direct product/service experiences in various aspects of marketing ( Chung et al., 2018 ). Specifically, it overcomes the limitations of online shopping by allowing prospects to try on products, such as makeup ( Smink et al., 2019 ; Hsu et al., 2021 ; Javornik et al., 2021 ), eyewear ( Pantano et al., 2017 ; Yim et al., 2017 ; Yim and Park, 2019 ), clothing ( Huang and Liu, 2014 ; Huang and Liao, 2017 ; Plotkina and Saurel, 2019 ), shoes ( Hilken et al., 2018 ; Plotkina et al., 2021 ), and furniture ( Rauschnabel et al., 2019 ; Kowalczuk et al., 2021 ; Qin et al., 2021b ) virtually without having to interact physically with them. Major online retailing platforms, such as Amazon ( McLean and Wilson, 2019 ), JingDong ( Fan et al., 2020 ), Alibaba ( Fan et al., 2020 ), and eBay ( Banerjee and Longstreet, 2016 ), as well as leading brands, such as Tiffany & Co. ( Whang et al., 2021 ), L’Oréal ( Hilken et al., 2017 ), Sephora ( Smink et al., 2019 ), Nike ( Hilken et al., 2018 ), Converse ( Whang et al., 2021 ), Zara ( Yuan et al., 2021 ), IKEA ( McLean and Wilson, 2019 ; Qin et al., 2021b ), Mini ( Carmigniani et al., 2011 ), and Lego ( Hinsch et al., 2020 ), have devoted lots of efforts to introduce various forms of AR. They strive to enhance consumers’ vicarious experience of physical products in online settings and make it more immersive, interactive, informative, enjoyable, and satisfactory ( Yim et al., 2017 ). Furthermore, AR advertising has significant advantages over traditional advertising. AR empowered advertisements are more informative, novel, entertaining, and complex, which leads to positive consumer responses and helps advertising campaigns stand out ( Feng and Xie, 2018 ; Yang et al., 2020 ; Sung, 2021 ).

Second, AR offers a novel and fantastic on-site experience ( Barhorst et al., 2021 ). The application of AR creates augmented stores ( Bonetti et al., 2019 ), restaurants ( Heller et al., 2019a ; Batat, 2021 ), museums ( tom Dieck et al., 2016 ; He et al., 2018 ; Zhuang et al., 2021 ), and art galleries ( tom Dieck et al., 2018b ; Tussyadiah et al., 2018 ). Retail giants, such as Lowes ( Chalimov, 2021 ) and Machine-A ( Chitrakorn, 2021 ), engage consumers and offer interaction by incorporating AR-supported features into their mobile apps and serving consumers in innovative ways. Furthermore, both established and novel brands, such as Kate Spade, Charlotte Tilbury, Timberland, Lily, Philip, Lego, and Toys-R-Us, offer consumers a plethora of interactive experiences. The interactive experiences include learning more about products, creating unique and customizable products, and virtually trying on products by installing in-store AR displays or adding AR empowered features to the brand’s mobile apps ( Chalimov, 2021 ). AR augmented stores can produce extra brand value, simplify consumers’ decision-making process, stimulate brand engagement, and lead to stronger consumer purchase desire ( Bonetti et al., 2019 ; Cuomo et al., 2020 ). AR-empowered restaurant services affect consumers’ perceptions of restaurant experiences ( Batat, 2021 ) and promote the choice of high-value products ( Heller et al., 2019a ). Moreover, augmented reality applications, especially those built upon wearable devices, affect tourists’ destination visit intention ( Chung et al., 2015 ). They can also help tourists feel more enjoyable ( Tussyadiah et al., 2018 ), enhance their experiences with tourist destinations ( tom Dieck et al., 2018a ; Jiang et al., 2019 ), and increase their willingness to pay more ( He et al., 2018 ).

Methodology

This research adopts the systematic literature review approach to avoid the well-known limitations of literature selection in narrative reviews and expert reviews ( Tranfield et al., 2003 ; Kitchenham et al., 2009 ) and synthesize the existing research findings in a transparent and reproducible way ( Snyder, 2019 ). Following the guidelines for the systematic review approach ( Webster and Watson, 2002 ; Denyer and Tranfield, 2009 ; Paul and Criado, 2020 ), we conducted a review of AR marketing to identify relevant themes for this field. The guidelines suggest five steps for producing a systematic review that is both reproducible and transparent ( Snyder, 2019 ). The five steps include question formulation, study location, study selection and evaluation, analysis and synthesis, and results reporting and using ( Denyer and Tranfield, 2009 ).

Question Formulation

Question formulation is crucial for a well-conducted systematic review. To obtain a deeper understanding of the AR marketing literature, we conduct a pilot search in the first stage. Based on the findings of the pilot search, we establish the research scope, formulate research questions, and clarify the inclusion and exclusion criteria. The pilot search leads us to the central questions of this research: what are the roles of AR in marketing? and how does AR contribute to marketing? Specifically, we propose four research questions: (RQ1) How is AR marketing defined in the literature? (RQ2) What are the characteristics of the AR marketing literature? (RQ3) What are the major application areas investigated by the AR marketing literature? and (RQ4) What are the focal themes examined in each application area?

Study Location

We perform the first search with the term “augmented reality” in the Web of Science (WOS) core collections, which return 9,145 journal articles on the subject of AR. However, most papers come from WOS categories such as Computer Science, Engineering, Medical, Education, and other fields not related to marketing. Therefore, we perform a second search in the WOS core collections using the following query applied to the title, abstract, and keywords: [“augmented reality” AND (marketing OR consumer OR customer)]. As this research focuses on the most relevant studies of AR marketing, we keep the papers from the five WOS categories, such as Business, Management, Hospitality Leisure Sport Tourism, Computer Science Information Systems, and Computer Science Interdisciplinary Applications ( Alves et al., 2016 ). Furthermore, we limit the publication year to “1990–2021,” language to English, and document type to Article. From the query, we get 341 journal articles.

To ensure that no major AR marketing articles are ignored in the analysis, we use a “snowball” technique. Specifically, we review citations from the key studies in the 341 journal articles retrieved in the previous search and identify more keywords related to AR marketing. After that, we perform the third search using the following query applied to the title, abstract, and keywords: [“augmented reality” AND (marketing OR consumer OR customer OR retail* OR advertis* OR brand* OR touris*)]. There are multiple possible terms under the root word. Hence, some of the words in the query are followed by a wildcard. Meanwhile, we also limit the five WOS categories in the previous search, the publication year to “1990–2021,” language to “English,” and document type to “Article.” From the query, we obtain 442 journal articles.

Study Selection and Evaluation

For the 442 journal articles, we conduct manual screening of the titles, keywords, abstracts, and text under the following three inclusion and exclusion criteria:

(1) The study should focus on AR. Thus, we not only exclude the journal articles that discuss VR, XR, AI, and emerging innovative techniques but their combinations with AR.

(2) We focus on AR applications in the marketing context. Therefore, we exclude the articles that discuss the technical details of AR and AR-empowered systems and the application of AR in the non-marketing context.

(3) We aim to shed light on the findings of empirical studies. Hence, we exclude the conceptual papers and review papers.

After removing 343 journal articles, there are 99 journal articles included in the final set for further content analysis. During the screening process, two authors first identify potentially relevant articles independently. Then, we discuss the conflicts to obtain the journal articles for final analysis so that the agreement (Cohen’s Kappa coefficient) is larger than 0.85. Figure 1 presents the process used to select the journal articles for final analysis.

Figure 1. The process of selecting the journal articles for final analysis.

Analysis and Synthesis

The authors manually develop a data extraction process to report the main characteristics of the journal articles, such as publication year, publication journal, application areas, application context, AR type, research design, research method, and theoretical lenses. Then, two authors independently code the selected journal articles in themes, which offer us a more comprehensive understanding.

Descriptive Analysis

In this section, we summarize the 99 journal articles on AR marketing by examining the general characteristics such as publication year, publication outlet, research design, and research method, as well as the AR relevant characteristics such as application area, application context, AR type, and theoretical lenses.

Publication Year

As presented in Figure 2 , the number of journal articles published on AR marketing keeps increasing from 2014 to 2021. Specifically, the first article is published in 2014 ( Huang and Liu, 2014 ). Then, the number of journal articles increases rapidly from 2015 to 2018. After that, the number of articles goes up slowly from 2019 to 2020 and surges in 2021. In particular, the number of journal articles published in 2021 is more than two times the number of journal articles published in 2020. The fast growth of publications is consistent with the proliferation of AR applications in marketing practices. Specifically, the number of global mobile AR users reached 200 million in 2015. It will grow to 1.1 billion in 2022 and 1.7 billion in 2024 ( Alsop, 2021 ).

Figure 2. Publication year.

Publication Journal

The 99 articles are published in 43 journals of which 25 journals only have one article and nine journals just have two articles. Table 1 presents the nine journals that have three articles or above. Among them, Journal of Retailing and Consumer Services has 22 articles, which ranks first in all journals. The number of journal articles published in this journal accounts for more than one-fifth of all publications. Journal of Business Research has nine articles, which ranks second in all journals. Computers in Human Behavior and Technological Forecasting and Social Change tie for the third place with four articles each.

Table 1. Publication journal.

Figure 3 shows the research design used in the selected AR marketing literature. We first analyze the research design by the broad category, that is, quantitative, qualitative, and mixed research design. The quantitative research design incorporates quantitative methods such as surveys and experiments. The qualitative research design adopts qualitative methods such as interviews and focus groups. Finally, the mixed research design uses both quantitative and qualitative methods ( Sreejesh and Mohapatra, 2014 ). As presented in Figure 3A , the quantitative methodology dominated the field. Specifically, 85.86% of the journal articles adopts the quantitative research design, 11.11% of the journal articles uses the qualitative research design. Only 3.03% of the corpus takes the mixed research design.

Figure 3. Research design. (A) Distribution of qualitative, quantitative, and mixed studies. (B) Composition of research methodologies.

Second, a fine-grained examination of the research design by single-method vs. multi-method and single-study vs. multi-study demonstrates that a considerate proportion of the journal articles use the multi-method or multi-study research design. Specifically, the single-method research design or multi-method research design refers to whether there are one or multiple research methods, such as survey, experiment, interview, and focus group, in the research design. Moreover, the single-study research design or multi-study research design refers to whether there are one or multiple studies in the research design. It is noteworthy that a single study can only use one research method. However, a single-method research design may have one or multiple studies. Thus, from the perspectives of single-method vs. multi-method and single-study vs. multi-study, we have three types of research design: single-method single-study, single-method multi-study, and multi-method multi-study. As presented in Figure 3B , most journal articles (75.76%) use the single-method single-study research design. Only about a quarter of the journal articles (16.16% + 8.08% = 24.24%) adopt the single-method multi-study or multi-method multi-study research design. The proportion of the journal articles using the single-method multi-study or multi-method multi-study research design among the journal articles using the quantitative research design (17.65% + 4.71% = 22.35%) is slightly higher than that among the journal articles using the qualitative research design (9.09% + 9.09% = 18.18%).

Research Method

Table 2 presents the number and the ratio of the journal articles that use different research methods in the selected AR marketing literature. The most popular research methods are the experiment and survey, which are adopted by 43.44 and 39.39% of the journal articles, respectively. The statistics are consistent with the fact that most research in this field is consumer/tourist-oriented. Specifically, the survey studies can be the online survey, the offline survey, and those performed by survey companies. The experiment studies can be the lab experiment, the online experiment, and the field experiment. Among the qualitative methods, the interview is more popular than the focus group. In multi-method studies, the combination of experiment and survey is the most commonly used. Furthermore, scholars often use the combinations of the interview and one of the other three methods.

Table 2. Research design.

Application Area

We identify retailing, tourism, and advertising as the three major application areas of AR marketing. Specifically, AR marketing research originates from retail. The studies in the first journal article and most of the early journal articles are conducted in retail. Moreover, the number of journal articles that shed light on AR in retail has increased significantly over the last few years. Tourism is the second application area of AR marketing. However, the increase in the number of journal articles on AR in tourism keeps steady after the first years. Advertising is the latest application area of AR in marketing. The first journal article on AR in advertising is published in 2018. Nevertheless, the number of journal articles on this topic remained limited until 2021.

Figure 4 presents the number and ratio of the journal articles for the three application areas. First, retail is the earliest application area that attracted the most attention. There are 65 journal articles (65.66%) that shed light on AR in retailing. Second, scholarly works on AR in tourism appears shortly after those on AR in retail. Tourism ranks second in terms of the number of journal articles among the three application areas. There are 26 journal articles (26.26%) that have gained insight into AR in tourism. Finally, advertising is the newest application area that has received the least attention. Specifically, only eight journal articles (8.08%) have examined AR in advertising.

Figure 4. Application area.

Application Context

The application context of AR includes both online settings and on-site scenarios. In online settings, AR enriches consumers’/tourists’ experience and improves their satisfaction with online retail, virtual tourism, and online advertising ( Chung et al., 2018 ). In on-site scenarios, AR increases the attractiveness of physical stores, restaurants, museums, and art galleries by offering novel and fantastic experience to consumers/tourists ( Barhorst et al., 2021 ).

Figure 5 presents the distribution of application context in the selected AR marketing literature. While most journal articles investigate the AR application in online settings, a quarter of the journal articles gain insight into AR applications in offline scenarios. First, the studies in 66 articles (66.67%) have focused on the AR application in online settings. The majority of existing studies on the AR application in retail and all prior studies on the AR application in advertising used online settings. Second, the studies in 25 journal articles (25.25%) concentrate on the application of AR in offline scenarios. A considerable proportion of the studies focus on the application of AR in tourism used offline scenarios.

Figure 5. Application context.

Augmented Reality Type

The AR applications that consumers interact with are built upon stationary devices (e.g., AR mirrors and PC), mobile devices (e.g., smartphones and tablets), wearable devices (e.g., headsets and smart glasses), and somatosensory devices (e.g., Kinect) ( Rauschnabel, 2018 ). Accordingly, the application of AR in marketing has different types, such as web-based AR, mobile AR, somatosensory device-based AR, wearable AR, and on-site AR.

Early applications of AR in marketing are web-based. In particular, consumers experience the products such as sunglasses, watches, makeup, clothes, shoes, and furniture through web-based AR applications (e.g., virtual try-on) installed on their PCs ( Huang and Liu, 2014 ; Huang and Liao, 2015 ; Huang and Tseng, 2015 ). Specifically, they also need to have webcams connected to their PCs. Later on, with the widespread use of mobile devices that have innovative sensors (e.g., smartphones and tablets) and the availability of economic and high-speed mobile internet, mobile AR apps have gained popularity rapidly due to their convenience and low cost. Nowadays, AR is predominantly available in more and more mobile apps ( eMarketer, 2020 ). Meanwhile, AR is also in more sophisticated forms. Specifically, consumers need to experience AR using smart glasses (e.g., HoloLens) ( Carrozzi et al., 2019 ; Heller et al., 2019b ) or somatosensory devices (e.g., Kinect and depth sensors) ( Huang and Liao, 2017 ; Huang, 2018 ; Huang et al., 2019 ). Furthermore, it is worth noting that AR is prevalent in both online settings and on-site scenarios ( Barhorst et al., 2021 ).

Figure 6 presents the distribution of AR types for the selected AR marketing literature. First, mobile AR is the dominant AR type. The studies in 57 journal articles (57.58%) use mobile AR as the research context. Second, the popularity of web-based AR follows mobile AR. The studies in 18 journal articles (18.18%) use web-based AR. Third, somatosensory device-based AR, wearable AR, and on-site AR are rare AR types. Only several journal articles use these AR types. Finally, no clear AR type is claimed in 11 journal articles. A closer look at the relationship between application areas and AR types shows that mobile AR and web-based AR are prevalent across the three application areas of retail, tourism, and advertising. Specifically, mobile AR is the dominant AR type. However, on-site AR only exist in retailing, wearable AR only exists in tourism, and somatosensory device-based AR has not appeared in advertising.

Figure 6. Augmented reality type.

Theoretical Lenses

Extant AR marketing literature builds upon a wide range of theory lenses. Table 3 summarizes the nine popular theoretical lenses used in at least three journal articles. Specifically, Technology Acceptance Model (TAM) is the most popular one. Altogether, 15 journal articles use TAM in the studies. Meanwhile, Stimulus-Organism-Response (S-O-R) Framework is also a widely used theoretical framework in AR marketing literature, which has appeared in seven journal articles. Furthermore, Self-referencing Theory, Use and Gratification Theory (UGT), Equity Theory, Flow Theory, Theory of Reasoned Action (TRA), and Unified Theory of Acceptance and the Use of Technology (UTAUT) are also well accepted theoretical perspectives.

Table 3. Theoretical lenses.

Thematic Analysis

To have a fine-grained understanding of the AR marketing literature, we summarize the focal themes in the three application areas (e.g., retail, tourism, and advertising) based on the outcome variables. Table 4 presents the outcome variables and their categories. To begin with, the literature on AR in retail examines technology-related, product-related, and brand-related outcome variables. Second, the literature on AR in tourism investigates the technology-related and tourist destination-related outcome variables. Finally, the outcome variables explored in the literature on AR in advertising include advertisement-related, brand-related, and product-related outcome variables.

Table 4. Outcome variable.

Retail is the earliest application area of AR in marketing. Recently, the recognition and adoption of AR marketing by retail giants, leading on-site retailers, and well-known consumer brands have increased significantly. Both industrial practice and academic research have provided evidence for the potentials of AR to entertain, educate, and engage consumers. Specifically, AR can transform online and on-site experience, inspire brand love, facilitate pre-purchase product fit evaluation, boost product sales, and enhance post-purchase consumption experience ( Tan et al., 2022 ). Extant literature on the application of AR in retail has investigated the effects of AR use and various AR characteristics on a set of technology-related, product-related, and brand-related outcome variables and shed light on the underlying mechanisms of these effects. Some articles focus on a specific category of outcome variables (i.e., technology-related, product-related, or brand-related outcome variables); others examine more than one category of outcome variables.

Table 5 presents the popular AR characteristics examined in the literature on AR in retail. The characteristics include interactivity, augmentation, informativeness, vividness, novelty, and aesthetics. Interactivity is the most widely investigated AR characteristic. It refers to the capability of a technological system to enable users to interact easily, control, manipulate, and be involved with the content. Augmentation, also called augmentation quality, is the most unique characteristic of AR that offers an immersive consumer experience. It describes the extent to which the digital objects are integrated into a person’s real-world environment and the ability to enable users to move the digital objects naturally. Informativeness describes the degree to which the provided information is beneficial for better decision-making of consumers. Vividness refers to the ability of AR to combine the sensory experience of real objects (e.g., that can be seen and touched) with the non-sensory imaginary objects (i.e., those created in an individual’s mind) to create a clear image of a product or experience for consumers. Novelty describes the newness, uniqueness, specificness, and unusualness of the AR-enriched information that are presented to consumers. Aesthetics describes the visual appeal of AR-enriched objects or AR empowered environments.

Table 5. Augmented reality characteristics examined in the literature on AR in retail.

Technology-Related Outcome Variables

Technology-related outcome variables examined in the literature on AR in retail include consumers’ attitude toward, satisfaction with, adoption/use intention of, continued use/reuse intention of, and recommendation intention of AR technology/AR retail application (e.g., web-based AR retail application and mobile AR retail app). Among these outcome variables, consumers’ attitude toward and reuse intention of AR technology/AR retail application are the most popular. It is noteworthy that many journal articles examined two or more responses simultaneously. In particular, consumer attitude toward AR technology/AR retail application is frequently investigated together with their adoption/use intention of, recommend intention of, and reuse use intention of AR technology/AR retail application.

Consumers’ attitude toward AR technology/AR retail application refers to their feelings associated with using it ( Pantano et al., 2017 ; Rese et al., 2017 ; Yim et al., 2017 ; Plotkina and Saurel, 2019 ; Yim and Park, 2019 ; Park and Yoo, 2020 ; Smink et al., 2020 ; Daassi and Debbabi, 2021 ; Qin et al., 2021b ). Consumers’ satisfaction with AR technology/AR retail application describes their accumulative feelings when interacting with it repetitively within a period ( Poushneh and Vasquez-Parraga, 2017 ; Chiu et al., 2021 ). Consumers’ adoption/use intention of AR technology/AR retail application refers to their willingness to adopt/use it ( Pantano et al., 2017 ; Rese et al., 2017 ; Yim and Park, 2019 ; Bonnin, 2020 ; Park and Yoo, 2020 ; Qin et al., 2021b ). Consumers’ continued use/reuse intention of AR technology/AR retail application describes their willingness to use it again in the future ( Javornik, 2016 ; Pantano et al., 2017 ; Chiu et al., 2021 ; Daassi and Debbabi, 2021 ; Hsu et al., 2021 ; Kowalczuk et al., 2021 ; Nikhashemi et al., 2021 ). Consumers’ recommendation intention for AR technology/AR retail application refers to their willingness to share the information about it with friends privately or on social media publicly ( Javornik, 2016 ; Pantano et al., 2017 ; Park and Yoo, 2020 ; Smink et al., 2020 ).

The literature on AR in retail has two primary streams. The first stream of literature sheds light on the effects of AR use and delves into the underlying mechanisms. The second stream of literature gains insight into the impacts of specific AR characteristics and reveal how these impacts take place. First, AR use, that is, the inclusion of AR-empowered product presentation and interaction capabilities in retail applications has positive effects on consumer responses to the AR technology/AR retail application (i.e., the web-based AR application and mobile AR app). Specifically, AR use can stimulate favorable consumer attitude toward ( Plotkina and Saurel, 2019 ; Yim and Park, 2019 ; Smink et al., 2020 ; Daassi and Debbabi, 2021 ), increase their satisfaction with ( Poushneh and Vasquez-Parraga, 2017 ), adoption/use intention of ( Yim and Park, 2019 ; Bonnin, 2020 ), reuse/continued use intention of ( Daassi and Debbabi, 2021 ), and recommendation intention ( Smink et al., 2020 ) of the AR technology/AR retail application. These benefits are achieved through the utilitarian value and hedonic value ( Poushneh and Vasquez-Parraga, 2017 ; Plotkina and Saurel, 2019 ; Yim and Park, 2019 ; Bonnin, 2020 ) that consumers experienced while using the AR retail application.

Second, extant literature examines the impacts of specific AR characteristics such as interactivity, augmentation, informativeness, vividness, novelty, and aesthetics on consumers’ attitudes toward ( Pantano et al., 2017 ; Rese et al., 2017 ; Yim et al., 2017 ; Park and Yoo, 2020 ; Qin et al., 2021b ), satisfaction with ( Chiu et al., 2021 ), adoption/use intention of ( Pantano et al., 2017 ; Rese et al., 2017 ; Park and Yoo, 2020 ; Qin et al., 2021b ), reuse/continued use intention of ( Javornik, 2016 ; Pantano et al., 2017 ; Chiu et al., 2021 ; Hsu et al., 2021 ; Kowalczuk et al., 2021 ; Nikhashemi et al., 2021 ), and recommendation intention of ( Javornik, 2016 ; Hilken et al., 2017 ; Pantano et al., 2017 ; Park and Yoo, 2020 ) the AR technology/AR retail application.

Furthermore, compared with the studies focused on the effects of AR use, research on the impacts of AR characteristics delves deeper into the underlying mechanisms of how AR characteristics influence consumers’ responses to the AR technology/AR retail application. Except for the evaluation of the utilitarian value and hedonic value ( Hilken et al., 2017 ; Pantano et al., 2017 ; Rese et al., 2017 ; Yim et al., 2017 ; Hsu et al., 2021 ; Nikhashemi et al., 2021 ; Qin et al., 2021b ), this stream of literature also proposes and validates a variety of psychological mechanisms, such as affective responses and cognitive responses ( Kowalczuk et al., 2021 ), flow ( Javornik, 2016 ), inspiration ( Rauschnabel et al., 2019 ; Nikhashemi et al., 2021 ), and mental image ( Park and Yoo, 2020 ).

Product-Related Outcome Variables

As shown in Table 4 , product-related outcome variables investigated in the literature on AR in retail include consumers’ product attitude, product purchase intention, willingness to pay a price premium, and WOM intention. The majority of the journal articles use consumers’ product purchase intention as the outcome variable. Some articles also examined consumers’ responses to the AR technology/AR retail application at the same time. Consumers’ product attitude refers to their feelings about a product ( van Esch et al., 2019 ; Fan et al., 2020 ). Consumers’ product purchase intention describes their willingness to purchase the product they experience in the AR retail application ( Javornik, 2016 ; Plotkina and Saurel, 2019 ; Smink et al., 2019 ). Consumers’ willingness to pay a price premium refers to their intention to pay a higher price for a product ( Nikhashemi et al., 2021 ). Consumers’ WOM intention refers to their willingness to say positive things about the product to friends, relatives, and other people ( Hilken et al., 2017 ).

Similar to the studies on the impacts of AR on technology-related outcomes, literature on this theme also has two streams. The first stream of literature sheds light on the effects of AR use. The second stream of literature gains insight into the impacts of AR characteristics. First, AR experience/use stimulates the consumer purchase intention by increasing cognitive control ( Whang et al., 2021 ), eliciting higher self-brand connection ( Baek et al., 2018 ), and strengthening the utilitarian value and hedonic value perception ( Plotkina and Saurel, 2019 ; Smink et al., 2019 ). Furthermore, the literature also reveals the boundary conditions of how AR experience/use impacts their purchase intention. For instance, Whang et al. (2021) show that peer opinions moderate the impacts of AR experience on the consumer cognitive control and purchase intention.

Second, AR characteristics such as interactivity ( Hilken et al., 2017 ; Yim et al., 2017 ; Kowalczuk et al., 2021 ; Nikhashemi et al., 2021 ), vividness ( Hilken et al., 2017 ; Yim et al., 2017 ; Nikhashemi et al., 2021 ), augmentation ( Javornik, 2016 ; Fan et al., 2020 ; Poushneh, 2021 ), informativeness ( Kowalczuk et al., 2021 ), novelty ( Hilken et al., 2017 ; Nikhashemi et al., 2021 ), quality ( Kowalczuk et al., 2021 ; Nikhashemi et al., 2021 ), reality congruence ( Kowalczuk et al., 2021 ), anthropomorphism ( van Esch et al., 2019 ), and sensory control modality ( Heller et al., 2019b ) affect product-related outcomes. The majority of the studies focus on the consumer product purchase intention. Some studies also shed light on the impacts of AR characteristics on consumers’ product attitudes ( van Esch et al., 2019 ; Fan et al., 2020 ), willingness to pay a price premium ( Nikhashemi et al., 2021 ), and WOM intention ( Hilken et al., 2017 ). These studies explain the impacts using consumers’ experience of the utilitarian benefits and hedonic benefits ( Hilken et al., 2017 ; Poushneh and Vasquez-Parraga, 2017 ; Yim et al., 2017 ; Nikhashemi et al., 2021 ), consumers’ affective responses and cognitive responses ( Javornik, 2016 ; Kowalczuk et al., 2021 ), sense of presence ( Hilken et al., 2017 ), sense of immersion ( Yim et al., 2017 ), mental imagery ( Heller et al., 2019b ), inspiration ( Nikhashemi et al., 2021 ), and flow ( Javornik, 2016 ).

Brand-Related Outcome Variables

Compared with the AR marketing literature investigating technology-related and product-related outcome variables, the studies examining brand-related outcome variables are both new and limited in quantity. As presented in Table 4 , the brand-related outcome variables include consumers’ brand attitude, perceived brand personality, and brand purchase intention. Among them, consumers’ brand attitude is the most popular one. Moreover, the brand-related outcome variables are usually investigated with the product-related outcome variables (e.g., product purchase intention) and technology-related outcomes (e.g., reuse intention).

Consumers’ brand attitude refers to their feelings about a brand ( Rauschnabel et al., 2019 ; Smink et al., 2019 , 2020 ; van Esch et al., 2019 ). Consumers’ perceived brand personality describes their systematic and enduring perception of a set of human traits that serve as the foundation of brand relational consequences and brand equity ( Plotkina et al., 2021 ). Consumers’ brand purchase intention refers to their willingness to buy the products of a specific brand ( Smink et al., 2020 ).

Similar to AR marketing literature on product-related and technology-related outcome variables, the journal articles that investigate the effects of AR on brand attitude can be categorized into two groups. First, AR use (i.e., online product presentation with AR) enhances consumers’ brand attitude by eliciting their perception of spatial presence, personalization, and utilitarian and hedonic benefits ( Smink et al., 2019 , 2020 ). However, AR use can also be harmful to consumers’ brand attitudes because it may elicit the perception of intrusiveness ( Smink et al., 2019 ).

Second, AR characteristics such as augmentation and anthropomorphism influence consumers’ brand attitudes. In particular, augmentation drives changes in consumers’ brand attitudes through inspiration ( Rauschnabel et al., 2019 ). Anthropomorphism (i.e., endowing AR with human characteristics) influences consumers’ attitudes toward the brand by boosting confidence, increasing the perceived transaction convenience and innovativeness, and decreasing the perceptions of barriers to AR use ( van Esch et al., 2019 ). Furthermore, AR types such as goal and location affect consumers’ perceived brand personality. The impact is mediated by consumers’ perceived AR app experience and attitudes toward the AR app and moderated by consumer characteristics such as IT innovativeness and shopping orientation ( Plotkina et al., 2021 ).

Tourism is an emerging application area of AR marketing. Different from retail in which increasing product sales is the central point, the primary concern for tourism is enhancing visitors’ experience. AR is valuable for the tourism industry in multiple ways, such as economic, experiential, social, epistemic, cultural and historical, and educational ( tom Dieck and Jung, 2017 ). The application of AR in tourism transforms tourists’ experience by providing more interactive, enjoyable, personalized, and context-aware tourism experiences, which further increases tourists’ satisfaction and expands target markets ( Jung et al., 2015 ; tom Dieck and Jung, 2017 ; Jiang et al., 2019 ). Therefore, more and more business entities in tourism, such as tourism destinations ( Lacka, 2020 ; Huang and Liu, 2021 ), heritage tourism sites ( tom Dieck and Jung, 2018 ; Tsai et al., 2020 ), museums ( He et al., 2018 ), art galleries ( tom Dieck et al., 2016 , 2018b ), protected areas ( Jiang et al., 2019 ), science festivals ( tom Dieck et al., 2018a ), theme parks ( Jung et al., 2015 ), and restaurants ( Heller et al., 2019a ; Batat, 2021 ), adopt AR applications in offline environments and online settings.

The first journal article on AR in tourism was published in 2015, which is just one year after the publication of the first journal article on AR in retail. The number of journal articles exploring AR in tourism keeps increasing over the past few years. Prior studies on AR in tourism examined the influence of AR on various technology-related and tourist destination-related outcomes. Some journal articles also delved into the underlying psychological and behavioral mechanisms. Besides, several journal articles gained insight into the broader themes, such as the perceived value of AR for the tourism industry ( tom Dieck and Jung, 2017 ; Cranmer et al., 2020 ) and the AR business models in the tourism industry ( Cranmer et al., 2020 ).

The technology-related outcome variables examined in the literature on AR in tourism are similar to those investigated in the literature on AR in retail. As presented in Table 4 , the outcome variables include tourists’ attitude toward, adoption of, satisfaction with, and recommendation intention of the AR technology/AR tourism application. Tourists’ attitude toward the AR technology/AR tourism application refers to their feelings about the AR technology/AR tourism application ( Wu et al., 2013 ; Jung et al., 2018 ; Paulo et al., 2018 ; tom Dieck and Jung, 2018 ; Shin and Jeong, 2021 ). Tourists’ adoption of the AR technology/AR tourism application describes their willingness to use the AR technology/AR tourism application ( tom Dieck and Jung, 2018 ). Tourists’ satisfaction with the AR technology/AR tourism application refers to their overall feelings while interacting with it constantly within a period ( Jung et al., 2015 ). Tourists’ recommendation intention for the AR technology/AR tourism application describes their desire to publicly or privately share the information about it ( Jung et al., 2015 ).

First, scholars have proposed improved models of the well-recognized models, such as the Technology Acceptance Model (TAM), Unified Theory of Acceptance and Usage of Technology (UTAUT2), and Task Technology Fit (TTF). These models can better explain the determinants of the adoption of AR in tourism by incorporating new antecedents or combining existing models ( Wu et al., 2013 ; Jung et al., 2018 ; Paulo et al., 2018 ; tom Dieck and Jung, 2018 ). Particularly, tourists’ perceptions of usefulness and ease of use of AR technology have significant positive impacts on their attitude toward AR technology. Moreover, tourists’ motivations to adopt AR, such as hedonic motivation, utilitarian motivation, and self-presentation motivation, have significant positive effects on their attitudes toward AR tourism applications ( Shin and Jeong, 2021 ). In addition, tourists’ attitudes toward ( Shin and Jeong, 2021 ; Zhuang et al., 2021 ) and subjective norms of ( Zhuang et al., 2021 ) AR technology positively impact their intention to use it.

Second, the three quality dimensions of the AR tourism application (i.e., content quality, personalized service quality, and system quality) affect tourists’ satisfaction with and recommendation intention of it. The effect of the quality dimensions on tourists’ intention to recommend the AR tourism application is mediated by their satisfaction with it. Furthermore, this effect is more prominent for the tourists with high innovativeness than for those with low innovativeness ( Jung et al., 2015 ).

Tourist Destination-Related Outcome Variables

Tourist destination-related outcome variables include tourists’ knowledge acquisition of, visit intention of, satisfaction with, memory of, and WOM generation for tourist destinations, as well as tourists’ choice of products and willingness to pay a price premium in tourist destinations. Specifically, AR enriches tourists’ sensory, affective, behavioral, social, and intellectual experiences ( Heller et al., 2019a ). The enriched experiences lead to tourists’ better knowledge acquisition of tourist destinations, increased intention to visit tourism destinations, improved satisfaction with and memory of the tourist destination, choice of higher value products, and increased willingness to pay a price premium. In conclusion, the application of AR in tourism increases the overall well-being of tourists ( Batat, 2021 ).

Tourists’ knowledge acquisition of a tourist destination refers to their learning of new, interesting, or necessary things about it ( tom Dieck et al., 2018b ; Lacka, 2020 ). Tourists’ intention to visit a tourist destination describes their desire to visit it ( Chung et al., 2018 ; He et al., 2018 ; Lacka, 2020 ). Tourists’ satisfaction with a tourist destination refers to how much they enjoy visiting it using the AR tourist application ( tom Dieck et al., 2018a ). Tourists’ memory of a tourist destination describes what stays in their minds after visiting it using AR tourist applications ( tom Dieck et al., 2018a ). Tourists’ choice of higher value products refers to their decision to buy products of higher prices ( Heller et al., 2019a ). Tourists’ willingness to pay a price premium describes their desire to pay a higher price that exceeds the benchmark price ( Huang, 2021 ).

The studies delving into the underlying mechanisms of the effects reveal that they are achieved by creating an immersive experience, stimulating tourist engagement, and increasing processing fluency ( Heller et al., 2019a ; Tsai et al., 2020 ). Moreover, the effects are heterogeneous across tourists with different visual processing styles and sensation-seeking tendencies, and products with different contextuality ( Heller et al., 2019a ). More nuanced investigations into the impacts of specific AR characteristics provides a deeper understanding of how AR affects tourists. For instance, the two AR empowerment features, such as environmental embedding and simulated physical control, foster immersion and increase the willingness to pay more by generating a restorative experience ( Huang, 2021 ). Moreover, the three dimensions of technology embodiment (i.e., ownership, location, and agency) affect tourists’ enjoyment and enhance their experience ( Tussyadiah et al., 2018 ). The three key features of humanizing experiences in the AR tourism application (i.e., anthropomorphism, self-representation, and intimacy) lead to a more prominent effect on the brand love of tourism destinations ( Huang and Liu, 2021 ). In addition, both the information types (i.e., dynamic verbal vs. visual cues) and the augmenting immersive scenes (i.e., high vs. low virtual presence) influence tourists’ purchase intentions and willingness to pay more. Specifically, dynamic verbal cues lead to a higher level of willingness to pay more than dynamic visual cues. The effect is more prominent in high virtual presence environments ( He et al., 2018 ).

Advertising

Advertising is the latest and fast-growing application area of AR marketing. The application of AR in advertising is mobile apps based ( Yang et al., 2020 ; Sung, 2021 ) or in the form of online AR advertisement videos ( Feng and Xie, 2018 ). Compared with traditional print advertising, radio advertising, and TV broadcast advertising, AR advertising is more informative, novel, entertaining, and complex ( Feng and Xie, 2018 ; Yang et al., 2020 ). AR-enabled immersive, interactive, and personalized experience elicits positive consumer responses and helps advertising campaigns stand out ( Sung, 2021 ).

A variety of advertisement characteristics affect the consumers’ affective, cognitive, and behavioral response to AR advertisements. The characteristics of AR advertisements include AR advertisement type (e.g., quick response hypermedia and app response hypermedia) ( Uribe et al., 2021 ), AR interaction type (e.g., instrumental and hedonic) ( Tsai et al., 2020 ), and advertisement context (e.g., realistic and imaginative) ( Tsai et al., 2020 ). In addition, product type (i.e., think and feel) ( Tsai et al., 2020 ) and consumer personality traits (i.e., extraversion, openness, agreeableness, conscientiousness, and neuroticism) ( Srivastava et al., 2021 ; Uribe et al., 2021 ) also have influence on the consumers’ responses to AR advertisements. Specifically, AR advertisements enhance consumer physiological responses ( Pozharliev et al., 2021 ), boost their engagement ( Sung, 2021 ), and facilitate social experience sharing among consumers ( Sung, 2021 ). These desirable effects further stimulate positive attitudes toward AR advertisements ( Yang et al., 2020 ), increase the efficacy of advertising campaigns ( Feng and Xie, 2018 ), strengthen consumer-brand connections ( Pozharliev et al., 2021 ), increase the brand liking ( Tsai et al., 2020 ), and stimulate product purchase intentions ( Pozharliev et al., 2021 ; Sung, 2021 ).

Advertisement-Related Outcome Variables

Advertisement-related outcome variables examined in AR advertising literature is consumers’ attitudes toward AR advertisements. It refers to consumers’ feelings toward AR advertisements ( Feng and Xie, 2018 ; Yang et al., 2020 ; Uribe et al., 2021 ). Overall, AR advertising has many advantages over traditional ones. Specifically, AR advertising leads to positive attitudes toward the advertisements. This effect is mediated by consumers’ perceived enjoyment and informativeness ( Uribe et al., 2021 ). The content characteristics of AR advertisements, such as informativeness, novelty, entertainment, and complexity, affect consumers’ attitudes toward AR advertisements. Moreover, irritation, value, and believability of AR advertisements serially mediate the effects of the content characteristics of AR advertisements and consumers’ attitudes toward them ( Feng and Xie, 2018 ). Compared with traditional advertisements without AR, advertisements with AR can increase consumers’ curiosity about the advertisements, which in turn attract their visual attention toward the advertisements and bolster their attitudes toward the advertisements ( Yang et al., 2020 ). Besides, AR interaction type, advertisement context, and product type affect the perceived informativeness of AR ads. Telepresence mediates the effects ( Tsai et al., 2020 ).

Brand-related outcome variables examined in the literature on AR in advertising include consumers’ brand attitude and brand liking. Brand attitude and brand liking are consumers’ feelings about a brand ( Phua and Kim, 2018 ; Tsai et al., 2020 ; Uribe et al., 2021 ). AR advertisements have a positive impact on consumers’ attitudes toward the brand. Consumers’ perception of the advertisements’ entertainment value partially mediates the effect ( Uribe et al., 2021 ). Moreover, self-brand congruity, self-referencing, and perceived humor significantly influence consumers’ post-use brand attitude toward the advertised brand ( Phua and Kim, 2018 ). AR interaction type (i.e., instrumental vs. hedonic), advertisement context (i.e., realistic vs. imaginative), and product type (think vs. feel) impose significant impacts on brand liking. Telepresence plays the role of mediator in the relationship ( Tsai et al., 2020 ).

The product-related outcome variable examined in the literature on AR in advertising is consumers’ product purchase intention or willingness to pay. Consumers’ product purchase intention or willingness to pay refers to their desire to buy the advertised product ( Phua and Kim, 2018 ; Pozharliev et al., 2021 ; Uribe et al., 2021 ). Compared with traditional advertising, AR advertising improves consumers’ attitudes toward advertisements, enhances their emotional responses (i.e., physiological arousal), and leads to higher product purchase intention or willingness to pay ( Pozharliev et al., 2021 ; Uribe et al., 2021 ). The positive effect of AR advertising on consumers’ product purchase intention is partly mediated by their entertainment value perception of advertisements ( Uribe et al., 2021 ) or fully mediated by their emotional responses (i.e., physiological arousal) ( Pozharliev et al., 2021 ). Furthermore, self-brand congruity, self-referencing, and perceived humor affect consumers’ product purchase intention. Self-brand congruity interacted with the other two factors to influence brand attitude, while the three factors interacted in pairs to affect consumers’ product purchase intention ( Phua and Kim, 2018 ).

Contributions and Conclusion

This study makes two important contributions to research in AR marketing. First, we delve into the factors specific to AR marketing research. In addition to the shared aspects such as publication year, publication journal, research design, and research method, we shed light on the factors specific to AR marketing such as application area, application context, AR type, and theoretical lenses. Our analyses show that retail, tourism, and advertising are the major application area of AR marketing research. Specifically, retail is the earliest and most popular application area, advertising is the newest and least investigated application area. Next, most prior studies investigated AR applications in online settings, but only a small portion of the literation examined AR applications in on-site scenarios. Third, mobile AR applications and web-based AR applications are the most prevalent AR type in the three application areas. On-site AR applications, wearable AR applications, and somatosensory device-based AR applications have received little scholarly attention in some of the application areas. Finally, TAM, S-O-R Framework, Self-Referencing Theory, UGT, Equity Theory, Flow Theory, TRA, and UTAUT are the most prevalent theoretical lenses in AR marketing research. These findings offer more comprehensive and integrated perspectives to understand the state-of-the-art of AR marketing research.

Second, we identify the focal themes in the three application areas to illustrate the current status of scholarly works. We obtain the focal themes by the outcome variables used in the empirical studies. The outcome variables describe the effects of AR use in general and specific AR characteristics on the AR technology/AR application, products, brands, tourist destinations, and advertisement campaigns. Our analyses show that technology-related variables, product-related variables, and brand-related variables are the shared outcome variables examined in the literature of more than one application area. Tourist destination-related and advertisement-related outcome variables are studied in the literature on a single application area. Specifically, technology-related outcome variables include consumers’/tourists’ attitudes toward, satisfaction with, adoption/use intention of, continued use/reuse intention of, and recommendation intention of the AR technology/AR applications. Product-related variables include consumers’ product attitudes, product purchase intention, willingness to pay a price premium, and WOM intention. Brand-related outcome variables include consumers’ brand attitudes, perceived brand personality, brand liking, and brand purchase intention. Tourist destination-related variables include tourists’ knowledge acquisition of, intention to visit, satisfaction with, the memory of, and WOM generation for tourist destinations. The advertisement-related outcome variable is consumers’ attitudes toward advertisements. These findings provide a clear guideline to grasp the main streams of the AR marketing literature.

Future Research Agenda

A systematic literature review integrates research papers in a comprehensive, structured, and analytical way. Therefore, it can identify the gaps in extant literature ( Paul and Criado, 2020 ) and highlight the understudied areas that need further attention ( Snyder, 2019 ). We discuss the important but uncovered topics that flow directly from our literature analysis. Then, we put forward the topics that the authors value but have not been investigated in detail by the extant literature.

The Effects of Augmented Reality on More Outcome Variables and Mediating Variables

As the methods used in most literature are the survey and lab experiment, the outcome variables examined in the literature are self-reported ones collected through scales. With the proliferation of AR applications, the availability of more data collection methods and data analysis techniques will increase significantly. Thus, future research can investigate additional outcome variables and mediating variables obtained from the AR application systems in natural settings ( Pantano et al., 2017 ; Smink et al., 2020 ; Castillo and Bigne, 2021 ; Javornik et al., 2021 ; Poushneh, 2021 ; Qin et al., 2021a ; Tan et al., 2022 ) or measured via consumer neuroscience methods ( Jung et al., 2021 ; Pozharliev et al., 2021 ). For instance, except for the examination of the effects of AR on product purchase intention, WOM intention, recommendation intention, and personal data disclosure intention, future research can shed light on the effects of AR on actual purchase behavior, WOM behavior, recommendation behavior, personal data disclosure, post-purchase product satisfaction, customer retention, and product return rate ( Smink et al., 2019 ; Kowalczuk et al., 2021 ; Qin et al., 2021a ). In this vein, we can gain broader insight into more nuanced consumer response and behavior; and obtain a deeper understanding of the affective, cognitive, and social processes underlying the effects of AR on consumer response and behavior.

Furthermore, most existing AR research examined consumers’ immediate experiences and behavioral intentions toward AR. Future research can explore the persistent impacts of AR adoption and design features on consumers’ motivation, experiences, responses, and behavior in various contexts using longitudinal approaches ( He et al., 2018 ; Carrozzi et al., 2019 ; Huang et al., 2019 ; Zhang et al., 2019 ; Barhorst et al., 2021 ; Batat, 2021 ; Hsu et al., 2021 ; Javornik et al., 2021 ; Uribe et al., 2021 ). Overall, a more comprehensive understanding of what AR brings to various fields of marketing will enable us to better incorporate this novel and potentially disruptive technology in the service frontline design and operation.

The Heterogenous Effects of AR

The effects of AR are heterogeneous across consumers with different characteristics, products/services in different categories, and scenarios in different contexts. Although existing literature examines the heterogeneous effects of AR across some general characteristics of consumers, products/services, and contexts, future studies can delve deeper into more nuanced effects of AR regarding consumers, products, services, and contexts with different AR relevant characteristics.

With the fast-growing the application of AR in marketing, consumers using these applications, and products/services offered through these application, future research can delve deeper into the heterogenous effects of AR on consumers’ responses and behavior across consumer characteristics such as gender ( tom Dieck et al., 2018a ; Smink et al., 2019 ; Batat, 2021 ; Chen et al., 2021 ; Daassi and Debbabi, 2021 ; Javornik et al., 2021 ; Yuan et al., 2021 ), age groups (e.g., children, middle-aged people, and elder) ( Jung et al., 2015 ; Pantano et al., 2017 ; Plotkina and Saurel, 2019 ; Smink et al., 2019 ; Batat, 2021 ; Chiu et al., 2021 ; Daassi and Debbabi, 2021 ; Kowalczuk et al., 2021 ; Plotkina et al., 2021 ; Qin et al., 2021b ; Yuan et al., 2021 ), educational level ( He et al., 2018 ; Yuan et al., 2021 ), occupation ( Song et al., 2019 ), culture background ( Jung et al., 2015 , 2021 ; Rese et al., 2017 ; Rauschnabel, 2018 ; Jiang et al., 2019 ; Plotkina and Saurel, 2019 ; Jessen et al., 2020 ; Chen et al., 2021 ; Chiu et al., 2021 ; Javornik et al., 2021 ; Lim et al., 2021 ; Plotkina et al., 2021 ; Qin et al., 2021a ; Yuan et al., 2021 ), personality traits ( Tussyadiah et al., 2018 ; Park and Stangl, 2020 ; Uribe et al., 2021 ), cognitive style ( Fan et al., 2020 ), processing style ( Heller et al., 2019b ), innovativeness ( Huang and Liao, 2015 ; Huang, 2019 ; Smink et al., 2019 ; Yim and Park, 2019 ; Daassi and Debbabi, 2021 ), expertise regarding the products/services (e.g., novice vs. experienced) ( He et al., 2018 ; Jung et al., 2021 ), need for touch (e.g., high vs. low) ( Hilken et al., 2017 ; Huang, 2019 ; Plotkina and Saurel, 2019 ), need for vision (e.g., high vs. low) ( Huang and Liao, 2017 ), technology awareness and enthusiasm ( Yang et al., 2020 ), familiarity with AR technology ( Park and Yoo, 2020 ), and privacy sensitivity ( Smink et al., 2020 ; Daassi and Debbabi, 2021 ).

Future research can also examine the heterogeneous effects of AR on consumers’ responses and behavior regarding different product or service characteristics such as product type (e.g., hedonic vs. functional) ( Chen et al., 2021 ; Pozharliev et al., 2021 ), product category ( Pantano et al., 2017 ; Song et al., 2019 ; Park and Yoo, 2020 ; Barhorst et al., 2021 ; Castillo and Bigne, 2021 ; Daassi and Debbabi, 2021 ; Hsu et al., 2021 ; Kowalczuk et al., 2021 ; Plotkina et al., 2021 ; Whang et al., 2021 ), product size (small sized vs. large sized) ( Yim et al., 2017 ), product novelty (highly specialized vs. newly developed products) ( Hilken et al., 2017 ), level of body involvement (e.g., high, moderate, and low) ( Yim and Park, 2019 ; Daassi and Debbabi, 2021 ), and brand awareness ( Song et al., 2019 ).

Moreover, future research can investigate the effects of AR on consumers’ responses and behavior under the contexts with different characteristics such as customer experience stages (e.g., pre-adoption vs. post-adoption, pre-purchase vs. post-purchase, pre-trip vs. post-trip) ( Park and Stangl, 2020 ), noise levels in the ambient environment (e.g., high-noise vs. low-noise environment) ( Yang et al., 2020 ), choice situations ( Kowalczuk et al., 2021 ), and privacy of the environment (i.e., public vs. private) ( Javornik, 2016 ; Carrozzi et al., 2019 ; Castillo and Bigne, 2021 ).

The Effects of Specific Augmented Reality Design Elements and Features

As both the industry practice and academic investigation of AR marketing are still in the infant stage, most existing studies focus on the effects of AR use or AR characteristics. In this regard, deeper investigations of the impacts of sophisticated AR design features in marketing applications on the outcome variables that describe consumers’ experiences of and responses to the AR technology/AR application, products/services, brands, tourist destinations, and advertisements are needed ( McLean and Wilson, 2019 ).

Particularly, to provide enriched information and offer rapid responses to consumers, further research needs to focus on the AR design elements and features that can increase the realisticness, authenticity, vividness, novelty, interactivity, and efficiency ( Huang and Liu, 2014 ; Pantano et al., 2017 ; Bonnin, 2020 ; Jessen et al., 2020 ; Barhorst et al., 2021 ). For instance, current AR marketing studies build upon AR applications that augment consumers’ visual and auditory perceptions of products and services. With the emergence of the AR technology that can enrich more sensory experiences such as tactile, gustatory, and olfactory, new AR applications using it can provide multi-sensory feedback ( Heller et al., 2019b ). Thus, scholars can seek to investigate the effects of AR applications incorporating multi-sensory augmentation and feedback capabilities ( Huang and Liao, 2017 ; Heller et al., 2019a , b ; Sung, 2021 ).

In conclusion, academic research can help better understand how AR design elements and features will affect consumers’ motivations, experiences, responses, and behavior regarding products/services, brands, and product/service providers. Also, product/service providers can figure out ways of improving AR applications and better satisfying the needs of consumers. Finally, the stakeholders can reach a win–win situation in which both consumers will derive high experiences value and product/service providers can achieve high-revenue profit ( Huang and Liu, 2014 ).

The Dark Side of Augmented Reality Application in Marketing

Most extant AR marketing literature focus on the bright side of AR use and the positive effects of AR characteristics. However, little studies discuss the dark side of AR application in marketing. To provide enriched personalized services (e.g., consumer movement detection and synchronized and accurate response provision), AR applications need to collect, process, store, and transmit a variety of consumer data such as the face, body, and personal space ( Huang et al., 2019 ; Smink et al., 2019 ). Thus, potential ethical issues regarding privacy, surveillance, and security risk need more investigations in the future ( Rauschnabel, 2018 ; Carrozzi et al., 2019 ; Smink et al., 2019 ; Chang, 2021 ; Huang and Liu, 2021 ; Javornik et al., 2021 ; Lim et al., 2021 ). For instance, consumers’ privacy concerns may act as a boundary condition and strengthen/weaken the effects of AR use or AR characteristics on their motivations, experiences, response, and behaviors ( Lim et al., 2021 ). Moreover, an underwhelming AR experience will harm consumers’ product/service perception and damage brand equity ( Rauschnabel et al., 2019 ). Another potential outcome of the application of AR is vicarious consumption. Specifically, instead of interacting with the physical elements of a brand, consumers may only build connections with the brand in a computer-mediated environment ( Rauschnabel et al., 2019 ). Therefore, future research needs to hold a more balanced and critical perspective to examine when AR applications will backfire and lead to undesired spill-over effects ( Rauschnabel et al., 2019 ).

Limitations

Although this research has many meaningful contributions, we acknowledge that this research still has several limitations. These limitations provide opportunities for further investigation. To begin with, the journal articles included in this research are extracted and selected according to our criteria. Thus, we may miss some valuable materials. For instance, we use the WOS core collections as the data extraction source to ensure the high quality of the literature analyzed in this research. Future research can extract literature from more databases such as Scopus, Elsevier, Emerald, Wiley, and Google Scholar to incorporate information from conference proceedings, research reports, working papers, theses and dissertations, books, magazines, white papers, and industry reports. Including more sources and casting the net wider help to gain additional insights. Also, as AR marketing research is still in the infant stage, we use descriptive analysis and thematic analysis in the systematic review. Moreover, the literature analysis is based on the authors’ expertise and understanding. Therefore, the results of this research may have limited generalizability. With the increase in the number of publications on AR marketing, future research can gain more insights into the increased literature by using analytic techniques such as meta-analysis, bibliometric analysis, and text mining.

Author Contributions

ZD and JL designed the research, collected the data, and conducted the data analysis. ZD, JL, and TW contributed to the drafting and revision of the manuscript. All authors approved the submitted manuscript.

This research was supported by the National Natural Science Foundation of China (71901030).

Conflict of Interest

The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Publisher’s Note

All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article, or claim that may be made by its manufacturer, is not guaranteed or endorsed by the publisher.

Alsop, T. (2021). Global Mobile Augmented Reality (AR) Users 2019-2024. Available online at: https://www.statista.com/statistics/1098630/global-mobile-augmented-reality-ar-users (accessed on May 12, 2022).

Google Scholar

Alves, H., Fernandes, C., and Raposo, M. (2016). Social media marketing: a literature review and implications. Psychol. Market. 33, 1029–1038. doi: 10.1002/mar.20936

CrossRef Full Text | Google Scholar

Baek, T. H., Yoo, C. Y., and Yoon, S. (2018). Augment yourself through virtual mirror: the impact of self-viewing and narcissism on consumer responses. Int. J. Advert. 37, 421–439. doi: 10.1080/02650487.2016.1244887

Banerjee, S., and Longstreet, P. (2016). Mind in eBay, body in Macy’s: dual consciousness of virtuo-physical consumers and implications for marketers. J. Res. Interact. Market. 10, 288–304. doi: 10.1108/JRIM-05-2015-0036

Barhorst, J. B., McLean, G., Shah, E., and Mack, R. (2021). Blending the real world and the virtual world: exploring the role of flow in augmented reality experiences. J. Bus. Res. 122, 423–436. doi: 10.1016/j.jbusres.2020.08.041

Batat, W. (2021). How augmented reality (AR) is transforming the restaurant sector: investigating the impact of “Le Petit Chef” on customers’ dining experiences. Technol. Forecast. Soc. Change 172:121013. doi: 10.1016/j.techfore.2021.121013

Baytar, F., Chung, T., and Shin, E. (2020). Evaluating garments in augmented reality when shopping online. J. Fashion Market. Manag. 24, 667–683. doi: 10.1108/JFMM-05-2018-0077

Bell, D. R., Gallino, S., and Moreno, A. (2018). Offline showrooms in omnichannel retail: demand and operational benefits. Manag. Sci. 64, 1629–1651. doi: 10.1287/mnsc.2016.2684

PubMed Abstract | CrossRef Full Text | Google Scholar

Bonetti, F., Pantano, E., Warnaby, G., and Quinn, L. (2019). Augmenting reality: fusing consumers’ experiences and interactions with immersive technologies in physical retail settings. Int. J. Technol. Market. 13, 260–284. doi: 10.1504/IJTMKT.2019.104592

Bonnin, G. (2020). The roles of perceived risk, attractiveness of the online store and familiarity with AR in the influence of AR on patronage intention. J. Retail. Consum. Serv. 52:101938. doi: 10.1016/j.jretconser.2019.101938

Carmigniani, J., Furht, B., Anisetti, M., Ceravolo, P., Damiani, E., and Ivkovic, M. (2011). Augmented reality technologies, systems and applications. Multimed. Tools Applic. 51, 341–377. doi: 10.1007/s11042-010-0660-6

Carrozzi, A., Chylinski, M., Heller, J., Hilken, T., Keeling, D. I., and de Ruyter, K. (2019). What’s mine is a hologram? How shared augmented reality augments psychological ownership. J. Interact. Market. 48, 71–88. doi: 10.1016/j.intmar.2019.05.004

Castillo, S. M. J., and Bigne, E. (2021). A model of adoption of AR-based self-service technologies: a two country comparison. Int. J. Retail Distrib. Manag. 49, 875–898. doi: 10.1108/IJRDM-09-2020-0380

Chalimov, A. (2021). Bringing Augmented Reality to Your Retail App. Available online at: https://easternpeak.com/blog/bringing-augmented-reality-to-your-retail-app (accessed May 13, 2022).

Chang, V. (2021). An ethical framework for big data and smart cities. Technol. Forecast. Soc. Change 165:120559. doi: 10.1016/j.techfore.2020.120559

Chen, J. V., Ruangsri, S., Ha, Q. A., and Widjaja, A. E. (2021). An experimental study of consumers’ impulse buying behaviour in augmented reality mobile shopping apps. Behav. Inform. Technol. doi: 10.1080/0144929X.2021.1987523

Chitrakorn, K. (2021). An Augmented Reality Shopping Boutique Is Here. Available online at: https://www.voguebusiness.com/consumers/augmented-reality-store-machine-a-institute-of-fashion-design (accessed May 13, 2022).

Chiu, C. L., Ho, H. C., Yu, T., Liu, Y., and Mo, Y. (2021). Exploring information technology success of augmented reality retail applications in retail food chain. J. Retail. Consum. Serv. 61:102561. doi: 10.1016/j.jretconser.2021.102561

Chung, N., Han, H., and Joun, Y. (2015). Tourists’ intention to visit a destination: the role of augmented reality (AR) application for a heritage site. Comp. Hum. Behav. 50, 588–599. doi: 10.1016/j.chb.2015.02.068

Chung, N., Lee, H., Kim, J. Y., and Koo, C. (2018). The role of augmented reality for experience-influenced environments: the case of cultural heritage tourism in Korea. J. Travel Res. 57, 627–643. doi: 10.1177/0047287517708255

Cipresso, P., Giglioli, I. A. C., Raya, M. A., and Riva, G. (2018). The past, present, and future of virtual and augmented reality research: a network and cluster analysis of the literature. Front. Psychol. 9:2086. doi: 10.3389/fpsyg.2018.02086

Cranmer, E. E., tom Dieck, M. C., and Fountoulaki, P. (2020). Exploring the value of augmented reality for tourism. Tour. Manag. Perspect. 35:100672.

Cuomo, M. T., Tortora, D., Festa, G., Ceruti, F., and Metallo, G. (2020). Managing omni-customer brand experience via augmented reality: a qualitative investigation in the Italian fashion retailing system. Qual. Mark. Res. 23, 427–445. doi: 10.1108/QMR-11-2017-0142

Daassi, M., and Debbabi, S. (2021). Intention to reuse AR-based apps: the combined role of the sense of immersion, product presence and perceived realism. Inform. Manag. 58:103453. doi: 10.1016/j.im.2021.103453

Denyer, D., and Tranfield, D. (2009). “Producing a systematic review,” in The Sage Handbook of Organizational Research Methods , eds D. A. Buchanan and A. Bryman (Thousand Oaks, CA: Sage Publications Ltd), 671–689.

eMarketer (2020). US Virtual and Augmented Reality Users 2020. Available online at: https://www.emarketer.com/content/us-virtual-and-augmented-reality-users-2020 (accessed March 26, 2022).

Fan, X., Chai, Z., Deng, N., and Dong, X. (2020). Adoption of augmented reality in online retailing and consumers’ product attitude: a cognitive perspective. J. Retail. Consum. Serv. 53:101986. doi: 10.1016/j.jretconser.2019.101986

Feng, Y., and Xie, Q. (2018). Measuring the content characteristics of videos featuring augmented reality advertising campaigns. J. Res. Interact. Market. 12, 489–508. doi: 10.1108/JRIM-01-2018-0027

Ferrari, V., Klinker, G., and Cutolo, F. (2019). Augmented reality in healthcare. J. Healthc. Eng. 2019:9321535. doi: 10.1155/2019/9321535

Han, X., Wang, F., Lv, S., and Han, W. (2021). Mechanism linking AR-based presentation mode and consumers’ responses: a moderated serial mediation model. J. Theor. Appl. Electron. Commer. Res. 16, 2694–2707. doi: 10.3390/jtaer16070148

He, Z., Wu, L., and Li, X. R. (2018). When art meets tech: the role of augmented reality in enhancing museum experiences and purchase intentions. Tour. Manag. 68, 127–139. doi: 10.1016/j.tourman.2018.03.003

Heller, J., Chylinski, M., de Ruyter, K., Mahr, D., and Keeling, D. I. (2019a). Let me imagine that for you: transforming the retail frontline through augmenting customer mental imagery ability. J. Retail. 95, 94–114. doi: 10.1016/j.jretai.2019.03.005

Heller, J., Chylinski, M., de Ruyter, K., Mahr, D., and Keeling, D. I. (2019b). Touching the untouchable: exploring multi-sensory augmented reality in the context of online retailing. J. Retail. 95, 219–234. doi: 10.1016/j.jretai.2019.10.008

Hilken, T., de Ruyter, K., Chylinski, M., Mahr, D., and Keeling, D. I. (2017). Augmenting the eye of the beholder: exploring the strategic potential of augmented reality to enhance online service experiences. J. Acad. Market. Sci. 45, 884–905. doi: 10.1007/s11747-017-0541-x

Hilken, T., Heller, J., Chylinski, M., Keeling, D. I., Mahr, D., and de Ruyter, K. (2018). Making omnichannel an augmented reality: the current and future state of the art. J. Res. Interact. Market. 12, 509–523. doi: 10.1108/JRIM-01-2018-0023

Hinsch, C., Felix, R., and Rauschnabel, P. A. (2020). Nostalgia beats the wow-effect: inspiration, awe and meaningful associations in augmented reality marketing. J. Retail. Consum. Serv. 53:101987. doi: 10.1016/j.jretconser.2019.101987

Hsu, S. H. Y., Tsou, H. T., and Chen, J. S. (2021). “Yes, we do. Why not use augmented reality?” customer responses to experiential presentations of AR-based applications. J. Retail. Consum. Serv. 62:102649. doi: 10.1016/j.jretconser.2021.102649

Huang, T. L. (2018). Creating a commercially compelling smart service encounter. Serv. Bus. 12, 357–377. doi: 10.1007/s11628-017-0351-8

Huang, T. L. (2019). Psychological mechanisms of brand love and information technology identity in virtual retail environments. J. Retail. Consum. Serv. 47, 251–264. doi: 10.1016/j.jretconser.2018.11.016

Huang, T. L. (2021). Restorative experiences and online tourists’ willingness to pay a price premium in an augmented reality environment. J. Retail. Consum. Serv. 58:102256. doi: 10.1016/j.jretconser.2020.102256

Huang, T. L., and Liao, S. L. (2015). A model of acceptance of augmented-reality interactive technology: the moderating role of cognitive innovativeness. Electron. Commer. Res. 15, 269–295. doi: 10.1007/s10660-014-9163-2

Huang, T. L., and Liao, S. L. (2017). Creating e-shopping multisensory flow experience through augmented-reality interactive technology. Int. Res. 27, 449–475. doi: 10.1016/j.ausmj.2020.06.010

Huang, T. L., and Liu, B. S. (2021). Augmented reality is human-like: how the humanizing experience inspires destination brand love. Technol. Forecast. Soc. Change 170:120853. doi: 10.1016/j.techfore.2021.120853

Huang, T. L., and Liu, F. H. (2014). Formation of augmented-reality interactive technology’s persuasive effects from the perspective of experiential value. Int. Res. 24, 82–109. doi: 10.1108/IntR-07-2012-0133

Huang, T. L., Mathews, S., and Chou, C. Y. (2019). Enhancing online rapport experience via augmented reality. J. Serv. Market. 33, 851–865. doi: 10.1108/JSM-12-2018-0366

Huang, T. L., and Tseng, C. H. (2015). Using augmented reality to reinforce vivid memories and produce a digital interactive experience. J. Electron. Commer. Res. 16, 307–328.

Javornik, A. (2016). “It’s an illusion, but it looks real!” Consumer affective, cognitive and behavioural responses to augmented reality applications. J. Market. Manag. 32, 987–1011. doi: 10.1080/0267257X.2016.1174726

Javornik, A., Marder, B., Pizzetti, M., and Warlop, L. (2021). Augmented self-the effects of virtual face augmentation on consumers’ self-concept. J. Bus. Res. 130, 170–187. doi: 10.1016/j.jbusres.2021.03.026

Jessen, A., Hilken, T., Chylinski, M., Mahr, D., Heller, J., Keeling, D. I., et al. (2020). The playground effect: how augmented reality drives creative customer engagement. J. Bus. Res. 116, 85–98. doi: 10.1016/j.jbusres.2020.05.002

Jiang, S., Scott, N., and Tao, L. (2019). Antecedents of augmented reality experiences: potential tourists to Shangri-La Potatso National Park, China. Asia Pac. J. Tour. Res. 24, 1034–1045. doi: 10.1080/10941665.2019.1653949

Jung, T., Chung, N., and Leue, M. C. (2015). The determinants of recommendations to use augmented reality technologies: the case of a Korean theme park. Tour. Manag. 49, 75–86. doi: 10.1016/j.tourman.2015.02.013

Jung, T. H., Bae, S., Moorhouse, N., and Kwon, O. (2021). The impact of user perceptions of AR on purchase intention of location-based AR navigation systems. J. Retail. Consum. Serv. 61:102575. doi: 10.1016/j.jretconser.2021.102575

Jung, T. H., Lee, H., Chung, N., and tom Dieck, M. C. (2018). Cross-cultural differences in adopting mobile augmented reality at cultural heritage tourism sites. Int. J. Contemp. Hosp. Manag. 30, 1621–1645. doi: 10.1108/IJCHM-02-2017-0084

Kitchenham, B., Brereton, O. P., Budgen, D., Turner, M., Bailey, J., and Linkman, S. (2009). Systematic literature reviews in software engineering–a systematic literature review. Inf. Softw. Technol. 51, 7–15. doi: 10.1016/j.infsof.2008.09.009

Kowalczuk, P., Siepmann, C., and Adler, J. (2021). Cognitive, affective, and behavioral consumer responses to augmented reality in e-commerce: a comparative study. J. Bus. Res. 124, 357–373. doi: 10.1016/j.jbusres.2020.10.050

Lacka, E. (2020). Assessing the impact of full-fledged location-based augmented reality games on tourism destination visits. Curr. Issues Tour. 23, 345–357. doi: 10.1080/13683500.2018.1514370

Lim, X. J., Cheah, J. H., Ng, S. I., Basha, N. K., and Soutar, G. (2021). The effects of anthropomorphism presence and the marketing mix have on retail app continuance use intention. Technol. Forecast. Soc. Change 168:120763. doi: 10.1016/j.techfore.2021.120763

Manchanda, M., and Deb, M. (2021). On m-commerce adoption and augmented reality: a study on apparel buying using m-commerce in Indian context. J. Int. Commer. 20, 84–112.

McLean, G., and Wilson, A. (2019). Shopping in the digital world: examining customer engagement through augmented reality mobile applications. Comp. Hum. Behav. 101, 210–224. doi: 10.1016/j.chb.2019.07.002

Nee, A. Y., Ong, S. K., Chryssolouris, G., and Mourtzis, D. (2012). Augmented reality applications in design and manufacturing. CIRP Ann. 61, 657–679. doi: 10.1016/j.cirp.2012.05.010

Nikhashemi, S. R., Knight, H. H., Nusair, K., and Liat, C. B. (2021). Augmented reality in smart retailing: a (n)(A) symmetric approach to continuous intention to use retail brands’ mobile AR apps. J. Retail. Consum. Serv. 60:102464. doi: 10.1016/j.jretconser.2021.102464

Pantano, E., Rese, A., and Baier, D. (2017). Enhancing the online decision-making process by using augmented reality: a two country comparison of youth markets. J. Retail. Consum. Serv. 38, 81–95. doi: 10.1016/j.jretconser.2017.05.011

Park, M., and Yoo, J. (2020). Effects of perceived interactivity of augmented reality on consumer responses: a mental imagery perspective. J. Retail. Consum. Serv. 52:101912. doi: 10.1016/j.jretconser.2019.101912

Park, S., and Stangl, B. (2020). Augmented reality experiences and sensation seeking. Tour. Manag. 77:104023. doi: 10.1016/j.tourman.2019.104023

Paul, J., and Criado, A. R. (2020). The art of writing literature review: what do we know and what do we need to know? Int. Bus. Rev. 29:101717. doi: 10.1016/j.ibusrev.2020.101717

Paulo, M. M., Rita, P., Oliveira, T., and Moro, S. (2018). Understanding mobile augmented reality adoption in a consumer context. J. Hosp. Tour. Technol. 9, 142–157. doi: 10.1108/JHTT-01-2017-0006

Phua, J., and Kim, J. J. (2018). Starring in your own snapchat advertisement: influence of self-brand congruity, self-referencing and perceived humor on brand attitude and purchase intention of advertised brands. Telemat. Inform. 35, 1524–1533. doi: 10.1016/j.tele.2018.03.020

Plotkina, D., Dinsmore, J., and Racat, M. (2021). Improving service brand personality with augmented reality marketing. J. Serv. Market. doi: 10.1108/JSM-12-2020-0519 [Epub ahead of print].

Plotkina, D., and Saurel, H. (2019). Me or just like me? The role of virtual try-on and physical appearance in apparel M-retailing. J. Retail. Consum. Serv. 51, 362–377. doi: 10.1016/j.jretconser.2019.07.002

Poushneh, A. (2018). Augmented reality in retail: a trade-off between user’s control of access to personal information and augmentation quality. J. Retail. Consum. Serv. 41, 169–176. doi: 10.1016/j.jretconser.2017.12.010

Poushneh, A. (2021). How close do we feel to virtual product to make a purchase decision? Impact of perceived proximity to virtual product and temporal purchase intention. J. Retail. Consum. Serv. 63:102717. doi: 10.1016/j.jretconser.2021.102717

Poushneh, A., and Vasquez-Parraga, A. Z. (2017). Discernible impact of augmented reality on retail customer’s experience, satisfaction and willingness to buy. J. Retail. Consum. Serv. 34, 229–234. doi: 10.1016/j.jretconser.2016.10.005

Pozharliev, R., De Angelis, M., and Rossi, D. (2021). The effect of augmented reality versus traditional advertising: a comparison between neurophysiological and self-reported measures. Market. Lett. 33, 113–128. doi: 10.1007/s11002-021-09573-9

PwC (2019). Seeing is Believing: How Virtual Reality and Augmented Reality are Transforming Business and the Economy. Available online at: https://www.pwc.com/gx/en/technology/publications/assets/how-virtual-reality-and-augmented-reality.pdf (accessed April 10, 2022).

Qin, H., Osatuyi, B., and Xu, L. (2021a). How mobile augmented reality applications affect continuous use and purchase intentions: a cognition-affect-conation perspective. J. Retail. Consum. Serv. 63:102680. doi: 10.1016/j.jretconser.2021.102680

Qin, H., Peak, D. A., and Prybutok, V. (2021b). A virtual market in your pocket: how does mobile augmented reality (MAR) influence consumer decision making? J. Retail. Consum. Serv. 58:102337. doi: 10.1016/j.jretconser.2020.102337

Rauschnabel, P. A. (2018). Virtually enhancing the real world with holograms: an exploration of expected gratifications of using augmented reality smart glasses. Psychol. Market. 35, 557–572. doi: 10.1002/mar.21106

Rauschnabel, P. A., Babin, B. J., tom Dieck, M. C., Krey, N., and Jung, T. (2022). What is augmented reality marketing? Its definition, complexity, and future. J. Bus. Res. 142, 1140–1150. doi: 10.1016/j.jbusres.2021.12.084

Rauschnabel, P. A., Felix, R., and Hinsch, C. (2019). Augmented reality marketing: how mobile AR-apps can improve brands through inspiration. J. Retail. Consum. Serv. 49, 43–53. doi: 10.1016/j.jretconser.2019.03.004

Rauschnabel, P. A., Rossmann, A., and tom Dieck, M. C. (2017). An adoption framework for mobile augmented reality games: the case of Pokémon Go. Comp. Hum. Behav. 76, 276–286. doi: 10.1016/j.chb.2017.07.030

Rese, A., Baier, D., Geyer-Schulz, A., and Schreiber, S. (2017). How augmented reality apps are accepted by consumers: a comparative analysis using scales and opinions. Technol. Forecast. Soc. Change 124, 306–319. doi: 10.1016/j.techfore.2016.10.010

Saleem, M., Kamarudin, S., Shoaib, H. M., and Nasar, A. (2021). Retail consumers’ behavioral intention to use augmented reality mobile apps in Pakistan. J. Int. Commer. doi: 10.1080/15332861.2021.1975427

Saprikis, V., Avlogiaris, G., and Katarachia, A. (2021). Determinants of the intention to adopt mobile augmented reality apps in shopping malls among university students. J. Theor. Appl. Electron. Commer. Res. 16, 491–512. doi: 10.3390/jtaer16030030

Shin, H. H., and Jeong, M. (2021). Travelers’ motivations to adopt augmented reality (AR) applications in a tourism destination. J. Hosp. Tour. Technol. 12, 454–470. doi: 10.1108/JHTT-08-2018-0082

Smink, A. R., Frowijn, S., van Reijmersdal, E. A., van Noort, G., and Neijens, P. C. (2019). Try online before you buy: how does shopping with augmented reality affect brand responses and personal data disclosure. Electron. Commer. Res. Appl. 35:100854. doi: 10.1016/j.elerap.2019.100854

Smink, A. R., Reijmersdal, E., Noort, G. V., and Neijens, P. C. (2020). Shopping in augmented reality: the effects of spatial presence, personalization and intrusiveness on app and brand responses. J. Bus. Res. 118, 474–485. doi: 10.1016/j.jbusres.2020.07.018

Snyder, H. (2019). Literature review as a research methodology: an overview and guidelines. J. Bus. Res. 104, 333–339. doi: 10.1016/j.jbusres.2019.07.039

Song, H. K., Baek, E., and Choo, H. J. (2019). Try-on experience with augmented reality comforts your decision: focusing on the roles of immersion and psychological ownership. Inform. Technol. People 33, 1214–1234. doi: 10.1108/ITP-02-2019-0092

Sreejesh, S., and Mohapatra, S. (2014). Mixed Method Research Design: An Application in Consumer-Brand Relationships (CBR). Cham: Springer International Publishing.

Srivastava, A., Dasgupta, S. A., Ray, A., Bala, P. K., and Chakraborty, S. (2021). Relationships between the “Big Five” personality types and consumer attitudes in Indian students toward augmented reality advertising. Aslib J. Inform. Manag. 73, 967–991. doi: 10.1108/AJIM-02-2021-0046

Sung, E. C. (2021). The effects of augmented reality mobile app advertising: viral marketing via shared social experience. J. Bus. Res. 122, 75–87. doi: 10.1016/j.jbusres.2020.08.034

Tan, Y. C., Chandukala, S. R., and Reddy, S. K. (2022). Augmented reality in retail and its impact on sales. J. Market. 86, 48–66. doi: 10.1177/0022242921995449

tom Dieck, M. C., and Jung, T. (2018). A theoretical model of mobile augmented reality acceptance in urban heritage tourism. Curr. Issues Tour. 21, 154–174. doi: 10.1080/13683500.2015.1070801

tom Dieck, M. C., Jung, T., and Han, D. I. (2016). Mapping requirements for the wearable smart glasses augmented reality museum application. J. Hosp. Tour. Technol. 7, 230–253. doi: 10.1108/JHTT-09-2015-0036

tom Dieck, M. C., and Jung, T. H. (2017). Value of augmented reality at cultural heritage sites: a stakeholder approach. J. Dest. Market. Manag. 6, 110–117. doi: 10.1016/j.jdmm.2017.03.002

tom Dieck, M. C., Jung, T. H., and tom Dieck, D. (2018b). Enhancing art gallery visitors’ learning experience using wearable augmented reality: generic learning outcomes perspective. Curr. Issues Tour. 21, 2014–2034. doi: 10.1080/13683500.2016.1224818

tom Dieck, M. C., Jung, T. H., and Rauschnabel, P. A. (2018a). Determining visitor engagement through augmented reality at science festivals: an experience economy perspective. Comp. Hum. Behav. 82, 44–53. doi: 10.1016/j.chb.2017.12.043

Tranfield, D., Denyer, D., and Smart, P. (2003). Towards a methodology for developing evidence-informed management knowledge by means of systematic review. Br. J. Manag. 14, 207–222. doi: 10.1111/1467-8551.00375

Tsai, W. H. S., Tian, S. C., Chuan, C. H., and Li, C. (2020). Inspection or play? A study of how augmented reality technology can be utilized in advertising. J. Interact. Advert. 20, 244–257. doi: 10.1080/15252019.2020.1738292

Tussyadiah, I. P., Jung, T. H., and tom Dieck, M. C. (2018). Embodiment of wearable augmented reality technology in tourism experiences. J. Travel Res. 57, 597–611. doi: 10.1177/0047287517709090

Uribe, R., Labra, R., and Manzur, E. (2021). Modeling and evaluating the effectiveness of AR advertising and the moderating role of personality traits. Int. J. Advert. doi: 10.1080/02650487.202

van Esch, P., Arli, D., Gheshlaghi, M. H., Andonopoulos, V., von der Heidt, T., and Northey, G. (2019). Anthropomorphism and augmented reality in the retail environment. J. Retail. Consum. Serv. 49, 35–42. doi: 10.1016/j.jretconser.2019.03.002

Webster, J., and Watson, R. T. (2002). Analyzing the past to prepare for the future: writing a literature review. MIS Q. 26, 13–23. doi: 10.2307/4132319

Whang, J. B., Song, J. H., Choi, B., and Lee, J. H. (2021). The effect of augmented reality on purchase intention of beauty products: the roles of consumers’ control. J. Bus. Res. 133, 275–284. doi: 10.1016/j.jbusres.2021.04.057

Wu, H. K., Lee, S. W. Y., Chang, H. Y., and Liang, J. C. (2013). Current status, opportunities and challenges of augmented reality in education. Comp. Educ. 62, 41–49. doi: 10.1016/j.compedu.2012.10.024

Wu, X., and Lai, I. K. W. (2021). The acceptance of augmented reality tour app for promoting film-induced tourism: the effect of celebrity involvement and personal innovativeness. J. Hosp. Tour. Technol. 12, 454–470. doi: 10.1108/JHTT-03-2020-0054

Yang, S., Carlson, J. R., and Chen, S. (2020). How augmented reality affects advertising effectiveness: the mediating effects of curiosity and attention toward the ad. J. Retail. Consum. Serv. 54:102020. doi: 10.1016/j.jretconser.2019.102020

Yim, M. Y. C., Chu, S. C., and Sauer, P. L. (2017). Is augmented reality technology an effective tool for e-commerce? An interactivity and vividness perspective. J. Interact. Market. 39, 89–103. doi: 10.1016/j.intmar.2017.04.001

Yim, M. Y. C., and Park, S. Y. (2019). “I am not satisfied with my body, so I like augmented reality (AR)”: consumer responses to AR-based product presentations. J. Bus. Res. 100, 581–589. doi: 10.1016/j.jbusres.2018.10.041

Yuan, C., Wang, S., Yu, X., Kim, K. H., and Moon, H. (2021). The influence of flow experience in the augmented reality context on psychological ownership. Int. J. Advert. 40, 922–944. doi: 10.1080/02650487.2020.1869387

Zhang, T., Wang, W. Y. C., Cao, L., and Wang, Y. (2019). The role of virtual try-on technology in online purchase decision from consumers’ aspect. Int. Res. 29, 529–551. doi: 10.1108/IntR-12-2017-0540

Zhuang, X., Hou, X., Feng, Z., Lin, Z., and Li, J. (2021). Subjective norms, attitudes, and intentions of AR technology use in tourism experience: the moderating effect of millennials. Leisure Stud. 40, 392–406. doi: 10.1080/02614367.2020.1843692

Keywords : augmented reality, marketing, retailing, tourism, advertising, technology, brand, tourist destination

Citation: Du Z, Liu J and Wang T (2022) Augmented Reality Marketing: A Systematic Literature Review and an Agenda for Future Inquiry. Front. Psychol. 13:925963. doi: 10.3389/fpsyg.2022.925963

Received: 22 April 2022; Accepted: 20 May 2022; Published: 16 June 2022.

Reviewed by:

Copyright © 2022 Du, Liu and Wang. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY) . The use, distribution or reproduction in other forums is permitted, provided the original author(s) and the copyright owner(s) are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.

*Correspondence: Zhao Du, [email protected]

Disclaimer: All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article or claim that may be made by its manufacturer is not guaranteed or endorsed by the publisher.

An official website of the United States government

The .gov means it’s official. Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

The site is secure. The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

• Publications
• Account settings

Preview improvements coming to the PMC website in October 2024. Learn More or Try it out now .

• Advanced Search
• Journal List
• Sensors (Basel)

In-Depth Review of Augmented Reality: Tracking Technologies, Development Tools, AR Displays, Collaborative AR, and Security Concerns

Toqeer ali syed.

1 Faculty of Computer and Information Systems, Islamic University of Madinah, Medina 42351, Saudi Arabia

Muhammad Shoaib Siddiqui

Hurria binte abdullah.

2 School of Social Sciences and Humanities, National University of Science and Technology (NUST), Islamabad 44000, Pakistan

3 Malaysian Institute of Information Technology, Universiti Kuala Lumpur, Kuala Lumpur 50250, Malaysia

4 Department of Computer Science, Bacha Khan University Charsadda, Charsadda 24420, Pakistan

Abdallah Namoun

Ali alzahrani, adnan nadeem, ahmad b. alkhodre.

Augmented reality (AR) has gained enormous popularity and acceptance in the past few years. AR is indeed a combination of different immersive experiences and solutions that serve as integrated components to assemble and accelerate the augmented reality phenomena as a workable and marvelous adaptive solution for many realms. These solutions of AR include tracking as a means for keeping track of the point of reference to make virtual objects visible in a real scene. Similarly, display technologies combine the virtual and real world with the user’s eye. Authoring tools provide platforms to develop AR applications by providing access to low-level libraries. The libraries can thereafter interact with the hardware of tracking sensors, cameras, and other technologies. In addition to this, advances in distributed computing and collaborative augmented reality also need stable solutions. The various participants can collaborate in an AR setting. The authors of this research have explored many solutions in this regard and present a comprehensive review to aid in doing research and improving different business transformations. However, during the course of this study, we identified that there is a lack of security solutions in various areas of collaborative AR (CAR), specifically in the area of distributed trust management in CAR. This research study also proposed a trusted CAR architecture with a use-case of tourism that can be used as a model for researchers with an interest in making secure AR-based remote communication sessions.

1. Introduction

Augmented reality (AR) is one of the leading expanding immersive experiences of the 21st century. AR has brought a revolution in different realms including health and medicine, teaching and learning, tourism, designing, manufacturing, and other similar industries whose acceptance accelerated the growth of AR in an unprecedented manner [ 1 , 2 , 3 ]. According to a recent report in September 2022, the market size of AR and VR reached USD 27.6 billion in 2021, which is indeed estimated to reach USD 856.2 billion by the end of the year 2031 [ 4 ]. Big companies largely use AR-based technologies. For instance, Amazon, one of the leading online shopping websites, uses this technology to make it easier for customers to decide the type of furniture they want to buy. The rise in mobile phone technology also acted as an accelerator in popularizing AR. Earlier, mobile phones were not advanced and capable enough to run these applications due to their low graphics. Nowadays, however, smart devices are capable enough to easily run AR-based applications. A lot of research has been done on mobile-based AR. Lee et al. [ 5 ] developed a user-based design interface for educational purpose in mobile AR. To evaluate its conduct, fourth-grade elementary students were selected.

The adoption of AR in its various perspectives is backed up by a prolonged history. This paper presents an overview of the different integrated essential components that contribute to the working framework of AR, and the latest developments on these components are collected, analyzed, and presented, while the developments in the smart devices and the overall experience of the users have changed drastically [ 6 ]. The tracking technologies [ 7 ] are the building blocks of AR and establish a point of reference for movement and for creating an environment where the virtual and real objects are presented together. To achieve a real experience with augmented objects, several tracking technologies are presented which include techniques such as sensor-based [ 8 ], markerless, marker-based [ 9 , 10 ], and hybrid tracking technologies. Among these different technologies, hybrid tracking technologies are the most adaptive. As part of the framework constructed in this study, the simultaneous localization and mapping (SLAM) and inertial tracking technologies are combined. The SLAM technology collects points through cameras in real scenes while the point of reference is created using inertial tracking. The virtual objects are inserted on the relevant points of reference to create an augmented reality. Moreover, this paper analyzes and presents a detailed discussion on different tracking technologies according to their use in different realms i.e., in education, industries, and medical fields. Magnetic tracking is widely used in AR systems in medical, maintenance, and manufacturing. Moreover, vision-based tracking is mostly used in mobile phones and tablets because they have screen and camera, which makes them the best platform for AR. In addition, GPS tracking is useful in the fields of military, gaming, and tourism. These tracking technologies along with others are explained in detail in Section 3 .

Once the points of reference are collected after tracking, then another important factor that requires significant accuracy is to determine at which particular point the virtual objects have to be mixed with the real environment. Here comes the role of display technologies that gives the users of augmented reality an environment where the real and virtual objects are displayed visually. Therefore, display technologies are one of the key components of AR. This research identifies state-of-the-art display technologies that help to provide a quality view of real and virtual objects. Augmented reality displays can be divided into various categories. All have the same task to show the merged image of real and virtual content to the user’s eye. The authors have categorized the latest technologies of optical display after the advancements in holographic optical elements (HOEs). There are other categories of AR displays, such as video-based, eye multiplexed, and projected onto a physical surface. Optical see-through has two sub-categories, one is a free-space combiner and the other is a wave-guide combiner [ 11 , 12 ]. The thorough details of display technologies are presented in Section 4 .

To develop these AR applications, different tools are used depending on the type of application used. For example, to develop a mobile-based AR application for Android or iOS, ARToolKit [ 13 ] is used. However, FLARToolKit [ 14 ] is used to create a web-based application using Flash. Moreover, there are various plug-ins available that can be integrated with Unity [ 15 ] to create AR applications. These development tools are reviewed in Section 6 of this paper. Figure 1 provides an overview of reviewed topics of augmented reality in this paper.

Overview of AR, VR, and collaborative AR applications, tools, and technologies.

After going through a critical review process of collaborative augmented reality, the research has identified that some security flaws and missing trust parameters need to be addressed to ensure a pristine environment is provided to the users. Hackers and intruders are always active to exploit different vulnerabilities in the systems and software, but the previous research conducted on collaborative augmented reality did not depict reasonable efforts made in this direction to make secure collaboration. To address the security flaws and to provide secure communication in collaborative augmented reality, this research considered it appropriate to come up with a security solution and framework that can limit danger and risks that may be posed in the form of internal and external attacks. To actualize the secure platform, this study came up with an architecture for presenting a secure collaborative AR in the tourism sector in Saudi Arabia as a case study. The focus of the case study is to provide an application that can guide tourists during their visit to any of the famous landmarks in the country. This study proposed a secure and trustful mobile application based on collaborative AR for tourists. In this application, the necessary information is rendered on screen and the user can hire a guide to provide more information in detail. A single guide can provide the services to a group of tourists visiting the same landmark. A blockchain network was used to secure the applications and protect the private data of the users [ 16 , 17 ]. For this purpose, we performed a thorough literature review for an optimized solution regarding security and tracking for which we studies the existing tracking technologies and listed them in this paper along with their limitations. In our use case, we used a GPS tracking system to track the user’s movement and provide the necessary information about the visited landmark through the mobile application.

Observing the fact that AR operates in an integrated fashion that combines different technologies including tracking technologies, display technologies, AR tools, collaborative AR, and applications of AR has encouraged us to explore and present these conceptions and technologies in detail. To facilitate researchers on these different techniques, the authors have explored the research previously conducted and presented it in a Venn diagram, as shown in Figure 2 . Interested investigators can choose their required area of research in AR. As can be seen in the diagram, most research has been done in the area of tracking technologies. This is further divided into different types of tracking solutions including fiducial tracking, video-based tracking, and inertial tracking. Some papers lie in several categories for, example some papers such as [ 18 , 19 , 20 ] fall in both the fiducial tracking and sensor categories. Similarly, computer vision and display devices have some common papers, and inertial tracking and video-based tracking also have some papers in common. In addition, display devices share common papers with computer vision, mobile AR, design guidelines, tool-kits, evaluation, AR tags, and security and privacy of AR. Furthermore, visualization has different papers in common with business, interior design, and human–robot communication. While education shares some paper with gaming, simulation, medicine, heritage, and manufacturing. In short, we have tried to summarize all papers and further elaborate in their sections for the convenience of the reader.

Classification of reviewed papers with respect to tracking, display, authoring tools, application, Collaborative and security.

Contribution: This research presents a comprehensive review of AR and its associated technologies. A review of state-of-the-art tracking and display technologies is presented followed by different essential components and tools that can be used to effectively create AR experiences. The study also presents the newly emerging technologies such as collaborative augmented reality and how different application interactions are carried out. During the review phase, the research identified that the AR-based solutions and particularly collaborative augmented reality solutions are vulnerable to external intrusion. It is identified that these solutions lack security and the interaction could be hijacked, manipulated, and sometimes exposed to potential threats. To address these concerns, this research felt the need to ensure that the communication has integrity; henceforth, the research utilizes the state-of-the-art blockchain infrastructure for the collaborating applications in AR. The paper further proposes complete secure framework wherein different applications working remotely have a real feeling of trust with each other [ 21 ].

Outline : This paper presents the overview of augmented reality and its applications in various realms in Section 2 . In Section 3 , tracking technologies are presented, while a detailed overview of the display technologies is provided in Section 4 . Section 6 apprises readers on AR development tools. Section 7 highlights the collaborative research on augmented reality, while Section 8 interprets the AR interaction and input technologies. The paper presents the details of design guidelines and interface patterns in Section 9 , while Section 10 discusses the security and trust issues in collaborative AR. Section 12 highlights future directions for research, while Section 13 concludes this research.

2. Augmented Reality Overview

People, for many years, have been using lenses, light sources, and mirrors to create illusions and virtual images in the real world [ 22 , 23 , 24 ]. However, Ivan Sutherland was the first person to truly generate the AR experience. Sketchpad, developed at MIT in 1963 by Ivan Sutherland, is the world’s first interactive graphic application [ 25 ]. In Figure 3 , we have given an overview of the development of AR technology from the beginning to 2022. Bottani et al. [ 26 ] reviews the AR literature published during the time period of 2006–2017. Moreover, Sereno et al. [ 27 ] use a systematic survey approach to detail the existing literature available on the intersection of computer-supported collaborative work and AR.

Augmented reality advancement over time for the last 60 years.

2.1. Head-Mounted Display

Ens et al. [ 28 ] review the existing work on design exploration for mixed-scale gestures where the Hololens AR display is used to interweave larger gestures with micro-gestures.

2.2. AR Towards Applications

ARToolKit tracking library [ 13 ] aimed to provide the computer vision tracking of a square marker in real-time which fixed two major problems, i.e., enabling interaction with real-world objects and secondly, the user’s viewpoint tracking system. Researchers conducted studies to develop handheld AR systems. Hettig et al. [ 29 ] present a system called “Augmented Visualization Box” to asses surgical augmented reality visualizations in a virtual environment. Goh et al. [ 30 ] present details of the critical analysis of 3D interaction techniques in mobile AR. Kollatsch et al. [ 31 ] introduce a system that creates and introduces the production data and maintenance documentation into the AR maintenance apps for machine tools which aims to reduce the overall cost of necessary expertise and the planning process of AR technology. Bhattacharyya et al. [ 32 ] introduce a two-player mobile AR game known as Brick, where users can engage in synchronous collaboration while inhabiting the real-time and shared augmented environment. Kim et al. [ 33 ] suggest that this decade is marked by a tremendous technological boom particularly in rendering and evaluation research while display and calibration research has declined. Liu et al. [ 34 ] expand the information feedback channel from industrial robots to a human workforce for human–robot collaboration development.

2.3. Augmented Reality for the Web

Cortes et al. [ 35 ] introduce the new techniques of collaboratively authoring surfaces on the web using mobile AR. Qiao et al. [ 36 ] review the current implementations of mobile AR, enabling technologies of AR, state-of-art technology, approaches for potential web AR provisioning, and challenges that AR faces in a web-based system.

2.4. AR Application Development

The AR industry was tremendously increasing in 2015, extending from smartphones to websites with head-worn display systems such as Google Glass. In this regard, Agati et al. [ 18 ] propose design guidelines for the development of an AR manual assembly system which includes ergonomics, usability, corporate-related, and cognition.

AR for Tourism and Education: Shukri et al. [ 37 ] aim to introduce the design guidelines of mobile AR for tourism by proposing 11 principles for developing efficient AR design for tourism which reduces cognitive overload, provides learning ability, and helps explore the content while traveling in Malaysia. In addition to it, Fallahkhair et al. [ 38 ] introduce new guidelines to make AR technologies with enhanced user satisfaction, efficiency, and effectiveness in cultural and contextual learning using mobiles, thereby enhancing the tourism experience. Akccayir et al. [ 39 ] show that AR has the advantage of placing the virtual image on a real object in real time while pedagogical and technical issues should be addressed to make the technology more reliable. Salvia et al. [ 40 ] suggest that AR has a positive impact on learning but requires some advancements.

Sarkar et al. [ 41 ] present an AR app known as ScholAR. It introduces enhancing the learning skills of the students to inculcate conceptualizing and logical thinking among sevemth-grade students. Soleiman et al. [ 42 ] suggest that the use of AR improves abstract writing as compared to VR.

2.5. AR Security and Privacy

Hadar et al. [ 43 ] scrutinize security at all steps of AR application development and identify the need for new strategies for information security, privacy, and security, with a main goal to design and introduce capturing and mapping concerns. Moreover, in the industrial arena, Mukhametshin et al. [ 44 ] focus on developing sensor tag detection, tracking, and recognition for designing an AR client-side app for Siemen Company to monitor the equipment for remote facilities.

3. Tracking Technology of AR

Tracking technologies introduce the sensation of motion in the virtual and augmented reality world and perform a variety of tasks. Once a tracking system is rightly chosen and correctly installed, it allows a person to move within a virtual and augmented environment. It further allows us to interact with people and objects within augmented environments. The selection of tracking technology depends on the sort of environment, the sort of data, and the availability of required budgets. For AR technology to meet Azuma’s definition of an augmented reality system, it must adhere to three main components:

• it combines virtual and the real content;
• it is interactive in real time;
• is is registered in three dimensions.

The third condition of being “registered in three dimensions” alludes to the capability of an AR system to project the virtual content on physical surroundings in such a way that it seems to be part of the real world. The position and orientation (pose) of the viewer concerning some anchor in the real world must be identified and determined for registering the virtual content in the real environment. This anchor of the real world may be the dead-reckoning from inertial tracking, a defined location in space determined using GPS, or a physical object such as a paper image marker or magnetic tracker source. In short, the real-world anchor depends upon the applications and the technologies used. With respect to the type of technology used, there are two ways of registering the AR system in 3D:

• Determination of the position and orientation of the viewer relative to the real-world anchor: registration phase;
• Upgrading of viewer’s pose with respect to previously known pose: tracking phase.

In this document, the word “tracking” would define both phases as common terminology. There are two main types of tracking techniques which are explained as follows (depicted in Figure 4 ).

Categorization of augmented reality tracking techniques.

3.1. Markerless Tracking Techniques

Markerless tracking techniques further have two types, one is sensor based and another is vision based.

3.1.1. Sensor-Based Tracking

Magnetic Tracking Technology: This technology includes a tracking source and two sensors, one sensor for the head and another one for the hand. The tracking source creates an electromagnetic field in which the sensors are placed. The computer then calculates the orientation and position of the sensors based on the signal attenuation of the field. This gives the effect of allowing a full 360 range of motion. i.e., allowing us to look all the way around the 3D environment. It also allows us to move around all three degrees of freedom. The hand tracker has some control buttons that allow the user to navigate along the environment. It allows us to pick things up and understand the size and shape of the objects [ 45 ]. In Figure 5 we have tried to draw the tracking techniques to give a better understanding to the reader.

Augmented reality tracking techniques presentation.

Frikha et al. [ 46 ] introduce a new mutual occlusion problem handler. The problem of occlusion occurs when the real objects are in front of the virtual objects in the scene. The authors use a 3D positioning approach and surgical instrument tracking in an AR environment. The paradigm is introduced that is based on monocular image-based processing. The result of the experiment suggested that this approach is capable of handling mutual occlusion automatically in real-time.

One of the main issues with magnetic tracking is the limited positioning range [ 47 ]. Orientation and position can be determined by setting up the receiver to the viewer [ 48 ]. Receivers are small and light in weight and the magnetic trackers are indifferent to optical disturbances and occlusion; therefore, these have high update rates. However, the resolution magnetic field declines with the fourth power of the distance, and the strength of magnetic fields decline with the cube of the distance [ 49 ]. Therefore, the magnetic trackers have constrained working volume. Moreover, magnetic trackers are sensitive to environments around magnetic fields and the type of magnetic material used and are also susceptible to measurement jitter [ 50 ].

Magnetic tracking technology is widely used in the range of AR systems, with applications ranging from maintenance [ 51 ] to medicine [ 52 ] and manufacturing [ 53 ].

Inertial Tracking: Magnetometers, accelerometers, and gyroscopes are examples of inertial measurement units (IMU) used in inertial tracking to evaluate the velocity and orientation of the tracked object. An inertial tracking system is used to find the three rotational degrees of freedom relative to gravity. Moreover, the time period of the trackers’ update and the inertial velocity can be determined by the change in the position of the tracker.

Advantages of Inertial Tracking: It does not require a line of sight and has no range limitations. It is not prone to optical, acoustic, magnetic, and RE interference sources. Furthermore, it provides motion measurement with high bandwidth. Moreover, it has negligible latency and can be processed as fast as one desires.

Disadvantages of Inertial Tracking: They are prone to drift of orientation and position over time, but their major impact is on the position measurement. The rationale behind this is that the position must be derived from the velocity measurements. The usage of a filter could help in resolving this issue. However, the issue could while focusing on this, the filter can decrease the responsiveness and the update rate of the tracker [ 54 ]. For the ultimate correction of this issue of the drift, the inertial sensor should be combined with any other kind of sensor. For instance, it could be combined with ultrasonic range measurement devices and optical trackers.

3.1.2. Vision-Based Tracking

Vision-based tracking is defined as tracking approaches that ascertain the camera pose by the use of data captured from optical sensors and as registration. The optical sensors can be divided into the following three categories:

• visible light tracking;
• 3D structure tracking;
• infrared tracking.

In recent times, vision-based tracking AR is becoming highly popular due to the improved computational power of consumer devices and the ubiquity of mobile devices, such as tablets and smartphones, thereby making them the best platform for AR technologies. Chakrabarty et al. [ 55 ] contribute to the development of autonomous tracking by integrating the CMT into IBVS, their impact on the rigid deformable targets in indoor settings, and finally the integration of the system into the Gazebo simulator. Vision-based tracking is demonstrated by the use of an effective object tracking algorithm [ 56 ] known as the clustering of static-adaptive correspondences for deformable object tracking (CMT). Gupta et al. [ 57 ] detail the comparative analysis between the different types of vision-based tracking systems.

Moreover, Krishna et al. [ 58 ] explore the use of electroencephalogram (EEG) signals in user authentication. User authentication is similar to facial recognition in mobile phones. Moreover, this is also evaluated by combining it with eye-tracking data. This research contributes to the development of a novel evaluation paradigm and a biometric authentication system for the integration of these systems. Furthermore, Dzsotjan et al. [ 59 ] delineate the usefulness of the eye-tracking data evaluated during the lectures in order to determine the learning gain of the user. Microsoft HoloLens2’s designed Walk the Graph app was used to generate the data. Binary classification was performed on the basis of the kinematic graphs which users reported of their own movement.

Ranging from smartphones to laptops and even to wearable devices with suitable cameras located in them, visible light tracking is the most commonly used optical sensor. These cameras are particularly important because they can both make a video of the real environment and can also register the virtual content to it, and thereby can be used in video see-through AR systems.

Chen et al. [ 60 ] resolve the shortcomings of the deep learning lightning model (DAM) by combining the method of transferring a regular video to a 3D photo-realistic avatar and a high-quality 3D face tracking algorithm. The evaluation of the proposed system suggests its effectiveness in real-world scenarios when we have variability in expression, pose, and illumination. Furthermore, Rambach et al. [ 61 ] explore the details pipeline of 6DoF object tracking using scanned 3D images of the objects. The scope of research covers the initialization of frame-to-frame tracking, object registration, and implementation of these aspects to make the experience more efficient. Moreover, it resolves the challenges that we faced with occlusion, illumination changes, and fast motion.

3.1.3. Three-Dimensional Structure Tracking

Three-dimensional structure information has become very affordable because of the development of commercial sensors capable of accomplishing this task. It was begun after the development of Microsoft Kinect [ 62 ]. Syahidi et al. [ 63 ] introduce a 3D AR-based learning system for pre-school children. For determining the three-dimensional points in the scene, different types of sensors could be used. The most commonly used are the structured lights [ 64 ] or the time of flight [ 65 ]. These technologies work on the principle of depth analysis. In this, the real environment depth information is extracted by the mapping and the tracking [ 66 ]. The Kinect system [ 67 ], developed by Microsoft, is one of the widely used and well-developed approaches in Augmented Reality.

Rambach et al. [ 68 ] present the idea of augmented things: utilizing off-screen rendering of 3D objects, the realization of application architecture, universal 3D object tracking based on the high-quality scans of the objects, and a high degree of parallelization. Viyanon et al. [ 69 ] focus on the development of an AR app known as “AR Furniture" for providing the experience of visualizing the design and decoration to the customers. The customers fit the pieces of furniture in their rooms and were able to make a decision regarding their experience. Turkan et al. [ 70 ] introduce the new models for teaching structural analysis which has considerably improved the learning experience. The model integrates 3D visualization technology with mobile AR. Students can enjoy the different loading conditions by having the choice of switching loads, and feedback can be provided in the real-time by AR interface.

3.1.4. Infrared Tracking

The objects that emitted or reflected the light are some of the earliest vision-based tracking techniques used in AR technologies. Their high brightness compared to their surrounding environment made this tracking very easy [ 71 , 72 ]. The self-light emitting targets were also indifferent to the drastic illumination effects i.e., harsh shadows or poor ambient lighting. In addition, these targets could either be transfixed to the object being tracked and camera at the exterior of the object and was known as “outside-looking-in” [ 73 ]. Or it could be “inside-looking-out”, external in the environment with camera attached to the target [ 74 ]. The inside-looking-out configuration, compared to the sensor of the inside-looking-out system, has greater resolution and higher accuracy of angular orientation. The inside-looking-out configuration is used in the development of several systems [ 20 , 75 , 76 , 77 ], typically with infrared LEDs mounted on the ceiling and a head-mounted display with a camera facing externally.

3.1.5. Model-Based Tracking

The three-dimensional tracking of real-world objects has been the subject of researchers’ interest. It is not as popular as natural feature tracking or planner fiducials, however, a large amount of research has been done on it. In the past, tracking the three-dimensional model of the object was usually created by the hand. In this system, the lines, cylinders, spheres, circles, and other primitives were combined to identify the structure of objects [ 78 ]. Wuest et al. [ 79 ] focus on the development of the scalable and performance pipeline for creating a tracking solution. The structural information of the scene was extracted by using the edge filters. Additionally, for the determination of the pose, edge information and the primitives were matched [ 80 ].

In addition, Gao et al. [ 81 ] explore the tracking method to identify the different vertices of a convex polygon. This is done successfully as most of the markers are square. The coordinates of four vertices are used to determine the transformation matrix of the camera. Results of the experiment suggested that the algorithm was so robust to withstand fast motion and large ranges that make the tracking more accurate, stable, and real time.

The combination of edge-based tracking and natural feature tracking has the following advantages:

• It provides additional robustness [ 82 ].
• Enables spatial tracking and thereby is able to be operated in open environments [ 83 ].
• For variable and complex environments, greater robustness was required. Therefore, they introduced the concept of keyframes [ 84 ] in addition to the primitive model [ 85 ].

Figen et al. [ 86 ] demonstrate of a series of studies that were done at the university level in which participants were asked to make the mass volume of buildings. The first study demanded the solo work of a designer in which they had to work using two tools: MTUIs of the AR apps and analog tools. The second study developed the collaboration of the designers while using analog tools. The study has two goals: change in the behavior of the designer while using AR apps and affordances of different interfaces.

Developing and updating the real environment’s map simultaneously had been the subject of interest in model-based tracking. This has a number of developments. First, simultaneous localization and map building (SLAM) was primarily done for robot navigation in unknown environments [ 87 ]. In augmented reality, [ 88 , 89 ], this technique was used for tracking the unknown environment in a drift-free manner. Second, parallel mapping and tracking [ 88 ] was developed especially for AR technology. In this, the mapping of environmental components and the camera tracks were identified as a separate function. It improved tracking accuracy and also overall performance. However, like SLAM, it did not have the capability to close large loops in the constrained environment and area ( Figure 6 ).

Hybrid tracking: inertial and SLAM combined and used in the latest mobile-based AR tracking.

Oskiper et al. [ 90 ] propose a simultaneous localization and mapping (SLAM) framework for sensor fusion, indexing, and feature matching in AR apps. It has a parallel mapping engine and error-state extended Kalman filter (EKF) for these purposes. Zhang et al.’s [ 91 ] Jaguar is a mobile tracking AR application with low latency and flexible object tracking. This paper discusses the design, execution, and evaluation of Jaguar. Jaguar enables a markerless tracking feature which is enabled through its client development on top of ARCoreest from Google. ARCore is also helpful for context awareness while estimating and recognizing the physical size and object capabilities, respectively.

3.1.6. Global Positioning System—GPS Tracking

This technology refers to the positioning of outdoor tracking with reference to the earth. The present accuracy of the GPS system is up to 3 m. However, improvements are available with the advancements in satellite technology and a few other developments. Real-time kinematic (RTS) is one example of them. It works by using the carrier of a GPS signal. The major benefit of it is that it has the ability to improve the accuracy level up to the centimeter level. Feiner’s touring machine [ 92 ] was the first AR system that utilized GPS in its tracking system. It used the inclinometer/magnetometer and differential GPS positional tracking. The military, gaming [ 93 , 94 ], and the viewership of historical data [ 95 ] have applied GPS tracking for the AR experiences. As it only has the supporting positional tracking low accuracy, it could only be beneficial in the hybrid tracking systems or in the applications where the pose registration is not important. AR et al. [ 96 ] use the GPS-INS receiver to develop models for object motion having more precision. Ashutosh et al. [ 97 ] explore the hardware challenges of AR technology and also explore the two main components of hardware technology: battery performance and global positioning system (GPS). Table 1 provides a succinct categorization of the prominent tracking technologies in augmented reality. Example studies are referred to while highlighting the advantages and challenges of each type of tracking technology. Moreover, possible areas of application are suggested.

3.1.7. Miscellaneous Tracking

Yang et al. [ 98 ], in order to recognize the different forms of hatch covers having similar shapes, propose tracking and cover recognition methods. The results of the experiment suggest its real-time property and practicability, and tracking accuracy was enough to be implemented in the AR inspection environment. Kang et al. [ 99 ] propose a pupil tracker which consists of several features that make AR more robust: key point alignment, eye-nose detection, and infrared (NIR) led. NIR led turns on and off based on the illumination light. The limitation of this detector is that it cannot be applied in low-light conditions.

Summary of tracking techniques and their related attributes.

Moreover, Bach et al. [ 118 ] introduce an AR canvas for information visualization which is quite different from the traditional AR canvas. Therefore, dimensions and essential aspects for developing the visualization design for AR-canvas while enlisting the several limitations within the process. Zeng et al. [ 119 ] discuss the design and the implementation of FunPianoAR for creating a better AR piano learning experience. However, a number of discrepancies occurred with this system, and the initiation of a hybrid system is a more viable option. Rewkowski et al. [ 120 ] introduce a prototype system of AR to visualize the laparoscopic training task. This system is capable of tracking small objects and requires surgery training by using widely compatible and inexpensive borescopes.

3.1.8. Hybrid Tracking

Hybrid tracking systems were used to improve the following aspects of the tracking systems:

• Improving the accuracy of the tracking system.
• Coping with the weaknesses of the respective tracking methods.
• Adding more degrees of freedom.

Gorovyi et al. [ 108 ] detail the basic principles that make up an AR by proposing a hybrid visual tracking algorithm. The direct tracking techniques are incorporated with the optical flow technique to achieve precise and stable results. The results suggested that they both can be incorporated to make a hybrid system, and ensured its success in devices having limited hardware capabilities. Previously, magnetic tracking [ 109 ] or inertial trackers [ 110 ] were used in the tracking applications while using the vision-based tracking system. Isham et al. [ 111 ] use a game controller and hybrid tracking to identify and resolve the ultrasound image position in a 3D AR environment. This hybrid system was beneficial because of the following reasons:

• Low drift of vision-based tracking.
• Low jitter of vision-based tracking.
• They had a robust sensor with high update rates. These characteristics decreased the invalid pose computation and ensured the responsiveness of the graphical updates [ 121 ].
• They had more developed inertial and magnetic trackers which were capable of extending the range of tracking and did not require the line of sight. The above-mentioned benefits suggest that the utilization of the hybrid system is more beneficial than just using the inertial trackers.

In addition, Mao et al. [ 122 ] propose a new tracking system with a number of unique features. First, it accurately translates the relative distance into the absolute distance by locating the reference points at the new positions. Secondly, it embraces the separate receiver and sender. Thirdly, resolves the discrepancy in the sampling frequency between the sender and receiver. Finally, the frequency shift due to movement is highly considered in this system. Moreover, the combination of the IMU sensor and Doppler shift with the distributed frequency modulated continuous waveform (FMCW) helps in the continuous tracking of mobile due to multiple time interval developments. The evaluation of the system suggested that it can be applied to the existing hardware and has an accuracy to the millimeter level.

The GPS tracking system alone only provides the positional information and has low accuracy. So, GPS tracking systems are usually combined with vision-based tracking or inertial sensors. The intervention would help gain the full pose estimation of 6DoF [ 123 ]. Moreover, backup tracking systems have been developed as an alternative when the GPS fails [ 98 , 124 ]. The optical tracking systems [ 100 ] or the ultrasonic rangefinders [ 101 ] can be coupled with the inertial trackers for enhancing efficiency. As the differential measurement approach causes the problem of drift, these hybrid systems help resolve them. Furthermore, the use of gravity as a reference to the inertial sensor made them static and bound. The introduction of the hybrid system would make them operate in a simulator, vehicle, or in any other moving platform [ 125 ]. The introduction of accelerators, cameras, gyroscopes [ 126 ], global positioning systems [ 127 ], and wireless networking [ 128 ] in mobile phones such as tablets and smartphones also gives an opportunity for hybrid tracking. Furthermore, these devices have the capability of determining outdoor as well as indoor accurate poses [ 129 ].

3.2. Marker-Based Tracking

Fiducial Tracking: Artificial landmarks for aiding the tracking and registration that are added to the environment are known as fiducial. The complexity of fiducial tracking varies significantly depending upon the technology and the application used. Pieces of paper or small colored LEDs were used typically in the early systems, which had the ability to be detected using color matching and could be added to the environment [ 130 ]. If the position of fiducials is well-known and they are detected enough in the scene then the pose of the camera can be determined. The positioning of one fiducial on the basis of a well-known previous position and the introduction of additional fiducials gives an additional benefit that workplaces could dynamically extend [ 131 ]. A QR code-based fudicial/marker is also proposed by some researchers for marker-/tag-based tracking [ 115 ]. With the progression of work on the concept and complexity of the fiducials, additional features such as multi-rings were introduced for the detection of fiducials at much larger distances [ 116 ]. A minimum of four points of a known position is needed for determining for calculating the pose of the viewer [ 117 ]. In order to make sure that the four points are visible, the use of these simpler fiducials demanded more care and effort for placing them in the environment. Examples of such fiducials are ARToolkit and its successors, whose registration techniques are mostly planar fiducial. In the upcoming section, AR display technologies are discussed to fulfill all the conditions of Azuma’s definition.

3.3. Summary

This section provides comprehensive details on tracking technologies that are broadly classified into markerless and marker-based approaches. Both types have many subtypes whose details, applications, pros, and cons are provided in a detailed fashion. The different categories of tracking technologies are presented in Figure 4 , while the summary of tracking technologies is provided in Figure 7 . Among the different tracking technologies, hybrid tracking technologies are the most adaptive. This study combined SLAM and inertial tracking technologies as part of the framework presented in the paper.

Steps for combining real and virtual content.

4. Augmented Reality Display Technology

For the combination of a real and the virtual world in such a way that they both look superimposed on each other, as in Azuma’s definition, some technology is necessarily required to display them.

4.1. Combination of Real and the Virtual Images

Methods or procedures required for the merging of the virtual content in the physical world include camera calibration, tracking, registration, and composition as depicted in Figure 7 .

4.2. Camera vs. Optical See Through Calibration

It is a procedure or an optical model in which the eye display geometry or parameters define the user’s view. Or, in other words, it is a technique of complementing the dimensions and parameters of the physical and the virtual camera.

In AR, calibration can be used in two ways, one is camera calibration, and another is optical calibration. The camera calibration technique is used in video see-through (VST) displays. However, optical calibration is used in optical see-through (OST) displays. OST calibration can be further divided into three umbrellas of techniques. Initially, manual calibration techniques were used in OST. Secondly, semi-automatic calibration techniques were used, and thirdly, we have now automatic calibration techniques. Manual calibration requires a human operator to perform the calibration tasks. Semi-automatic calibration, such as simple SPAAM and display relative calibration (DRC), partially collect some parameters automatically, which usually needed to be done manually in earlier times by the user. Thirdly, the automatic OST calibration was proposed by Itoh et al. in 2014 with the model of interaction-free display calibration technique (INDICA) [ 132 ]. In video see through (VST), computer vision techniques such as cameras are used for the registration of real environments. However, in optical see through (OST), VST calibration techniques cannot be used as it is more complex because cameras are replaced by human eyes. Various calibration techniques were developed for OST. The author evaluates the registration accuracy of the automatic OST head-mounted display (HMD) calibration technique called recycled INDICA presented by Itoh and Klinker. In addition, two more calibration techniques called the single-point active alignment method (SPAAM) and degraded SPAAM were also evaluated. Multiple users were asked to perform two separate tasks to check the registration and the calibration accuracy of all three techniques can be thoroughly studied. Results show that the registration method of the recycled INDICA technique is more accurate in the vertical direction and showed the distance of virtual objects accurately. However, in the horizontal direction, the distance of virtual objects seemed closer than intended [ 133 ]. Furthermore, the results show that recycled INDICA is more accurate than any other common technique. In addition, this technique is also more accurate than the SPAAM technique. Although, different calibration techniques are used for OST and VST displays, as discussed in [ 133 ], they do not provide all the depth cues, which leads to interaction problems. Moreover, different HMDs have different tracking systems. Due to this, they are all calibrated with an external independent measuring system. In this regard, Ballestin et al. propose a registration framework for developing AR environments where all the real objects, including users, and virtual objects are registered in a common frame. The author also discusses the performance of both displays during interaction tasks. Different simple and complex tasks such as 3D blind reaching are performed using OST and VST HMDs to test their registration process and interaction of the users with both virtual and real environments. It helps to compare the two technologies. The results show that these technologies have issues, however, they can be used to perform different tasks [ 134 ].

Non-Geometric Calibration Method

Furthermore, these geometric calibrations lead to perceptual errors while converting from 3D to 2D [ 135 ]. To counter this problem, parallax-free video see-through HMDs were proposed; however, they were very difficult to create. In this regard, Cattari et al. in 2019 proposes a non-stereoscopic video see-through HMD for a close-up view. It mitigates perceptual errors by mitigating geometric calibration. Moreover, the authors also identify the problems of non-stereoscopic VST HMD. The aim is to propose a system that provides a view consistent with the real world [ 136 , 137 ]. Moreover, State et al. [ 138 ] focus on a VST HMD system that generates zero eye camera offset. While Bottechia et al. [ 139 ] present an orthoscope monocular VST HMD prototype.

4.3. Tracking Technologies

Some sort of technology is required to track the position and orientation of the object of interest which could either be a physical object or captured by a camera with reference to the coordinate plan (3D or 2D) of a tracking system. Several technologies ranging from computer vision techniques to 6DoF sensors are used for tracking the physical scenes.

4.4. Registration

Registration is defined as a process in which the coordinate frame used for manifesting the virtual content is complemented by the coordinate frame of the real-world scene. This would help in the accurate alignment of the virtual content and the physical scene.

4.5. Composition

Now, the accuracy of two important steps, i.e., the accurate calibration of the virtual camera and the correct registration of the virtual content relative to the physical world, signifies the right correspondence between the physical environment and the virtual scene which is generated on the basis of tracking updates. This process then leads to the composition of the virtual scene’s image and can be done in two ways: Optically (or physically) or digitally. The physical or digital composition depends upon the configuration and dimensions of the system used in the augmented reality system.

4.6. Types of Augmented Reality Displays

The combination of virtual content in the real environment divides the AR displays into four major types, as depicted in Figure 8 . All have the same job to show the merged image of real and virtual content to the user’s eye. The authors have categorized the latest technologies of optical display after the advancements in holographic optical elements HOEs. There are other categories of AR display that arealso used, such as video-based, eye multiplexed, and projection onto a physical surface.

Types of augmented reality display technologies.

4.7. Optical See-Through AR Display

These kinds of displays use the optical system to merge the real scenes and virtual scene images. Examples of AR displays are head-up display HUD systems of advanced cars and cockpits of airplanes. These systems consist of the following components: beam splitters, which can be of two forms, combined prisms or half mirrors. Most beam splitters reflect the image from the video display. This reflected image is then integrated with a real-world view that can be visualized from the splitter. For half mirrors as a beam splitter, the working way is somewhat different: the real-world view is reflected on the mirror rather than the image of the video display. At the same time, the video display can also be viewed from the mirror. The transport projection system is semi-transparent optical technology used in optical display systems. Their semi-transparent property allows the viewer to witness the view at the back of the screen. Additionally, this system uses diffused light to manifest the exhibited image. Examples of semi-display optical systems are transparent projection film, transparent LCDs, etc. Optical combiners are used for the combination of virtual and real scene images. Optical see-through basically has two sub-categories, one is a free-space combiner and the other is a wave-guide combiner [ 140 ]. Additionally, now the advancement of technology has enabled technicians to make self-transparent displays. This self-transparent feature help in the miniaturization and simplification of the size and structure of the optical see-through displays.

4.7.1. Free-Space Combiners

Papers related to free space combiners are discussed here. Pulli et al. [ 11 ] introduce a second-generation immersive optical see-through AR system known as meta 2. It is based on an optical engine that uses the free-form visor to make a more immersive experience. Another traditional geometric display is ultra-fast high-resolution piezo linear actuators combined with Alvarez’s lens to make a new varifocal optical see-through HMD. It uses a beamsplitter which acts as an optical combiner to merge the light paths of the real and virtual worlds [ 12 ]. Another type of free-space combiner is Maxwellian-type [ 112 , 113 , 114 , 141 ]. In [ 142 ], the author employs the random structure as a spatial light modulator for developing a light-field near-eye display based on random pinholes. The latest work in [ 143 , 144 ] introduces an Ini-based light field display using the multi-focal micro-lens to propose the extended depth of the field. To enhance the eyebox view there is another technique called puppil duplication steering [ 145 , 146 , 147 , 148 , 149 , 150 ]. In this regard, refs. [ 102 , 151 ] present the eyebox-expansion method for the holographic near-eye display and pupil-shifting holographic optical element (PSHOE) for the implementation. Additionally, the design architecture is discussed and the incorporation of the holographic optical element within the holographic display system is discussed. There is another recent technique similar to the Maxwellian view called pin-light systems. It increases the Maxwellian view with larger DoFs [ 103 , 104 ].

4.7.2. Wave-Guide Combiner

The waveguide combiner basically traps light into TIR as opposed to free-space, which lets the light propagate without restriction [ 104 , 105 , 106 ]. The waveguide combiner has two types, one is diffractive waveguides and another is achromatic waveguides [ 107 , 152 , 153 , 154 , 155 ].

4.8. Video-Based AR Displays

These displays execute the digital processes as their working principle [ 156 ]. To rephrase, the merging of the physical world video and the virtual images, in video display systems, is carried out by digital processing. The working of the video-based system depends upon the video camera system by which it fabricates the real-world video into digital. The rationale behind this system is that the composition of the physical world’s video or scenario with the virtual content could be manifested digitally through the operation of a digital image processing technique [ 157 ]. Mostly, whenever the user has to watch the display, they have to look in the direction of the video display, and the camera is usually attached at the back of this display. So, the camera faces the physical world scene. These are known as “video see-through displays" because in them the real world is fabricated through the digitization (i.e., designing the digital illusion) of these video displays. Sometimes the design of the camera is done in such a way that it may show an upside-down image of an object, create the illusion of a virtual mirror, or site the image at a distant place.

4.9. Projection-Based AR Display

Real models [ 158 ] and walls [ 159 ] could be example of projection-based AR displays. All the other kinds of displays use the display image plan for the combination of the real and the virtual image. However, this display directly overlays the virtual scene image over the physical object. They work in the following manner:

• First, they track the user’s viewpoint.
• Secondly, they track the physical object.
• Then, they impart the interactive augmentation [ 160 ].

Mostly, these displays have a projector attached to the wall or a ceiling. This intervention has an advantage as well as a disadvantage. The advantage is that this does not demand the user to wear something. The disadvantage is that it is static and restricts the display to only one location of projection. For resolving this problem and making the projectors mobile, a small-sized projector has been made that could be easily carried from one place to another [ 161 ]. More recently, with the advancement of technology, miniaturized projectors have also been developed. These could be held in the hand [ 162 ] or worn on the chest [ 163 ] or head [ 164 ].

4.10. Eye-Multiplexed Augmented Reality Display

In eye-multiplexed AR displays, the users are allowed to combine the views of the virtual and real scenes mentally in their minds [ 72 ]. Rephrased, these displays do not combine the image digitally; therefore, it requires less computational power [ 72 ]. The process is as follows. First, the virtual image gets registered to the physical environment. Second, the user will get to see the same rendered image as the physical scene because the virtual image is registered to the physical environment. The user has to mentally configure the images in their mind to combine the virtual and real scene images because the display does not composite the rendered and the physical image. For two reasons, the display should be kept near the viewer’s eye: first, the display could appear as an inset into the real world, and second, the user would have to put less effort into mentally compositing the image.

The division of the displays on the basis of the position of the display between the real and virtual scenes is referred to as the “eye to world spectrum”.

4.11. Head-Attached Display

Head-attached displays are in the form of glasses, helmets, or goggles. They vary in size from smaller to bigger. However, with the advancement of technology, they are becoming lighter to wear. They work by displaying the virtual image right in front of the user’s eye. As a result, no other physical object can come between the virtual scene and the viewer’s eye. Therefore, the third physical object cannot occlude them. In this regard, Koulieris et al. [ 165 ] summarized the work on immersive near-eye tracking technologies and displays. Results suggest various loopholes within the work on display technologies: user and environmental tracking and emergence–accommodation conflict. Moreover, it suggests that advancement in the optics technology and focus adjustable lens will improve future headset innovations and creation of a much more comfortable HMD experience. In addition to it, Minoufekr et al. [ 166 ] illustrate and examine the verification of CNC machining using Microsoft HoloLens. In addition, they also explore the performance of AR with machine simulation. Remote computers can easily pick up the machine models and load them onto the HoloLens as holograms. A simulation framework is employed that makes the machining process observed prior to the original process. Further, Franz et al. [ 88 ] introduce two sharing techniques i.e., over-the-shoulder AR and semantic linking for investigating the scenarios in which not every user is wearing HWD. Semantic linking portrays the virtual content’s contextual information on some large display. The result of the experiment suggested that semantic linking and over-the-shoulder suggested communication between participants as compared to the baseline condition. Condino et al. [ 167 ] aim to explore two main aspects. First, to explore complex craniotomies to gauge the reliability of the AR-headsets [ 168 ]. Secondly, for non-invasive, fast, and completely automatic planning-to-patient registration, this paper determines the efficacy of patient-specific template-based methodology for this purpose.

4.12. Head-Mounted Displays

The most commonly used displays in AR research are head-mounted displays (HMDs). They are also known as face-mounted displays or near-eye displays. The user puts them on, and the display is represented right in front of their eyes. They are most commonly in the form of goggles. While using HMDs, optical and video see-through configurations are most commonly used. However, recently, head-mounted projectors are also explored to make them small enough to wear. Examples of smart glasses, Recon Jet, Google glass, etc., are still under investigation for their usage in head-mounted displays. Barz et al. [ 169 ] introduce a real-time AR system that augments the information obtained from the recently attended objects. This system is implemented by using head-mounted displays from the state-of-the-art Microsoft HoloLens [ 170 ]. This technology can be very helpful in the fields of education, medicine, and healthcare. Fedosov et al. [ 171 ] introduce a skill system, and an outdoor field study was conducted on the 12 snowboards and skiers. First, it develops a system that has a new technique to review and share personal content. Reuter et al. [ 172 ] introduce the coordinative concept, namely RescueGlass, for German Red Cross rescue dog units. This is made up of a corresponding smartphone app and a hands-free HMD (head-mounted display) [ 173 ]. This is evaluated to determine the field of emergency response and management. The initial design is presented for collaborative professional mobile tasks and is provided using smart glasses. However, the evaluation suggested a number of technical limitations in the research that could be covered in future investigations. Tobias et al. [ 174 ] explore the aspects such as ambiguity, depth cues, performed tasks, user interface, and perception for 2D and 3D visualization with the help of examples. Secondly, they categorize the head-mounted displays, introduce new concepts for collaboration tasks, and explain the concepts of big data visualization. The results of the study suggested that the use of collaboration and workspace decisions could be improved with the introduction of the AR workspace prototype. In addition, these displays have lenses that come between the virtual view and the user’s eye just like microscopes and telescopes. So, the experiments are under investigation to develop a more direct way of viewing images such as the virtual retinal display developed in 1995 [ 175 ]. Andersson et al. [ 176 ] show that training, maintenance, process monitoring, and programming can be improved by integrating AR with human—robot interaction scenarios.

4.13. Body-Attached and Handheld Displays

Previously, the experimentation with handheld display devices was done by tethering the small LSDs to the computers [ 177 , 178 ]. However, advancements in technology have improved handheld devices in many ways. Most importantly, they have become so powerful to operate AR visuals. Many of them are now used in AR displays such as personal digital assistants [ 179 ], cell phones [ 180 ], tablet computers [ 181 ], and ultra-mobile PCs [ 182 ].

4.13.1. Smartphones and Computer tablets

In today’s world, computer tablets and smartphones are powerful enough to run AR applications, because of the following properties: various sensors, cameras, and powerful graphic processors. For instance, Google Project Tango and ARCore have the most depth imaging sensors to carry out the AR experiences. Chan et al. [ 183 ] discuss the challenges faced while applying and investigating methodologies to enhance direct touch interaction on intangible displays. Jang et al. [ 184 ] aim to explore e-leisure due to enhancement in the use of mobile AR in outdoor environments. This paper uses three methods, namely markerless, marker-based, and sensorless to investigate the tracking of the human body. Results suggested that markerless tracking cannot be used to support the e-leisure on mobile AR. With the advancement of electronic computers, OLED panels and transparent LCDs have been developed. It is also said that in the future, building handheld optical see-through devices would be available. Moreover, Fang et al. [ 185 ] focus on two main aspects of mobile AR. First, a combination of the inertial sensor, 6DoF motion tracking based on sensor-fusion, and monocular camera for the realization of mobile AR in real-time. Secondly, to balance the latency and jitter phenomenon, an adaptive filter design is introduced. Furthermore, Irshad et al. [ 186 ] introduce an evaluation method to assess mobile AR apps. Additionally, Loizeau et al. [ 187 ] explore a way of implementing AR for maintenance workers in industrial settings.

4.13.2. Micro Projectors

Micro projectors are an example of a mobile phone-based AR display. Researchers are trying to investigate these devices that could be worn on the chest [ 188 ], shoulder [ 189 ], or wrist [ 190 ]. However, mostly they are handheld and look almost like handheld flashlights [ 191 ].

4.13.3. Spatial Displays

Spatial displays are used to exhibit a larger display. Henceforth, these are used in the location where more users could get benefit from them i.e., public displays. Moreover, these displays are static, i.e., they are fixed at certain positions and can not be mobilized.

The common examples of spatial displays include those that create optical see-through displays through the use of optical beamers: half mirror workbench [ 192 , 193 , 194 , 195 ] and virtual showcases. Half mirrors are commonly used for the merging of haptic interfaces. They also enable closer virtual interaction. Virtual showcases may exhibit the virtual images on some solid or physical objects mentioned in [ 196 , 197 , 198 , 199 , 200 ]. Moreover, these could be combined with the other type of technologies to excavate further experiences. The use of volumetric 3D displays [ 201 ], autostereoscopic displays [ 202 ], and other three-dimensional displays could be researched to investigate further interesting findings.

4.13.4. Sensory Displays

In addition to visual displays, there are some sensors developed that work with other types of sensory information such as haptic or audio sensors. Audio augmentation is easier than video augmentation because the real world and the virtual sounds get naturally mixed up with each other. However, the most challenging part is to make the user think that the virtual sound is spatial. Multi-channel speaker systems and the use of stereo headphones with the head-related transfer function (HRTF) are being researched to cope with this challenge [ 203 ]. Digital sound projectors use the reverberation and the interference of sound by using a series of speakers [ 204 ]. Mic-throughand hear-through systems, developed by Lindeman [ 205 , 206 , 206 ], work effectively and are analogous to video and optical see-through displays. The feasibility test for this system was done by using a bone conduction headset. Other sensory experiences are also being researched. For example, the augmentation of the gustatory and olfactory senses. Olfactory and visual augmentation of a cookie-eating scene was developed by Narumi [ 207 ]. Table 2 gives the primary types of augmented reality display technologies and discusses their advantages and disadvantages.

A Summary of Augmented Reality Display Technologies.

4.14. Summary

This section presented a comprehensive survey of AR display technologies. These displays not only focused on combing the virtual and real-world scenes of visual experience but also other ways of combining the sensory, olfactory, and gustatory senses are also under examination by researchers. Previously, head-mounted displays were most commonly in practice; however, now handheld devices and tablets or mobile-based experiences are widely used. These things may also change in the future depending on future research and low cost. The role of display technologies was elaborated first, thereafter, the process of combining the real and augmented contents and visualizing these to users was elaborated. The section elaborated thoroughly on where the optical see-through and video-based see-through are utilized along with details of devices. Video see-through (VST) is used in head-mounted displays and computer vision techniques such as cameras are used for registration of real environment, while in optical see-through (OST), VST calibration techniques cannot be used due to complexity, and cameras are replaced by human eyes. The optical see-through is a trendy approach as of now. The different calibration approaches are presented and analyzed and it is identified after analysis, the results show that recycled INDICA is more accurate than other common techniques presented in the paper. This section also presents video-based AR displays. Figure 8 present a classified representation of different display technologies pertaining to video-based, head-mounted, and sensory-based approaches. The functions and applications of various display technologies are provided in Table 2 Each of the display technologies presented has its applicability in various realms whose details are summarized in the same Table 2 .

5. Walking and Distance Estimation in AR

The effectiveness of AR technologies depends on the perception of distance of users from both real and virtual objects [ 214 , 215 ]. Mikko et al. performed some experiments to judge depth using stereoscopic depth perception [ 216 ]. The perception can be changed if the objects are on the ground or off the ground. In this regard, Carlos et al. also proposed a comparison between the perception of distance of these objects on the ground and off the ground. The experiment was done where the participant perceived the distance from cubes on the ground and off the ground as well. The results showed that there is a difference between both perceptions. However, it was also shown that this perception depends on whether the vision is monocular or binocular [ 217 ]. Plenty of research has been done in outdoor navigation and indoor navigation areas with AR [ 214 ]. In this regard, Umair et al. present an indoor navigation system in which Google glass is used as a wearable head-mounted display. A pre-scanned 3D map is used to track an indoor environment. This navigation system is tested on both HMD and handheld devices such as smartphones. The results show that the HMD was more accurate than the handheld devices. Moreover, it is stated that the system needs more improvement [ 218 ].

6. AR Development Tool

In addition to the tracking and display devices, there are some other software tools required for creating an AR experience. As these are hardware devices, they require some software to create an AR experience. This section explores the tools and the software libraries. It will cover both the aspects of the commercially available tools and some that are research related. Different software applications require a separate AR development tool. A complete set of low-level software libraries, plug-ins, platforms, and standalones are presented in Figure 9 so they can be summarized for the reader.

Stack of development libraries, plug-ins, platforms, and standalone authoring tools for augmented reality development.

In some tools, computer vision-based tracking (see Section 3.1.2 ) is preferred for creating an indoor experience, while others utilized sensors for creating an outdoor experience. The use of each tool would depend upon the type of platform (web or mobile) for which it is designed. Further in the document, the available AR tools are discussed, which consist of both novel tools and those that are widely known. Broadly, the following tools will be discussed:

• Low-level software development tools: needs high technological and programming skills.
• Rapid prototyping: provides a quick experience.
• Plug-ins that run on the existing applications.
• Standalone tools that are specifically designed for non-programmers.
• Next generation of AR developing tools.

6.1. Low-Level Software Libraries and Frameworks

Low-level software and frameworks make the functions of display and core tracking accessible for creating an AR experience. One of the most commonly used AR software libraries, as discussed in the previous section, is ARToolKit. ARToolKit is developed by Billing Hurst and Kato that has two versions [ 219 ]. It works on the principle of a fiducial marker-based registration system [ 220 ]. There are certain advances in the ARToolKit discussed related to the tracking in [ 213 , 221 , 222 , 223 , 224 ]. The first one is an open-source version that provides the marker-based tracking experience, while the second one provides natural tracking features and is a commercial version. It can be operated on Linux, Windows, and Mac OS desktops as it is written in the C language. It does not require complex graphics or built-in support for accomplishing its major function of providing a tracking experience, and it can operate simply by using low-level OpenGL-based rendering. ARToolKit requires some additional libraries such as osgART and OpenScene graph library so it can provide a complete AR experience to AR applications. OpenScene graph library is written in C language and operates as an open-source graph library. For graphic rendering, the OpenScene graph uses OpenGL. Similarly, the osgART library links the OpenScene graph and ARToolKit. It has advanced rendering techniques that help in developing the interacting AR application. OsgART library has a modular structure and can work with any other tracking library such as PTAM and BazAR, if ARtoolkit is not appropriate. BazAR is a workable tracking and geometric calibration library. Similarly, PTAM is a SLAM-based tracking library. It has a research-based and commercial license. All these libraries are available and workable to create a workable AR application. Goblin XNA [ 208 ] is another platform that has the components of interactions based on physics, video capture, a head-mounted AR display on which output is displayed, and a three-dimensional user interface. With Goblin XNA, existing XNA games could be easily modified [ 209 ]. Goblin XNA is available as a research and educational platform. Studierstube [ 210 ] is another AR system through which a complete AR application can be easily developed. It has tracking hardware, input devices, different types of displays, AR HMD, and desktops. Studierstube was specially developed to subsidize the collaborative applications [ 211 , 212 ]. Studierstube is a research-oriented library and is not available as commercial and workable easy-to-use software. Another commercially available SDK is Metaio SDK [ 225 ]. It consists of a variety of AR tracking technologies including image tracking, marker tracking, face tracking, external infrared tracking, and a three-dimensional object tracking. However, in May 2015, it was acquired by Apple and Metaio products and subscriptions are no longer available for purchase. Some of these libraries such as Studierstube and ARToolKit were initially not developed for PDAs. However, they have been re-developed for PDAs [ 226 ]. It added a few libraries in assistance such as open tracker, pocketknife for hardware abstraction, KLIMT as mobile rendering, and the formal libraries of communication (ACE) and screen graphs. All these libraries helped to develop a complete mobile-based AR collaborative experience [ 227 , 228 ]. Similarly, ARToolKit also incorporated the OpenScene graph library to provide a mobile-based AR experience. It worked with Android and iOS with a native development kit including some Java wrapping classes. Vuforia’s Qualcomm low-level library also provided an AR experience for mobile devices. ARToolKit and Vuforia both can be installed as a plug-in in Unity which provides an easy-to-use application development for various platforms. There are a number of sensors and low-level vision and location-based libraries such as Metaio SDK and Droid which were developed for outdoor AR experience. In addition to these low-level libraries, the Hit Lab NZ Outdoor AR library provided high-level abstraction for outdoor AR experience [ 229 ]. Furthermore, there is a famous mobile-based location AR tool that is called Hoppala-Augmentation. The geotags given by this tool can be browsed by any of the AR browsers including Layar, Junaio, and Wikitude [ 230 ].

ARTag is designed to resolve the limitations of ARToolkit. This system was developed to resolve a number of issues:

• Resolving inaccurate pattern matching by preventing the false positive matches.
• Enhancing the functioning in the presence of the imbalanced lightening conditions.
• Making the occlusion more invariant.

However, ARTag is no longer actively under development and supported by the NRC Lab. A commercial license is not available.

6.3. Wikitude Studio

This is also a web-based authoring tool for creating mobile-based AR applications. It allows the utilization of computer vision-based technology for the registration of the real world. Several types of media such as animation and 3D models can be used for creating an AR scene. One of the important features of Wikitude is that the developed mobile AR content can be uploaded not only on the Wikitude AR browser app but also on a custom mobile app [ 231 ]. Wikitude’s commercial plug-in is also available in Unity to enhance the AR experience for developers.

6.4. Standalone AR Tools

Standalone AR tools are mainly designed to enable non-programmer users to create an AR experience. A person the basic computer knowledge can build and use them. The reason lies in the fact that most AR authoring tools are developed on a graphical user interface. It is known as a standalone because it does not require any additional software for its operation. The most common and major functions of standalone are animation, adding interactive behaviors, and construction. The earliest examples of the standalone tools are AMIRE [ 232 ] and CATOMIR [ 233 ]. However, AMIRE and CATOMIR have no support available and are not maintained by the development team.

This standalone AR authoring tool has the advantage of quickly adding to the development of the AR experience. BuildAR has important characteristics. This allows the user to add video, 3D models, sound, text, and images. It has both arbitrary images and the square marker for which it provides computer vision-based tracking. They use the format of proprietary file format for saving the content developed by the user. BuildAR viewer software can be downloaded for free and it helps in viewing the file. However, BuildAR has no support available and the exe file is not available on their website.

Limitation: It does not support adding new interactive features. However, Choi et al. [ 234 ] have provided a solution to this constraint. They have added the desktop authoring tool that helps in adding new interactive experiences.

6.5. Rapid Prototyping/Development Tools

In order to cope with the limitation of low-level libraries, another more fast and more rapid AR application development tool is required. The major idea behind the development of rapid prototyping was that it rapidly shows the user the prototype before executing the hard exercise of developing the application. In the following paragraphs, a number of different tools are explained for developing rapid prototyping. For the creation of multimedia content, Adobe Flashis one of the most famous tools. It was developed on desktop and web platforms. Moreover, the web desktop and mobile experiences can be prototyped by it. Flash developers can use the FLARManager, FLARToolKit, or any other plug-ins for the development of AR experience. Porting the version of ARToolKit over the flash on the web creates the AR experience. Its process is so fast that just by writing a few lines, the developer can:

• Activate their camera.
• The AR markers could be viewed in a camera.
• The virtual content could be overlaid and loaded on the tracked image.

FLARToolkit is the best platform for creating AR prototyping because it has made it very easy for being operated by anyone. Anyone who has a camera and flash-enabled web browser can easily develop the AR experience. Alternatives to Flash: According to the website of Adobe, it no longer supports Flash Player after 31 December 2020 and blocked Flash content from running in Flash Player beginning 12 January 2021. Adobe strongly recommends all users immediately uninstall Flash Player to help protect their systems. However, some AR plug-ins could be used as an alternative to Flash-based AR applications. For instance, Microsoft Silverlight has the SLARToolKit. HTML5 is also recently used by researchers for creating web-based AR experiences. The major benefit of using HTML5 is that the interference of the third-party plug-in is not required. For instance, the AR natural feature tracking is implemented on WebGL, HTML5, and JavaScript. This was developed by Oberhofer and was viewable on mobile web browsers and desktops. Additionally, the normal HTML, with few web component technologies, has been used by Ahn [ 235 ] to develop a complete mobile AR framework.

6.6. Plug-ins to Existing Developer Tools

For the creation of AR experiences, the software libraries require tremendous programming techniques. So, plug-ins could be used as an alternative. Plug-ins are devices that could be plugged into the existing software packages. The AR functionality is added to the software packages that to the existing two-dimensional or three-dimensional content authoring tools. If the user already knows the procedure of using authoring tools that are supported by plug-ins, then AR plug-ins for the non-AR authoring tools are useful. These tools are aimed at:

• AR tracking and visualization functions for the existing authoring tools.
• It depends on the content authoring function supplied by the main authoring tool.

There are certain tools available as plug-ins and standalone through which AR applications can be built comparatively simply. These are commercial and some of them are freely available. As discussed earlier, Vuforia can be installed as a plug-in in Unity [ 236 ] and also has a free version. However, with complete support of tools certain amount needs to be paid. Similarly, ARtoolkit is available standalone and a plug-in for Unity is available. It is freely available for various platforms such as Android, iOS, Linux, and Windows. Moreover, ARCore and ARKit are also available for Android and iOS, respectively, and can work with Unity and Unreal authoring tools as a plug-in. ARCore is available and free for developers. MAXST and Wikitude also can work in integration with Unity, though they have a licensing price for the commercial version of the software. MAXST had a free version as well. All these tools, the abovementioned libraries, and standalone tools are depicted in Figure 9 . Cinema 4D, Maya, Trimble SketchUp 3D modeling software, 3Ds Max, and many others were created by a number of plug-ins that acted as authoring tools for three-dimensional content. While 3D animation and modeling tools are not capable of providing interactive features, it is very productive in creating three-dimensional scenes. SketchUp can utilize the AR plug-in by creating a model for the content creators. This model is then viewable in the AR scene provided by a free AR media player. The interactive three-dimensional graphic authoring tools are also available for the creation of highly interactive AR experiences, for instance, Wizard [ 237 ], Quest3D [ 238 ], and Unity [ 236 ]. All of these authoring tools have their own specific field of operation; however, Unity can be utilized to create a variety of experiences. The following are examples that justify the use of Unity over different solutions available:

• The AR plug-in of the Vuforia tracking library can be used with Unity 3D. This integration will help Vuforia in the creation of AR applications for the android or iOS platform.
• Similarly, the ARToolkit for Unity also provides marker-based experiences. It provides both image and marker-based AR visualization and tracking.

In such integrations, the highly interactive experiences are created by the normal Unity3D scripting interface and visual programming. Limitations of AR plug-ins: The following are the limitations accrued with the AR plug-in:

• The need for proprietary software could arise for the content produced by the authoring tool. The design provided by the authoring tools could restrict the user’s interactive and interface designs.
• Moreover, the authoring tools can also restrict the configurations of hardware or software within a certain limit.

Moreover, Nebeling et al. [ 239 ] reviewed the issues with the authoring tools of AR/VR. The survey of the tools has identified three key issues. To make up for those limitations, new tools are introduced for supporting the gesture-based interaction and rapid prototyping of the AR/VR content. Moreover, this is done without having technical knowledge of programming, gesture recognition, and 3D modeling. Mladenov et al. [ 240 ] review the existing SDKs and aim to find the most efficient SDK for the AR applications used in industrial environments. The paper reveals that currently available SDKs are very helpful for users to create AR applications with the parameters of their choice in industrial settings.

6.7. Summary

This section presents a detailed survey of different software and tools required for creating an AR experience. The section outlines hardware devices used in AR technology and various software to create an AR experience. It further elaborates on the software libraries required and covers bother the aspects of the commercially available tools. Table 3 provides a stack of software libraries, plug-ins, supported platforms, and standalone authoring tools. The figure also presents details of whether the mentioned tools are active or inactive. As an example, BazAR is used in tracking and geometric calibration. It is an open-source library for Linux or windows available under research-based GPL and can be used for research to detect an object via camera, calibrate it, and initiate tracking to put a basic virtual image on it; however, this library is not active at the present. Commercially used AR tools such as plug-ins have the limitations of only working efficiently in the 2D GUI and become problematic when used for 3D content. The advancement of technology may bring about a change in the authoring tools by making them capable of being operated for 3D and developing more active AR interfaces.

A summary of development and authoring tools for augmented reality application development.

7. Collaborative Research on Augmented Reality

In general, collaboration in augmented reality is the interaction of multiple users with virtual objects in the real environment. This interaction is regardless of the users’ location, i.e., they can participate remotely or have the same location. In this regard, we have two types of collaborative AR: co-located collaborative AR and remote collaborative AR. We mention it further in Figure 10 .

Collaborative augmented reality research domains.

7.1. Co-Located Collaborative AR

In this type of collaborative AR, the users interact with the virtual content rendered in the real environment while sharing the same place. The participant are not remote in such case. In this regard, Wells et al. [ 241 ] aim to determine the impact on the co-located group activities by varying the complexity of AR models using mobile AR. The paper also discusses different styles of collaborative AR such as:

• hlActive Discussion: A face-to-face discussion including all participants.
• Single Shared view: The participants focus on a single device.
• Disjoint and Shared View: Two to three participants focus on a single device.
• Disjoint and Distributed View: One to two people focus on their devices while the others are discussing.
• Distributed View: Participants focus on their devices with no discussion.
• Distributive View with Discussion: Participants focus on their devices while discussing in the group.

In this paper, the author did not contribute to the technology of co-located collaborative AR, but rather performed analysis on the effectiveness of different collaborative AR.

Grandi et al. [ 242 ] target the development of design approaches for synchronous collaboration to resolve complex manipulation tasks. For this, purpose fundamental concepts of design interface, human collaboration, and manipulation are discussed. This research the spiral model of research methodology which involves the development, planning, analysis, and evaluation. In addition, Dong et al. [ 243 ] introduce “ARVita”, a system where multiple users can interact with virtual simulations of engineering processes by wearing a head-mounted display. This system uses a co-located AR technique where the users are sitting around a table.

7.1.1. Applications of Co-located Collaborative AR

Kim et al. [ 244 ] propose a PDIE model to make a STEAM educational class while incorporating AR technology into the system. Furthermore, the “Aurasma” application is used to promote AR in education. In addition, Kanzanidis et al. [ 245 ] focus on teaching mobile programming using synchronous co-located collaborative AR mobile applications in which students are distributed in groups. The result showed that the students were satisfied with this learning methodology. Moreover, Chang et al. [ 246 ] explore the use of a mobile AR (MAR) application to teach interior design activities to students. The results identified that the students who were exposed to MAR showed more effectiveness in learning than those who were taught traditionally. Lastly, Sarkar et al. [ 247 ] discuss three aspects of synchronous co-located collaboration-based problem-solving: first, students’ perspectives on AR learning activities, either in dyads or individually are determined; second, the approach adopted by students while problem-solving is determined; third, the students’ motivation for using ScholAR is determined. Statistical results suggested that 90.4% students preferred the collaborative AR experience, i.e., in dyads. Meanwhile, motivation level and usability scores were higher for individual experiences. Grandi et al. [ 248 ] introduce the design for the collaborative manipulation of AR objects using mobile AR. This approach has two main features. It provides a shared medium for collaboration and manipulation of 3D objects as well as provides precise control of DoF transformations. Moreover, strategies are presented to make this system more efficient for users in pairs. Akccayir et al. [ 249 ] explore the impact of AR on the laboratory work of university students and their attitudes toward laboratories. This study used the quasi-experimental design with 76 participants—first year students aged 18–20 years. Both qualitative and quantitative methods were used for the analyses of data. A five-week implementation of the experiment proved that the use of AR in the laboratory significantly improved the laboratory skills of the students. However, some teachers and students also discussed some of the negative impacts of other aspects of AR. Rekimoto et al. [ 250 ] propose a collaborative AR system called TransVision. In this system, two or more users use a see-through display to look at the virtual objects rendered in a real environment using synchronous co-located collaborative AR. Oda et al. [ 251 ] propose a technique for avoiding interference for hand-held synchronous co-located collaborative AR. This study is based on first-person two-player shooting AR games. Benko et al. [ 87 ] present a collaborative augmented reality and mixed reality system called “VITA” or “Visual Interaction Tool For Archaeology”. They have an off-site visualization system that allows multiple users to interact with a virtual archaeological object. Franz et al. [ 88 ] present a system of collaborative AR for museums in which multiple users can interact in a shared environment. Huynh et al. [ 252 ] introduce art of defense (AoD), a co-located augmented reality board game that combines handheld devices with physical game pieces to create a unique experience of a merged physical and virtual game. Nilsson et al. [ 253 ] focus on a multi-user collaborative AR application as a tool for supporting collaboration between different organizations such as rescue services, police, and military organizations in a critical situation.

7.1.2. Asynchronous Co-Located Collaborative AR

Tseng et al. [ 254 ] present an asynchronous annotation system for collaborative augmented reality. This system can attribute virtual annotations with the real world due to a number of distinguishing capabilities, i.e., playing back, placing, and organizing. Extra context information is preserved by the recording of the perspective of the annotator. Furthermore, Kashara et al. [ 255 ] introduce “Second Surface”, an asynchronous co-located collaborative AR system. It allows the users to render images, text, or drawings in a real environment. These objects are stored in the data server and can be accessed later on.

7.2. Remote Collaborative AR

In this type of collaborative AR, all the users have different environments. They can interact with virtual objects remotely from any location. A number of studies have been done in this regard. Billinghurst et al. [ 256 ] introduce a wearable collaborative augmented reality system called “WearCom” to communicate with multiple remote people. Stafford et al. [ 257 ] present God-like interaction techniques for collaboration between outdoor AR and indoor tabletop users. This paper also describes a series of applications for collaboration. Gauglitz et al. [ 258 ] focus on a touchscreen interface for creating annotations in a collaborative AR environment. Moreover, this interface is also capable of virtually navigating a scene reconstructed live in 3D. Boonbrahm et al. [ 259 ] aim to develop a design model for remote collaboration. The research introduces the multiple marker technique to develop a very stable system that allows users from different locations to collaborate which also improves the accuracy. Li et al. [ 260 ] suggest the viewing of a collaborative exhibit has been considerably improved by introducing the distance-driven user interface (DUI). Poretski et al. [ 261 ] describe the behavioral challenges faced in interaction with virtual objects during remote collaborative AR. An experiment was performed to study users’ interaction with shared virtual objects in AR. Clergeaud et al. [ 262 ] tackle the limitations of collaboration in aerospace industrial designs. In addition, the authors propose prototype designs to address these limitations. Oda et al. [ 263 ] present the GARDEN (gesturing in an augmented reality depth-mapped environment) technique for 3D referencing in a collaborative augmented reality environment. The result shows that this technique is more accurate than the other comparisons. Muller et al. [ 85 ] investigate the influence of shared virtual landmarks (SVLs) on communication behavior and user experience. The results show that enhancement in user experience when SVLs were provided. Mahmood et al. [ 264 ] present a remote collaborative system for co-presence and sharing information using mixed reality. The results show improvements in user collaborative analysis experience.

7.2.1. Applications of Remote Collaborative AR

Munoz et al. [ 265 ] present a system called GLUEPS-AR to help teachers in learning situations by integrating AR and web technologies i.e., Web 2.0 tools and virtual learning environments (VLEs) [ 266 ]. Bin et al. [ 267 ] propose a system to enhance the learning experience of the students using collaborative mobile augmented reality learning application (CoMARLA). The application was used to teach ICT to students. The results showed improvement in the learning of the students using CoMARLA. Dunleavy et al. [ 268 ] explore the benefits and drawbacks of collaborative augmented reality simulations in learning. Moreover, a collaborative AR system was proposed for computers independent of location, i.e., indoor or outdoor. Maimone et al. [ 269 ] introduce a telepresence system with real-time 3D capture for remote users to improve communication using depth cameras. Moreover, it also discusses the limitations of previous telepresence systems. Gauglitz et al. [ 270 ] present an annotation-based remote collaboration AR system for mobiles. In this system, the remote user can explore the scene regardless of the local user’s camera position. Moreover, they can also communicate through annotations visible on the screen. Guo et al. [ 271 ] introduce an app, known as Block, that enables the users to collaborate irrespective of their geographic position, i.e., they can be either co-located or remote. Moreover, they can collaborate either asynchronously or synchronously. This app allows users to create structures that persist in the real environment. The result of the study suggested that people preferred synchronous and collocated collaboration, particularly one that was not restricted by time and space. Zhang et al. [ 272 ] propose a collaborative augmented reality for socialization app (CARS). This app improves the user’s perception of the quality of the experience. CARS benefits the user, application, and system on various levels. It reduces the use of computer resources, end-to-end latency, and networking. Results of the experiment suggest that CARS acts more efficiently for users of cloud-based AR applications. Moreover, on mobile phones, it reduces the latency level by up to 40%. Grandi et al. [ 242 ] propose an edge-assisted system, known as CollabAR, which combines both collaboration image recognition and distortion tolerance. Collaboration image recognition enhances recognition accuracy by exploiting the “spatial-temporal" correlation. The result of the experiment suggested that this system has significantly decreased the end-to-end system latency up to 17.8 ms for a smartphone. Additionally, recognition accuracy for images with stronger distortions was found to be 96%.

7.2.2. Synchronous Remote Collaborative AR

Lien et al. [ 273 ] present a system called “Pixel-Point Volume Segmentation” in collaborative AR. This system is used for object references. Moreover, one user can locate the objects with the help of circles drawn on the screen by other users in a collaborative environment. Huang et al. [ 274 ] focus on sharing hand gestures and sketches between a local user and a remote user by using collaborative AR. The system is named “HandsinTouch”. Ou et al. [ 275 ] present the DOVE (drawing over video environment) system, which integrates live-video and gestures in collaborative AR. This system is designed to perform remote physical tasks in a collaborative environment. Datcu et al. [ 276 ] present the creation and evaluation of the handheld AR system. This is done particularly to investigate the remote forensic and co-located and to support team-situational awareness. Three experienced investigators evaluated this system in two steps. First, it was investigated with one remote and one local investigator. Secondly, with one remote and two local investigators. Results of the study suggest the use of this technology resolves the limitation of HMDs. Tait et al. [ 277 ] propose the AR-based remote collaboration that supports view independence. The main aim of the system was to enable the remote user to help the local user with object placement. The remote user uses a 3D reconstruction of the environment to independently find the local user’s scene. Moreover, a remote user can also place the virtual cues in the scene visible to the local user. The major advantage of this system is that it allows the remote user to have an independent scene in the shared task space. Fang et al. [ 278 ] focus on enhancing the 3D feel of immersive interaction by reducing communication barriers. WebRTC, a real-time video communication framework, is developed to enable the operator site’s first-hand view of the remote user. Node.js and WebSocket, virtual canvas-based whiteboards, are developed which are usable in different aspects of life. Mora et al. [ 279 ] explain the CroMAR system. The authors aim to help the users in crowd management who are deployed in a planned outdoor event. CroMAR allows the users to share viewpoints via email, and geo-localized tags allow the users to visualize the outdoor environment and rate these tags. Adcock et al. [ 280 ] present three remote spacial augmented reality systems “Composite Wedge”, “Vector Box”, and “Eyelight” for off-surface 3D viewpoints visualization. In this system, the physical world environment of a remote user can be seen by the local user. Lincoln et al. [ 281 ] focus on a system of robotic avatars of humans in a synchronous remote collaborative environment. It uses cameras and projectors to render a humanoid animatronic model which can be seen by multiple users. This system is called “Animatronic Shader Lamps Avatars”. Komiyama et al. [ 282 ] present a synchronous remote collaborative AR system. It can transition between first person and third person view during collaboration. Moreover, the local user can observe the environment of the remote user. Lehment et al. [ 283 ] present an automatically aligned videoconferencing AR system. In this system, the remote user is rendered and aligned on the display of the local user. This alignment is done automatically regarding the local user’s real environment without modifying it. Oda et al. [ 284 ] present a remote collaborative system for guidance in a collaborative environment. In this system, the remote expert can guide a local user with the help of both AR and VR. The remote expert can create virtual replicas of real objects to guide a local user. Piumsomboon et al. [ 285 ] introduce an adaptive avatar system in mixed reality (MR) called “Mini Me” between a remote user using VR and a local user using AR technology. The results show that it improves the overall experience of MR and social presence. Piumsomboon et al. [ 286 ] present “CoVAR”, a collaboration consisting of both AR and VR technologies. A local user can share their environment with a remote VR user. It supports gestures, head, and eye gaze to improve the collaboration experience. Teo et al. [ 287 ] present a system that captures a 360 panorama video of one user and shares it with the other remote user in a mixed reality collaboration. In this system, the users communicate through hand gestures and visual annotation. Thanyadit et al. [ 288 ] introduce a system where the instructor can observe students in a virtual environment. The system is called “ObserVAR” and uses augmented reality to observe students’ gazes in a virtual environment. Results show that this system is more improved and flexible in several scenarios. Sodhi et al. [ 289 ] present a synchronous remote collaborative system called “BeThere” to explore 3D gestures and spatial input. This system enables a remote user to perform virtual interaction in the local user’s real environment. Ong et al. [ 290 ] propose a collaborative system in which 3D objects can be seen by all the users in a collaborative environment. Moreover, the changes made to these objects are also observed by the users. Butz et al. [ 84 ] present EMMIE (environment management for multi-user information environments) in a collaborative augmented reality environment in which virtual objects can be manipulated by the users. In addition, this manipulation is visible to each user of this system.

7.2.3. Asynchronous Remote Collaborative AR

Irlitti et al. [ 291 ] explore the challenges faced during the use of asynchronous collaborative AR. Moreover, the author further discusses how to enhance communication while using asynchronous collaborative AR. Quasi-systems do not fulfill Azuma’s [ 292 ] definition of AR technology. However, they are very good at executing certain aspects of AR as other full AR devices are doing. For instance, mixed-space collaborative work in a virtual theater [ 268 ]. This system explained that if someone wants two groups to pay attention to each other, a common spatial frame of reference should be created to have a better experience of social presence. In the spatially aware educational system, students were using location-aware smartphones to resolve riddles. This was very useful in the educational system because it supported both engagement and social presence [ 245 , 265 , 269 ]. However, this system did not align the 3D virtual content in the virtual space. Therefore, it was not a true AR system. In order to capture a remote 3D scene, Fuchs and Maimone [ 293 ] developed an algorithm. They also developed a proof of concept for teleconferencing. For capturing images, RGB-D cameras were used. The remote scene was displayed on the 3D stereoscopic screen. These systems were not fully AR, but they still exhibited a very good immersion. Akussah et al. [ 294 ] focus on developing a marker-based collaborative augmented reality app for learning mathematics. First, the system focuses on individual experience and later on expands it to collaborative AR.

7.3. Summary

This section provides comprehensive details on collaborative augmented reality which is broadly classified into co-located collaborative AR, where participants collaborate with each other in geographically the same location, and remote collaboration. The applications of both approaches are presented as well. Co-located collaborative AR is mostly adopted in learning realms for sharing information, for example, in museums. On the other hand, in remote collaborative AR the remote user can explore the scene regardless of the local user’s camera position. The applications of this technology are mostly found in education.

8. AR Interaction and Input Technologies

The interaction and input technologies are detailed in this section. There are a number of input methods that are utilized in AR technologies. First, multimode and 3D interfaces such as speech, gesture and handheld wands. Second, the mouse, and keyboard traditional two-dimensional user interfaces (UI). The type of interaction task needed for the interface defines which input method would be utilized in the application. A variety of interfaces have been developed: three-dimensional user interfaces, tangible user interfaces, multimedia interfaces, natural user interfaces, and information browsers.

8.1. AR Information Browsers

Wikitude and Navicam are one of the most popular examples of AR information browsers. The only problem with AR browsers is that they cannot provide direct interaction with the virtual objects.

8.2. Three-Dimensional User Interfaces

A three-dimensional user interface uses the controllers for providing the interaction with virtual content. By using the traditional 3D user interface techniques, we can directly interact with the three-dimensional object in the virtual space. There are a number of 3D user interface interaction techniques as follows: 3D motion tracking sensors are one of the most commonly used devices for AR interaction. The motion tracking sensors allow the following functions: tracking the parts of the user’s body and allow pointing as well as the manipulation of the virtual objects [ 295 ]. Haptic devices are also used for interacting with AR environments [ 296 , 297 , 298 ]. They mainly used as 3D pointing devices. In addition, they provide tactile and forces feedback. This will create the illusion of a physical object existing in the real world. Thereby, it helps in complementing the virtual experience. They are used in training, entertainment, and design-related AR applications.

8.3. Tangible User Interface

The tangible user interface is one of the main concepts of human–computer interface technology research. In this, the physical object is used for interaction [ 299 ]. It bridges the gap between the physical and the virtual object [ 300 ]. Chessa et al. incorporated grasping behavior in a virtual reality systems [ 301 ], while Han et al. presented and evaluated hand interaction techniques using tactile feedback (haptics) and physical grasping by mapping a real object with virtual objects [ 302 ].

8.4. Natural User Interfaces in AR

Recently, more accurate gesture and motion-based interactions for AR and VR applications have become extensively available due to the commercialization of depth cameras such as Microsoft Kinect and technical advances. Bare-hand interaction with a virtual object was made possible by the introduction of a depth camera. It provided physical interaction by tracking the dexterous hand motion. For instance, the physical objects and the user’s hands were recognized by the use of Kinect Camera, designed by the Microsoft HoloDesk [ 299 ]. The virtual objects were shown on the optical see-through AR workbench. It also allowed the users to interact with the virtual objects presented on the AR workbench. The user-defined gestures have been categorized into sets by the Piumsomboon [ 300 ]. This set can be utilized in AR applications for accomplishing different tasks. In addition, some of the mobile-based depth-sensing cameras are also under investigation. For instance, the SoftKinetic and Myo gesture armband controller. SodtKinetic is aimed at developing hand gesture interaction in mobile phones and wearable devices more accurately, while the Myo gesture armband controller is a biometric sensor that provides interaction in wearable and mobile environments.

8.5. Multimodal Interaction in AR

In addition to speech and gesture recognition, there are other types of voice recognition are being investigated. For example, the whistle-recognition system was developed by Lindeman [ 303 ] in mobile AR games. In this, the user had to whistle the right length and pitch to intimidate the virtual creatures in the game. Summary: The common input techniques and input methods have been examined in this section. These included simple information browsers and complex AR interfaces. The simple ones have very little support for the interaction and virtual content, while the complex interfaces were able to recognize even the speech and gesture inputs. A wide range of input methods are available for the AR interface; however, they are needed to be designed carefully. The following section delineates the research into the interface pattern, design, and guideline for AR experiences.

9. Design Guidelines and Interface Pattern

The previous section detailed the wide range of different AR input and interaction technologies; however, more rigorous research is required to design the AR experience. This section explores the interface patterns and design guidelines to develop an AR experience. The development of new interfaces goes through four main steps. First, the prototype is demonstrated. Second, interaction techniques are adopted from the other interface metaphors. Third, new interface metaphors are developed that are appropriate to the medium. Finally, the formal theoretical models are developed for modeling the interaction of users. In this regard, Wang et al. [ 304 ] employ user-centered AR instruction (UcAI) in procedural tasks. Thirty participants were selected for the experiment while having both the control and experiment groups. The result of the experiment suggested that introduction of UcAI increased the user’s spatial cognitive ability, particularly in the high-precision operational task. This research has the potential of guiding advanced AR instruction designs to perform tasks of high cognitive complexity. For instance, WIMP (windows, icons, menus, and pointers) is a very well-known desktop metaphor. In development, it has gone through all of these stages. There are methods developed that are used to predict the time taken by the mouse will select an icon of a given size. These are known as formal theoretical models. Fitts law [ 305 ] is among those models that help in determining the pointing times in the user interfaces. There are also a number of virtual reality interfaces available that are at the third stage with reference to the techniques available. For example, the manipulation and selection in immersive virtual worlds can be done by using the go-go interaction method [ 306 ]. On the other hand, as evident in the previous section, AR interfaces have barely surpassed the first two stages. Similarly, a number of AR interaction methods and technologies are available; however, by and large, they are only the extensions or versions of the existing 3D and 2D techniques present in mobiles, laptops, or AR interfaces. For instance, mobile phone experiences such as the gesture application and the touch screen input are added to AR. Therefore, there is a dire need to develop AR-specific interaction techniques and interface metaphors [ 307 ]. A deeper analysis and study of AR interfaces will help in the development of the appropriate metaphor interfaces. AR interfaces are unique in the sense that they need to develop close interaction between the real and the virtual worlds. A researcher, MacIntyre, has argued that the definition and the fusion of the virtual and real worlds are required for creating an AR design [ 308 ]. The primary goal of this is to depict the physical objects and user input onto the computer-generated graphics. This is done by using a suitable interaction interface. As a result, an AR design should have three components:

• The physical object.
• The virtual image to be developed.
• An interface to create an interaction between the physical world and the virtual objects.

Use of design patterns could be an alternative technique to develop the AR interface design. These design patterns are most commonly used in the fields of computer science and design interface. Alexander has defined the use of design patterns in the following words: “Each pattern describes a problem that occurs over and over again in our environment, and then describes the core of the solution to that problem in such a way that you can use this solution a million times over, without ever doing it the same way twice” [ 309 , 310 ]. The pattern language approach could be used to enhance AR development, as suggested by Reicher [ 311 ]. This idea has evolved from the earlier research works of MacWilliam [ 312 ]. This approach has two main functionalities. First, it is more focused on the software engineering aspect. Secondly, it suggests ways to develop complex AR systems by combining different modules of design patterns. So, they describe each pattern by the number of its aspects such as name, motivation, goal, description, consequences, known project usage, and general usability. One of the most notable examples of it is the DWARF framework [ 313 ]. DWARF is a component-based AR framework that is developed through the design pattern approach. In contrast to the pattern language approach, the user experience of design in the AR handheld device could be used for developing designs. This was described by Xu and the main concern was pre-patterns. Pre-patterns are the components that bridge the gap between the game design and the interaction design. For determining the method of using of design patterns, seamful design could be used. This suggests that the designer should integrate the AR handheld game design and the technology in such a way that they should blend into each other. Some users need more attention for designing effective AR experiences; therefore, the designing of special needs is another intervention to resolve this discrepancy. For instance, as pointed out by Rand and Maclntyre [ 314 ], in designing an AR system for the age group of 6–9, the developmental stages of the children should be accounted for in it. The research has also suggested that a powerful educational experience could be created through the use of AR. In addition to this development, it was also stated that the developmental stages of the students should be considered [ 315 , 316 ]. However, there is no extensive research that suggests the development of AR experiences for children [ 317 ]. Radu, in his paper, has determined the key four areas that should be considered while designing AR for children: attention, motor, special, logic, and memory abilities [ 318 ].

10. Security, Trust, and Collaborative AR

Security is very important in augmented reality, especially in collaborative augmented reality. While using collaborative AR applications, the data are exposed to external attacks, which increases concerns about security relating to AR technologies. Moreover, if the users who share the same virtual collaborative environments are unknown to each other, it also elevates these issues. In [ 319 ], the basic premise of the research is that the developed abstraction device not only improves the privacy but also the performance of the AR apps, which lays the groundwork for the development of future OS support for AR apps. The results suggested that the prototype enables secure offloading of heavyweight, incurs negligible overhead, and improves the overall performance of the app. In [ 320 ], the authors aim to resolve security and privacy challenges in multi-user AR applications. They have introduced an AR-sharing module along with systematized designs and representative case studies for functionality and security. This module is implemented as a prototype known as ArShare for the HoloLens. Finally, it also lays the foundation for the development of fully fledged and secure multi-user AR interaction. In [ 321 ], the authors used AR smart glasses to detail the “security and safety” aspect of AR applications as a case study. In the experiment, cloud-based architecture is linked to the oil extractor in combination with Vuzix Blade smart glasses. For security purposes, this app sends real-time signals if a dangerous situation arrives. In [ 322 ], deep learning is used to make the adaptive policies for generating the visual output in AR devices. Simulations are used that automatically detect the situation and generate policies and protect the system against disastrous malicious content. In [ 323 ], the authors discussed the case study of challenges faced by VR and AR in the field of security and privacy. The results showed that the attack reached the target of distance 1.5 m with 90 percent accuracy when using a four-digit password. In [ 324 ], the authors provide details and goals for developing security. They discuss the challenges faced in the development of edge computing architecture which also includes the discussion regarding reducing security risks. The main idea of the paper is to detail the design of security measures for both AR and non-AR devices. In [ 325 ], the authors presented that the handling of multi-user outputs and handling of data are demonstrated are the two main obstacles in achieving security and privacy of AR devices. It further provides new opportunities that can significantly improve the security and privacy realm of AR. In [ 326 ], the authors introduce the authentication tool for ensuring security and privacy in AR environments. For these purposes, the graphical user password is fused with the AR environments. A doodle password is created by the touch-gesture-recognition on a mobile phone, and then doodles are matched in real-time size. Additionally, doodles are matched with the AR environment. In [ 327 ], the authors discussed the immersive nature of augmented reality engenders significant threats in the realm of security and privacy. They further explore the aspects of securing buggy AR output. In [ 328 ], the authors employ the case study of an Android app, “Google Translator”, to detect and avoid variant privacy leaks. In addition, this research proposes the foundational framework to detect unnecessary privacy leaks. In [ 329 ], the authors discuss the AR security-related issues on the web. The security related vulnerabilities are identified and then engineering guidelines are proposed to make AR implementation secure. In [ 330 ], the past ten years of research work of the author, starting from 2011, in the field of augmented reality is presented. The main idea of the paper is to figure out the potential problems and to predict the future for the next ten years. It also explains the systematization for future work and focuses on evaluating AR security research. In [ 331 ], the authors presented various AR-related security issues and identified managing the virtual content in the real space as a challenge in making AR spaces secure for single and multi-users. The authors in [ 332 ] believe that there is a dire need of cybersecurity risks in the AR world. The introduction of systemized and universal policy modules for the AR architecture is a viable solution for mitigating security risks in AR. In [ 333 ], the authors discuss the challenge of enabling the different AR apps to augment the user’s world experience simultaneously, pointing out the conflicts between the AR applications.

11. Summary

In this paper, the authors have reviewed the literature extensively in terms of tracking and displays technology, AR, and collaborative AR, as can be seen in Figure 10 . It has been observed that collaborative AR has further two classifications i.e., co-located AR and remote collaboration [ 334 ]. Each of these collocated and remote collaborations has two further types i.e., synchronous and asynchronous. In remote collaborative AR, there are a number of use cases wherein it has been observed that trust management is too important a factor to consider because there are unknown parties that participate in remote activities to interact with each other and as such, they are unknown to each other as well [ 21 , 335 , 336 , 337 , 338 ]. There has been a lack of trust and security concerns during this remote collaboration. There are more chances of intrusion and vulnerabilities that can be possibly exploited [ 331 , 339 , 340 ]. One such collaboration is from the tourism sector, which has boosted the economy, especially in the pandemic era when physical interactors were not allowed [ 341 ]. To address these concerns, this research felt the need to ensure that the communication has integrity and for this purpose, the research utilized state-of-the-art blockchain infrastructure for collaborative applications in AR. The paper has proposed a complete secure framework wherein different applications working remotely are having a real feeling of trust in each other [ 17 , 342 , 343 ]. The participants within the collaborative AR subscribed to a trusted environment to further make interaction with each other in a secure fashion while their communication was protected through state-of-the-art blockchain infrastructure [ 338 , 344 ]. A model of such an application is shown in Figure 11 .

A model of blockchain-based trusted and secured collaborative AR system.

Figure 12 demonstrates the initiation of the AR App in step 1, while in step 2 of Figure 12 , the blockchain is initiated to record transactions related to sign-up, record audio calls, proceed with payment/subscription, etc. In step 3, when the transaction is established, AR is initiated, which enables the visitor to receive guidance from the travel guide. The app creates a map of the real environment. The created map and the vision provide a SLAM, i.e., SLAM provides an overall vision and details of different objects in the real world. Inertial tracking controls the movement and direction in the augmented reality application. The virtual objects are then placed after identifying vision and tracking. In a collaborative environment, the guides are provided with an option of annotation so they can circle a particular object or spot different locations and landmarks or point to different incidents [ 16 ].

Sharing of the real-time environment of CAR tourist app for multiple users [ 16 ].

12. Directions for Research

The commercialization efforts of companies have made AR a mainstream field. However, for the technology to reach its full potential, the number of research areas should be expanded. Azuma has explained the three major obstacles in the way of AR: interface limitation, technological limitations, and the issue of social acceptance. In order to overcome these barriers, the two major models are developed: first, Roger’s innovation diffusion theory [ 345 ] and the technology acceptance model (developed by Martinez) [ 346 ]. Roger has explained the following major restriction towards the adoption of this technology: limited computational power of AR technology, social acceptance, no AR standards, tracking inaccuracy, and overloading of information. The main research trends in display technology, user interface, and tracking were identified by Zho by evaluating ten years of ISMAR papers. The research has been conducted in a wide number of areas except for social acceptance. This section aims at exploring future opportunities and ongoing research in the field of AR, particularly in the four key areas: display, tracking, interaction, and social acceptance. Moreover, there are a number of other topics including evaluation techniques, visualization methods, applications, authoring and content-creating tools, rendering methods, and some other areas.

13. Conclusions

This document has detailed a number of research papers that address certain problems of AR. For instance, AR tracking techniques are detailed in Section 3 . Display technologies, such as VST and OST, and its related calibration techniques in Section 4 , authoring tools in Section 6 , collaborative AR in Section 7 , AR interaction in Section 8 , and design guidelines in Section 9 . Finally, promising security and trust-related papers are discussed in the final section. We presented the problem statement and a short solution to the problem is provided. These aspects should be covered in future research and the most pertinent among these are the hybrid AR interfaces, social acceptance, etc. The speed of research is significantly increasing, and AR technology is going to dramatically impact our lives in the next 20 years.

Acknowledgments

Thanks to the Deanship of Research, Islamic University of Madinah. We would like to extend special thanks to our other team members (Anas and his development team at 360Folio, Ali Ullah and Sajjad Hussain Khan) who participated in the development, writeup, and finding of historical data. Ali Ullah has a great ability to understand difficult topics in AR, such as calibration and tracking.

Funding Statement

This project is funded by the Deputyship For Research and Innovation, Ministry of Education, Kingdom of Saudi Arabia, under project No (20/17), titled Digital Transformation of Madinah Landmarks using Augmented Reality.

Author Contributions

Conceptualization of the paper is done by T.A.S. Sections Organization, is mostly written by T.A.S. and S.J.; The protype implementation is done by the development team, however, the administration and coordination is performed by A.A., A.N. (Abdullah Namoun) and A.B.A.; Validation is done by A.A. and A.N. (Adnan Nadeem); Formal Analysis is done by T.A.S. and S.J.; Investigation, T.A.S.; Resources and Data Curation, is done by A.N. (Adnan Nadeem); Writing—Original Draft Preparation, is done by T.A.S. and S.J., Writing—Review & Editing is carried out by H.B.A.; Visualization is mostly done by T.A.S. and M.S.S.; Supervision, is done by T.A.S.; Project Administration, is done by A.A.; Funding Acquisition, T.A.S. All authors have read and agreed to the published version of the manuscript.

Conflicts of Interest

The authors declare no conflict of interest.

Disclaimer/Publisher’s Note: The statements, opinions and data contained in all publications are solely those of the individual author(s) and contributor(s) and not of MDPI and/or the editor(s). MDPI and/or the editor(s) disclaim responsibility for any injury to people or property resulting from any ideas, methods, instructions or products referred to in the content.

The Influence of Augmented Reality on the Consumer Purchasing Process

• Conference paper
• First Online: 28 May 2024
• Cite this conference paper

• Federica Murmura   ORCID: orcid.org/0000-0001-8443-8680 10 ,
• Laura Bravi   ORCID: orcid.org/0000-0002-1733-7043 10 ,
• Giada Pierli   ORCID: orcid.org/0000-0001-8656-9772 10 ,
• Gilberto Santos   ORCID: orcid.org/0000-0001-9268-3272 11 &
• Fabio Musso   ORCID: orcid.org/0000-0002-5189-2956 10  

Part of the book series: Springer Proceedings in Business and Economics ((SPBE))

Included in the following conference series:

• International Conference on Quality Innovation and Sustainability

Augmented reality (AR) has recently gained the attention of both scholars and practitioners, thanks to its ability to provide captivating and immersive experiences that lead users to fully engage in the proposed content. Despite the increasing use of AR by companies and individuals, there are still few empirical studies investigating its influence on consumer behavior; therefore, the present study intends to analyze the different behaviors and attitudes of consumers during their purchasing process in front of AR technologies. To this end, an online questionnaire was submitted to individuals belonging to different generational cohorts, obtaining 337 responses. From the results, it emerged that most of the sample expressed a more than positive opinion regarding the help that AR technologies could give to the consumer when he/she chooses and buys a product. Understanding the characteristics of the product, having a unique experience with it during the purchasing process, and making choices with more awareness and fastness are some of the positive aspects that were identified by the respondents.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

• Available as PDF
• Read on any device
• Instant download
• Own it forever
• Available as EPUB and PDF
• Compact, lightweight edition
• Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Sima, V., Gheorghe, I. G., Subić, J., & Nancu, D. (2020). Influences of the industry 4.0 revolution on the human capital development and consumer behavior: A systematic review. Sustainability, 12 (10), 4035.

Article   Google Scholar  

Kazmi, S. H. A., Ahmed, R. R., Soomro, K. A., Hashem, E., Akhtar, H., & Parmar, V. (2021). Role of augmented reality in changing consumer behavior and decision making: Case of Pakistan. Sustainability, 13 (24), 14064.

McLean, G., & Wilson, A. (2019). Shopping in the digital world: Examining customer engagement through augmented reality mobile applications. Computers in Human Behavior, 101 , 210–224.

Rauschnabel, P. A., Babin, B. J., tom Dieck, M. C., Krey, N., & Jung, T. (2022). What is augmented reality marketing? Its definition, complexity, and future. Journal of Business Research, 142 , 1140–1150.

Lavingia, K., & Tanwar, S. (2020). Augmented reality and industry 4.0. In A. Nayyar & A. Kumar (Eds.), A roadmap to industry 4.0: Smart production, sharp business and sustainable development (Advances in Science, Technology & Innovation) (pp. 143–155). Springer.

Chapter   Google Scholar  

Uddin, S.-A., Hanna, G., Ross, L., Molina, C., Urakov, T., Johnson, P., Kim, T., & Drazin, D. (2021). Augmented reality in spinal surgery: Highlights from augmented reality lectures at the emerging technologies annual meetings. Cureus, 13 (10).

Google Scholar  

ABI Research. The digital enterprise ecosystem: Augmented reality spotlight (2019). Available online: https://www.abiresearch.com/blogs/2019/10/01/digital-enterprise-ecosystem-augmented-reality-spotlight/ . Accessed 27 Jan 2023.

Grand View Research Inc. The global augmented reality market size (2021). Available online: https://www.grandviewresearch.com/industry-analysis/augmented-reality-market . Accessed 27 Jan 2023.

Hilken, T., de Ruyter, K., Chylinski, M., Mahr, D., & Keeling, D. I. (2017). Augmenting the eye of the beholder: Exploring the strategic potential of augmented reality to enhance online service experiences. Journal of the Academy of Marketing Science, 45 (6), 884–905.

Kumar, H. (2021). Augmented reality in online retailing: A systematic review and research agenda. International Journal of Retail & Distribution Management, 50 (4), 537–559.

Riar, M., Xi, N., Korbel, J. J., Zarnekow, R., & Hamari, J. (2022). Using augmented reality for shopping: A framework for AR induced consumer behavior, literature review and future agenda. Internet Research .

İpek, İ. (2020). Understanding consumer behavior in technology-mediated spaces. In B. George & J. Paul (Eds.), Digital transformation in business and society (pp. 169–189). Palgrave Macmillan.

Qin, Y. S. (2020). Fostering brand–consumer interactions in social media: The role of social media uses and gratifications. Journal of Research in Interactive Marketing, 14 (3), 337–354.

Cheung, M. L., Pires, G. D., Rosenberger, P. J., Leung, W. K., & Sharipudin, M. N. S. (2021). The role of consumer-consumer interaction and consumer-brand interaction in driving consumer-brand engagement and behavioral intentions. Journal of Retailing and Consumer Services, 61 , 102574.

Roy, S. K., Gruner, R. L., & Guo, J. (2022). Exploring customer experience, commitment, and engagement behaviours. Journal of Strategic Marketing, 30 (1), 45–68.

Ahmed, R. R., Streimikiene, D., Berchtold, G., Vveinhardt, J., Channar, Z. A., & Soomro, R. H. (2019). Effectiveness of online digital media advertising as a strategic tool for building brand sustainability: Evidence from FMCGs and services sectors of Pakistan. Sustainability, 11 (12), 3436.

Rasool, A., Shah, F. A., & Islam, J. U. (2020). Customer engagement in the digital age: A review and research agenda. Current Opinion in Psychology, 36 , 96–100.

Tom Dieck, M. C., & Han, D. I. D. (2022). The role of immersive technology in customer experience management. Journal of Marketing Theory and Practice, 30 (1), 108–119.

Yim, M. Y.-C., Chu, S.-C., & Sauer, P. L. (2017). Is augmented reality technology an effective tool for Ecommerce? An interactivity and vividness perspective. Journal of Interactive Marketing, 39 (1), 89–103.

Romano, B., Sands, S., & Pallant, J. I. (2021). Augmented reality and the customer journey: An exploratory study. Australasian Marketing Journal, 29 (4), 354–363.

Azuma, R. T. (1997). A survey of augmented reality. Presence: Teleoperators & Virtual Environments, 6 (4), 355–385.

Cao, M., Zheng, L., Jia, W., & Liu, X. (2020). Fast monocular visual odometry for augmented reality on smartphones. IEEE Consumer Electronics Magazine .

Wedel, M., Bigné, E., & Zhang, J. (2020). Virtual and augmented reality: Advancing research in consumer marketing. International Journal of Research in Marketing, 37 (3), 443–465.

Kang, H. J., Shin, J., & Ponto, K. (2020). How 3D virtual reality stores can shape consumer purchase decisions: The roles of informativeness and playfulness. Journal of Interactive Marketing, 49 , 70–85.

Batat, W. (2021). How augmented reality (AR) is transforming the restaurant sector: Investigating the impact of “Le Petit Chef” on customers’ dining experiences. Technological Forecasting and Social Change, 172 , 121013.

Rejeb, A., Rejeb, K., & Treiblmaier, H. (2021). How augmented reality impacts retail marketing: A state-of-the-art review from a consumer perspective. Journal of Strategic Marketing, 31 (3), 718.

Rese, A., Baier, D., Geyer-Schulz, A., & Schreiber, S. (2017). How augmented reality apps are accepted by consumers: A comparative analysis using scales and opinions. Technological Forecasting and Social Change, 124 , 306–319.

Duarte, P., Silva, S., & Ferreira, M. B. (2018). How convenient is it? Delivering online shopping convenience to enhance customer satisfaction and encourage e-WOM. Journal of Retailing and Consumer Services, 44 , 161–169.

El-Jarn, H., & Southern, G. (2020). Can co-creation in extended reality technologies facilitate the design process? Journal of Work-Applied Management, 12 (2), 191–205.

Malhotra, M. K., & Grover, V. (1998). An assessment of survey research in POM: From constructs to theory. Journal of Operations Management, 16 (4), 407–425.

Nunnally, J. C., & Bernstein, I. H. (1994). Psychometric theory (3rd ed.). McGraw-Hill.

Suh, K. S., & Lee, Y. E. (2005). The effects of virtual reality on consumer learning: An empirical investigation. Management Information Systems Quarterly, 29 , 673–697.

Poushneh, A., & Vasquez-Parraga, A. Z. (2017). Discernible impact of augmented reality on retail customer’s experience, satisfaction and willingness to buy. Journal of Retailing and Consumer Services, 34 , 229–234.

Poncin, I., & Mimoun, M. S. B. (2014). The impact of “e-atmospherics” on physical stores. Journal of Retailing and Consumer Services, 21 (5), 851–859.

Garaus, M., Wagner, U., & Kummer, C. (2015). Cognitive fit, retail shopper confusion, and shopping value: Empirical investigation. Journal of Business Research, 68 (5), 1003–1011.

Scholz, J., & Smith, A. N. (2016). Augmented reality: Designing immersive experiences that maximize consumer engagement. Business Horizons, 59 (2), 149–161.

Heller, J., Chylinski, M., de Ruyter, K., Mahr, D., & Keeling, D. I. (2019a). Let me imagine that for you: Transforming the retail frontline through augmenting customer mental imagery ability. Journal of Retailing, 95 (2), 94–114.

Heller, J., Chylinski, M., de Ruyter, K., Mahr, D., & Keeling, D. I. (2019b). Touching the untouchable: Exploring multi-sensory augmented reality in the context of online retailing. Journal of Retailing, 95 (4), 219–234.

Alimamy, S., & Nadeem, W. (2021). Is this real? Cocreation of value through authentic experiential augmented reality: The mediating effect of perceived ethics and customer engagement. Information Technology & People, 35 (2), 577–599.

Wood, S. L. (2002). Future fantasies: A social change perspective of retailing in the 21st century. Journal of Retailing, 78 (1), 77–83.

Ratchford, B. T., Talukdar, D., & Lee, M. S. (2001). A model of consumer choice of the internet as an information source. International Journal of Electronic Commerce, 5 (3), 7–21.

Download references

Author information

Authors and affiliations.

Department of Economics, Society, Politics, Carlo Bo University of Urbino, Urbino, Italy

Federica Murmura, Laura Bravi, Giada Pierli & Fabio Musso

School of Design, Campos do IPCA, Barcelos, Portugal

Gilberto Santos

You can also search for this author in PubMed   Google Scholar

Corresponding author

Correspondence to Laura Bravi .

Editor information

Editors and affiliations.

Industrial Engineering, Universidade Lusófona, Lisbon, Portugal

Escola Superior de Ciências Empresariais, Instituto Politécnico de Setúbal, Setúbal, Portugal

Tiago Pinho

Vítor Barbosa

Escola Superior de Ciências Empresariais, Instituto Politécnico de Viana do Castelo, Viana do Castelo, Portugal

Luís Barreto

Sandrina B. Moreira

Pedro Pardal

João Nabais

Carlos Mata

Rights and permissions

Reprints and permissions

Copyright information

© 2024 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this paper

Cite this paper.

Murmura, F., Bravi, L., Pierli, G., Santos, G., Musso, F. (2024). The Influence of Augmented Reality on the Consumer Purchasing Process. In: Reis, J., et al. Driving Quality Management and Sustainability in VUCA Environments . ICQUIS 2023. Springer Proceedings in Business and Economics. Springer, Cham. https://doi.org/10.1007/978-3-031-52723-4_5

Download citation

DOI : https://doi.org/10.1007/978-3-031-52723-4_5

Published : 28 May 2024

Publisher Name : Springer, Cham

Print ISBN : 978-3-031-52722-7

Online ISBN : 978-3-031-52723-4

eBook Packages : Business and Management Business and Management (R0)

Share this paper

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Publish with us

Policies and ethics

• Find a journal
• Track your research

Augmented Reality

Ieee account.

• Change Username/Password
• Update Address

Purchase Details

• Payment Options
• Order History
• View Purchased Documents

Profile Information

• Communications Preferences
• Profession and Education
• Technical Interests
• US & Canada: +1 800 678 4333
• Worldwide: +1 732 981 0060
• Contact & Support
• About IEEE Xplore
• Accessibility
• Terms of Use
• Nondiscrimination Policy
• Privacy & Opting Out of Cookies

A not-for-profit organization, IEEE is the world's largest technical professional organization dedicated to advancing technology for the benefit of humanity. © Copyright 2024 IEEE - All rights reserved. Use of this web site signifies your agreement to the terms and conditions.

Best Apps and Sites for Augmented Reality

Augmented reality offers opportunities for immersive learning

iOS and Android AR Apps

Ios ar apps, websites for ar, recent updates.

This article was updated May 24, 2024

Why should teachers integrate augmented reality (AR) apps and sites into their curricula? With manipulable 3D visuals, augmented reality apps and sites inject a wow factor into any subject, increasing kids’ engagement and enthusiasm for learning. In addition, recent research suggests that AR can foster greater empathy in users. Many of these immersive technology apps and sites are free or inexpensive.

To learn more about how to use augmented reality tools in your classroom, check out What Is Augmented Reality? and Tools and Apps to Bring Augmented Reality into Your Classroom before diving into the top augmented reality tools below.

My Very Hungry Caterpillar Based on Eric Carle’s The Very Hungry Caterpillar book, the My Very Hungry Caterpillar app is ideal for encouraging a love of nature and nurture in kids up to age five. Users feed and care for the virtual caterpillar, allowing him to eventually transform into a butterfly. Along the way, they grow flowers, paint pictures, and hunt for buried treasure. This one’s a gem. iOS Android

RakugakiAR An award-winning app that animates scanned drawings and doodles, such that the characters spring to life. Users can then engage with their creations through virtual activities such as playing and feeding. For only $0.99, create your own virtual reality. Check out this brief video to see how it works. iOS Android

solAR - Solar System in AR A stunning AR app that allows users to bring elements of the solar system into close view for investigation and learning. The app uses your device’s camera to superimpose digital planets and moons on Earth-based scenarios, adjust the scale, and explore all the features. Free with in-app purchases. iOS Android

Assemblr EDU This 3D and AR imaging platform allows students to explore, produce, edit, and annotate 3D and augmented reality creations, which can then be shared with teachers and classmates. Pre-made content and drag-n-drop controls makes it easy for users to get hooked right away. Free basic plan plus three paid levels provides a price point for every budget iOS Android

3DBear AR This super-creative AR design app offers lesson plans, challenges, 3D models, social media sharing, and 3D printing capability. The 3DBear website provides video tutorials, curriculum, and distance learning resources for educators. Great for PBL, design and computational thinking. Free and paid plans, with a 30-day free trial. iOS Android

Tech & Learning Newsletter

Tools and ideas to transform education. Sign up below.

Quiver - 3D Coloring App Quiver turns a special coloring book into a 3D augmented reality experience. Users print AR-enabled blank coloring sheets through the app or the website, then color each with crayons, colored pencils, or oil pastels. Activate the quiver app to bring the pages to life. Free app, free and paid coloring packs. iOS Android

SkyView Explore the Universe One of the most popular AR astronomy apps, SkyView Explore the Universe makes it easy to identify galaxies, stars, constellations, planets, and satellites in the night sky, while the AR mode allows users to spot objects day or night. Great for out-of-the-way places, as no WiFi, data signal, or GPS is required. Try the Time Travel setting to see the sky of ancient and future ages. $1.99. iOS Android From the same developer, the similar SkyView Lite is free.

Augment A simple app for viewing 3D AR models. Users upload their own 3D models and can manipulate each one in various ways with the app. Augment works well for design-related subjects and presentations; resources are available for students and teachers to get started. Free.

Froggipedia An impressive AR anatomy app that covers every phase of frog life, from a single-celled egg to a full-grown frog. Dissect and examine the AR frog’s lifelike internal organs using a finger or Apple pencil with this 2018 iPad App of the Year. $3.99

Sky Guide Winner of the Apple Design Award 2014, Sky Guide allows users to instantly locate stars, planets, satellites, and other celestial objects in the present, past, or future. Augmented Reality mode makes it easier to visualize and identify constellations. Works with or without WiFi, cellular service, or GPS. $2.99

Wonderscope This highly engaging interactive story app puts kids at the center of the unfolding action, allowing them to move around, become part of the story, and explore details by tapping on objects. Free for the first story; additional stories are $4.99 each

East of the Rockies A beautiful story app from the National Film Board of Canada. Written by acclaimed author Joy Kogawa, this interactive tale follows the journey of a Japanese-Canadian family interned during WWII. Free teachers guide in English and French. $3.99

McGraw Hill AR Online Explore topics ranging from V8 engines to exponential growth to the features of glaciers in this fine collection of 3D interactive augmented reality animations. Each interactive begins with an overview of the subject, then provides an opportunity for users to manipulate objects and activate functions. Take the quiz following each exercise to demonstrate comprehension. Free, no account required.

CoSpaces Edu A complete 3D, coding, and AR/VR platform for education , CoSpaces Edu provides online tools for teachers and students to create and explore their own augmented worlds. Features include lesson plans and an extensive gallery of CoSpaces created by teachers, students, and the CoSpacesEdu team. AR requires iOS or Android device and free app. Free basic plan for up to 29 students.

Lifeliqe A standards-aligned K-12 science curriculum with lesson plans and eye-popping, sophisticated interactive 3D models. Integrates with Google Drive, Google Classroom, and Microsoft Teams for easy sharing and saving. Prices start at $8 per student annually. 14-day free trial with premium content.

• How To Setup Virtual Reality Or Augmented Reality In Schools For Free
• Catchy Words: How to Use It to Teach
• Best Free AI Quiz Generators

Diana has been Tech & Learning's web editor and contributor since 2010, dedicated to ferreting out the best free tech tools for teachers.

NoRedInk: How to Use It to Teach Writing

GPT-4o: What Educators Need to Know

What Is Scratch And How Does It Work? What's New?

Most Popular