Virtual eyes can rearrange your body: Adaptation to visual displacement in see-through, head-mounted displays

Among the most critical issues in the design of immersive virtual environments are those that deal with the problem of technologically induced intersensory conflict and one of the results, sensorimotor adaptation. An experiment was conducted to support the design of a prototype see-through, head-mounted display (HMD). When wearing video see-through HMDs in augmented reality systems, subjects see the world around them through a pair of head-mounted video cameras. The study looked at the effects of sensory rearrangement caused by a HMD design that displaced the user's "virtual" eye position forward (165 mm) and above (62 mm) toward the spatial position of the cameras. The position of the cameras creates images of the world that are slightly downward and inward from normal. Measures of hand-eye coordination and speed on a manual pegboard task revealed substantial perceptual costs of the eye displacement initially, but also evidence of adaptation. Upon first wearing the video see-through HMD, subjects' pointing errors increased significantly along the spatial dimensions displaced (the y dimension, above-below the target, and z dimension, in front-behind the target). Speed of performance on the pegboard task decreased by 43% compared to baseline performance. Pointing accuracy improved by approximately 33% as subjects adapted to the sensory rearrangement, but it did not reach baseline performance. When subjects removed the see-through HMD, there was evidence that their hand-eye coordination had been altered. Negative aftereffects were observed in the form of greater errors in pointing accuracy compared to baseline. Although these aftereffects are temporary, the results may have serious practical implications for the use of video see-through HMDs by users (e.g., surgeons) who depend on very accurate hand-eye coordination.