Multimodal Scene Creation

Considering only one modality in product quality research is often insufficient. When using the product, the user always perceives it with all sensory modalities. The multimodal measurement laboratory of the Chair of Acoustic and Haptic Engineering offers the possibility to present multimodal scenes and thus to address auditory, tactile and visual modalities. In addition, a haptic reproduction on special force-feedback-systems such as Cybergrasp gloves can be realized. Such multimodal scenes are the basis for of virtual environments.

Immersion into Virtual Realities

However, the sensory perception, multimodal interaction effects, synchronicity thresholds between the modalities as well as the experiences and expectations of the user must be taken into account when designing virtual environments. The goal of any virtual environment is to present the environment as coherently as possible so that users react or interact to the environment as they would in everyday life. There are two factors that influence the user's perception or behavior: "place illusion" and "plausibility illusion". The place illusion describes the impression of being placed in an environment, and the plausibility illusion refers to the content of the depicted environment and is elicited when the scene meets the user's expectations.

Design of Multimodal Scenes

Essentially, there are two ways in which you can design scenes: the "authentic approach" or the "plausible approach." The authentic approach tries to evoke the percepts in the virtual environment that would be evoked in the corresponding real situation. For this purpose, the scene must be reproduced with high accuracy in the virtual environment. This approach is very complex and places high demands on recording and playing back of the scenes. The main problem with this approach, however, is that the user's experience is not taken into account. In order to decide whether a virtual environment is plausible, the user compares the scene presented with his previous experiences regarding similar scenes. It follows that a physically correct performance does not necessarily have to be plausible. When different sensory modalities are addressed with contradictory stimuli, the evoked perceptual event can be dominated by one modality. For example, since the perceptual events occurring in a scene are closely related to the experiences of this scene context, the expectations of a tactile stimulus could be dominated by the visual modality. It is even possible that the expectations of a perceptual event in the context of a particular situation are shaped by media experience and are thus independent of the real situation. The pursuit of the plausible approach to the design of scenes therefore makes it possible to even generate scenes that have no equivalent in a real existing scene. The approach also allows for a potential reduction in the amount of data needed to generate the scene, since the scene does not have to be presented physically correct but merely in a plausible way. Approaches to scene design, which are based on the perception of humans, thus seem to be particularly suitable for the plausible reproduction in virtual environments

Creating a Synthesis Model for Plausible Scene Presentation

Since the concept of plausibility is closely related to expectations, methods are developed at the Chair to quantify these expectations by characterizing them with perceptual attributes. The relationship between the stimuli presented and the perceptual attributes they produce can be modeled. This should make it possible to systematically generate stimuli that meet the expectations of the user and thus cause the plausibility illusion.

Multimodal Scenes as Content for Virtual Environments

Virtual environments offer advantages for many applications. A user can be placed in any environment with which he can safely interact. Controlled changes in certain aspects can be used to investigate their influence on the perception or behavior of the user. These benefits are used in a variety of fields, such as driving simulator or flight simulator training, treatment of anxiety disorders, entertainment and research and development.