Memory Stairs: Defining an Expressive Grammar for Emotionally Evocative Virtual Environments

The Expanse of Virtual Reality

Content

The immersive environment as a distinct new medium has yet to develop a unique and significant grammar.  It currently functions on expectations mediated by existing forms of interactivity and entertainment, such as movies, theater, games, architecture, television and performance art, borrowing from each as needed.  This is because its first few decades have emphasized development of increasingly sophisticated hardware and software in support of technology demands, to the exclusion of serious content creation that would facilitate an emerging grammatical structure.

I contend that there is an emerging grammar that exists for such environments, and will support this by investigating ideas from recent semiotic theories, especially those that pertain to media such as films and photography and the social dimension that informs how we learn to “read” such media.  I will look at the basic structural elements unique to virtual environments, including world building, auditory events, smells, lighting and interaction, for their contribution to an overall meaning expressed as an emotional impact on people experiencing such worlds. I will then create content that reveals the unique grammatical potential inherent in virtual environments. Memories, with their strong linkages to emotional response, are my focal point for the design framework of the virtual environments.

	Image from the second Memory: Just New
	Image from the Forgotten House Memory

Practice

The "Memory Stairs" art work will inform the theoretical components of my research by providing a practical testbed to explore these ideas. The installation will consist of a physical spiral staircase serving as the first-person interface to a series of virtual scenarios.  Stepping on each stair will trigger a unique emotionally evocative “memory experience”.  First-person visitors will wear a head-mounted display (HMD) and other interface accoutrements necessary to hear, see and smell the multi-sensory experience.  The simulation images being viewed by the person in the HMD will also be projected for spectators on a large curved screen in a VR theater I have access to in Los Angeles.  I hope to port the work to a similar installation space in London.

Methodology

The design process for my virtual environments consists of determining a “memory” scenario, such as a childhood birthday party. The geometric models that support that scenario are then built, using a 3D software, then textured and lit.  Active objects in the scenario are animated, for example, the candles the visitor might blow out on the cake.  Sounds must be generated as well, and all these components integrated with programming code to run as a real-time simulation.  This process is repeated for each “memory”.  All memories are finally integrated into a seamless real-time virtual experience.

As these memories are completed and people experience them, their choices and reactions will be used to improve the experiences so they are more responsive, or provide a stronger emotional impact. These changes will be more than technical; they will involve artistic expertise, much as a cinematographer tweaks the mood of a scene by choreographing the lighting and arranging the set.  This is unexplored territory for virtual environments.

 Research Plan

The research will investigate the following areas:

a)      Development of the grammar in related media: early theatre, films, location-based entertainment and performance; comparison of these developments to the rise of virtual environment practice.

b) A detailed historical survey of artist-created virtual environments that evoke emotional responses.

c)      How semiotics have informed traditional media and its potential relevance to virtual environments. I am especially interested in whether the VE scenarios function as signs or mythologies in the sense that Barthes assigns to these terms. (Barthes, 1957)

d)   Contributions from other disciplines and discourses will be examined, from cognitive science to feminist theory.

e)    The idea of virtual environments as a ritual, liminal space that can encompass both the rational and the emotional aspects of our human nature.

f) The role of memories, associative reasoning and subjective effects, especially from the standpoint of what the participant in the virtual world contributes to the perceived emotional intensity. I will relate this to semiotic discussions of decoding and participant interpretation. (Hall 1973, Eco, 1979)

The first phase of work includes deconstructing existing virtual environments and analyzing their structure and efficacy within current semiotic narrative models. (Barthes 1977, Eaton, 1981, Bolter, 1991).  Concurrently I will develop the technology framework (equipment, code, model creation, production pipelines) for my practice. This will be followed by design and creation of the first "stair memories", informed by the findings from my initial analysis.   Early feedback from participants who experience these memories will be derived from interviews and questionnaires and this information, along with logged records of the individual user’s interaction choices, will help determine the emotional valence of the virtual environment experiences.

The second phase of work will take the results of the above investigations and incorporate them into the continuing design of the “Memory Stairs” scenarios.  At this stage I will begin to formulate a critical discourse relating my findings about the developing grammar.  The iterative process of creating the environments, participant feedback, and discourse will continue as the full installation version of “Memory Stairs” takes shape.

The last stage of research begins with the installation of “Memory Stairs” in a museum space.  Participant interviews and interactions will be collected and analysed and the findings factored into my final assessment of the emerging grammar within the written dissertation.

Bibliography