A.) Abstract:

Free access to the virtual space needs to be improved and facilitated by enabling real-time integration of human participants into the three-dimensional space. Using an "in-house" "3D Key System" developed by Sommerer & Mignonneau and first shown at the "Tokyo Metropolitan Museum of Photography" 1995, participating parties enter the virtual space in real-time and interact with it in three dimensions.

Virtual space is understood as a place of interaction and integration, where human-human communication and interaction can be performed and visualized.

The set-up, construction and development of such a system is research topic of Laurent Mignonneau and Christa Sommerer within the MIC Media Integration and Communication Laboratories.

"MIC Exploration Space" shall be used for the communication between participants through images and extended integration of real-time interaction in virtual space.

The place functions as a new form of integration space, where real-time integration of development pictures, artificial life , as well as artistic features will display an evolving environment, that can be topic related and refer to and represent the individuality of the participant.

In the future sound as well as precise gesture recognition shall be implemented as well.

Integrating the human participants on both sides of the interaction set-up into a common virtual space, a shared environment shall be created, which functions as a place of exchange and interaction. The advantage of having a shared 3 dimensional environment is, that it lets the participants interact with each other in a more efficient, natural and creative way.

2) Background:

Sommerer & Mignonneau have been working in the field of "Interactivity" as art form since 4 years. There works and installations have been presented since 1993 in 33 exhibitions all around the world: Germany, Austria, France, Spain, Italy, Poland, Finland, Belgium, Monaco, Canada, USA, Korea and Japan.

There works received 8 international awards since 1992. A permanent interactive computer installation, called "Trans Plant" is installed at the " Tokyo Metropolitan Museum of Photography" in Tokyo.A second permanent installation is installed at the "AEC Ars Electronica Center" in Linz Austria in 1996. In 1997 a new interactive computer installation, called " Life Spacies", related to their research at ATR MIC Laboratories, will be installed at the NTT-ICC InterCommunication Center in Tokyo Japan, to be opened in summer 1997.

3) Interactive computer installation:

In order to provide an insight into topics that have been relevant for the development of interactive computer installations by Sommerer & Mignonneau, three of the total five installations shall be described here in more details.

3.1) Interactive Plant Growing:

The project " Interactive Plant Growing" was developed and programmed by Laurent Mignonneau and Christa Sommerer at Institut für Neue Medien, Frankfurt, Germany 1992/93.

It deals with the sensitive relationship between 5 real plants and 5 or more human viewers, who can initiate and control by their hands a high developed 3 dimensional real time growth on the screen of a Silicon Graphics computer. By producing a sensitive interaction with the real plants, the viewers become part of the installation. They decide how this interaction is translated to the screen and how the growth takes place on the screen.

In the installation the electrical potential difference between viewers and plants is directly transformed into electrical signals, which conduct the way how the virtual 3 D growth on the screen takes place. By this the viewer performs a dialog with the real plants; he can see the interpretation of this dialog in real time on the video screen.

The viewers can stop, continue, deform, cross and rotate the plants, develop new forms and thus produce always unexpected new combinations and variations of growth, which only depend on their interactive dialog with the real plants . The growing processes are programmed to be very flexible; they are not predetermined, so that the result on the screen is always new and different, depending on the interaction viewer - plant.

3.2) A-Volve:

Developed by Sommerer & Mignonneau 1994/95, A-Volve was supported by ICC-Intercommunication Center NTT, Japan and NSCA, National Center for Supercomputing Applications , Illinois, USA. It was realized at ATR Advanced Telecommunications Research Laboratories, Japan in collaboration with Dr. Tom Ray, HIP Lab.

In the interactive real-time environment "A-Volve" visitors interact with virtual creatures in the space of a water filled glass pool. These virtual creatures are products of evolutionary rules and influenced by human creation and decision.

Designing any kind of shape and profile with their finger on a touch screen, visitors will "bear" virtual three dimensional creatures, that are automatically "alive" and swim in the real water of the pool.The movement and behavior of the virtual creature is decided by its form, how the viewer was designing it on the touch screen. Behavior in space is, so to speak, an expression of form. Form is an expression of adaptation to the environment.

Form and movement are closely connected, the creatures capability to move will decide its fitness in the pool.

The fittest creature will survive longest and will be able to mate and reproduce. The creatures will compete by trying to get as much energy as possible. Thus predator creatures will hunt for prey creatures, trying to kill them.

The creatures also interact with the visitors, by reacting to their hands movement in the water. If a visitor tries to catch a creature, it will try to flee or stays still, if it gets caught. Thus the visitor is able to influence the evolution by for example protection preys against predators. If two strong creatures meet, they can create an offspring and a new creature can be born. It carries the genetic code of its parents. Mutation and cross-over provides a nature-like reproduction mechanism, that follows the genetic rules of Mendel. This newly born offspring will now also react and live in the pool, interacting with visitors and other creatures.

Algorithms, developed by Mignonneau and Sommerer ensure smooth and natural movements and "animal-like" behavior of the creatures. None of the creatures is pre calculated, they are all born exclusively in real time through the interaction of the visitors and the interaction of the creatures. Thus a great variety of forms will be possible, representing human and evolutionary rules.

By closely connecting the real natural space of the water to the unreal virtual living space of the creatures, "A-Volve" minimizes the borders between "real" and "unreal", creating a further step (after "Interactive Plant Growing") in the search of "Natural Interfaces" and "Real-Time Interaction".

3.4) " Trans Plant ":

"Trans Plant" is an interactive computer installation, developed by Sommerer & Mignonneau in 1995 for the "Tokyo Metropolitan Museum of Photography" and displayed there for a period of 3 years. "Trans Plant" was realized at ATR Advanced Telecommunications Research Laboratories, Japan.

In "Trans Plant" visitors enter a semi-circled room and become part of a virtual jungle which starts to surround them. As the visitor steps forward into the installation space, he will see himself inside a projection screen in front of him. By walking freely and without any devices, he soon will discover that grass is growing, wherever he walks, following each step and movement he does.

When stopping and staying still, trees and bushes will grow on the place where the visitor currently stands. Changing the speed and frequency of his movements, the visitor thus will create a biotope, that is full of different plant species. The size, color and shape of these plants depends only on the size of the person. Small children usually will create different plants than their parents, but by folding out the arms, the size of the plants can be increased. If moving the body slightly backwards or forwards the color density can be changed as well.

As each visitor will create different plants, he will bring up his/her own personal forest, that is an expression of his personal attention and feeling for the virtual space. As the growth gets more and more dense and the space more and more full of different plant species , the visitor will more and more engulf into this virtual jungle: he will enter deeper and deeper into the growth, and appear as a 3 dimensional person in the virtual space.

A NEW '3D Video KEY' system

A new system of '3D Video KEY' was developed by Mignonneau for this application, which allows the visitor for the first time, to enter the virtual space without any devices, but still be displayed as a 3D person.

This allows him, for the first time, to be in front of plants, behind them, to cross them or to walk on them. All movement and exploration is in real-time, the comparison of 3D space and the 3D position of the visitor has therefore no delay.

The feeling of really being inside the 3D space is enhanced by the fact, that the visitor sees his own full image inside the virtual jungle and thus is able to explore the virtual space very naturally and freely.

He can touch the plants, hide between leaves, have one arm in front of a trunk or the other one behind a bush.

Several people can be displayed at the same time inside the jungle, but one person will lead the interaction. Thus people will start to hide between the plants, grow different personal types of plants and so create environments, that are different from person to person.

The aim of "Trans Plant" is, to create a personal environment, where visitors find themselves freely interacting with the virtual space, becoming part of it and essentially creating this space by themselves.

4.) "MIC Exploration Space":

Whereas all the above described installations emphasize the aspect of human-environment interaction ( human-plants, human-virtual creatures..), the "MIC Exploration Space" additionally emphasizes the communication between remotely located participants.

"MIC Exploration Space" features the following characteristics:

4.1) Human-Human Communication:

Human to human communication is probably one of the most advanced and challenging types of communication. In the case of "MIC Exploration Space" two steps of human to human communication shall be explored.

a) Non verbal communication: Human - Human Image Communication

Human to human communication shall be enriched by the virtual environment. Communication between remotely located participants will enable the participants to interact in the same virtual environment by communicating with each other through gestures and movements.

By sharing the same virtual environment, participants will start to discover each other through their environments.

The environment has to be designed in a way, that it allows minute changes and sensitive modulations of the visitors movements and body interactions.

Thus the evolvable and flexible virtual space will be a direct interpretation of the participants personality and their behaviour in space.

b) Verbal and acoustic communication:

Sound as shall be implemented. As the participants talk to each other, their conversations shall be linked and interpreted in the visual environment.

4.2) Human-Environment Communication = Image Communication:

As in the previous works, where human-image communication has been one of the main aspects of the works, Image Communication will be applied and integrated in the "MIC Exploration Space " as well.

This will allow the participant to interact with his environment, even if there is no other second participant. Images and the three dimensional algorithms will reflect the personal state of the participants individuality.

When sharing the virtual space with another or other person/s, the environment shall reflect these change as well and adapt itself to all new situations.

4.3) Real-Time Interaction:

Real-time interaction is essential when designing interactive exhibits.Immediate feedback is critical in order to provide a satisfying interaction between the participant and the virtual environment. Yet the feedback has to be not too linear, since the action-reaction mechanism could otherwise be discovered too easily and the interaction would become too obvious.

Basically here lies one of the main differences between games, or purely technical demonstrations and an art piece.

Whereas games emphasize the easy to understand "press a button- get an effect" interaction, technical demonstrations, such as flight simulators or other high tech virtual reality applications can be very elaborate.

Still they do lead the participant in a predefined path of action and reaction code. They are designed in a way, that every visitor will be getting the same type of results, when encoding the systems rules.

The artistic interactive computer installation is on the other hand much more concentrating on the intellectual and emotional effect it has on the participants.

At the first glance an artistic project might even look similar to a technical demonstration, but when reflecting about the concept and the issues transported with the work, we clearly will be able to see the difference.

Yet is has to be said, that it is an ongoing goal of the avant garde art, to distinguish itself from the purely technical demonstration and to not get caught up in the technical aspect of the work, but to clearly define concepts and content.

In designing appropriate interfaces we need to consider, that freedom of movement, freedom of decision and human sensibility have to be taken in account.

4.4) Unencumbered Interaction through "3 D Video Key System":

To avoid uncomfortable devices and to facilitate the access to the virtual space, unencumbered human-human interaction is essential .

The "3D Video Key" system hardware interface by Mignonneau 95, will allow the keying of the viewers video image into the three-dimensional space, pixel by pixel.

The detection of the participants position in 3 dimensions will be solely performed by a camera detection system.

Thus the participants will be able to freely create and explore the virtual space by simply moving in the real physical space. Each physical body position refers to the virtual body position, displaying the participants virtual body position on a large projection screen in front of him . As it is very easy for the participant to identify himself with his image on the screen, he immediately will be able to interact in the virtual environment.

4.5) Multilayered Interaction:

Interaction has to be rich and multi layered, in order to provide interesting and involving interaction.

To provide a rich environment, the participants gestures and individual body movements, as well as body size and color of cloths e.t.c. will be linked to characteristic features in the evolving environment.

To avoid the wide spread "press button - get an effect" interaction, a special protocol will be created, that is able to encode personal characteristics such as, frequency of movements, gestures , body size, body color and speech.

The representation of the participants individuality and its translation into the virtual environment should be subtle and with a extendible variety. In order to create a human environment, the virtual world has to be evolvable and sensitive to minute changes.

As in "Trans Plant", where each persons behavior is linked to special events on the screen, "MIC Exploration Space" shall also emphasize on the richness of the environment.

4.6) Interaction between remotely located participants:

A key aspect of "MIC Exploration Space" is the interaction between remotely located participants. Visitors on both sides can communicate with each other through gestures, movement and sound and thus share a common virtual environment.

4.7) Evolving Artificial Life Environment:

In order to achieve rich interaction environments it is necessary to use evolution for the creation of open-ended environments.

As in "A-Volve" where virtual creatures interact with each other and with the visitors, in MIC Exploration Space participants again shall be able to interact with and communicate in an environment, that is not static and predetermined, but able to evolve over time.

Like the virtual creatures in "A-Volve", graphical 3 dimensional representations of the participants individual actions and communication processes shall be provided, that use evolution of images and object data base for a rich and open-ended exploration of "MIC Exploration Space" .

4.8) Art:

Art is often described by scientists as unprecise and incomprehensible for linear oriented research. "Interactive Art" in the past years has proved, that "non linear" interpretation of human - human and human - environment communication is often very successful, as it leaves the participants the freedom of interaction, communication and interpretation. It certainly has to be said that some of the interactions still are in their early stages and need to be explored and refined.

As we are achieving more and more sophisticated interfaces to link real action and communication to virtual space, and as different medium, such as gesture, sound and speech recognition systems can be integrated, interaction and communication will become more and more complex.

It is the target of art, i.e. interactive art, to represent and enhance human emotional understanding and to integrate human decision and human personality in the interactive environment. In contrast to the conventional art work, that represented an one way communication path (from the artist trough the artwork to a relatively passive and perceiving audience), interactive art emphasizes integration, interaction and communication.

This is the point, where research on interaction and communication and art can profit from each other, as they have a common target: to enhance and enable human-human and human-(art)work communication.

5. ) Conclusion:

"MIC Exploration Space " is a ongoing research project, that will evolve over time. In its first and early version it was shown at the ATR open house in November 95 and at the Siggraph 1996 Art Show "The Bridge".

"MIC Exploration Space" (c) 96-97, Christa Sommerer & Laurent Mignonneau,

ATR Media Integration and Communications Research Lab, Kyoto Japan