The pioneering technology plus the thought-provoking narratives–I will definitely visit the Future of Story Telling again next year! I have gained the most immersive experience of a virtual reality film, Miyubi (2017), even though I was sitting in the heat of a little pavilion of the “old castle” built in 1834 on the remote Staten Island an-hour- away from the south ferry.

The Music Hall held most of the events. Photo Credit to Annie.

The 40-minute VR film Miyubi (2017) had me inhabit in a Japanese toy robot named Miyubi, who was gifted to a young boy Dennis on his birthday in 1982 suburban America. Over the year, I witness the fracture of the family, the malfunction of Miyubi, and the arrival of a new robot. At last, Miyubi was reprogrammed by children in an underdeveloped region and rebooted among other rusty machines besides a donkey.

In this story, VR did not exist for the sake of technological novelty only but created an indispensable environment for an engaging story based on an in-body experience. It was through VR that I embodied Miyubi, seeing through his eyes/screen, and identifying with him, because there was no time lapse between my movement and that of Miyubi. Whenever I turned my head, the vision changed accordingly, and in one scene I could see me panning my robotic head through the reflection on the diegetic TV screen. In Toni Dove’s words, I was “haunting” the robot, and saw “my traces [were] left on the screen” (Dove 64).

Such illusion of body control was exemplary in the beginning when the birthday boy Dennis was eager to have Miyubi pronouncing his name. At the sight of the encouraging facial close to me and his demo of pronouncing “D-E-N-N-I-S”, I could not help utter this name at his request, while Miyubi produced a robotic sound at the exact same time. As I was wearing the headphone, my voice was canceled out and Miyubi’s voice was clearly captured. It felt like as if I was producing the mechanical sound by activating my own muscles and vibrating my vocal folds.

Miyubi’s first meet with the family. Courtesy of Sundance Institute.

I wondered why I had a strong bond with Miyubi and moved by what happened to him. I reckoned that in addition to movement synchronicity and sound identification, Miyubi was a perfect container for me to fill my own feelings and interpretations. He remained silent for most of the time, so that I would not be bothered by a voice different from mine and embedded with the emotions that I did not have, such as the high-pitch cry of a boy in the VR horror film Night Night, which constantly detached me from the character and the story. Miyubi’s silence made me feel comfortable occupying his body, even though I doubted if robots really felt the way I did.

I suppose this was the paradox of Object-Oriented Ontology: I tried to sense what the robot would feel by entering the story from an object’s perspective, but I ended up humanizing the object, not to mention that the story itself was scripted by a human. As long as the robotic narrative could not automatically grow like that of “a seed crystal suspended in liquid” (Barker 71), I doubt if the human could witness a narrative beyond our epistemological frames.

At least, the designer attempted to resemble an object’s point of view to offer a new insight. Miyubi’s vision had boundaries marked by a red rectangle with an indication of remaining battery in the lower left, similar to the frame of DV camera. Such a visual design was extended to the editing style. Each sequence ended with the battery dying and fading to black and started with the rebooting and self-examination, which showed the date and memory health of Miyubi. It was more than indicating the passage of time for film narration, but also suggested the weariness of the robot which foreshadows his obsolescence.

Miyubi finally tasted the obsolescence shared by Grandpa. Grandpa was the only one who detested the novelty at first when Dennis showed off Miyubi to the classmates of his math skills and the little sister planned to hold a wedding ceremony with Miyubi. Later, as Miyubi started to have increasing technical glitches–his vision fell seriously; he could not pronounce the right name; he was unbearable compared with the newly arrived robot of higher artificial intelligence­ capable of expressing affection through complicated sentence–grandpa became the best friend of Miyubi.

Grandpa(left) empathized with Miyubi. Courtesy of Sundance Institute.

Once again, I put my own sensitivity into Miyubi’s body. As grandpa took Miyubi for a ride to escape the house that no longer needs them, I geared towards him, watching in silence how grandpa mumbled an age-old story. Being the sole company for a lonely man as a broken-down robot was such a melancholic moment. The pain of not wanted was intense for me, both as a human and as a robot. I was not watching the discarded Cowboy doll Woody in Toy Story (1995), I lived those beautiful memories through Miyubi, and the experience of losing affection was so real.

My impulse to preserve and recycle the elder robot did not result from the rational thinking of eco-friendliness, but from the sensitivity of a robot. I rejoiced in the rebirth the poor yet creative children gave to Miyubi, but also reflected on the problematic chain of the globalized industry, where the first-world kid consumed the new toy robot, and the third world children recycled the rusty one and made do with it.

I loved Miyubi because it put me in the shoes of a robot, who I can fully identify with while exploring a new perspective, whether it was the genuine objective vision or not. The issue of aging and globalization lying behind the family comedy made the story thicker. I was glad to immerse, amaze, and reflect in one piece, and the passive immersion by suspending one’s own will criticized by Huhtamo (2) did not happen to this beautiful and heartrending piece.

References

Barker, Timothy. “Objects and interaction.” Digital Creativity 22.2 (2011): 65-77.

Dove, Toni. “Swimming in Time: Performing Programmes, Mutable Movies–Notes on a Process in Progress.” Performance and Place. Palgrave Macmillan UK, 2006. 60-74.

Huhtamo, Erkki. “Seeking Deeper Contact: Interactive Art as Metacommentary.” Convergence 1.2 (1995): 81-104.

————— & More In Fost For Prize ——————

Intimacy, a multi-person VR, was tailored to two people who wish to know the unspoken thoughts of a couple in different stages of a relationship, such as the first meet on a train, the casual talk at home, or a nervous dating at the bar. The users could hear the conversation between two people, plus an inner voice from one of them, like the double lines in the balcony conversion in Anny Hall (1977). To my disappointment, the multi-person VR did not guarantee interaction between the users, except for the instruction before the program started “think about the person sitting next to you”, and users had no control over the character. Yet it indeed sparkled conversation after the show because people hoped to hear the other part of the story. However, the scope of intimacy is limited, which was confined to a heterosexual relationship, and assumed that the perspectives were largely determined by gender.

Fall in Love mimicked a dating program in the forest setting, where people asked questions to know more about each other. Here I could only ask the person as if it was my turn to do so. I picked one question of the two presented to me over the microphone, and the dating partner would respond to me in a natural voice, thanks to the technology of “natural language processing”. If I got bored, I could say “it was nice meeting you, but I got to go.” A signal for leaving the system as well as in the real life.Everything was made smoothly, except that the voice recognization system could not properly register my pronunciation, which led the partner to repeat over and over again with a confused facial expression “sorry, there is too much noise out there, could you speak up a little?”

Presentation ideas

Mathew’s video last year intrigues our group’s curiosity as he says “the dehumanization of Sally”. The high threshold his team sets pressures me to think about ways of presentation beside a short video. I came up with an interactive PPT, where people could vote for the parts they are interested in or a live performance inspired by the interactive furniture in Sally. But time is limited and our major aim is to show a sense of what Sally feels like, so I design a PPT in the conversational framework, which is collaborated by all of us and starts with Chen’s video teaser. I really love my passionate and efficient group members, enjoy watching Sally twice together and meet three times to sparkle each other’s ideas.

Chen Gong edited the film; Matthew Alan Lester and I love it!

The interface of DVD-ROM

When I first encountered Sally through the DVD-ROM in the old Mac (Legacy OSX 10.6 iMac), I was quite confused. There are four sections, “Play the program,” “Sally sings,” “Artificial Changelings” and “Spectropia”, but the menu does not show their connections and tell me where to start.

Disoriented as I feel, I am glad that we explored by ourselves before watching the explanatory video. Not acknowledging the standard process, we had to try our own instincts against the artist’s program, and negotiate the unexpected before we deemed it as a technical glitch or an artistic design. When Sally became silent, it could be that she was processing our input, was broken down at the question of “do you like ice-cream”, or maybe silence is one of her expressions.

The technical glitches draw my attention to the interface. Every time I wanted to start a conversation, I had to hold the “ESC” key, which kept reminding us of the boundary between Sally and me. I like Toni’s graceful performance on the stage, where she controlled the speed or perspective of the film with her hand movement detected by the movement sensor as if she was playing an instrument.

The stage version of the transparent interface is Toni’s idea media, where our body in movement, in space and in time is extended into the screen (Dove 65). However, for our easy access, Toni has to make compromises in the preservation of this hybrid form of installation and cinema. The mesmerizing effect of Sally on a huge screen has to be reduced to a video-game experience on the computer.

Furniture Talk, Database Cinema, Object-Oriented Philosophy

We love the furniture talk best. We discovered in the second viewing that each time we click on the furniture; they offer different conversations. The chair boasts about the massive production of chairs, but with another click, he anxiously asks “am I unique?” Even though we catch different phrases, any combination won’t fail to describe an economy powered and destroyed by consumerism in the 1920s, or show the life philosophy of Sally. I am convinced that the narration and database can be compatible (Kinder 348), from the forerunner of database cinema, Vertov and Bunuel, to Toni Dove.

The malfunction of voice recognition limits us to interacting through clicking instead of saying the name “chair”, which gives an impression that there is more interaction between the furniture themselves, than the objects and us. Such lack of interactivity is also felt in our conversation with Sally, who has few responses available and ends up telling her story no matter what we choose to say.

Interestingly, the object-oriented philosophy relieves me from my self-centered frustration by offering a new perspective on “activity”: “It is not that ‘user’ merely uses a passive technology; rather the technology and the machine, as the context of interactivity are active. Any moment of the interactive experience…is constituted by the active engagement of the context and its protocols in the object” (Barker 76-77).

Sally, a cyborg?

Having watched humanized machines talking naturally, I wonder why Sally moves and sounds in a jerky way. I owe such mechanic style to the insufficient AI development in 2003, until Toni gives us another reason during her NYU lecture on September 20th.

Toni explained she deliberately broke everything in Sally. Sally’s words were broken down to phonemes to make her look mechanical and funny. Toni said, “If she speaks in a seamless manner, how could users tell the difference between a live performance and a machine?” Besides, the broken-down Sally is adaptive to new settings. Toni and her team can quickly input sentence for Sally to pronounce, instead of filming speeches ahead, thus making Sally more improvisational and independent.

But Sally also displays distinct femininity. She claims to be a bubble dancer in 1920s, and she announces a vivid personality through her constant blow of kisses and roll of eyes, which are short videos restored in her database. Toni reduces the fluidity of the gestures by deleting a few frames and enable them to stay in harmony with her broken speech. As a result, Sally becomes an uncanny combination of human and non-human features.

I suppose Toni does not aim for a perfect human resemblance, but a “cyborg” to reflect on what woman means in the context of new technology since women protagonists prevail all her works. Sally boldly incorporates oppositional characters, a hyper-feminine appearance, and a robotic manner, flirting like a flapper but sometimes irresponsive and making no sense, historically situated in the Great Depression but preserved in a new technology of DVD-ROM. This hybrid “cyborg” woman is born from the history but not burdened by its restriction. She “transgresses the boundaries, potent fusions, and dangerous possibilities” (Haraway 154) and reflects on herself by her rebirth into a robotic body.

In the context of the Spectropia project, which Sally belongs to, the cyborg Sally fits well as the image of a historical ghost summoned to the high-tech future. She is a restored human body in a mechanical container, a product of wonder, demystify and mystify the past at the same time.

Sally in Spectropia

Reference:

Barker, Timothy. “Objects and interaction.” Digital Creativity 22.2 (2011): 65-77.

Dove, Toni. “Swimming in Time: Performing Programmes, Mutable Movies–Notes on a Process in Progress.” Performance and Place. Palgrave Macmillan UK, 2006. 60-74.

Haraway, Donna. Simians, Cyborgs, and Women; The Reinvention of Nature. New York: Routledge, 1991.

Kinder, Marsha. “Designing a Database Cinema.” Future cinema: The cinematic imaginary after film (2003): 346-353.