Miyubi@Future of Story Telling 2017

Interactive Cinema

The pioneering technology plus the thought-provoking narratives–I will definitely visit the Future of Story Telling again next year! I have gained the most immersive experience of a virtual reality film, Miyubi (2017), even though I was sitting in the heat of a little pavilion of the “old castle” built in 1834 on the remote Staten Island an-hour- away from the south ferry.

IMG_6438.JPGThe Music Hall held most of the events. Photo Credit to Annie.

The 40-minute VR film Miyubi (2017) had me inhabit in a Japanese toy robot named Miyubi, who was gifted to a young boy Dennis on his birthday in 1982 suburban America. Over the year, I witness the fracture of the family, the malfunction of Miyubi, and the arrival of a new robot. At last, Miyubi was reprogrammed by children in an underdeveloped region and rebooted among other rusty machines besides a donkey.

In this story, VR did not exist for the sake of technological novelty only but created an indispensable environment for an engaging story based on an in-body experience. It was through VR that I embodied Miyubi, seeing through his eyes/screen, and identifying with him, because there was no time lapse between my movement and that of Miyubi. Whenever I turned my head, the vision changed accordingly, and in one scene I could see me panning my robotic head through the reflection on the diegetic TV screen. In Toni Dove’s words, I was “haunting” the robot, and saw “my traces [were] left on the screen” (Dove 64).

Such illusion of body control was exemplary in the beginning when the birthday boy Dennis was eager to have Miyubi pronouncing his name. At the sight of the encouraging facial close to me and his demo of pronouncing “D-E-N-N-I-S”, I could not help utter this name at his request, while Miyubi produced a robotic sound at the exact same time. As I was wearing the headphone, my voice was canceled out and Miyubi’s voice was clearly captured. It felt like as if I was producing the mechanical sound by activating my own muscles and vibrating my vocal folds.


Miyubi’s first meet with the family. Courtesy of Sundance Institute.

I wondered why I had a strong bond with Miyubi and moved by what happened to him. I reckoned that in addition to movement synchronicity and sound identification, Miyubi was a perfect container for me to fill my own feelings and interpretations. He remained silent for most of the time, so that I would not be bothered by a voice different from mine and embedded with the emotions that I did not have, such as the high-pitch cry of a boy in the VR horror film Night Night, which constantly detached me from the character and the story. Miyubi’s silence made me feel comfortable occupying his body, even though I doubted if robots really felt the way I did.

I suppose this was the paradox of Object-Oriented Ontology: I tried to sense what the robot would feel by entering the story from an object’s perspective, but I ended up humanizing the object, not to mention that the story itself was scripted by a human. As long as the robotic narrative could not automatically grow like that of “a seed crystal suspended in liquid” (Barker 71), I doubt if the human could witness a narrative beyond our epistemological frames.

At least, the designer attempted to resemble an object’s point of view to offer a new insight. Miyubi’s vision had boundaries marked by a red rectangle with an indication of remaining battery in the lower left, similar to the frame of DV camera. Such a visual design was extended to the editing style. Each sequence ended with the battery dying and fading to black and started with the rebooting and self-examination, which showed the date and memory health of Miyubi. It was more than indicating the passage of time for film narration, but also suggested the weariness of the robot which foreshadows his obsolescence.

Miyubi finally tasted the obsolescence shared by Grandpa. Grandpa was the only one who detested the novelty at first when Dennis showed off Miyubi to the classmates of his math skills and the little sister planned to hold a wedding ceremony with Miyubi. Later, as Miyubi started to have increasing technical glitches–his vision fell seriously; he could not pronounce the right name; he was unbearable compared with the newly arrived robot of higher artificial intelligence­ capable of expressing affection through complicated sentence–grandpa became the best friend of Miyubi.


Grandpa(left) empathized with Miyubi. Courtesy of Sundance Institute.

Once again, I put my own sensitivity into Miyubi’s body. As grandpa took Miyubi for a ride to escape the house that no longer needs them, I geared towards him, watching in silence how grandpa mumbled an age-old story. Being the sole company for a lonely man as a broken-down robot was such a melancholic moment. The pain of not wanted was intense for me, both as a human and as a robot. I was not watching the discarded Cowboy doll Woody in Toy Story (1995), I lived those beautiful memories through Miyubi, and the experience of losing affection was so real.

My impulse to preserve and recycle the elder robot did not result from the rational thinking of eco-friendliness, but from the sensitivity of a robot. I rejoiced in the rebirth the poor yet creative children gave to Miyubi, but also reflected on the problematic chain of the globalized industry, where the first-world kid consumed the new toy robot, and the third world children recycled the rusty one and made do with it.

I loved Miyubi because it put me in the shoes of a robot, who I can fully identify with while exploring a new perspective, whether it was the genuine objective vision or not. The issue of aging and globalization lying behind the family comedy made the story thicker. I was glad to immerse, amaze, and reflect in one piece, and the passive immersion by suspending one’s own will criticized by Huhtamo (2) did not happen to this beautiful and heartrending piece.


Barker, Timothy. “Objects and interaction.” Digital Creativity 22.2 (2011): 65-77.

Dove, Toni. “Swimming in Time: Performing Programmes, Mutable Movies–Notes on a Process in Progress.” Performance and Place. Palgrave Macmillan UK, 2006. 60-74.

Huhtamo, Erkki. “Seeking Deeper Contact: Interactive Art as Metacommentary.” Convergence 1.2 (1995): 81-104.

————— & More In Fost For Prize ——————

Intimacy, a multi-person VR, was tailored to two people who wish to know the unspoken thoughts of a couple in different stages of a relationship, such as the first meet on a train, the casual talk at home, or a nervous dating at the bar. The users could hear the conversation between two people, plus an inner voice from one of them, like the double lines in the balcony conversion in Anny Hall (1977). To my disappointment, the multi-person VR did not guarantee interaction between the users, except for the instruction before the program started “think about the person sitting next to you”, and users had no control over the character. Yet it indeed sparkled conversation after the show because people hoped to hear the other part of the story. However, the scope of intimacy is limited, which was confined to a heterosexual relationship, and assumed that the perspectives were largely determined by gender.


Fall in Love mimicked a dating program in the forest setting, where people asked questions to know more about each other. Here I could only ask the person as if it was my turn to do so. I picked one question of the two presented to me over the microphone, and the dating partner would respond to me in a natural voice, thanks to the technology of “natural language processing”. If I got bored, I could say “it was nice meeting you, but I got to go.” A signal for leaving the system as well as in the real life.Everything was made smoothly, except that the voice recognization system could not properly register my pronunciation, which led the partner to repeat over and over again with a confused facial expression “sorry, there is too much noise out there, could you speak up a little?”

Sally or the Bubble Burst-Vivian

Interactive Cinema

Presentation ideas

Mathew’s video last year intrigues our group’s curiosity as he says “the dehumanization of Sally”. The high threshold his team sets pressures me to think about ways of presentation beside a short video. I came up with an interactive PPT, where people could vote for the parts they are interested in or a live performance inspired by the interactive furniture in Sally. But time is limited and our major aim is to show a sense of what Sally feels like, so I design a PPT in the conversational framework, which is collaborated by all of us and starts with Chen’s video teaser. I really love my passionate and efficient group members, enjoy watching Sally twice together and meet three times to sparkle each other’s ideas.

Chen Gong edited the film; Matthew Alan Lester and I love it!

The interface of DVD-ROM

When I first encountered Sally through the DVD-ROM in the old Mac (Legacy OSX 10.6 iMac), I was quite confused. There are four sections, “Play the program,” “Sally sings,” “Artificial Changelings” and “Spectropia”, but the menu does not show their connections and tell me where to start.

Disoriented as I feel, I am glad that we explored by ourselves before watching the explanatory video. Not acknowledging the standard process, we had to try our own instincts against the artist’s program, and negotiate the unexpected before we deemed it as a technical glitch or an artistic design. When Sally became silent, it could be that she was processing our input, was broken down at the question of “do you like ice-cream”, or maybe silence is one of her expressions.

The technical glitches draw my attention to the interface. Every time I wanted to start a conversation, I had to hold the “ESC” key, which kept reminding us of the boundary between Sally and me. I like Toni’s graceful performance on the stage, where she controlled the speed or perspective of the film with her hand movement detected by the movement sensor as if she was playing an instrument.

The stage version of the transparent interface is Toni’s idea media, where our body in movement, in space and in time is extended into the screen (Dove 65). However, for our easy access, Toni has to make compromises in the preservation of this hybrid form of installation and cinema. The mesmerizing effect of Sally on a huge screen has to be reduced to a video-game experience on the computer.

Furniture Talk, Database Cinema, Object-Oriented Philosophy

We love the furniture talk best. We discovered in the second viewing that each time we click on the furniture; they offer different conversations. The chair boasts about the massive production of chairs, but with another click, he anxiously asks “am I unique?” Even though we catch different phrases, any combination won’t fail to describe an economy powered and destroyed by consumerism in the 1920s, or show the life philosophy of Sally. I am convinced that the narration and database can be compatible (Kinder 348), from the forerunner of database cinema, Vertov and Bunuel, to Toni Dove.

The malfunction of voice recognition limits us to interacting through clicking instead of saying the name “chair”, which gives an impression that there is more interaction between the furniture themselves, than the objects and us. Such lack of interactivity is also felt in our conversation with Sally, who has few responses available and ends up telling her story no matter what we choose to say.

Interestingly, the object-oriented philosophy relieves me from my self-centered frustration by offering a new perspective on “activity”: “It is not that ‘user’ merely uses a passive technology; rather the technology and the machine, as the context of interactivity are active. Any moment of the interactive experience…is constituted by the active engagement of the context and its protocols in the object” (Barker 76-77).


Sally, a cyborg?

Having watched humanized machines talking naturally, I wonder why Sally moves and sounds in a jerky way. I owe such mechanic style to the insufficient AI development in 2003, until Toni gives us another reason during her NYU lecture on September 20th.

Toni explained she deliberately broke everything in Sally. Sally’s words were broken down to phonemes to make her look mechanical and funny. Toni said, “If she speaks in a seamless manner, how could users tell the difference between a live performance and a machine?” Besides, the broken-down Sally is adaptive to new settings. Toni and her team can quickly input sentence for Sally to pronounce, instead of filming speeches ahead, thus making Sally more improvisational and independent.

But Sally also displays distinct femininity. She claims to be a bubble dancer in 1920s, and she announces a vivid personality through her constant blow of kisses and roll of eyes, which are short videos restored in her database. Toni reduces the fluidity of the gestures by deleting a few frames and enable them to stay in harmony with her broken speech. As a result, Sally becomes an uncanny combination of human and non-human features.

I suppose Toni does not aim for a perfect human resemblance, but a “cyborg” to reflect on what woman means in the context of new technology since women protagonists prevail all her works. Sally boldly incorporates oppositional characters, a hyper-feminine appearance, and a robotic manner, flirting like a flapper but sometimes irresponsive and making no sense, historically situated in the Great Depression but preserved in a new technology of DVD-ROM. This hybrid “cyborg” woman is born from the history but not burdened by its restriction. She “transgresses the boundaries, potent fusions, and dangerous possibilities” (Haraway 154) and reflects on herself by her rebirth into a robotic body.

In the context of the Spectropia project, which Sally belongs to, the cyborg Sally fits well as the image of a historical ghost summoned to the high-tech future. She is a restored human body in a mechanical container, a product of wonder, demystify and mystify the past at the same time.

Sally in Spectropia


Barker, Timothy. “Objects and interaction.” Digital Creativity 22.2 (2011): 65-77.

Dove, Toni. “Swimming in Time: Performing Programmes, Mutable Movies–Notes on a Process in Progress.” Performance and Place. Palgrave Macmillan UK, 2006. 60-74.

Haraway, Donna. Simians, Cyborgs, and Women; The Reinvention of Nature. New York: Routledge, 1991.

Kinder, Marsha. “Designing a Database Cinema.” Future cinema: The cinematic imaginary after film (2003): 346-353.



VR Fest. # Jump into the Light

Interactive Cinema, Uncategorized

On Saturday, I started my first VR trip in the Big Apple. After all the hard work on sending out the friend requests, establishing an Interactive Cinephilia group on Facebook, and calling for a hang out in the lower-east side, I feel it worthwhile to meet several friends at #Jump into the Light, including Matthew, Yifan and some NYU alumni.



small building but fascinating (and hot!)



Due to the malfunctioning train on the weekends, I sadly missed the workshop on how to make easy VR projects. I decided to explore the stands and get the most of this hot and magic space.

Attraction 1: Live 3D VR Camera 


I put on the gadget and saw myself from above. It was an “out-of-body” experience, like the one I had in my dream. I waved my hand and there was no latency. With a click by the presenter, my vision transits between different locations of the VR Fest, the workshop, the stand and so on, thanks to the little cameras hanging on the ceiling.IMG_6048.JPG

According to SUOMETRY’s website,  it is a “computer vision technology company focused on developing ultra-fast and high-quality live action VR capture”. The following is how it works.

← the camera


I asked the application of this 360° immersive technology. The presenter told me that people can stream live concert and share with their friends who live far away. Or the doctors can give remote instructions to the construction team, what they want their hospital to look like, with the first-hand experience offered by the technique.

Such 360° immersive technology can be a huge hit in the travel market.



presentation on travel and VR

One presenter depicted the future work of a travel agent, who shows the different spots to the potential customers for more informed selection. With advanced haptic technology, people can feel the sand for themselves and see whether they are excited to go. A big market is blooming for VR, but the emphasis on marketability also indicates the fiscal anxiety of developers.


#360° immersive

Attraction 2: Doodle Ematic 


the Rule of Doodlematic

This is my favorite! Easy and magical. You can draw your game and take a picture. The software will automatically codify the drawings, and turn the still image to a playable game. Green stands for a player, Black for solid ground, Red for hazard, Purple for things that drop, Blue for the goal.

Here is my work. →


My game


If you can tell from the embarrassing drawing, there are two players (someone with a piglet), climbing up the hill to get the trophy filled with fodder. They have to jump over the red chef, the hazard of my game. Unfortunately, when I was playing it with a single hand (another hand was holding the camera), the piglet failed to pass the chief.

It is hard to make a winnable game! An elementary art teacher sitting next to me designed a sophisticated adventure of beating-dragon-and-save-the-princess, however, the channel was too small for the player to pass. Luckily the software has an option to “shrink” the icons, and even add rotation to the images.

By encouraging learning from the practice, the game displays its potential in education. In another type of game it offers, “touch to win”, a math teacher can encourage students to design 2+7=? quizzes for each other. An art teacher can inspire students to design a game based on a certain story, and level it up by letting students refine the game in photoshop.

Doodlematic renders codifying accessible to everyone, making it an easy tool for creative work of all ages. It breaks the wall between the paper and the screen, the material and the digital, the insider and the amateur. Fascinating!

#Education, Entertainment

IMG_6064.JPGAttraction 3: Small Wonders

Visitors in the festival recommend me to experience the delicate art in a nutshell. In this collaborative project between media lab, school, and artists, the intricate carvings are scanned through micro-CT (the same technology for scanning cancer), reproduced and dissected for appreciation and research.

With my “helmet” on, I could see the whole structure of the art piece and walk past several layers to focus on one figure. When I pointed the controller at the figure, a label floated beside the figure, saying it was a man in the hell. It is said that some hidden figures are seen by people for the first time.

I could also hear the introduction of the work from my headphone, accompanied with a religious chant, transporting myself to a tranquil museum setting.

The only unsatisfactory aspect is the visual texture. The model does not look wood, but cement covered with a thin layer of snow, glowing in red. The presenter explains that wood would seem even worse in this case, and the present material aims to show the contour of each carving. ↓



a snapshot of Small Wonders


#Museum, micro-CT

My Favorite: Les Trois Graces (Jean Francois Malouin, Canada)

It was such a creepy experience! In this art installation, I could move the arms and head of three nudes. When I moved one limb, other parts of that body flowed with it, gravitated to the force. If the limb touched other bodies, those passively responded to the force like an exhausted professional model, who let you manipulate and act according to your wish albeit in a slow manner. The nudes had a blank look. Their mouths opened a little sometimes when I dragged their arms to change the body positions.

Occasionally, I lost one hand in the virtual world, only to find it seized by a nude. I had to bring it back with my other hand. However, it was not a technical glitch. I finally lost both of my hands. As I looked back, I seemed to see an ironical face and a sense of anger in their blank look, their unstable body flowing towards me. I do not know if I project my own fear and guilt on their faces, or the artist changed their facial expression in a subtle way.

a Vimeo video from the artist

I felt relieved when the assistants helped me remove the headset and reorient myself in the real world. I could not forget the passive aggression on their faces and my sense of helplessness when I was exposed to the inescapable virtual world without a hand to manipulate or at least defend myself; it was an immersive nightmare. Maybe some ethical issue is involved, but I appreciate the controversial nature of art.

In retrospect, Les Trois Grâces became my favorite piece. I could not have reflected deeply on the power relationship between the viewer and the subject, if I did not have the embedded experience. The installation subverts the active and the passive and gives us an opportunity to examine our desire and fear.



The label


The label of Les Trois Graces also highlighted the issue of boundaries, how the boundaries between the self and other could collapse. The three nude women influence the shape of each other, and we, the manipulators are deprived of our virtual hands by the women in the virtual world. Even though we still have our physical hands, the terror of being out of control is the same. In this sense, the virtual evades the real.

The artist of Les Trois Graces makes good use of the specific media of VR. JF Maloulin transforms the classical 2-D painting of the static, harmless beauties into the creepy 3-D art installation of movable, threatening modern nudes. Their deliberate passiveness and the innocent seizure of my hands make it a thought-provoking piece on the image of nude women. Manipulated as the beautiful women are, they can turn alive whenever you look into their blank eyes and scare ourselves with our own psychological projections.


The classic: Raphaël’s Les Trois Grâces. →

#Art installation, women, body


Sept. 18th 2017