Game Technology Inspires Creativity in Zurich

ETH Zurich researchers in game technology collaborate with Disney Research Zurich to bring a bit of sparkle into the lives of children and adults. Reminding us of the intrinsic need for play, they provide novel digital technologies to enhance traditional experiences in a new realm called, “augmented creativity.” When artist Edvard Munch, experimenting with different media, produced four versions of The Scream, he inspired innumerable creative adaptations, making the composition among the most well-known in the world. Imagine an app that enables the “inner child” in all of us to create real-time adaptations of portraits – changing the size of the subject’s eyes, altering the hairstyle, reshaping the nose or mouth, and, with just the tap of a finger, transforming the texture and color of a painting. Bob Sumner, who heads up ETH Zurich’s Game Technology Center, finds that “some of the most satisfying endeavors in life happen when you think about putting children first.” This is exactly what Sumner and his research team did when they developed their latest prototype: an app that uses digital technology to make the experience of visiting an art museum more playful and interesting – perhaps even magical. Fun Is the Future We have all seen it: technology so intuitive that even toddlers can figure out how to swipe a mobile phone screen in search of their favorite game app. In Zurich, and other cities around the world, teenagers favor communicating with their friends via Snapchat rather than actually speaking face-to-face. Commuter trains are filled with silent adults – heads down, eyes fixed on small displays. When physical interaction with the environment is such an important part of the human experience, we have to ask, “As passive consumers of digital content, are we losing touch with reality?” Researchers at ETH Zurich – the Swiss Federal Institute of Technology Zurich – in collaboration with Disney Research, envision a magical world in which augmented reality enhances creative play in a novel concept they refer to as augmented creativity. “We are living in an interesting time,” says Sumner. “Like the development of the smartphone, I believe augmented creativity is the next era.” Rather than passively consuming digital content, Sumner conceptualizes augmented reality as a vehicle for providing a magical overlay to enhance creative play. For Sumner, the potential for gaming extends far beyond entertainment: “Fun is the future,” he says. Gaming for Insight For more than a decade, Swiss game developers have been sweeping up awards at international gaming conferences and garnering the attention of Swiss Federal Councilors, who called for an initiative to support game development in Switzerland. As a result, the Game Technology Center opened its doors in 2015. “Gaming really has the potential to help us study human behavior and apply this knowledge to some of the world’s most difficult problems,” says Fabio Zünd, Managing Director of the Game Technology Center. Take the massive multiplayer game Gnome Trader, for example, where players buy and sell food from gnomes who magically appear in mailboxes all over Zurich in a virtual trading economy driven by supply and demand. “It’s more than just a game,” says Zünd. “Data gathered from player’s choices within the virtual economy provide us with insight into the economics of real-world financial trading.” The team also works with Minecraft, a game that can be played on multiple platforms, where young players not only build their own virtual communities, but they also administer the game’s servers. As server admins they learn how to exercise decision-making skills that foster positive behaviors in a virtual environment, while at the same time affording researchers insight into how society shapes real-world communities. The Brain on Play Gamification involves more than just virtual environments and a motivation to win. Studies on reinforcement and learning identify how our neurons work together to make decisions – revealing that physiological sensation, kinesthetic adaptation, and risk calculation light up the brain at play as the neurotransmitter dopamine signals pathways that reward behavior. Stéphane Magnenat confirms, “Humans have an intrinsic curiosity for learning.” Magnenat, a Senior Researcher who works with Sumner and Zünd, explores algorithms to build games that optimize learning in a magical state of flow or focused concentration – much like the magic we experience when listening to inspiring music or engrossed in playing a video game. According to Magnenat, the “flow point” – the equilibrium between the level of challenge and personal skill – is the point where the most efficient learning takes place. Brain Runners, a specially developed brain–computer interface (BCI) game, recently demonstrated the concept of optimized flow at the world’s first Cybathlon, held in Zurich. The BCI game, developed by researchers at the ETH Zurich Sensory Motor Systems Lab in collaboration with René Bauer and Ulrich Goetz, Zurich University of the Arts (ZHdK), enabled competitors with spinal cord injuries to control the movement of an avatar in a real-time video game. Using only concentrated thoughts, electrodes connected to signal amplifiers measured players’ brain signals and transmitted this information to their avatars, propelling them along the racetrack alongside other competitors. The Magic of Algorithms While controlling an avatar with your thoughts may seem magical, wait until you learn about the technologies that make 2D coloring book characters come to life. Zünd revealed that the research team has developed two unique technologies in collaboration with Disney Research Zurich to enable their augmented-reality coloring book. The first was a texture transfer process that generates a real-time transfer from a flat 2D surface to a 3D model by creating a lookup map which copies in existing pixels from a drawing. It allows coloring enthusiasts to see a full 3D view of their characters and alter the view with new colors and textures in real time. The team developed the new algorithm specifically for artists to generate and map a figure automatically. Second, since traditional tracking algorithms for augmented reality only work with flat image markers, the team created a deformable surface-tracking algorithm that accommodates the curvature of a page bound within a book. There is great potential for this algorithm to impact further applications in the visual computing field. Demonstrating the coloring book, filled with octopuses, sumo wrestlers, and elephants, Zünd selected a two-dimensional outline of a character that the team named Hannah, and with a felt-tip marker, he added texture and color to her dress. When Zünd held the tablet over the coloring book page, Hannah magically came alive in full 3D form. She looked down at the new color on her dress – and with an animated expression that can only be described as pure delight, she began to dance and spin around. Music to My Ears Exploring more than just visual experiences and decision-making processes, researchers at the Game Technology Center have also delved into auditory senses, creating an app that brings both coloring book and adapted portrait caricatures together to create music. “The important thing is that you don’t have to be a musician or know anything about musical theory, you can combine instruments or music styles arbitrarily and, in a playful way, explore a song,” says Zünd. When a smartphone or tablet’s camera is pointed at a small instrument card, it begins to play a song. Users can put a whole band together by adding other cards – a calypso drum, for example – to experiment with or enhance a musical style. Markers on the cards, on the coloring book, and on the portraits, represent an instrument or even a voice, so when Hannah joined the band, we heard a woman’s voice chime in. Just like at a live concert, the volume increases or decreases depending on how far away your device is from the band. The interactive nature of augmented creativity allows children to exploit the intuitive interaction capabilities of this prototype and explore the world of music. Augmented Creativity In 2050? For thousands of years, humans have been developing tools to express themselves creatively. The Lascaux cave paintings, estimated to be more than 17,000 years old, are some of the earliest forms of creative human expression and also among the earliest images to be shared digitally when the world began accessing the Internet. “More recently, since the advent of computers we have created digital tools that seem like magic compared to what was possible only 15 years ago,” says Sumner. So, what will augmented creativity mean in 2050? Will life and learning be all fun and games? More importantly, will technology still seem magical? Let’s hope so.