What is Gesture-Based Computing?

Thanks in part to the Nintendo Wii, the Apple iPhone and the iPad, many people now have some immediate experience with gesture-based computing as a means for interacting with a computer. The proliferation of games and devices that incorporate easy and intuitive gestural interactions will certainly continue, bringing with it a new era of user interface design that moves well beyond the keyboard and mouse. While the full realization of the potential of gesture-based computing remains several years away, especially in education, its significance cannot be underestimated, especially for a new generation of students accustomed to touching, tapping, swiping, jumping, and moving as a means of engaging with information.

It’s almost a cliché to say it, but the first exposure to gesture-based computing for many people may have occurred over a decade ago when they saw Tom Cruise in Minority Report swatting information around in front of him by swinging his arms. The fact that John Underkoffler, who designed the movie’s fictional interface, presented a non-fiction version of it, called the G-Speak, in a TED Talk in 2010, fittingly asserts the growing relevance and promise of gesture-based computing. The G-Speak tracks hand movements and allows users to manipulate 3D objects in space. This device, as well as SixthSense, which was developed by Pranav Mistry while at the MIT Media Lab and uses visual markers and gesture recognition to allow interaction with real-time information, has ignited the cultural imagination regarding the implications for gesture-based computing. This imagination is further fueled by the Kinect system for the Xbox, which continues to explore the potential of human movement in gaming. In short, gesture-based computing is moving from fictional fantasy to lived experience.

INSTRUCTIONS: Enter your responses to the questions below. This is most easily done by moving your cursor to the end of the last item and pressing RETURN to create a new bullet point. Please include URLs whenever you can (full URLs will automatically be turned into hyperlinks; please type them out rather than using the linking tools in the toolbar).

Please "sign" your contributions by marking with the code of 4 tildes (~) in a row so that we can follow up with you if we need additional information or leads to examples- this produces a signature when the page is updated, like this: - Sam Sam Jul 21, 2011

(1) How might this technology be relevant to the museums you know best?

  • - sheila.carey sheila.carey Aug 26, 2011 The most obvious application of this technology in museums is in multi-touch tables. These are especially attractive to young people, as seen by how young users enthusiastically use tablet computers.
  • The true power of gesture-based computing is that it frees users from being connected to a device. I'd argue that multitouch is a bastard child of real gestural interfaces, because you have to be in contact with the screen surface. The Kinect offers the promise of being able to interact without any physical contact. - ed.rodley ed.rodley Aug 30, 2011 - lorna.obrien lorna.obrien Aug 30, 2011

(2) What themes are missing from the above description that you think are important?

  • - sheila.carey sheila.carey Aug 26, 2011 I think an important aspect of this technology is the fact that it can be used by several users making the usage social, particularly in the case of multitouch tables, but even other gesture-based technologies often attract other users. I saw this at first hand at a conference where one person was using Kinect in the exhibit hall and a few others were gathered around watching.
  • The social aspect *is* a great feature. I'd add the whole body interaction potential. Running, jumping, dancing, could all be control inputs for future interactives. The idea of someone sitting/standing motionless in front of a display feels pretty staid in comparison. - ed.rodley ed.rodley Aug 30, 2011
  • Museums working with artists already working in this area, to develop these ideas and the technology and experiment. - lorna.obrien lorna.obrien Aug 30, 2011
  • A hacked Kinect in a museum in a space where there are crowds brings up interesting challenges. Technical limitations necessitate separating out the "social" experience to one or two people for the most part—what does that mean in terms of a group experience? - allegra.burnette allegra.burnette Aug 31, 2011

(3) What do you see as the potential impact of this technology on education and interpretation in museums?

  • Integrating kinesthetic learning into the more sedate kinds of learning typically done in museums. - ed.rodley ed.rodley Aug 30, 2011
  • Interaction and personal kinetics can be an effective tool for exploring scientific and engineering concepts and designs; years ago a experienced a cave environment which allowed you to poke at a tornado and see what happens. - len.steinbach len.steinbach Sep 1, 2011
  • Another perspective here.

(4) Do you have or know of a project working in this area?

Please share information about related projects in our Horizon Project sharing form.