Humans experience the world through our bodies. We form understandings for how to interact with environments through bodily senses. The advent of new low-cost, low-power sensing technologies create the potential for new modes of interaction and communication. New interface ecosystems can sense, recognize, respond to, and represent nuances in our environments and in our bodies. The Interface Ecology Lab is emphasizing processes of human expression, ideation, and social interaction as we develop sensory interfaces that involve embodied awareness of the human body and the physical world.

body-based diagramming

We are investigating how body-based interfaces support creativity and expression in design. Design processes are supported by embodied representations, including gestures, tangibles, and diagrams, which have been found to help people think. A diagram is a design thinking tool that enables and stimulates imagination, facilitating conceptualization. Diagrams mediate exploration of relationships between concepts, using ambiguous visual representations to foster varied, flexible interpretations. We use a form of diagramming called, information composition.

We are developing a new creativity support environment, Body-based IdeaMÂCHÉ, that enables designers to express, collect, organize, and reflect upon ideas using pen + touch interaction. The kinematic chain model for bimanual interaction serves as a basis for the design of new gestural interaction techniques. In the kinematic chain model, interactions with the non-preferred hand function as a frame of reference for actions with the preferred hand. For example, when drawing on paper, the non-preferred hand positions the paper for drawing with a pencil in the preferred hand. We seek to support expression and ideation by enabling designers to fluidly switch between and manipulate parameters of diagram transformations with their hands.

cross-surface interaction

We are developing new methods for cross-surface interaction, that is interaction across multi-touch surfaces. Embodied cognition is key. We inform our designs through the use of culturally based design, a method that draws from the embodied experiences of activities embedded in culture. By mimicking these well established activities, interactions activate people's embodied mental models. This makes the interactions intuitive and familiar, as they draw from prior experiences. Our culturally based designs draw a variety of contexts, ranging from card playing to sharing and exchanging photographs.

Our long term objective is to give people interactive experiences in which embodied gestures performed by the human hand are mapped to actions in ways that are natural, meaningful and intuitive.


ZeroTouch is a high performance multi-point sensor that detects visual hulls, within a plane. Applications include free-air interaction, person tracking in rooms, automobile windshields, hover sensing (e.g., over capacitive multi-touch), and multi-touch.
EcoTUIODriver is our own open source TUIO to Windows 7/8 Touch bridge. It uses virtual touch drivers and TUIO bridge services to convert TUIO touch messages to Windows touch events. EcoTUIODriver supports TUIO bridging on up to 5 monitors simultaneously. The EcoTUIODriver source can be downloaded from github or you can pull down the installer here.

publications and exhibitions

Webb, A.M., Pen + Touch Diagramming to Stimulate Design Ideation, to appear in Proc. Creativity and Cognition 2015.
Lupfer, N., Hamiliton, W. Webb, A.M., Linder, R., Edmonds, E., and Kerne, A., The Art.CHI Gallery: An Embodied Iterative Curation Experience, Interactivity Exhibit, Proc CHI 2015 EA.
Webb, A.M. and Kerne, A., Embodying Diagramming through Pen + Touch Gestures, CHI 2014 Workshop: Gesture Interaction Design: Communication and Cognition, Toronto, Canada.
Fei, S., Webb, A.M., Kerne, A., Qu, Y., and Jain, A., Peripheral Array of Tangible NFC Tags: Positioning Portals for Embodied Trans-Surface Interaction, Proc. Interactive Tabletops and Surfaces 2013, [29%]. [video]
Hamilton, W., Kerne, A., and Robbins, T., High-Performance Pen + Touch Modality Interactions: A Real-Time Strategy Game eSports Context, Proc. UIST 2012, Cambridge, MA, USA, OCT 2012.
Kerne, A., Hamilton, W., Toups, Z. Culturally Based Design: Embodying Trans-Surface Interaction in Rummy Proc CSCW 2012, in press [top 9%].
Damaraju, S., Kerne, A., Comparing Multi-Touch Interaction Techniques for Manipulation of an Abstract Parameter Space, accepted to Proc. International Conference on Multimodal Interaction (ICMI) 2011.
Moeller, J. and Kerne, A., ZeroTouch: A Zero-Thickness Optical Multi-Touch Force Field, CHI Interactivity -- Extended Abstracts of the 29th International Conference on Human factors in computing systems, May 7-12, 2011, Vancouver, BC, CA. [video]
Moeller, J., Lupfer, N., Hamilton, B., Lin, H., Kerne, A., intangibleCanvas: Free-Air Finger Painting on a Projected Canvas , CHI Works-in-Progress -- Extended Abstracts of the 29th International Conference on Human factors in computing systems, May 7-12, 2011, Vancouver, BC, CA.
Moeller, J. and Kerne, A., Scanning FTIR: Unobtrusive Multi-Touch Sensing through Waveguide Transmissivity Imaging, Proceedings of the 4th International Conference on Tangible, Embedded and Embodied Interaction, Cambridge, MA, Janurary 25-27 2010.
Damaraju, S., Kerne, A. Multitouch Gesture Learning and Recognition System. , Extended Abstracts of IEEE Workshop on Tabletops and Interactive Surfaces, Amsterdam, Netherlands, 1-3 October 2008.
Webb, A., Kerne, A., Koh, E., Joshi, P., Park, Y., Graeber, R., Choreographic Buttons: Promoting Social Interaction through Human Movement and Clear Affordances, Proc ACM Multimedia 2006, 451-460 [16%].
Toups, Z., Overby, K., Kerne, A., Graeber, R., Cooper, T., Alley, E., Censor Chair, ACM SIGCHI Intl Conf on Advances in Computer Entertainment, June 2006, Hollywood.

Alley, E. Cooper, T., Graeber, R., Kerne, A., Overby, K., Toups, Z., Censor Chair: Exploring Censorship and Social Presence through Psychophysiological Sensing, Proc ACM Multimedia 2005, 922-929.
Stenner, J., Kerne, A., Williams, Y., Playas: Homeland Mirage, ISEA/Zero One, Aug 2006: San Jose.

Stenner, J., Kerne, A., Williams, Y., Playas: Homeland Mirage, Proc ACM Multimedia 2005, 1057-1058.
Schiphorst, T., Kozel, S., Andersen, K., Jaffe, N., Mah, S., Kerne, A., Lovell, R., Tolmie, J., whisper. Dutch Electronic Arts Festival (DEAF) 2003, Rotterdam, The Netherlands.