body-based interaction

We humans experience the world through our bodies. We form understandings for how to interact with environments through bodily senses. The advent of new low-cost, low-power sensing technologies create the potential for new modes of interaction and communication. New interface ecosystems can sense, recognize, respond to, and represent nuances in our environments and in our bodies. The Interface Ecology Lab is emphasizing processes of human expression, ideation, and social interaction as we develop sensory interfaces that involve embodied awareness of the human body and the physical world.

body-based diagramming

We are investigating how body-based interfaces support creativity, expression, and design. We are particularily interested in the context of ideation in design. Design processes are supported by embodied representations, including gestures, tangibles, and diagrams, which have been found to help people think.

A diagram is a design thinking tool that enables and stimulates imagination, facilitating conceptualization. Diagrams mediate exploration of relationships between concepts, using ambiguous visual representations to foster varied, flexible interpretations. We use a form of diagramming called, free-form curation.

We are developing a new creativity support environment, Body-based IdeaMÂCHÉ, that enables designers to express, collect, organize, and reflect upon ideas using pen + touch interaction. The kinematic chain model for bimanual interaction serves as a basis for the design of new gestural interaction techniques. In the kinematic chain model, interactions with the non-preferred hand function as a frame of reference for actions with the preferred hand. For example, when drawing on paper, the non-preferred hand positions the paper for drawing with a pencil in the preferred hand. We seek to support expression and ideation by enabling designers to fluidly switch between and manipulate parameters of diagram transformations with their hands.

collaborative design ideation

We are investigating landscape architecture students' individual and collaborative ideation practices and experiences in design studio education. We have won a Surface Hub for Research grant from Microsoft Research, for developing new and collaborative pen+touch interaction techniques and integrating them into a design environment in order to show how Surface Hub can provide a basis for advancing design ideation.

To effectively build interactive environments that support the needs of visual designers, we need to understand designers' current processes of printing, drawing, and pinning media as they develop ideas, in project development, and during critique sessions in studios. We are looking at the media they use, the interactions their hands and bodies become involved in with that media, and how they interweave individual and collaborative work to create design products.

Our larger goal is to support individual and collaborative ideation in design studios using interactive surfaces. We will design and develop interaction techniques, integrated with software infrastructure, to support multiple users collaborating synchronously on Microsoft Surface Hub.

We have also worked on innovative hardware designs to support body-based interaction, such as the ZeroTouch sensor, which detects visual hulls, in a plane, in free-air or in proxmity to a surface.

publications and exhibitions

Webb, A. M., Fowler, H., Kerne, A., Newman, G., Kim, J.-H., Mackay, W. E. Interstices: Sustained Spatial Relationships between Hands and Surfaces Reveal Anticipated Action, Proc. of ACM CHI Conference on Human Factors in Computing Systems 2019, Glasgow, UK.
Webb, A.M., Kerne, A., Brown, Z., Kim, J.H., Kellogg, E., LayerFish: Bimanual Layering with a Fisheye In-Place, Proc. ACM International Conference on Interactive Surfaces and Spaces 2016, Niagara Falls, Canada. [28%].
Webb, A.M., Pahud, M., Hinckley, K., Buxton, B., Wearables as Context for Guiard-abiding Bimanual Touch, Proc. ACM Symposium on User Interface Software and Technology 2016, Tokyo, Japan. [20.6%].
Webb, A.M., Pen + Touch Diagramming to Stimulate Design Ideation, Proc. ACM Creativity and Cognition 2015, Graduate Symposium, 331-332.
Lupfer, N., Hamiliton, W. Webb, A.M., Linder, R., Edmonds, E., and Kerne, A., The Art.CHI Gallery: An Embodied Iterative Curation Experience, Interactivity Exhibit, Proc CHI 2015 EA.
Webb, A.M. and Kerne, A., Embodying Diagramming through Pen + Touch Gestures, CHI 2014 Workshop: Gesture Interaction Design: Communication and Cognition, Toronto, Canada.
Fei, S., Webb, A.M., Kerne, A., Qu, Y., and Jain, A., Peripheral Array of Tangible NFC Tags: Positioning Portals for Embodied Trans-Surface Interaction, Proc. Interactive Tabletops and Surfaces 2013, [29%]. [video]
Hamilton, W., Kerne, A., and Robbins, T., High-Performance Pen + Touch Modality Interactions: A Real-Time Strategy Game eSports Context, Proc. UIST 2012, Cambridge, MA, USA, OCT 2012.
Kerne, A., Hamilton, W., Toups Dugas, P. O. Culturally Based Design: Embodying Trans-Surface Interaction in Rummy Proc CSCW 2012, in press [top 9%].
Damaraju, S., Kerne, A., Comparing Multi-Touch Interaction Techniques for Manipulation of an Abstract Parameter Space, accepted to Proc. International Conference on Multimodal Interaction (ICMI) 2011.
Moeller, J. and Kerne, A., ZeroTouch: A Zero-Thickness Optical Multi-Touch Force Field, CHI Interactivity -- Extended Abstracts of the 29th International Conference on Human factors in computing systems, May 7-12, 2011, Vancouver, BC, CA. [video]
Moeller, J., Lupfer, N., Hamilton, B., Lin, H., Kerne, A., intangibleCanvas: Free-Air Finger Painting on a Projected Canvas , CHI Works-in-Progress -- Extended Abstracts of the 29th International Conference on Human factors in computing systems, May 7-12, 2011, Vancouver, BC, CA.
Moeller, J. and Kerne, A., Scanning FTIR: Unobtrusive Multi-Touch Sensing through Waveguide Transmissivity Imaging, Proceedings of the 4th International Conference on Tangible, Embedded and Embodied Interaction, Cambridge, MA, Janurary 25-27 2010.
Damaraju, S., Kerne, A. Multitouch Gesture Learning and Recognition System. , Extended Abstracts of IEEE Workshop on Tabletops and Interactive Surfaces, Amsterdam, Netherlands, 1-3 October 2008.
Webb, A., Kerne, A., Koh, E., Joshi, P., Park, Y., Graeber, R., Choreographic Buttons: Promoting Social Interaction through Human Movement and Clear Affordances, Proc ACM Multimedia 2006, 451-460 [16%].
Toups Dugas, P. O., Overby, K., Kerne, A., Graeber, R., Cooper, T., Alley, E., Censor Chair, ACM SIGCHI Intl Conf on Advances in Computer Entertainment, June 2006, Hollywood.

Alley, E. Cooper, T., Graeber, R., Kerne, A., Overby, K., Toups Dugas, P. O., Censor Chair: Exploring Censorship and Social Presence through Psychophysiological Sensing, Proc ACM Multimedia 2005, 922-929.
Stenner, J., Kerne, A., Williams, Y., Playas: Homeland Mirage, ISEA/Zero One, Aug 2006: San Jose.

Stenner, J., Kerne, A., Williams, Y., Playas: Homeland Mirage, Proc ACM Multimedia 2005, 1057-1058.
Schiphorst, T., Kozel, S., Andersen, K., Jaffe, N., Mah, S., Kerne, A., Lovell, R., Tolmie, J., whisper. Dutch Electronic Arts Festival (DEAF) 2003, Rotterdam, The Netherlands.