Scott R. Klemmer, Björn Hartmann, Leila Takayama

How Bodies Matter: Five Themes for Interaction Design

Our physical bodies play a central role in shaping human experience in the world, understanding of the world, and interactions in the world.
This paper draws on theories of embodiment—from psychology, sociology, and philosophy—synthesizing five themes we believe are particularly salient for interaction design: thinking through doing, performance, visibility, risk, and thick practice.
We introduce aspects of human embodied engagement in the world with the goal of inspiring new interaction design approaches and evaluations that better integrate the physical and computational worlds.

The richness of human knowledge and understanding is far deeper than the set of knowledge we can produce a symbolic account of. As Polanyi puts it, “we know more than we can tell” [56, p. 4]. To elucidate this assertion, consider riding a bicycle: one is simultaneously navigating, balancing, steering, and pedaling; yet it is not possible for bicyclists to articulate all of the nuances of an activity that they successfully perform. Perhaps the most remarkable aspect of this is that riding a bicycle is just one of thousands of activities that our bodies can do.

Contrast the richness, subtlety, and coordination of tasks at several levels of concern that bicycling offers with the graphical user interface that we use today. One of the most sweeping — and unintended — transformations that the desktop computing paradigm has brought about is the extent to which the physical performance of work has homogenized. For certain activities, such as writing this paper, the keyboard interaction paradigm appropriately leverages our bimanual dexterity. But, with a keyboard and mouse interface, the use of our bodies for writing a paper is the same as for editing photographs. And playing music. And communicating with friends and family. And anything else that one might want computation for.

This paper presents five themes that we believe are particularly salient for designing and evaluating interactive systems. The first, thinking through doing, describes how thought (mind) and action (body) are deeply integrated and how they co-produce learning and reasoning. The second, performance, describes the rich actions our bodies are capable of, and how physical action can be both faster and more nuanced than symbolic cognition. The first two themes primarily address inspanidual corporeality; the next two are primarily concerned with the social affordances. Visibility describes the role of artifacts in collaboration and cooperation. Risk explores how the uncertainty and risk of physical co-presence shapes interpersonal and human-computer interactions. The final theme, thickness of practice, suggests that because the pursuit of digital verisimilitude is more difficult than it might seem, embodied interaction is a more prudent path.

To be sure, this paper is not the first to posit that richer interaction paradigms are possible. What we hope to contribute to this discussion is a synthesis of theoretical and empirical work— drawn from psychology, sociology, and philosophy — that provides insight for both ideation and evaluation of interaction design that integrates the physical and computational worlds.

Thinking Through Doing

The evidence supports … an evolutionary view of human reason, in which reason uses and grows out of bodily capacities.
George Lakoff and Mark Johnson [38]
Direct physical interaction with the world is a key constituting factor of cognitive development during childhood. The importance of physical action as an active component of our cognition extends beyond early developmental stages. This section reviews the connection between thinking and doing as uncovered by educational theorists, gesture researchers, and cognitive scientists. Cumulatively, their empirical work point towards a common nexus of perception, cognition, and action. Unlike theories of information processing and human cognition that focus primarily on thought as something that only happens in the head, theories and research of embodied cognition regard bodily activity as being essential to understanding human cognition [54]. These theories have important implications for designing interactive systems.

Learning through doing

Being able to move around in the world and interact with pieces of the world enables learning in ways that reading books and listening to words do not. Jean Piaget [55] posited that cognitive structuring requires both physical and mental activity. Particularly for infants in the sensorimotor stage of development, physical interaction in the world facilitates cognitive development. For example, locomotor experience increases spatial cognitive abilities in infants, such as understanding the concept of object permanence (i.e., that objects continue to exist even when they are not visible) [33]. In this very basic sense, humans learn about the world and its properties by interacting within it.

Pedagogies such as the Montessori method [48] employ bodily engagement with physical objects to facilitate active learning (see Figure 1). The use of tangible manipulatives has been shown to improve elementary school student understanding of mathematical concepts. Such educational methods nicely leverage the bodily basis of mathematical concepts for learning [39]. Physical reasoning can also play an important role in professional and higher education. An example is MIT’s Illuminating Light interface [69], which enables users to combine rapid creation of light reflection simulations by moving tangible objects on a tabletop surface (see Figure 2).

Figure 1 With Montessori blocks, concepts such as distinct numbers are represented through distinct physical sizes, shapes, and colors.

The Role of Gesture

Just as moving about in the world helps infants to learn about the physics of the world and consequences of actions, gesture plays a role in pre-linguistic communication for babies [31] as well as aids cognition and fully linguistic communication for adults. From studies of gesturing in face-to-face interactions, we know that people use gesture to conceptually plan speech production [2] and to communicate thoughts that are not easily verbalized [12].

While gesturing is normally thought of as having a purely communicative function, many studies suggest that gesture also plays a helpful role for the speaker: gesturing has been shown to lighten cognitive load for both adults and children [22]; even congenitally blind children gesture [32]. A less obvious point is that systems that constrain gestural abilities (e.g., having your hands stuck on a keyboard) are likely to hinder the user’s thinking and communication. Consider telephones: we have seen shifts from corded phones to cordless phones to mobile phones and mobile phone head-sets. Experimental studies demonstrated that more physical mobility increased user creativity and disclosure of personal information in microphone use [70]. These results suggest that less constraining interaction styles are likely to help users think and communicate.

Epistemic Action

Body engagement with physical and virtual environments constitutes another important aspect of cognitive work. We are familiar with people leaving keys or notes for them-selves in strategic locations to serve as later reminders.

Distinguishing pragmatic action—manipulating artifacts to directly accomplish a task—from epistemic action—manipulating artifacts to better understand the task’s context [34]—provides interpretation for such behavior. One might expect that the predominant task in Tetris is piece movement with the pragmatic effect of aligning the piece with the optimal available space. However, contrary to intuitions, the proportion of shape rotations later undone by backtracking increases (not decreases) with increasing Tetris-playing skill levels: players manipulate pieces to understand how different options would work [42].

These epistemic actions are one of many helpful ways in which a user’s environment may be appropriated to facilitate mental work [26, 51]. Analogous examples include moving lettered tiles into various arrangements for playing Scrabble [43] and using external representations for numeric tasks [78].

Figure 2 The tangible Illuminating Light workbench lets students learn about optical systems by designing them.

Thinking through prototyping

Iterative design practices provide another perspective on the importance of concrete, artifact-centered action in the world to aid thought. Reflective practice, the framing and evaluation of a design challenge by working it through, rather than just thinking it through, points out that physical action and cognition are interconnected [58]. Successful product designs result from a series of “conversations with materials.” Here, the “conversations” are interactions between the designer and the design medium — sketching on paper, shaping clay, building with foam core [59] (see Figure 3). The epistemic production of concrete prototypes provides the crucial element of surprise, unexpected realizations that the designer could not have arrived at without producing a concrete manifestation of her ideas.

The backtalk that artifacts provide helps uncover problems or generate suggestions for new designs. Prototypes thus become the “essential medium for information, interaction, integration, and collaboration” [60]. Beyond backtalk, creating intermediate tangible artifacts allows for expression of tacit knowledge. It also facilitates communication within a design team, with clients, or users, by providing a concrete anchor around which discussion can occur. Prototypes then present us with a different kind of embodiment: they themselves embody design ideas or specifications, render them concrete and, in doing so, inform the de-signer’s thinking (see Figure 3).

Our own fieldwork with design professionals underscores the centrality of thinking through prototyping. One architect estimated the number of tangible prototypes made for a building to be between 200 and 300 in his own practice. A design director stressed the importance of generating a wide range of different tangible and virtual prototypes. Because different styles and fidelities of artifacts yield different perspectives, externalizing ideas through a variety of prototypes affords a richer understanding of a design.

As a counterpoint, Schrage [60] cautions us against placing too much emphasis on the physicality of prototypes. In his view, the reliance of Detroit car manufacturers on high-fidelity clay models was a factor in their loss of market share to foreign firms who used more rapid software prototyping strategies. Thus concrete tangibility is no panacea, but an important ingredient of a successful prototyping practice.

On Representation

The representation of a task can radically affect our reason-ing abilities and performance. For example, the game of tic-tac-toe (opposing players mark X’s and O’s in a 3 × 3 grid) can be equivalently represented as a game of drawing numbered cards with the goal of selecting three that sum to 15 [51, 64]. From a computational perspective, these two problems are isomorphic. However, the tic-tac-toe representation is significantly easier to work with because the representational form of the problem makes visible the most relevant constraints implicit in the problem. As Simon writes, in mathematics, “solving a problem simply means representing a problem so as to make the solution transparent” [64, p. 153].

Tangibility offers both direct familiarity and a set of common metaphors to leverage in interaction. But some map-pings between the physical and the virtual work, while others do not. An example of an interactive system that successfully leverages our familiarity with everyday physics is the automotive drive-by-wire system that uses force feedback to alter driver perceptions of the road [68]. It discourages lane drifting by exerting forces on the wheel such that the driver has the impression that the driving lane is shaped like a shallow bathtub.

Perhaps the most common stated purpose of tangibility is that these interfaces provide “natural” mappings [14] and leverage our familiarity with the real world [15], e.g., virtual objects are positioned in virtual space by moving physical handles in physical space. These identifications are only possible for a restricted domain of systems so how does one interact with symbolic information that does not have an obvious physical equivalent? In a data- or technology-centric view of tangible interaction, the question of representation is equivalent to deciding on a reification strategy that turns bits into atoms. A body-centered view looks at how the actions that we perform with a system contribute to task transparency.

Figure 3 A paper sketch, physical mock-up, and final prototype, showing how the interface of SnuzieQ, an alarm clock, evolved through prototyping.


When compared to other human operated machinery (such as the automobile), today’s computer systems make extremely poor use of the potential of the human's sensory and motor systems. The controls on the average user's shower are probably better human-engineered than those of the computer on which far more time is spent.
Bill Buxton [8]
One of the most powerful human capabilities relevant to designers is the intimate incorporation of an artifact into bodily practice to the point where people perceive that artifact as an extension of themselves; they act through it rather than on it [16, 45, 56]. For example, experienced puppeteers can see through the eyes of their puppet and feel the ground through the puppet’s feet [75]. But what kinds of extensions are these interface artifacts? How do they enable or hinder thought and action? This section provides an account of the body’s ability for skillful performance. While much of the recent TUI literature has focused on “walk up and use” scenarios [28, 74] which require a low use threshold, this section describes how designing for skilled bodies can yield interfaces for expert performance. We describe the complexity and nuance of interaction that tangible artifacts can offer to bodies, especially to hands, to illustrate the benefits of rich physicality for skillful performance. Physical interfaces with dedicated (i.e., spatially multiplexed) controls and dedicated actions can leverage this skill to improve interaction speed and reliability [20].

Action-centered skills

The tacit knowledge that many physical situations afford plays an important role in expert behavior. We draw attention to the importance of tacit knowledge because computerization can, often accidentally, inhibit it. For example, Zuboff’s studies of paper plants found operators distrustful of recent computer mediation that interpreted plant conditions for them. Prior to this mediated experience, one plant operator could judge paper condition by his arm hair sensitivity to electricity in the atmosphere around a dry roller machine; another could judge pulp roll moisture content through a slap of the hand on the roll [79]. While enclosed control rooms provided physical protection from the fumes of the plant floor, the room full of computer monitors left plant operators at a loss for the rich sensory information they used to gather with their bodies. Physical tacit knowledge is an important part of professional skill.

In interaction design, calm technologies [73] like Jeremijenko’s Live Wire, which manifests the flow of Ethernet traffic through the twitch-ing of a cable suspended from a ceiling, explicitly take on the task of producing physical cues that can be tacitly understood. The Live Wire is designed for visual tacit knowledge; the next section explores manual tacit knowledge.


A natural place to start is with our hands, as they are simultaneously a means for complex expression and sensation: they allow for complicated movement but their skin also has the highest tactile acuity of our extremities. Significantly, the action and perception potentials of the hand are linked—most prehensile (grasping) actions use the hands as bidirectional modalities [7], exerting force and sensing pressure to adjust that force simultaneously. Active touch (see Figure 4) —where one manipulates the object they are investigating to control touch stimulation — is superior to passive touch in detecting shape and identity of objects [21]. In addition, many of the complex motions that we perform are bi-manual and asymmetric. Entire professions, such as surgeons, sculptors, jewelers, musicians and puppeteers rely almost exclusively on their hands as the principle organ of expression, yet such capabilities are seldom exploited in computer systems [75] (see Figure 5). Would you agree to have a doctor performing tele-surgery on you using only a mouse and keyboard?

Offering bimanual continuous input to computer systems allows users to speed up task performance, either through simultaneous action, or through maximizing efficiency of hand motion by distribution actions between two hands [9]. Tangible tokens such as Bricks [20] afford bimanual strategies without requiring them. Similarly, Brooks has developed combined haptic and visual interfaces that improve our understanding of spatial structures and forces for scientific visualization [5].

Figure 4 Gibson’s active touch shapes [21, p. 124].
Figure 5 The GUI’s mental model of a user [30].

Motor Memory

We are able to sense, store and recall our own muscular effort, body position and movement to build skill. It is this motor, or kinesthetic, memory [61] that is involved in knowing how to ride a bicycle, how to swim, how to improvise on the piano [67]. It is not available to introspection, but is reliable and robust. Traditional GUI interfaces employ the same bodily actions for a wide variety of tasks — this universality is both a strength and a weakness. It allows for control of any number of applications; however, for any given application, kinesthetic memory can only be lever-aged to a limited extent since the underlying actions are the same across applications.

Assigning dedicated actions to different functions of a user interface can take better advantage of kinesthetic memory. As Djajadiningrat et al. put it: “differentiation [in appearance and method of interaction] provides the ‘hooks’ for our perceptual-motor system to get a grip on a system’s functionality and to guide the user in his actions” [14]. Consistently dedicating physical movement to interface functions affords kinesthetic learning and memorization over prolonged use. Physical feedback can further help to distinguish commands kinesthetically.

Reflective reasoning is too slow

Beyond reliability and robustness of kinesthetic recall, speed of execution also favors bodily skill for a class of interactive systems that require tight integration of a human performer “in the loop.” Many daily actions such as driving a car or motorcycle, operating power tools, or engaging in athletic activities require complex yet rapid bodily responses for which planning through explicit cognition is simply too slow. These actions are learned skillful behavior, not reflexes, as they are voluntary and non-uniform in response. Norman termed this class of knowledge experiential cognition as opposed to reflective cognition [51], which is more flexible but requires more time.

Tangible interfaces that engage the body can leverage body-centric experiential cognition. To date, computer game controllers have been the most commercially successful example of such interfaces. Players of flight simulators increase their “grip” on the simulation using two -handed joystick plus throttle controllers; driving simulator players use foot pedals and table mounted wheels with force feedback to improve their vehicle control. The success of games and game controllers suggests that rich physical input devices may provide benefit in other domains as well.


The fact that the paper [air traffic control flight] strips are physically laid out in space and annotated directly (rather than indirectly through, for example, a keyboard) means that the activities of co-workers interacting with the strips can be perceived, providing mutual awareness for collaboration.
Abigail Sellen and Richard Harper [62]
We have discussed how increasing the richness of human performance benefits individual users; in this section, we examine how practices that are physically distinct support collaboration and coordination. The primary concern of this section is the extent to which the activities of a practice are made visible to colleagues and onlookers through the performance of the activity.

Situated Learning

How does one learns a craft or a profession? One method, as described earlier, is learning by doing. Another important method is learning by participating in a community of practice, such as the way that many trade practitioners learn (e.g., midwives, tailors, quartermasters, and butchers) [40]. We argue that an important, and rarely considered, aspect of interaction design is the way in which the interface enables this participation.

Whalen and Vinkhuyzen’s study of a call center for a copier company illustrates how workspaces can successfully support peripheral participation. At this call center, the most reliable phone operator was a veteran of eight years, but the second most reliable was a newcomer. Why? The new-comer sat across from the veteran. “…she could hear the veteran taking calls, asking questions and giving advice. And she began to do the same. She had also noticed that he had acquired a variety of pamphlets and manuals, so she began to build up her own stock” [6, p.132].

As an example of how the invisibility of work practice that the GUI has brought about inhibits peripheral participation, the first author was in a laundromat, working on his laptop. A child sat next to him while he was working, and looked at him, watching him work. After a few minutes, the child pulled from his backpack a game device with a similar clamshell form factor. The child watched to see what the author was doing, and then proceeded to copy those motions on the gaming device. With the graphical interface, there is no mechanism to be aware of the practices of experts; it all looks the same.

Figure 6 shows the Stanford Product Design loft — Barbie dolls, umbrellas, new ideas, old ideas, good ideas, and bad. These artifacts invite and ground discussion about activities in the space. Collocated, cluttered studios are hallmarks of art and design education. The studio model of education employs work practice transparency as a pedagogical technique, affording peer learning, discussion, and “constant critique of work in progress” [46]. This “technology” was introduced with the founding of the École des beaux-arts in Paris in 1819, and has endured for nearly 200 years.

Figure 6 The Stanford Product Design loft studios.
Figure 7 Butcher paper lines the wall of the Stanford meeting room.

Visibility Facilitates Coordination

In addition to supporting situated learning and peripheral participation, the production and manipulation of visible artifacts in the workplace facilitate coordination (e.g., [4, 11, 49, 62]). The visibility of a work practice manifests itself in the artifacts that the practice creates (see Figure 7). We see this in Heath and Luff’s account of paper medical records [25]. Paper medical records provide a platform for asynchronous coordination between hospital staff. They help organize work as staff leverage the many consequential properties of their colleagues’ handling of the records to gain richer insight into the history of the patient’s interaction with hospital — pencil means a note is tentative, worn means that a record has seen a lot of use, etc.

The visibility provided through collocated practice with task-specific artifacts is also successful in supporting synchronous collaboration, and can be especially useful in mission-critical systems. Mackay’s air traffic control studies [41] focused on the role of the paper flight strips that provide a hand-scale physical representation of air-planes. Her primary finding was that controllers coordinate the management of air traffic by coordinating the management of flight strips. As we are much less likely to ignore a colleague who presents a request by walking into our office than by sending an email (partially because “receipt” of the request is much more visible), Mackay found the physical act of handing a strip to have important properties not easily replicated in electronic systems. The social life of physical artifacts and their visibility facilitate distributing the cognitive work of groups (e.g., [26, 29]).

That’s what performance is about

The value we place in visibility of creative production is exemplified by live musical performance. While the music itself is more intricate and polished in studio recordings, audiences still pack concert venues because live performances permit listeners to witness the act of performance as well as co-produce the event (musician and audience respond to each other through mutual feedback). Think of the critical outrage when it became known that Milli Vanilli lip-synced. With the spread of software synthesis and sequencing, laptop performances of electronic music became common, where a lone musician sits behind a laptop, face hidden from the crowd by the LCD screen. Because performers sat motionless behind their computers (except for some mouse-clicking) the act of performance, although still taking place, was rendered invisible, and as a result audiences became both disengaged and suspicious — “How do I know the performer is not just checking his email?” As an antidote, Audiopad [53] reestablishes visibility of performance by creating a synthesis interface comprised of a projected tabletop display with several control pucks.

Verified Voting

One of the most surprising proponents of tangibility is the Verified Voting Foundation. Their assertion is that the only acceptable voting method is one that leaves a paper record. Their reason is that electronic voting machines “pose an unacceptable risk that errors or deliberate election-rigging will go undetected” [1]. The argument is not that touch-screen voting is less efficient, but that it is more difficult for one to tell when an electronic vote has been manipulated. Because tampering is made visible with physical systems, the Verified Voting Foundation suggests that they are more appropriate for catching attempted election fraud.


But where there is no risk and every commitment can be revoked without consequences, choice becomes arbitrary and meaningless.
Hubert Dreyfus [18]

Physical Action is Characterized by Risk

One’s unmediated experience of acting in the physical world is characterized by uncertainty and an awareness of corporeal vulnerability. Dreyfus [17] argues that this leads to a constant preparedness for danger and surprises, and that this readiness shapes one’s experience and interactions in the world. Individually, bodies can suffer harm if one chooses the wrong course of action (e.g., when using power tools), as the result of actions in the world cannot be un-done. Choosing an action requires commitment; carrying it out is an expression of this commitment. In social interactions, risk may not necessarily entail physical harm, but can also come from the imperative to act in the presence of others. As Watzlawick et al. note, “we cannot not communicate” — the absence of communicative effort is itself a message that is interpreted by one’s peers [71]. One cannot undo a social faux pas in face to face interactions; technology mitigates against this risk: one can delete sentences before sending them to friends over IM or email.

Risk is having to choose an action which cannot be undone while the consequences of the action are not fully knowable ahead of time.

Technologies of telepresence and digital design tools often strive to minimize or eliminate risk, e.g., flight simulators. Digital artifacts often do not exhibit commitment to actions; in fact, being able to index at random into the past of our creation through undo/redo and versioning may be the single most important characteristic that separates digital from physical interactions. Despite the obvious benefits of simulation and virtualization, retaining elements of risk in practice can be beneficial. With the challenges of risk come opportunities for more trusting, committed, responsible, and focused interactions in both social and individual activities.

Trust and Commitment

Because distance collaboration mitigates risk, there is less of an opportunity for building trust. “Even strong ties maintained at a distance through electronic communication are likely to be… diminished in strength compared with strong ties supported by physical proximity” [37]. Examples of problems with distant, electronic communication include flaming as observed on the Internet [65], which is attributed to the lack of social context cues. One could alternatively attribute these findings to decreased risk in computer-mediated communication as compared to face-to-face communication. On the other hand, it is important to remember that sometimes the elimination of the types of risks associated with face-to -face interaction can also lead to more open conversation and close emotional ties as described in online communities (e.g.,[13, 77]).

Though risk can make people feel more anxious about interactions with others, it can also engender the kind of trust necessary for successful distance collaborations. In reviewing the literature around both collocated and distance interactions, Olson & Olson [52] concluded that distance matters in deciding the outcome of collaborative work. Fortunately, problems that arise from distance collaborations may be mitigated by initial face-to-face contact [57].

Situations that involve more risk can also stimulate more committed involvement by participants of the interaction. In the context of writing, “Because the computer doesn’t permanently record what you write, you feel less committed when you type on it” [3, p. 155]. Likewise, painting in watercolor requires more commitment to each stroke than working in Adobe Illustrator; working with people face to face requires more commitment than in distance collaborations.

Personal responsibility

Making the consequences of decisions more directly visible to people alters the outcome of the decision-making process. There are situations where the decision-makers should not be subject to the overwhelming repercussions of their decisions, e.g., natural disaster response planning. How-ever, other scenarios suggest including the explicit aware-ness of risk into the decision-making scene. In Milgram’s studies on obedience to authority [47], physical proximity of the teacher to the learner significantly decreased levels of obedience to orders to inflict more pain upon the learner. Making the implications of one’s actions more visible (making risk more salient) increases one’s sense of personal responsibility for decisions, helping to overcome the human inclination for obedience to authority.


Situations of higher risk cause people to feel more emotion-ally negative and, therefore, more focused, paying closer attention to detail, while situations of low risk allow people to feel more emotionally positive, relaxed, curious, and creative [50, p. 26]. Instilling a higher sense of risk in the design of the interactive space helps people to focus. However, there are other times when divergent thinking, e.g., brainstorming, is more appropriate. One may better design for embodied interaction by designing the experience of risk in interactive systems to alter the emotional experience of user(s). An important caution with designing for risk is to avoid eliciting the combination of negative emotion with high arousal because this leads to closed-minded and often dangerous behavior, e.g., reflexively pushing on an emergency exit door that only opens inward [50, p. 28].

For a clearly corporeal example of designing with risk in mind, consider the Painstation [44]. This art project in-creases the amount of risk involved in the game of Pong through a shock, heat, and whip plate that each player places one hand upon. Not surprisingly, players stay more focused. While we do not advocate that shock plates be included with the next version of office productivity suites, this artwork elucidates Dreyfus’s point that risk, attention, and engagement are intertwined.

Thick Practice

Whilst the [electronic] system appears to have provided a more accurate and reliable record … it has failed to provide an adequate replacement for the dog-eared documents and ‘illegible scribbling’ that are the paper medical record cards. In the case of new technology … there are ‘bad’ organisational reasons for ‘good’ clinical records.
Christian Heath and Paul Luff [25]

It may seem a platitude, but it is worth repeating that, “if technology is to provide an advantage, the correspondence to the real world must break down at some point” [23]. Interaction design is simultaneously drawn in two directions. First, the promise of new technology is that it pro-vides previously unavailable functionality. Second, in designing almost any new technology, one is drawing on existing human understanding of the world. In the creation of the new, much technology formalizes some aspects of a work practice. System designers have often “paved paradise and put up a parking lot” — the goals were noble, but important invisible aspects of work practice were denied by the new technology (cf. [66]).

This section argues that interfaces that are the real world can obviate many of the difficulties of attempting to model all of the salient characteristics of a work process as practiced. This argument builds on Weiser’s exhortation to design for “embodied virtuality” rather than virtual reality [72]. Designing interactions that are the real world instead of ones that simulate or replicate it hedges against simulacra that have neglected an important practice.

A system that respects the primacy of physical practice is Final Scratch, which provides access to digital music through specially encoded vinyl records (see Figure 8). These vinyl records contain a time code instead of an audio signal. The system interposes a soundcard into the signal path between turntable and mixer to pick up the time code, link it to playback of digital music files on a laptop computer, and return that audio signal to the inputs of the mixing console. FinalScratch affords continuity of practice — skills acquired over years of practice still apply since the physical interface has not changed. DJs regard it as superior to competing digital control products (such as CD players with jog dials) because digital controls do not provide the sensory richness or the nuance of manipulation offered by the “real thing.”

Books with Voices [36] augments paper transcripts of oral histories with barcodes printed alongside the text. These can be scanned by a PDA to access original audio recordings. In retaining the printed paper page as the primary artifact around which interaction is structured, the system embraces existing reading practices, grafting digital media onto them.

The project of technology is the creation of increasingly malleable materials, and computation is perhaps the most malleable created so far. Given the techno-utopian ideology of computer science, it can seem heretical to suggest that one should undertake a project other than replacing the physical world. Clearly, the digital world can provide advantages. To temper that, we argue that because there is so much benefit in the physical world, we should take great care before unreflectively replacing it. More precisely, from a design perspective, solutions that carefully integrate the physical and digital worlds — leaving the physical world alone to the extent possible — are likely to be more successful by admitting the improvisations of practice that the physical world offers.

Figure 8 Final Scratch: encoded vinyl for digital music.

Related Work

New design considerations and design conversations emerge when our bodies are understood as more than just “Baby Bubbleheads” (i.e., the Model Human Processor [10]). We are not the first to undertake conceptual scaffold-ing in this area. We describe here two related areas of work: applying theory to HCI and generalizing the results of tangible interface research. We should also point out that there are other lenses through which one can reason about why bodies matter, such as aesthetics, which we do not cover in this paper.

Winograd and Flores introduced phenomenology — the philosophy of being-in-the-world—to the field of computer science as a caution against the then-prevalent symbolic view of cognition and intelligence [76]. Hayles traces the history of this view and how its DNA remains in current discourse and popular culture [24]. Weiser relied on Polanyi’s concept of tacit knowledge to develop his vision of ubiquitous computing [72]. More recently, Dourish suggests phenomenology and social science theory (specifically ethnomethodology, the study of the practical achievements of social actors) as constituting an appropriate uniting lens for social and tangible computing [15]. We draw from this work the focus on the human body and our experience of action, as well as the top-down approach of generating design concerns from theory. The project of this paper is distinct from this prior work in that our goal is to provide design themes, elucidated from the theoretical literature when appropriate, rather than provide an accessible entry for the HCI community into philosophy literature.

There have been several recent efforts to provide taxonomies for off-the-desktop interaction by surveying existing systems. These taxonomies have focused on characterizing the use of input and output technologies [35]; investigating the role of artifact physicality and interface metaphor [19]; conceptualizing tangibility in terms of tokens and constraints [63]; and the role of tangibility as a facilitator for collaboration [28]. This work largely represents a technology-centric view of interaction design: generalizing from systems is effective for finding commonalities, but — by definition — it is limited to describing what is already there. This paper contributes to this discussion by synthesizing theoretical results into themes that are both generative and evaluative.


Hollan and Stornetta [27] argue that the impact of electronic media should not be measured by how well they can approximate the affordances of face-to-face interaction, but rather how they can surpass the constraints of co-presence and co-location to offer value that motivate their use even if face-to-face communication is available [27]. Similarly, we should not just strive to approach the affordances of tangibility in our interfaces and interactions, but to go beyond what mere form offers. As Dourish notes, “Tangible computing is of interest precisely because it is not purely physical. It is a physical realization of a symbolic reality” [15, p. 207]. For a combination of virtual representations and physical artifacts to be successful and truly go beyond what each individual medium can offer, we need a thorough understanding what each can offer to us first. In this paper we developed our view of the affordances of physicality and concreteness for the design of interactive systems. We believe the five themes presented in this paper will be of value both generatively — helping designers come up with new solutions — and for evaluation — providing a rich set of axes for analyzing the benefits of systems.


We thank Jan Chong for introducing us to the notions of visibility and transparency; and Terry Winograd, Wendy Ju, and Erica Robles for insightful discussions about this work.