Human–Computer Interaction SeriesSpringerBriefs in Human-Computer Interaction

Series Editors

Desney Tan

Microsoft Research, Redmond, USA

Jean Vanderdonckt

Louvain School of Management, Université catholique de Louvain, Louvain-La-Neuve, Belgium

More information about this series at http://www.springer.com/series/15580

Carey Jewitt, Sara Price, Kerstin Leder Mackley, Nikoleta Yiannoutsou and Douglas Atkinson

Interdisciplinary Insights for Digital Touch Communication

../images/475670_1_En_BookFrontmatter_Figa_HTML.png

Carey Jewitt

UCL Knowledge Lab, Institute of Education, University College London, London, UK

Sara Price

UCL Knowledge Lab, Institute of Education, University College London, London, UK

Kerstin Leder Mackley

UCL Knowledge Lab, Institute of Education, University College London, London, UK

Nikoleta Yiannoutsou

UCL Knowledge Lab, Institute of Education, University College London, London, UK

Douglas Atkinson

UCL Knowledge Lab, Institute of Education, University College London, London, UK

ISSN 1571-5035e-ISSN 2524-4477

Human–Computer Interaction Series

ISSN 2520-1670e-ISSN 2520-1689

SpringerBriefs in Human-Computer Interaction

ISBN 978-3-030-24566-5e-ISBN 978-3-030-24564-1

https://doi.org/10.1007/978-3-030-24564-1

This book is an open access publication.

© The Editor(s) (if applicable) and The Author(s) 2020

Creative Commons

Open Access This book is licensed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license and indicate if changes were made.

The images or other third party material in this book are included in the book's Creative Commons license, unless indicated otherwise in a credit line to the material. If material is not included in the book's Creative Commons license and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder.

The use of general descriptive names, registered names, trademarks, service marks, etc. in this publication does not imply, even in the absence of a specific statement, that such names are exempt from the relevant protective laws and regulations and therefore free for general use.

The publisher, the authors, and the editors are safe to assume that the advice and information in this book are believed to be true and accurate at the date of publication. Neither the publisher nor the authors or the editors give a warranty, expressed or implied, with respect to the material contained herein or for any errors or omissions that may have been made. The publisher remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

This Springer imprint is published by the registered company Springer Nature Switzerland AG.

The registered company address is: Gewerbestrasse 11, 6330 Cham, Switzerland

This book is dedicated to Gunther Kress (1940–2019).

Acknowledgements

The research reported in this book is undertaken as a part of the InTouch project, a European Research Council Consolidator Award (Award Number: 681489).

The authors would like to thank the following people and organizations, especially those people who participated in the InTouch case study research:

Caroline Yan Zhang (Royal College of Art, UK) and Romain Meunier (UCL Institute of Making) for their facilitation of the rapid prototyping sessions as part of the Imagining Remote Personal Touch case study.

Dr. Emma Zhang and Professor Adrian Cheok for the loan of a pair of Kissenger devices used within the Imagining Remote Personal Touch case study.

The members of the Invisible Flock Artist Studio, who are collaborators on the Art of Remote Contact case study. The Remote Contact Exhibition was supported by FACT and Community Integrated Care, funded by Arts Council England and Leeds City Council, with the support of a Wellcome Trust Arts Award, in collaboration with Prof. Nadia Berthouze (UCL).

Owlet Baby Care, specifically Dr. Milena Adamian and Michelle Dangerfield, for the loan of four smart sock units to support the InTouch with Baby case study.

Dr. Aikaterini Fotopoulou , Professor Nadia Berthouze and Frederik Brudy , who are collaborators on the Tactile Emoticon case study, and the UCL Social Science Plus scheme, who provided additional funding for this collaboration.

Dr. Val Mitchell and Dr. Garrath T. Wilson, Loughborough University School of Design and Creative Arts, who are collaborators on the Designing Digital Touch case study.

Evy Samuelsson, InTouch Project administrator.

Lili Golmohammadi, a Doctoral Student on InTouch, particularly for her contribution to the development and design of the Designing Digital Touch Toolkit.

Contents

1 Introduction: Digital Touch Communication 1

1.1 Introduction 1

1.1.1 Touch Matters 2

1.1.2 Digital Touch 2

1.2 Situating This Book: A Social Revaluing of the Sensory and Multimodal 3

1.3 InTouch Digital Touch Communication 5

1.4 Overview of the Book 6

1.5 InTouch Case Studies 8

1.5.1 Imagining Remote Personal Touch 9

1.5.2 In Touch with Baby 10

1.5.3 The Art of Remote Contact 12

1.5.4 Tactile Emoticon 14

1.5.5 Designing Digital Touch 15

1.5.6 Virtual Touch 17

2 Interdisciplinary Explorations of Digital Touch 23

2.1 Introduction 23

2.2 A Multimodal and Sensory Lens on Digital Touch Communication 24

2.3 Interdisciplinary Dialogues of Digital Touch Communication 26

2.4 Prototyping 28

2.5 Conclusion 35

3 The Terrain of Digital Touch Communication 39

3.1 Introduction 39

3.2 Human-Human Digitally Mediated Touch Communication 42

3.3 Human-Robot and Robot-Human Touch 44

3.3.1 Affective and Social Robot Touch 44

3.3.2 Teleoperation 46

3.4 Human-Object Touch Communication 47

3.4.1 Object/Textile Handling 47

3.4.2 Education and Training 48

3.4.3 Disability and Rehabilitation 49

3.5 Conclusion 50

4 Social Norms of Touch 57

4.1 Introduction 57

4.2 Technology and Changing Social Norms 59

4.3 Digital Touch and Social Norms 59

4.3.1 Touching the Body 60

4.3.2 Types of Touch 62

4.3.3 The Materiality of Touch 65

4.3.4 Digital Touching 67

4.4 Conclusion 69

5 Touch Presence, Absence and Connection 73

5.1 Introduction 73

5.2 Connecting at a ‘Distance’: Questions of Presence 74

5.3 Connecting Through Touch 76

5.3.1 Tactile Emoticon 79

5.4 Beyond the Interface 81

5.4.1 Remote Contact 81

5.5 Touch Connection as a Bodily Way of Knowing 83

5.5.1 In Touch with Baby 84

5.6 Conclusion 86

6 Sociotechnical Imaginaries of Digital Touch 89

6.1 Introduction 89

6.2 The Sociotechnical Imaginary as a Design Resource 91

6.3 The Sociotechnical Imaginary as Methodological Resource 91

6.4 Making Legible Emergent Sociotechnical Imaginaries of Digital Touch 92

6.5 Generating New Methodological Routes to Imagining Digital Touch Futures 99

6.6 Conclusion 104

7 Digital Touch Ethics and Values 107

7.1 Introduction 107

7.2 What Is Ethical Touch? 108

7.3 Touch, Body and ‘Machine’ 110

7.3.1 Touching Bodies 110

7.3.2 Human Touch 112

7.3.3 Whose Bodies? 114

7.4 Consent, Trust and Control 115

7.5 A Note on Study Ethics 117

7.6 Conclusion 119

8 Closing Thoughts, Insights and Resources for Digital Touch Communication Research and Design 123

8.1 A Social Perspective on Digital Touch 123

8.2 Insights for Digital Touch Communication 124

8.2.1 Social Norms and Digital Touch 124

8.2.2 Touch Connections 125

8.2.3 Sociotechnical Imaginaries of Digital Touch 126

8.2.4 The Ethics of Touch 126

8.3 Methodologies for Digital Touch 127

8.4 An Emergent Research and Design Framework for Digital Touch Communication 128

List of Figures

Fig. 1.1 Rapid prototyping sessions and prototypes in the Imagining Remote Personal Touch Case Study 10

Fig. 1.2 The Owlet Smart Sock and bedtime re-enactments, In Touch with Baby case study 11

Fig. 1.3 Art of Remote Contact case study four exhibition artefacts. (Photo credit: Ed Waring) 13

Fig. 1.4 The prototype Tactile Emoticon device, designed to send heat, pressure and vibration between the two ‘mitt’ sections. (Diagram credit: Frederik Brudy) 15

Fig. 1.5 Designing Digital Touch Toolkit development 17

Fig. 1.6 Instances of touch in Virtual and Augmented Reality. From left to right: Saatchi Gallery – We live in an Ocean of Air – VR Experience by Marshmallow Laser Feast; Discussion with Dr. Isabel Van De Keere, Founder and CEO of “Immersive Rehab”; Demonstration of an exoskeleton glove by Dr. David Swapp, Manager of the Immersive VR Lab at UCL, London 18

Fig. 2.1 Prototyping to explore participants’ experiences, memories and imaginations of remote personal digital (touch) communication 31

Fig. 2.2 Prototyping session as ethnographic background and insights for the analysis of Design students’ final concept boards and videos. From top left to right: Amare by Betsy Cousins @betsyc_design; Puls by Joe Slatter https://www.behance.net/joeslatts82ca. Loughborough University design students exploring touch during rapid prototyping workshop 32

Fig. 2.3 Remote Contact exhibition, artistic provocations by Invisible Flock generate new digital touch experiences that can be observed and researched 33

Fig. 2.4 Iterative prototype development by Frederik Brudy as part of theTactile Emoticon case study 34

Fig. 2.5 Dr. David Swapp, Manager of the Immersive VR Lab at UCL, London, demonstrating touch affordances of an exoskeleton glove 35

Fig. 4.1 The ‘Haptic Chair’ prototype offers a whole-body sensorial touch experience to create a sense of a remote digital hug, Imagining Remote Personal Touch case study 61

Fig. 4.2 The Remote Contact exhibition ‘I wanna hold your hand’ artefact, traced the experience of a joint walk, holding hands, via a pair of digitally-enabled gloves. Embedded galvanic skin and pressure sensors and GPS linked to an Arduino plotter that mapped the data collected. (Photo Credit Ed Waring) 63

Fig. 4.3 The Kissenger, a prototype device for remote kissing, was used as a technological probe in the Imagining Remote Digital Touch case study 64

Fig. 4.4 The participants use the ‘heat dial’ of an early prototype Tactile Emoticon device to regulate the temperature of the tactile message 68

Fig. 5.1 TheI wanna hold your hand gloves and ‘rain’ artefacts prompted visitors to the Remote Contact exhibition to hold hands, often with strangers. (Photo credit: Ed Waring) 82

Fig. 5.2 Visitors using theMotion Prints artefact engaged in playful touch with therapy putty, themselves and one another as a way of being together and connecting through touch and shared memories of tactile experiences. (Photo credit: Ed Waring) 82

Fig. 5.3 Connecting parent and baby by positioning the Owlet Smart Sock correctly on baby’s foot to establish readings on the app|On the right: an example of night time disconnections 84

Fig. 6.1 Participants in the Imagining Remote Personal Communication case study engaged in prototyping to elicit their digital touch imaginaries 93

Fig. 6.2 Example of the use of vibration as a form of tactile correction in theDesigning Digital Touch case study|Personal Scrummaging Aid by Ben Cook: https://issuu.com/bencook11/docs/hd_portfolio (© Ben Cook) 94

Fig. 6.3 Participants interacting with theTactile Emoticon device 96

Fig. 6.4 Design-workshops and prototype ideation and iteration informed the final Tactile Emoticon Device 100

Fig. 6.5 The Art of Remote Contact case study exhibition – Remote Contact, provided an exploratory tactile space for new touch experiences 101

Fig. 6.6 The development of the Designing Digital Touch Toolkit. Developed in collaboration with Dr. Val Mitchell and Dr. Garrath T. Wilson, School of Design and Creative Arts, Loughborough University 103

Fig. 8.1 Initial InTouch Research and Design Framework for Digital Touch Communication 129

Fig. 8.2 An indication of the kinds of considerations and resources that might populate the Framework for Digital Touch Communication 130

About the Authors

Douglas Atkinson

is a Research Fellow at London College of Fashion, University of the Arts London, and guest lectures on fashion and digital technologies at a number of universities. His research interests include touch perception of physical and digital objects and the sensory and emotional experience of making, particularly in the context of off-shored production, materially detached from the designer. He is currently undertaking a PhD on the role of touch in the hands-on development of a garment, within the InTouch project. He was Research Associate on the ‘Digital Sensoria: Design Through Digital Perceptual Experience’ project and Co-investigator on the MIDAS ‘Methodological Innovation in Digital Arts and Social Sciences’ project.

Carey Jewitt

is Professor of Learning and Technology at UCL Knowledge Lab, University College London, and is currently Director of InTouch (funded by an ERC Consolidator Award). Her research interests include multimodal research theory and methods and methodological innovation, bringing this social perspective to touch and technology-mediated interaction more generally. She has led numerous research projects in these areas and previously led the large NCRM-funded project MODE ‘Multimodal Methods for Researching Digital Data and Environments’ (MODE.ioe.ac.uk). She is the Founding Editor of the journalVisual Communication (Sage) and has published extensively in her fields. Her most recent books includeIntroducing Multimodality (2016) andThe Routledge Handbook of Multimodal Analysis (2014).

Kerstin Leder Mackley

is a Senior Research Fellow on InTouch at the UCL Knowledge Lab, University College London. Her research interests are in sensory and visual ethnographic research approaches as applied to the study of everyday experiences and activities, emerging technologies and design futures. She has been a Research Fellow on a number of significant projects, including ‘LEEDR: Low Effort Energy Demand Reduction’ (Loughborough Uni.) and ‘TOTeM: Tales of Things and electronic Memory’ (Brunel Uni.). Her recent publications includeMaking Homes: Ethnography and Design (2017) with Pink, Moroşanu, Mitchell and Bhamra. She has published in a range of journals, notably inQualitative Research ,Media ,Culture & Society andVisual Studies .

Sara Price

is Professor of Digital Learning at UCL Knowledge Lab and Co-I on InTouch. Her research interests focus on the design, development and evaluation of emerging digital technologies for learning, teaching and training with attention to embodiment, how sensory and bodily interaction can be mediated through digital technology and the role of this in supporting new ways of thinking and meaning-making. She is UCL PI on the ‘Move2Learn’ project (Wellcome Trust, ESRC, NSF) and has co-led several other research projects, most recently ‘WeDraw’ (EU H2020). Her recent publications includeDigital Bodies: Creativity and Technology in the Arts and Humanities , with Sue Broadhurst (2017), andThe SAGE Handbook of Digital Technology Research (2013), with Carey Jewitt and Barry Brown.

Nikoleta Yiannoutsou

is a Research Fellow at the UCL Knowledge Lab, University College London. Her research interests lie at the intersection of education, psychology, design and technology studies, including the design and evaluation of emerging technologies (multisensory technologies, robotics, mobile technologies and digital games) for learning. She has been a Research Fellow on a number of projects, most recently ‘WeDraw: Exploiting the Best Sensory Modality for Learning Arithmetic and Geometrical Concepts Based on ICT Multisensory Technologies and Serious Games’ and ‘ER4STEM (EU) Educational Robotics for STEM’. Her recent publications includeExploring How Children Interact with 3D Shapes Using Haptic Technologies , with Johnson and Price (2018).