1 Introduction
Internet of Things (IOT) is a general term often used to describe digitally enhanced physical objects or spaces enriched with sensors and actuators, connected through a network, digitally controlled, and enable to exchange data. As the power and popularity of IoT technologies and applications increase, researchers face the challenge of making them accessible to and useful for people with disabilities. The focus of our research is to exploit IoT technology for children with Neurodevelopmental Disorder (NDD).
Neurodevelopmental Disorder is an umbrella term, used to identify different pathologies arising during the development period and characterised by the co-occurring deficits in cognitive, social and motor sphere [26]. Intellectual disabilities, Attention Deficit and Hyperactivity Disorders (ADHD), Autistic Spectrum Disorders (ASD) are the most frequent examples of NDD. These deficits affect deeply the life of these people and of their families, since patients can be unable to complete even simple daily tasks, which makes them strongly dependent on others to live [12].
NDD is chronic and patient’s improvements are generally very small and very slow. Still, it is acknowledged that intensive support from the childhood can help to alleviate the symptoms. Some therapies have been developed, but they must be deeply customized and constantly adjusted on the patient’s needs. Many therapeutic interventions have the goal of teaching some basic skills so that the patient can acquire a sort of, even minimum, autonomy in his/her daily life, e.g., through practices that promotes gross and fine motor coordination, attention and social interaction.
Among the many possible approaches to help these children multisensory interventions have a special role. These practices are grounded on two main concepts. First, the theory of sensory integration posits that the learning process depends on the ability of processing and integrating sensory information process and integrate them in order to plan and organize behaviour [9]. Second, most of the impairments associated to NDD are thought to originate from a sensory dysfunction, i.e., the fact that the sensory stimuli are badly processed and integrated. The result of this incomplete or distort process is the creation of an abnormal mental representation of the external world. This in turn may produce motor impairments and deficits in cognitive skills, like generalization, space awareness, language usage and social behaviour [13, 20, 24], and induces distress and discomfort, frequent concentration losses, and disengagement from the proposed activities. Multisensory interventions—integrated today in many programs both in therapeutic centres and in schools in US, Canada, Australia, and UK—attempt to improve the sensory discrimination, i.e., the ability to focus on and discriminate between different simultaneous stimuli, and sensory integration, i.e., the ability to interpret properly multiple sensory stimuli simultaneously.
Some kinds of multisensory treatments require a suitable space, called Multisensory Environment (MSE), a room intended to stimulate the vestibular, proprioceptive and tactile sense of the user, train the integration and identification of the different stimuli, and engage the user in useful activities.
In our research, multisensory environments meet Internet of Things (IoT) technology to offer a digitally enhanced space where sensory stimuli are originated from digitally enhanced objects (“smart objects”) or from the entire “smart environment” through multimedia digital projections, ambient sound, lights embedded in the physical space.
The solution we propose is called “Magic Room” (MR) where children with NDD can be involved in playful multisensory experiences that are specifically designed in order to match their needs and offer a much wider gamut of play opportunities than traditional MSEs [1]. The process of designing both the smart environment, the smart objects, and the activities to be performed inside the MR involved a local care centre and a set of NDD specialists who are experts in the use of traditional MSEs.
Physical environment where the magic room is installed (on the left). Multisensory effects in the magic room (projections, lights, bubbles, smart objects) (on the right)
2 Related Work
2.1 Traditional Multisensory Approaches
Multisensory approaches have been largely considered in past years and this has resulted into the adoption of two main approaches: one refers to objects and one refer to spaces.
Examples of Montessori toys and Hasbro’s Bop-It!
However, these toys are not customizable for the children: the set of components is fixed and modifications require the manufacturer to create a new version of the toy. Moreover, these toys have a limited amount of senses that can be stimulated simultaneously, mainly focused on visual. More sophisticated solutions can be achieved by considering the environment in which the child plays.
The expression MSE is often referred as the “snoezelen room”, for short “snoezelen”. This term is referred to a product present on the market and originally developed in the seventies in Holland; snoezeln is the contraction of two Dutch terms “snuffelen” (meaning “to discover or explore”) and “doezelen” (referring to a relaxed state). The goal of a Snoezelen is to offer a soothing, nonthreatening and relaxing environment that promotes a general feeling of restoration and refreshment by engaging people with NDD (with the close support of caregivers) with pleasurable, explorative experiences while keeping controlled the amount, intensity and quality of stimuli proposed.
Studies have been conducted to explore the therapeutic and educational effectiveness of Snoezelens and they report improvements of the ability to adapt to circumstances and the mitigation of some stereotypes during the sessions in the MSE [18, 20, 26].
However, “snoezelens” have limitations since they offer a restrained capability for the user to interact with artefacts producing a “cause” and receiving an appropriate stimulus as an “effect” to establish a case-effect relationship, fundamental in the development of cognitive skills. In addition to this limitation, another issue is linked to the creation of learning scenarios, sequences of combinations of stimuli from different sources, which is time-consuming and potentially risky for the session flow, since the stimuli in the snoezelen can be controlled only by the usage of physical buttons; this is not merely a problem of tiredness of the therapists, but also forces the caregiver to release attention from the user, who can in turn remove attention from his/her current task.
Several authors [2, 11, 21] therefore call for exploring new materials and solutions for MSEs.
2.2 Digitally Enhanced Multisensory Approaches
Researchers in the assistive robotic field have taken the path of designing robots able to sense touch, communicate through sound, movements and, in some cases, lights. Several examples have been realized in past years with specific therapeutic or educational purposes. Most of them are static robots [10, 14], with some notable exceptions that offer children the possibility to explore the physical space while experiencing a controlled multisensory experience. QueBall [25] is a spherical robot able to roll in order to move, sense touches on the surface and communicate with the user through a wide range of visual and sound stimuli. I-BLOCKS [6] offers a set of composable blocks equipped with sensors and actuators that can be connected to create interactive floors or walls. Teo [3] is a soft, huggable, mobile robot that can react to manipulation; it enables joint (child + robot) body movements in the space and joint control of multimedia contents on external displays.
Polipo [27] is an interactive smart toy devoted to training fine motor skills. Activities with this object are completely dependent to the presence of the therapist who provides support and decides which kind of movements the user has to perform. Polipo is equipped with four functions designed to train four different motion (press, pinch, slide and turn) that can be personalized to increase the difficulty of the action. It is also equipped with lights and speakers to play the preferred song of each child, which is directly customizable by the therapists, and is used to give positive reward to the child when completing a task.
Still, the cited works use one or more interactive devices in sedentary contexts, and do not investigate the learning potential of combining full-body interaction [4] and multisensory stimuli in the whole physical ambient.
The legacy of research on MSE has been collected by other researchers, mainly in the field of HCI and Assistive Technology, resulting in new approaches.
MapSense [7] is a multi-sensory interactive map that uses a touch-sensitive surface, tangibles, olfactory and gustatory stimuli, to help visually impaired children improving collaboration and memory skills. MEDIATE [22] is an example of interactive system that implements multisensory full-body interaction in the space, creating a sense of agency in children affected by NDD and to enhance non-repetitive actions in their behaviour. This is achieved through visual, tactile and aural stimuli, letting the user express him/herself through body movements: it contains, for example, an interactive floor able to generate sounds in reaction to the user’s footsteps. However MEDIATE does not integrate with smart objects, which is important to focus the attention of the child, to promote multimodal sensory integration and to trigger different possible behaviour of the user during the therapy. A different approach has been used by Sensorypaint [23], which is a multimodal multisensory system designed to let the player digitally paint using physical objects, body-based interaction and interactive audio.
Authors have compared the use of Sensory Paint with other MSEs through empirical studies, noticing that the combination of aural-visual stimuli and full-body multimodal interaction sustains engagement and helps develop different skills.
2.3 Customizable Technology
Several researchers in Assistive Technology (AT) pinpoint the importance of keeping both the caregivers and the care receivers “in the loop”, meaning starting from these stakeholders’ needs and defining technologies that can be customized to their specific and evolving needs during the therapeutic program.
A number of studies (e.g. [8, 28, 29]) embrace the concept of “user empowerment”, which can be expressed as “the users of the technology are empowered to create and modify it to solve their own problems … and they are involved in all design activities, including the development of prototypes” [19]. As discussed in [17], addressing the “user empowerment” requirement is fundamental in Assistive Technology to increase the success of an adoption process and the adoption rates consequently.
Important paradigms that are emerging in the AT field are the so called “Do it yourself” (DIY), and End User Development (EUD) [16, 28]. Both these approaches claim that it is important to provide forms of customization beyond parameters tuning, making possible for therapists or caregivers to build and personalize the technology they are using. In MapSense [7], for example, the tangibles integrated in the system’s interactive map are created by the educators and visually impaired children using 3D printing.
The robotic system reported in [29] provides a Scratch-like interface, letting the therapist to use a basic visual programming tool to design the behaviours of the robotic components according to the need of each care receiver. Now the challenge is to enlarge the power of DIY and EUD tools to smart ambients, instead of the single object present in the environment. An example is presented in [16], where authors describe a preliminary EUD tool for Ambient Assisted Living scenarios allowing the elderly and their caregivers to control and tailor personalized behaviours of different smart appliances in a “smartified” house.
3 The Magic Room
The Magic Room (MR) is an “open” smart environment designed to transform any regular room (which must satisfy only some minimal preconditions dimensions and aeration system) into a magic multisensory play space that offers enjoyable experiences to children with NDD, and helps them in learning and wellbeing.
Compared to commercially available solutions of traditional MSEs, the Magic Room is much more affordable, and can be extended easily both in terms of devices present in the smart environment and in terms of available stimuli and activities.
Schema of full MR’s content. (1) front projection, (2) floor projection, (3) computer, (4) audio system, (5) kinect sensor, (6) fed smart lights, (7) portable smart lights, (8) smart carpet, (9) magic ball, (10) olfactory machine, (11) bubble maker machine, (12) smart object, (13) smart dolphin and smart doll, (14) tube lamp
two projectors, one oriented towards the front wall and one oriented towards the floor
a Microsoft KinectTM sensor in order to detect the movements of the children playing in the room, the gesture they are performing and their position
an audio system composed of 5 speakers appropriately disposed across the room and a personal computer that controls and orchestrates the behaviour of these appliances
several smart objects: objects of different geometric shapes and materials (a cube, a pyramid, a cylinder, and a ball) and plush toys. These smart objects contain sensors and actuators in order to detect the child’s interactions with the smart object and with the smart environment; such sensors may vary from the different smart appliances but can be categorized as accelerometers, gyroscopes, tag readers, touch or pressor sensors, while actuators can be categorized as motors, controllable light actuators, sound emitters and vibration motors. The covering of smart objects is usually made by fluffy soft materials in order to be comfortable at touch and pleasant to the sight.
smart lights (portable lamps and light bulbs on the walls) that can be remotely controlled and can change both in colour and in brightness; we are currently using Philips HueTM lights that are easily retrievable on the market and are distributed with public API’s to control them.
a set of digitally controlled appliances: a bubble maker, to produce soap bubbles that are known to amuse children and tube lamp that illuminates a vortex of bubbles inside with changing colours, and is very attractive for children.
The enabling technology of the MR is based on a multi-layered software and hardware architecture integrating various sensors able to detect the behaviour of the children that are playing into MR and to communicate with smart objects of different nature and to respond with an orchestrated set of stimuli.
4 Children’s Activities in the Magic Room
The children’s activities in Magic Room consists of simple games that involve movements in the space and interactions with smart objects, smart lights, and multimedia contents [15]. All the activities have been designed in cooperation with therapists for local care centers to adapt both to their educational purpose and to the children’s needs.
It is important to notice that, independently of the complexity of the activity and the sensing capabilities of MR, the caregiver can keep the complete control: when the activity is running he/she has a remote control with which he/she can trigger events that are too complex or that MR is not able to sense, or can force the execution of some control “instructions” so to adapt the activity more efficiently to the child’s needs.
During the initial design phase, we acknowledged that a predefined fixed set of activities would have made the use of the Magic Room problematic: therapists’ goals change very frequently and activities must be constantly tuned for each child. Hence in the initial version of the MR there was no automatic control or orchestration of stimuli: MR was a gigantic “Wizard of Oz” where all the effects were activated or de-activated through buttons or sliders on a visual interface over the PC by an operator according to the children’s movements, positions and manipulation of objects. This has been of incredible value in a first exploratory study that has enabled us to simulate a countless number of interactive situations and tasks with the goal of finding the most suitable combination of stimuli.
However, this was not a appropriate solution on the long term: with one caregiver constantly focused on operating on the PC, children with NDD could not be properly controlled, nor they could receive the sufficient support to perform the task. Additionally when children understood that the “magic” was made by the caregiver at the PC, MR lost great part of its appeal.
To fulfil the therapist’s need of customization and simplification of the management of the smart environment, we developed a web tool used to empower the caregivers which enables them to define new activities.
Therapists have defined so far over 35 multisensory activities that are characterized by different levels of complexity and cognitive effort.
In the rest of this section we report examples of these activities, organized in groups according to their main learning goals: relaxation, visual-motor coordination (eye-hand coordination), gross motor skills, spatial relationships, shapes, sizes and colours, social reciprocity and turn taking, practical skills, affection and emotional bond, attention, concentration and memory span.
4.1 Relaxation
Relaxation activities obtained through soft lights and calm environment and music
4.2 Visual-motor Coordination (Eye-Hand Coordination)
Visual-motor coordination activities with polipo and the magic ball
Another type of activities relies upon the use of Polipo: the child is asked to perform fine manipulation of the elements present on the border of the toy to train in performing movements similar to the ones needed in the real life: the room projections display the situation in which that motion is needed and give visual feedback to reward the child increasing the motivation for him/her to continue.
4.3 Gross Motor Skills
Gross motor skills, which are larger movements a person makes with his arms, legs, feet, or entire body, are fundamental to perform every day functions, such as walking, running, and are also crucial for self-care operations like dressing.
An example of activities to train gross motion
4.4 Knowledge Skills (Spatial Relationships, Shapes, Sizes and Colours)
Activities to learn the basic geometrical shapes and the colours
A more complex activity is “Move to”, designed to train children to build associations between objects or shapes, and colours. A grid of images is projected on the floor (the images number depends on the chosen level of difficulty). Initially, each image shows an object with a dominant colour (for example a yellow sun, a red rose, a green tree) while the front projection presents a colour which is associated to one of the images; the child is asked to move to the image shown on the floor that has the projected color. When he or she reaches the right position, the image disappears and the lights in the room turn to the colour of the front projection. In a more difficult versions, the images show colored contours of objects only. Once the player has successfully chosen the correct item, he or she is asked to perform again the same action when the projected colour changes, until all images have been successfully chosen. The child is rewarded for his/her success with a “waterfall” of soap bubbles, while the tube lamp produces bubbles and changes colour, and clapping sounds are played.
4.5 Social Skills (Reciprocity and Turn Taking)
The goal of these kind of activities is teach children respect for the others and appropriate social behaviour, e.g., waiting while others are playing and the turn taking need in social interaction.
Children playing at Piera the frog story
4.6 Practical Skills
Activities in this category are devoted to promoting the understanding of some basic tasks in life social spaces such as crossing the road, taking a bus, or shopping. The physical room is transformed into an outdoor or indoor social environment, for example crossing the road with a policeman, by effect of sounds and fragments of realistic videos displayed on the walls and on the floor.
The interaction paradigm is the same as in the animated storytelling activities: the video proceeds if the child performs the right movement or gesture, or grasps the right object, according to social cues appearing on the current video scene.
In the “Bus stop” activity (Fig. 9- left), for example, a video of a real bus stop is shown frontally and children have to position near the signal so that the bus arrives, then people are shown to get off the bus; children have to wait until everyone has exited the bus then they are permitted to get up (simulated with the children that move towards the screen as if to enter the bus itself.
Activity to learn to take the bus (on the left) and the smart doll (on the right)
4.7 Affection and Emotions
To promote the capability of developing and externalizing feelings, various activities with the smart dolphin (inspired to pet therapy methods) help children to build an affective bond towards this toy. The physical room is transformed into a virtual aquarium by effect of virtual sea worlds displayed on the walls and on the floor, soft light effects, and smooth music, and “real” dolphins swim in the room. The child manipulates the smart dolphin and explores its affordances; the dolphin reacts to touch, pressure, vibration, position change with the light of its LEDs strips, soft vibrations, movements of the mouth or the eyes, real dolphin sounds. When it “falls asleep”, the lights are soft turned down and the video changes to a quite water space without animals.
Activities with the smart dolphin SAM
In another game, a digital dolphin is swimming horizontally across the water while some rocks are falling or sliding horizontally moved by the waterflow against it; the child has to make the digital dolphin avoiding the rocks; the game is completely controlled thanks to the orientation of the smart toy: the gyroscope inside it is able to detect which orientation the toy has and consequently move the digital dolphin so that he is able to avoid the dangers to reach it’s family.
5 The Magic Authoring Tool
To customize the Magic Room we provide caregivers with a tool able to define combinations of stimuli so that an infinite set of scenarios and activities can be created. This tool is the Magic Authoring Tool (MAT).
To define an activity MAT offers some primitives that are related to two concepts: what has been sensed by MR and what can be performed by its actuators. To define how Magic Room have to behave we offer a Rule Based descriptive language. Rules are used to describe “micro-tasks” and the effects of elementary interactions. To account for more structured situations and scenarios involving multiple interactions, rules are clustered in “Scenes” (sets of mutually exclusive rules). An “Activity” is a (ordered) set of Scenes. A “Session” is a (ordered) set of activities.
MAT interface: (1) stage area, (2) actions and sensing components, (3) multimedia contents, (4) simulation area
Another important aspect to take into account is that different activities require different multimedia contents, and these content should be frequently updated along the time to reduce the risk of boredom. To address this issue, MAT includes a multimedia database where the caregivers can upload the multimedia content they need (e.g., retrieving it from the web) and use for an existing or new activity.
Example of a rule realized in MAT
In particular, the system waits for two different possibilities: the user can press the touch sensor present in the left hand of the smart doll or the caregiver can signal that the user has pressed the wrong sensor. The latter case has a predetermined behaviour where red light is shown for 3 seconds. In correct cases the light of the room turns to green, an image of a contour of child with the left-hand coloured is shown on the front screen, and a clapping sound is played. After 5 seconds, so that the child can elaborate and process the reward, the clapping sound is stopped, the light turns white and the image of the contour of the same child with the right-hand coloured integrated with an explanatory text saying: “touch the right hand” are shown.
6 Conclusions
We performed an exploratory study to investigate strengths and weaknesses of the Magic Room for children with NDD. The study involved 19 children organized in 4 groups attending the care centre. Children are aged 8–13 and have different forms of NDD (Intellectual Disability, ASD, Down Syndrome, Prader-Willi syndrome) at different severity levels: “severe” (IQ = 30–35, 4 children), “moderate” (IQ = 35–50, 9 children), and “mild” (IQ = 50–70, 6 children). They used the MR in group with their 2 therapists for 2 or 3 sessions. Each session lasted for approximately 40 minutes and was video recorded. Our findings, based on the observations reported by the caregivers, the analysis of video-recordings (performed by therapists not participating in the session, and a final interview to the entire therapeutic team, indicate that the MR has a strong potential as learning environment for children with NDD.
The experiences in the MR have elicited functional performances, social behaviours, and emotional responses that either do not occur using traditional MSEs or require much more time to be achieved. For example, the “familiarization” with the new space was surprisingly short. Even if children with NDD are often suspicious and worried about the unknown and any new situation may be a source of distress, the participants to our study were not afraid to enter the MR and were immediately attracted by all its effects: the experience in the room was perceived pleasurable, as a kind of magic. The various stimuli, especially the lights, the projected animations, and the bubbles seemed to trigger interest as well as positive behaviours and emotions. After the first session some children explicitly asked to play again in the room.
Some improvements in the areas addressed by the various activities (communication/socialization, emotion, cognition, and motor) were observed in each child regardless the individual differences in intellectual functioning and adaptive behaviour. According to the therapists, these improvements can be ascribed to the richness of interactions and sensory stimuli offered by the many smart components, the capability of bridging the physical and the virtual world, and the strong role given to dynamic interactive lights. Obviously, these results are very preliminary and further research is needed. Improvements were not consistently present in all children and all sessions, and we have no empirical evidence of long term or generalization effects.
Still, the Magic Room may pave the ground towards new therapeutic interventions for children with NDD that we cannot even imagine now.