RELATED WORK

These are some work that has been undertaken that relates to my study.

Audio Math- This system presents the design, development and usability of an interactive virtual environment based on audio to develop and use short-term memory and to assist mathematics learning in children with visual disabilities. Children are required to identify numbers and carry out mathematical actions based on short-term auditory memory. It was reported that the sound feedback was limited in its quality and the way in which it is represented. It does not guide children to the right answer only provides confirmation when they've arrived at it.

Brewster et al compared user's ability to recall information about icons and menu items with the aid of earcons. Lemmens et al compared the use of earcons and auditory in visual categorization task

Term	Definition
Earcons	‘Non verbal audio message that are used in the computer/user interface to provide information to the user about some computer object, operation or interaction’ Blattner et al. Earcons are composed of motives, which are short, rhythmic sequences of pitches with variable intensity, timbre and register

MOTIVATION

We need a communication medium that involves more than the exchange of words, sounds is one of the better known disciplines that communicates powerfully through nonverbal means (Somliar 1994)

There is a long tradition of communicating through non-speech sound: like horns and bells in Europe: Hunting horns are an excellent example of signal type non-speech messages. These messages included warnings, cheering on the hounds, calls for aid, fanfares for each animal and so on. Drums in Africa is one of the most remarkable methods of communication is the talking drum of central Africa.

In this era of ubiquitous computing, we are equipped with technology that enables us to take advantage of the universal meaning of sounds to create rich soundscapes that can enhance and intensify computer interfaces. (Brewster et al 1998)

HYPOTHESIS

Because the display in interactive tabletops is visual, designers often focus on presenting information to user through the visual channel. Although visual feedback is the primary modality through which to communicate information in this environment, auditory feedback may also serve an important role. However, it is as yet unclear what that role is and how it might enhance user's experiences.

The designer of a co-located environment must consider that multiple users perceive auditory feedback and that this simultaneous perception might affect performance, understanding, and user experience. ( Hancock et al 2005)

This study is driven by two research questions.

H1: What effect does auditory cues have on children completing a task on an interactive surface?

H2: Does auditory cues serve as an important role in interactive tabletops?

USER FEEDBACK

I used a likert scale questionnaire after the experimental sessions as this type of questionnaire was probably the most suitable for children. It consisted of just 3 questions that had the open-ended question 'why' at the end of each question. Asking why at the end of question was a way to encourage participants to justify their decisions.

"Sounds help me a bit in completing the task, because it told me what to do"

"Yes sounds was helpful because it helped me get it right"

" The sounds made it a bit easy. It helped but I was still able to do it without sound"

"Sounds kept distracting me"

"Sounds were annoying. I could not concentrate when people kept touching; I keep getting all mixed up"

"I prefer session 1 with no sound because I want to do it by my-self instead of telling"

" I prefer session II, with the sound"

"I liked session 1&2"

Summary of answers to questions asked in the follow-up questionnaire after experiments

Informal interview

AA common complaint heard in the informal interview following the questionnaire was that sounds were distracting and annoying. Participants said although it helped them it was still annoying. A participant then expressed her opinion on how she thought the sound should be; she said she would prefer it if there were short sounds. Another participant agreed to this.

They also complained in the informal interview that task was beginning to get boring, as they were required to build the same stories over and over. Although participants was not given a choice to see if they would prefer visual cues over audio cues; two experimental participants mentioned they would have also liked visual only but no sounds, because sounds became annoying.

Below are pictures of participants in a lab at uni working on task on touchable surface.

SUMMARY OF COMBINED FINDINGS

I tried to placed together a table containing a summary of combined experimental findings. This may change or improve as I go along. The trade off act as a sort of counter balance.

	Benefits	Trade off
Audio + Visual off	Higher volumes of collaboration Critical thinking	No detection No alerting No affective response More time manipulating pieces
Audio only	Detection Alerting Affective response Increased awareness	Increased distraction Increased annoyance Decreased levels of concentration
Audio + Visual Present	More errors identified Affective response Alerting Detection	Directed & instructional

Summary of combined experimental findings

RESULTS OF STUDY

These are some general findings in experiments undertaken.

Findings of experiment I

In this experiment, experimental participants built a total of six stories although they were informed that there were five stories and five puzzle pieces to each story. they had two stories in the correct sequential order. Children spent more time manipulation pieces. There were high volumes of collaboration between participants, they helped each other and reasoned aloud.

During the duration of the task there were an unintended use. The design of the application highlights all tiles green when any number of tiles are placed together. One participant interpreted this as a cue, that objects were placed in the correct sequential order. She said, "Look, they are all green". She confirmed to other participants they placed their stories in their correct position by looking at them saying "yeah its green". They continued to build stories with this understanding.

Findings of experiment II

Participants achieved their task more accurately when audio cues were present. Performance did improve although participants reported sounds to be distracting and annoying. They placed three stories in the correct sequence. At the beginning of this experiment, when participants were finished building a story; they listened to audio cues provided which gave them confirmation as to whether the objects were placed in the correct sequence order. They re-checked by touching an object in the story built and listened again to the musical sound on scale. Experimental participants were able to use the sounds for functional feedback.They continued with this method until they built two stories, and then stopped. They proceeded to complete their task but ignored using sounds for functional feedback. Instead they relied on help from each other and reasoned among themselves as they did in the first experiment.

Findings of experiment III

Experimental participants perceived more errors when visual cues were provided with audio cues but they all became frustrated very quickly. This experiment did not improve participants performance on task; they complained that task was hard and said they could not do it anymore. On one occasion during the experiment one participant shouted,"Arrgh, this is boring now"

Experiment I		Experiment II		Experiment III
Condition	No audio + No visual	Condition	Audio Only	Condition	Audio + Visual

The order and conditions experiments are presented in

THE 3 CONDITIONS OF EXPERIMENTAL PROCEDURE

The focus was not on the direct comparison of visual vs. auditory cues, in experiment III both audio and visual cues was presented. It was intentionally decided to include just one session with visual cues with audio in order to better compare the effects of auditory cues on participant's performance. Participants were given a break at the end of experiment II.

Experiment I: No Audio + No Visual

This experiment also examined children's performance on task on an interactive surface. Audio and visual cues were turned off in this experiment. The experimental participants were required to manipulate virtual objects to build stories in the correct sequential order. The purpose of this experiment was to obtain a reference measure in which to compare the results of the other experiments.

Experiment II: Audio Only

This experiment investigated the effects auditory cues have on children completing a task when auditory cues were present. Affirmative auditory cues was provided under the following circumstances:

• When objects or puzzle pieces was were placed in the correct sequential order, a musical sound on scale were played in succession

Negative auditory cues was provided in a similar way, under the following circumstances:

• When objects or puzzles were placed in the incorrect sequential order, sound on scale naturally played in an incorrect sound succession

This was limited, as negative auditory cues provided no disparity in comparison to affirmative auditory cues; this did not serve as a very strong negative reactive cue.

Experiment III: Audio + Visual

In this experiment both audio and visual cues were present. The purpose of this experiment was to examine and compare the effect on participant's performance when visual and auditory cues were present. This experiment demonstrated an unexpected and very significant contribution.

Experiment II will allow the effectiveness of audio cues compared to non-audio to be determined, answering the first research question in my later blog under the section " Hypothesis." The second question will be answered by investigating participant's performance through out all experimental tasks. And paying attention to things that occur, reoccur and relate.

In all three experiments, task completion require the tight coupling of mental and physical operations.

EXPERIMENTAL PROCEDURE

All participants had used personal computer and most of them had experience using an interactive whiteboard. All participants had played a puzzle before, and all participants knew how to solve or place puzzles pieces.

The experiment consisted of three conditions: no audio or visual cues; audio cues only; and both audio and visual. The first two conditions will be counter-balanced with audio-visual condition, which was presented last.

Before sessions began, participants were instructed on how to complete task. In each trial, participants stood around the DT table to perform task. Exp

erimental participants were required to build five temporal stories using virtual puzzles on the DT. They had to manipulate the pieces by moving them into the correct sequence placement. The completion of the three trials was followed by a questionnaire and questionnaire was followed by an informal interview.

Apparatus Used - The MERL Diamond Touch Table

A 76 cm x 60 cm MERL Diamond Touch table, a multi-user-simultaneous-touch surface, was used for observation. The display image w

as projected from above at a resolution of 1024 by 768 pixels. Sound was presented through the computer speakers. The system was powered by using a computer.

Interaction is connected by the use of the fingers, hand or conductive object. A DT can process multiple inputs of up to 4 people. Each user stood on a separate receiver, this is a thin rectangular mat (60 cm x 30 cm) which was locat

ed on the floor, around just under the touch table. Sensors embedded in the screen of the tabletop transmit signals from the touchable surface through the user's body to the mat. This information is used by the tabletop to differentiate between different users and coordinate their inputs.

An iron frame was used to hold the projector that was mounted at a height approximately 122 cm from the table to the projector. The DT table was placed on the table that was approximately 45 cm in height, this height was suitable as children was able to reach all corner of the table top surface.

Iron frame used to hold projector

I probably think I will have to insert a photo of a proper set up of the Diamond Touch table in final thesis.

EXPERIMENTAL DESIGN

This follows from my individual project 'TransTime' but concentrating on the sound aspect.

The experimental task is in from of an activity, where participants collaboratively engage with each other to build temporal stories that use earcons.

The activity is designed for children aged 5-7. The application trains elementary learning to tell the passing of time and help develop diachronic thinking. It uses the appropriate stories that show how certain entities change qualitatively over the course of time whilst retaining their identity.

Stories consists of five life cycle stories, these are: activities during a typical day of school, human life - from an infant to old age, making a cake from ingredients, from seed to a tree and from caterpillar to butterfly.

A total of five puzzles completes each story. A series of virtual jigsaw puzzles /tiles are scattered in the interface; experimental participants are required to build stories by manipulating pieces, simply by moving pieces into their correct temporal sequence. Musical tone are associated with each piece to give auditory feedback about the correct sequence placement.

The sounds have a direct relationship to virtual objects, each objects were structured as follows: each object or tile has it's own timbre and the tiles belonging to a story were differentiated by rhythm, pitch or intensity. When a story is placed in the correct temporal sequence, each tile play its own sound in succession.

Interface of the activity on the MERL's Diamond Touch

Sounds were created using MIDI sound in GarageBand, it was then exported as a WAVE file (extension.wav) Squeak software (which is used to design activity in the MERL Diamond Touch) was used to split that wave file up into segments for the individual notes.

I had look at Roberts LA, Sikora, CA. Optimsing feedback signals for multimedia devices: Earcons vs. Auditory icons vs. Speech. Proceedings of IEA '97, Tampere 1997. Talks about mapping between the earcon and a function in the interface. It gave me insight on expectations in observation.

CHILDREN'S AUDITORY PERCEPTION & COGNITIVE DEVELOPMENT

The design for children's learning environment need to incorporate both the way children use their senses in the natural environment and take into account their age-dependent perceptual and cognitive abilities.

I believe since my writing focuses on children, sound and learning. I should therefore discuss on children's cognition and sensory developmental abilities relating to auditory perception. Something like this in my writings.

I have done a bit of reading in this area and was surprised to discover that children's ability to perceive sound relative to their level of cognitive development; I found in literature on music education!

I found this in the research of Morrongiello, B., Roes, C., (1990) Developmental changes in children's perception of musical sequences: Effects of musical training. Developmental Psychology, Vol 26.5, pp. 814-20

and

Gromko, J., Russell, C., (2003) Relationships among Young Children's Aural Perception, Listening Condition and Accurate Reasoning of Graphic Listening Maps. Journal of Research in Music Education, Vol 51.4,pp.333-342

The research of Morrongiello and Gromko may support my proposal in its result and findings where they noted that the use of sound augments responsive environments for children.

MORE ON SOUND

The audio used in my study uses a type of Western sound on scale as cues.

Western music uses a mathematical logarithmic scale. A logarithmic scale is a scale of measurement that uses the logarithm of physical quantity (this means measured and /or calculated and expressed numbers)

Some of our senses operate in a logarithmic manner, which makes logarithmic scales for these input quantities especially appropriate. In particular our sense of hearing perceives equal ratios of frequencies as equal differences in pitch. Studies in children and an isolated tribe have shown logarithmic scales to be the most natural display of numbers by humans.

On the interactive surface that my study focused on; each tile or object had a sound attached to it and each was differentiated by rhythm, pitch or intensity. There are several major sound parameters that can be dynamically varied in a single sound. These are: amplitude ( i.e., volume) frequency ( i.e., pitch); timbre ( i.e., soft/harsh); and phase ( i.e., (timbre/rhythm). Lessons from sonification ( that is the production of sound) suggests that the most intuitive mappings of information to sound rank as follows: amplitude; pitch; tempo and timbre. Research does provide us with specific aspects of sonification which must be considered in developing a model for sound feedback for children.

I did some reading from paper 'Exploring Ambient Sound Techniques in the Design of Responsive Environments for Children' by M. Dromeva, Professor A. Antle and Professor R. Wakkary. They developed the first known attempt to develop a model which uses sound (auditory displays) to support children in task-solving and related learning activities; which they specifically tailored to children's developmental needs and abilities. The model they developed uses sound feedback as an ambient display. I found this paper very relevant to my study and it shed light into issues of my own study.

I am indecisive as to whether I should briefly mention in my writing the logarithmic scale of Western sound and shed some light into the parameters of sound, since they have some relevance to my study.

Probably not. Not entirely sure, may be it is just useful information to know not necessarily to write.

Suggestions anyone?

Trying to fit appropriate writing is proving to be very challenging, may be a longer traditional thesis would be easier to write.

EXPLORING AUDITORY FEEDBACK

Auditory feedback has been extensively used to convey information in computer applications. For example Buxton in Speech, language and audition sited in Human Computer Interaction: Toward the Year2000; here he categorized non-speech audio messages into something like encoded messages and data, alarms and warning systems and status and monitoring indicators. The messages can be converted using either abstract synthetic sounds called earcons or naturally occurring sounds that may be related to an action or event called auditory icons.

Also in Brewster’s paper he compared users ability to recall information about icons and menu items with the help of earcons. He carried out an evaluation of earcons to see whether they are an effective means of communicating information in sound.

Initially I found information regarding earcons and auditory icon difficult to follow and had little understanding on how it links to my present study. I never heard the word ‘earcons’ and ‘auditory icon’ until I began reading Prof. Stephen Brewster’s website and materials.

After several reads I am now able to see how it links to my study in auditory/sound. Most of the references that I’ve been thorough focused on the specifics of sound design and codification. My study is not concerned with these specifics.

The aim of my research is to understand / investigate and inform the impact / effects auditory feedback has on children’s performance on task in an interactive tabletop and whether sound serves as an important role in Interactive tabletops?

My study differs from previous studies that investigated audio in that it is not focused on an individual application but rather it is intended to understand and inform the effects auditory feedback have on children performing any task in any touchable interfaces that uses auditory feedback.

I have also been doing some research into the kinds of auditory feedback of some systems. My study wishes to support the kinds of auditory feedback suggested by these systems below.

1. ShrEdt – a collaborative writing tool, handles cursor collisions; when a user tries to edit a document in the same place as another. Found in Dourish, P. and Bellotti, V. Awareness and coordination in shared workspaces. In Proc. CSCW 1992. ACM Press, 107 – 144

2. AROMA – an application that presents and abstraction of data captured to displays auditory cues about the remote presence of individuals in a distributed group Found in Pedersone, E.R. and Sokoler, T. AROMA: Abstract representation of presence supporting mutual awareness. In Proc. CHI 1997. ACM Press, 51-59

3. ARKola simulation – This application uses auditory icons to support awareness in a distributed collaborative bottling plant management application. Found in Gaver, W.W., Smith, R. B., and O’Shea, T. Effective sounds in complex systems: the ARKola simulation. In Proc. CHI 1991. ACM Press, 85 – 90

4. Aura – Provides for awareness of remote presence by providing “serendipitous information, via background audio cues, that is tied to people’s physical actions in the workplace”. Found in Mynatt, E. D., Back, M., Want, R., Baer, M. and Ellis, J. B. Designing audio aura. In Proc. CHI 1998. ACM Press/Addison – Wesley Publishing Co., 566 - 573

Audio feedback falls into three categories. These are: -

· Proactive

· Reactive [affirmative; negative]

· Ambient.

My study falls into category reactive auditory feedback meaning feedback provided to the user after (or as) the user performs an action. This type of feedback can be further classified as: -

- Affirmative – Affirmative reactions confirm the actions of the user (e.g. plays sound when correct)

- Negative – negative reactions indicate errors (e.g. highlights red when selected)

I have been reading, Finger talk: collaborative decision-making using talk and fingertip interaction around a tabletop display. Rogers, Y., Hazlewood, W., Blevis, E., and Lim, Y.-K. In Proc. CHI 2004: Extended abstracts. ACM Press, 1271 - 1274

I found this article very interesting and useful.

Research/Readings

Hi all!

Hope all is well.

What I have been doing so far -

I have been doing loads of research and reading around the title of my thesis. My thesis title is: "The Effects On Children Using Auditory Cues For Task On Tangible Tabletop Interface Environment"

I have been browsing through Prof. Stephen Brewster's website (suggested by Bob) and have found some useful information and journals on research into audio and haptic (touch-based) human computer interaction and mobile computing devices in the Glasgow Interaction Group. The group is a part of GIST ( Glasgow Interactive Systems Group).Prof. Stephen Brewster is an academic in the department of Computer Science at Glasgow university. Below links to Stephen Brewster's URL

http://www.dcs.gla.ac.uk/~stephen/

I have downloaded and read journals from the above page .These are:

• 'Maturation of Visual and Auditory Temporal Processing in School-Aged Children' by P. Dawn and D.V.M. Bishop
• 'The Design and Evaluation of an Auditory Enhances ScrollBar' by Stephen A. Brewster
• 'Realtime Digital Synthesis of Virtual Acoustic Environments' by E.M Wenzel & S. H. Foster
• 'Audio or Tactile Feedback: Which Modality When?' by Stephen Brewster et al
• 'Tac-tiles: Multimodal Pie Charts For Visually Imapred Users' by Stephen Brewster and Stephen A. Wal
• 'An Investigation of Using Music to Provide Navigation Cues' by G. Leplatre and Stephen Brewster.
• 'Investigating Background & Fore-ground Interactions Using Spatial Audio Cues' by Stephen Brewster & Y. Vazquez-Alvarez

While reading these journals, some information related to the work I am undertaking but felt the majority was not relevant and were a bit outdated. I contacted via phone a research fellow, Marilyn Rose McGee -Lennon a member of the Glasgow Multi-modal Interaction Group who works with Stephen Brewster. During our telephone conversation, I agreed when she told me that it is very easy to end up with loads of irrelevant information. I have been doing my research and felt this is exactly what was happening.

However she sent more target information relating to my study via e-mail. Below are the list of journals and and Online references that she sent me.

• 'Multisensory Roughness Perception of Virtual Surfaces: Effects of Correlated Cues' by J.Wiesenberger & G.L. Poling
• 'Evaluation the Use of Sound as a Navigational Aid on a Mobile Device - Discussion' by M.Crease & Y. Lau
• 'Haptics In Virtual Environments: Taxonomy, Research Status, and Challenges' by M. Srinivasan & C. Basdogan
• 'Humans Integrate visual and haptic Information in a statistically optimal fashion' by M.Ernts & M. Banks
• 'It's About Time: A Conceptual Framework for the Representation of Temporal Dynamics in Geographic Information Systems' by D. Peuquet
• 'Progress in Human Geography' by http://phg.sagepub.com
• 'Relative Performance using Haptic and/or Touch-Produced Auditory Cues in a Remote Absolute Texture Identification Task' by S.J. Lederman, A. Martin, C. Tong & R.L. Klatzky
• 'Seeing More: Visualizing Audio Cues' by T. Bergstrom & K. Karahalios
• 'The Effiective Combination of Haptic and Auditory Textural Information' by M. Rose McGee, P. Gray and S. Brewster
• 'Touching and Hearing GUI's Design Issues for the PC - Access System' by C. Ramstein, O.Martial, A. Dufresne, M. Carignan, P. Chasse and P. Mabileau
• http://www.dcs.gla.ac.uk/~stephen/bibliography.shtml

• http://www.neuroscience.ox.ac.uk/pubs/experimental-psychology-publications/HoSpence2005

  http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.5.7029

  http://www.pubmedcentral.nih.gov/articlerender.fcgi?artid=2632311

  http://cogprints.org/1509/0/wayfind.htm

The journals are a bit outdated but relevant and felt after reading can articulate bit on audio.

I contacted a PhD student Evi Indriasari Mansor from Manchester University who is now coming to the end of her studies. She does research into Touchable Surfaces (DT). She sent me information via e-mail; journals with research into Touchable Interfaces. Journals sent were:

• 'The fantasy Table' by Evi I. Mansor et al
• 'Read-It - Five-to-seven-year-old children learn to read in a tabletop environment' by R.J.W. Sluis et al School of User System Interaction
• 'Enforcing Cooperative Storytelling: First Studies' by A. Cappelletti, G. Gelmini, F. Pianesi, F. Rossi and M. Zancanaro
• 'Evaluation the Effects of Fluid Interface Components on Tabletop Collaboration' by U. Hinrichs, S. Carpendale and S.D. Scott
• 'Exploring the Effects of Group Size and Table Size on Interactions with Tabletop Shared-Display Groupware' by K. Ryall, C. Forlines, C. Shen and M.R. Morris
• ' Hands on What? Comparing Children's Mouse-based and Tangible-based Interaction' by A. N. Antle, M. Droumeva and D. Ha
• 'Little fingers on the tabletop: A usability evaluation in the Kindergarten' n Evi I. Mansor, A. De Angeli and o. De Bruijn
• 'Tabletop Sharing of Digital Photographs for the Elderly' by T. Aped, J. Kay and A. Quigley

I feel I have enough to talk about touchable surfaces and applications created. I have also researched and read very useful information regarding Interactive environments through ACM Portal. Some of these were:

• 'Exploring Non-Speech Auditory Feedback at an Interactive Multi-User Tabletop' by M. S. Hancock C. Shen, C. Forlines and K. Ryall
• 'Exploring Ambient Sound Techniques in the Design of Responsive Environments for Children' by M. Droumeva, A. Antle and R. Wakkary

I am behind schedule for my observation. I planned on carrying out my observation at the end of July. The problem I am experiencing at the moment is negotiating an a suitable date for all/most parents to bring their children in. I plan on carrying out my observation no later than Friday or early next week.

I think I have fitted appropriate questions for my methodology. I will be needing a second person to observe with me.

More in my next blog

Bye for now