User manual    
Design Evolution
   Prototype Implementation   
User population
   Usability goals   
Usability test procedure
   Reporting   
Usability evaluation

Computer Prototype




User manual or installation guide

User Manual and Installation Guide


Design Evolution

Confidential information

According to the goal number 1, we made initial setup to store user information for easy access during call time. However, some kinds of information especially bank information, are confidential so the user did not want to store that information. Therefore, we decided to change the form of ‘Setup’ mode and chose specific kinds of information which is less confidential but needs more time to type, such as home address, full name, date of birth etc. Of course, all this information is optional.

Ease of use

According to the goal number 4, we think that dividing the application by two parts, such as typing mode and phone call mode, is useful because the user can understand the application more. The users were initially not happy about the fact that they had to switch between so many screens. While it is still not possible to accomplish all the tasks without the different screens we have tried to make the switching between screens as intuitive as possible

Terminology

According to the goal number 6, we chose some terminologies based on program background too much, that was not familiar with normal user. When the user wanted to send a message, the user was supposed to push the button named ‘saved values’. However, it is not intuitive so we changed the name from ‘saved values’ to ‘type message’.

User interface design

According to the goal number 3, for making voice recognition better, before user search for business or entering phone number, the user was supposed to select the correct industry (bank, restaurant and hospital) because that ensure better voice recognition. So at first, we made 2 hierarchy structure, for example, first a user chooses the industry and then enters the phone number. However, in our initial testing we saw that the users did not always choose the industry intuitively because they did not know the importance. Therefore, we changed this and made it to show immediately after the user makes the call.

missing image                      missing image

Notifying conversation

According to group discussion, normally when we talk with someone, we can know if the someone is talking or not by sound, that is one of important parts in conversation. However, deaf person cannot hear the sound, so we want to know how can we make the deaf user notice the someone is talking or not. Therefore, we have added a small icon which blinks when the person on the other side of the phone is talking



Prototype Implementation

The Prototype can be accessed here

The password is "usersfirst" :)

for instructions on how to access on mobile, read the users manual above



User population

The users you will be dealing with are members of a non-profit organization called CHIP that provides various services to the deaf community of Montreal. http://hearhear.org/

The CHIP organization has been kind enough to let us conduct users testing on their members. We have personally not met these members before and therefor your interaction will be the first with them.

We made the following poster to introduce them to the application and invite them to the testing sessions

missing image

Most of the members of the organization are not pre lingually deaf (deaf before they could speak) but are either partially deaf, have developed deafness later in their lives or have had cochlear implants so have partial hearing. The age group varies and there are young adults as well as older members. We have been told that there will be people from multiple age groups available at the testing sessions.

The members of the organization are mostly oral communicators and most of them do not know much American Sign Language (ASL). They are proficient at Speech reading (lip reading) so that will be the best way to communicate with them



Usability goals

‘Setup’ Mode

Goal Number 1: The user is able to enter initial information easily.
The Setup process should be intuitive and promote confidence in the users to enter their personal information.

Quantitative measurement: The user proceeds to enter information within 10 seconds of application start up.

‘Dialer’ Mode

Goal Number 2: The dialer layout should be obvious.
The users should realize that they are in the dialer mode without any hesitation and so the dialer should be same as android

Quantitative measurement: The user immediately recognize the dialer and proceeds to enter number, i.e. if number known then no more than 5 seconds before first key press.

Goal Number 3: The desired service's number can be searched quickly.
The user should select the relevant industry and then easily be able to find the enterprise.

Quantitative measurement: The search starts with the first letter narrowing the search results down. From key press first options become immediately available(<1 sec). This portion has not been implemented as yet.

‘Call in Place’ Mode

Goal Number 4: The calling interface easily understandable.
The user should understand right away that they need to interact with the app as well as dialer.

Quantitative measurement: User switches back and forth between dialer and transcript screen atleast once.

Goal Number 5: The different operating modes should be easily distinguishable.
The user should immediately know which call state he is in i.e. ‘Machine Speaking’, ‘Human Speaking’ and ‘On Hold’

Quantitative measurement: There is no quantitative measure for this goal, the only measure is if the user is confused as to his current state.

Goal Number 6: The saved data should easy to access
The user can find pre-saved information easily and without frustration.

Quantitative measurement: User uses presaved responses atleast once

Goal Number 7: Responses should be fast in real time.
The use should be able to enter text while the call is in place easily and without frustration.

Quantitative measurement: Less than 10 seconds between operator completing sentence and user response.




Usability test procedure

1)How many examiners are required?

According to ‘why you only need to test with 5 users’. We want to have 5 users in each study and we believe two - three examiners will be enough to manage that.

2)What equipment will the examiners need?

We have made our high fidelity prototype on a web based application called proto.io.. The main reason we choose this kinds of prototype is that the proto.io “is ideally agnostic of operating system, and requires no special operating skills to install and launch” (according to hci course website).

Though our final application will be android based, this prototype can be used on any platform including Android, IOS or even any sort of PC.

The only constraint in using a mobile phone is that you need to have the Proto.io application installed in your phone. The application is available for free on both Android and Apple App stores.

Though we expect the test users to have smartphones, to avoid the hassle of having them download the Proto.io application we would like to suggest that the examiners download the Proto.io application on their phones and take it with them to the test

3)How should your prototype be handled?

The first thing after installing the Proto.io application and opening the SmartOp application that you should do is read the user manual to get a good understanding of the working of the application.

An important thing to realise with this prototype is that it is only a demonstration of the user interface and not the actual functionality (audio recognition or speech to text). After the user presses the Call button the whole application is a mere simulation of what a simple interaction might look like. We are aware that it is a simplification but for simulation purposes we believe that creating more complexity in this part of the design process would be counter productive.

The whole simulation runs in a very sequential manner with one input mapped to one output therefore if something unexpected is entered as input then the app will also do something unexpected.

Lastly, for better recording and reporting the information from the testing, please read and print out ‘reporting’ and ‘usability evaluation’ parts. Basically, other things also are very helpful to understand the application, so we recommend to read them carefully. If you have a question, do not hesitate and please send a mail us or you can use text message in Moodle.

 

4)How will you instruct your examiners to proceed?

Before you proceed with the testing on real users, we recommend you do a pilot test on the prototype yourself and understand how the application responds to different inputs.

Moreover, deaf people might not have much experience of making phone calls, so they maybe are not familiar with call system. So before starting the test, we think it is better to make them feel relaxed.

Unlike traditional user testing, we do not want to “discover the mistakes users make”, so you can answer the evaluators’ questions. However, you “should not be given help until they are clearly in trouble and have commented on the usability problem in question”. (according to How to conduct a Heuristic Evaluation, NIELSEN)

“Although adhering to the test plan is important to obtain meaningful results, often we need to modify the test plan to achieve overall test goals, so the test plan can be flexible”. (According to Fear and Loathing on the keyboard, Dickelman)

5)How should the examiners treat the test subjects, what should they tell them?

After a brief introduction, we recommend to say why we want to test you and what the goal is for sincerer attitude and better feedback. We prepared the sample sentences. “In order to make better application for people who are hard of hearing, we would like to get your feedback on our idea and all of your inputs is valued and will be taken seriously. However, do not be afraid of testing. This test is not for evaluating yourself but for knowing how you interact with the system.” (According to Fear and Loathing on the keyboard, Dickelman)

Since most of the users you will be meeting are oral communicators as opposed to manual communicators ( people who use sign language to communicate) they communicate through speech reading (commonly referred to as lip reading). Talk slowly and clearly and face the person you are talking to.

The more you have in writing is also better, powerpoints are a good way to easily communicate with them

6)What should the examiners avoid doing?

Talking very loudly is usually not helpful and can be seen as disrespectful. The examiners should not cover their mouth during talking with the users.

We have come to learn (the hard way, unfortunately) that it is very important to be politically correct when dealing with the deaf community. They do not see their deafness as a disability and the ones who use American Sign Language (ASL) believe that it is a better form of communication than oral methods. Therefore it is important to be aware of how you communicate with the deaf community, for example the use of the term “Hearing Impaired” is considered to be politically incorrect and rude.

7)What should the examiners avoid telling the test subjects?

According to Dickelman, you should avoid telling the usability goals that significantly ruin the user test. Moreover, as we mentioned, this user test is for evaluating the interaction with the system not for testing the user. So, you also should avoid telling some words which can give pressure to users.



Usability Evaluation


In addition to the Usability Test Procedures we would like you to do a formal evaluation of our usability. We recommend going through the system at least twice to ensure a general understanding of our system.


You have autonomy to pursue your personal heuristic evaluations but we have provided an example evaluation below. In the first pass of the evaluation you are expected to perform a cognitive walkthrough including a usability test, as well as a heuristic evaluation. In the second pass it is only required to perform the heuristic evaluation.


Cognitive Walkthrough including usability test:


The cognitive walkthrough involves going through each point in the operation of the application and answer the relevant points as well as complete the usability test.

You should guide the users to complete the simple task of activating their credit card using the application. To make this simpler for them provide them with the following information;

- Phone number to call: 18005617849

- Fake credit card number e.g. 4510 1560 9879 3425 (randomly generated 16 digits)


Operation

User Actions

Interface Support

Knowledge Required

Usability Test

Initializing Application.

User should perform initial setup.

It should be evident that user needs to provide an initial setup of his profile.

It should also inform the user as to how this information might be relevant.

Basic English literacy.

Experience with of mobile text forms and keyboard.

The user should start entering information within 15 seconds of initializing application.

Making a call.

In case of unknown number user should search through database to find service to call.

Else user should enter number in dialer

The interface is attempting to replicate the android dialer interface to make it clear to the user that he is in the make a call state.


Experience with making calls using android device.

The user should start searching though database within 15 seconds of being presented this screen.

If number known user should click on dialer within 10 seconds.


Call answered

User dials number to get to relevant extension.

User returns to original screen.

The user should know that the call has been answered by an automated service and is being presented options.

The dialer option should be evident to user.

User knows to switch back when done.

Proficient English Literacy.

Experience with making calls using android device.

Once user knows option to select he switches to dialer within 5 seconds.

User goes to previous screen after entering input within 10 seconds.

Presaved Messages

User enters text that he would like to communicate over the phone.

This portion requires demonstration.

But after demonstration the user should understand that he is prerecording responses.

Already covered in previous.

User should be able to switch between current message and previously recorded messages easily.

Real time text response.

User should type what they would like to say.

The interface will inform the user that a conversation is taking place.

Already covered in previous.

User should provide responses in real time.




Heuristic evaluation:


We have included selected heuristics from “10 Usability Heuristics for User Interface Design”

by Jakob Nielson. There is also included a justification as to why they are relevant for our system. On each pass evaluate the system in comparison to the list of heuristics. Please grade from 0- 5, where 0 is the lowest mark.



Visibility of system status:

Users should always be aware of the state that they are in i.e. they should know when they are on hold, talking to an automated system or talking to an operator.


System state never known <-------------------------> System state always known.


1                                 2                                 3                                4                                 5






Match between system and the real world:

The application is designed to replicate the service call experience and so at each screen users should intuitively know what to do and not have to rely on any external help.


User does not know next step <----------------------------------------> User knows what to do next

1                                 2                                 3                                4                                 5


User control and freedom

The user should be able to switch between dialer and transcription mode intuitively.

The user should be able to switch between presaved messages and real time responses easily.


User stays on current screen <-------------------------------> User switches easily and frequently

1                                 2                                 3                                4                                 5


Consistency and standards

All the icons in the application replicate the android icons and so the user should know what each icon means.


User does not recognize icons<-------------------------------> User recognizes all the icons

1                                 2                                 3                                4                                 5

Recognition rather than recall

The progression between states should follow a natural order and the user should not have to remember what the next state should be at any point.


User does not remember next state<-------------------------------> User can predict next state

1                                 2                                 3                                4                                 5


Flexibility and efficiency of use

Users should be making frequent use of presaved messages and should be able to add to them from their real time entered responses.


User does not remember next state<-------------------------------> User can predict next state

1                                 2                                 3                                4                                 5


Aesthetic and minimalist design

User is provided with the bare minimum of system information to read. Majority of the screen space should be shared by the user and operator conversation.


System text fills space<-------------------------------> User text fills space

1                                 2                                 3                                4                                 5

Help and documentation

The user manual should cover any issues that may arise.

No issue covered by manual <-------------------------------> All issues covered by manual

1                                 2                                 3                                4                                 5