Week 09 Evaluation Methods 1: Think Aloud, Interview & Survey HCI 2016 Spring Human-Computer Interaction+Design Lab _ Joonhwan Lee
Evaluation Methods 1 DECIDE Framework Think Aloud, Interview & Survey
Evaluation Methods (Evaluation) (user interaction) (implementation), 3
(Evaluation) 4
DECIDE: a framework to guide evaluation Determine the goals. Explore the questions. Choose the evaluation methods. Identify the practical issues. Decide how to deal with the ethical issues. Evaluate, analyze, interpret and present the data. 5
Determine the goals What are the high-level goals of the evaluation? Who wants it and why? The goals influence the methods used for the study. Goals vary and could be to: identify the best metaphor for the design check that user requirements are met check for consistency investigate how technology affects working practices improve the usability of an existing product 6
Explore the questions Questions help to guide the evaluation. The goal of finding out why some customers prefer to purchase paper airline tickets rather than e-tickets can be broken down into sub-questions: What are customers attitudes to e-tickets? Are they concerned about security? Is the interface for obtaining them poor? What questions might you ask about the design of a cell phone? 7
Choose the evaluation approach & methods The evaluation method influences how data is collected, analyzed and presented. Field studies typically: Observation and interviews. Involve users in natural settings. Do not involve controlled tests. Produce qualitative data. Lab Studies typically: Controlled observations or tests with users in lab settings. Heuristic evaluation by experts. 8
Identify practical issues For example, how to: Select users Find evaluators Select equipment Stay on budget Stay on schedule *** IRB 9
Decide about ethical issues Develop an informed consent form Participants have a right to: Know the goals of the study. Know what will happen to the findings. Privacy of personal information. Leave when they wish. Be treated politely. 10
Evaluate, interpret & present data Methods used influence how data is evaluated, interpreted and presented. The following need to be considered: Reliability: can the study be replicated? Validity: is it measuring what you expected? Biases: is the process creating biases? Scope: can the findings be generalized? Ecological validity: is the environment influencing the findings? 11
Lab Studies : : context (, ) 13
Field Studies : ( ) : 14
Observational Methods Think Aloud
Think Aloud,, : cognitive walkthrough :. 16
Think Aloud Users can voice the linguistic contents of WM Not the processes that t got it there 17
Think Aloud vs. CI Beyer & Holtzblatt CI think aloud CI think aloud usability evaluation think aloud CI. Think aloud in UEM (usability evaluation method):. working memory (verbalize) 18
Think Aloud Protocol Tal -aloud Protocols: When information in WM is already linguistic Information State 1 (stored in WM) Perceptual or cognitive process Information State 2 (stored in WM) Verbalize Verbalize info 1 info 2 19
Think Aloud Protocol Information State t 1: Information State t 2: Problem 2 + 4 Add 2 and 4 Problem 2 + 4 Answer? Answer 6 two plus four hmmm... six! 20
Think Aloud Protocol Think aloud Protocols: When info in WM is not already linguistic Info State 1: NOT in Perceptual or Info State 2: NOT in linguistic form cognitive process linguistic form Translation process Info State 1: Info in linguistic form Translation process Info State 2: Info in linguistic form Verbalize Verbalize info 1 info 2 21
Think Aloud Protocol Find a jigsaw puzzle piece Info State 1: See: Translate into words Find blue piece with 2 straight edges, then mentally rotate t Info State 2: Translate into words Info State t 1: Looking for the upper Info State 2: Blue right hand corner of the sky. see corner piece I see is sky pieces, sand piece the correct corner piece hmmm, wanna sky, the corner up here lot s of blue sky, not that one, that s sand there it is!!! uhhh two edges ummm 22
Think Aloud Instructions Main instruction: I m going to ask you to think aloud. What I mean by think aloud is that I want you to tell me everything that you are thinking from the time that you see the statement of the task until you finish the task. I would like you to talk aloud constantly from the time I give you the task until you have completed it. I don t want you to try to plan out what you say or try to explain to me what you are saying. Just act as if you are alone, speaking to yourself--just a little louder. 23
Query Techniques Interview Questionnaires
Interviews 1:1,, : (interviewer ), : CI... 25
Questionnaires :.. : /. 26
Questionnaires? scalar choice., open-ended scalar data scale ( : 1--2--3--4--5) usability evaluation demographic Experiment Pre-, Post- question 27
Workshop: Think Aloud Think Aloud Practice : 5/29/2016, 9PM 28
Workshop: Think Aloud ( ), UI,, ( ), / 29
Workshop: Think Aloud ( ) (,, /, ), 30
Workshop: Think Aloud ( ) / (? ),?,? ( )?, 31
Questions?