SEM 6 › Human Computer Interaction › Week 13
Human Computer Interaction · Week 13

Evaluation — Measuring Usability

Why evaluation is the core of user-centred design, matching evaluation to the prototype's functions and stage, qualitative → usability → quantitative feedback across the design life cycle, the three types of evaluation (qualitative, empirical, predictive), and four key terms (reliability, validity, generalisability, precision).

ITE37521 video · 22 min📄 Lecture video (no PDF for this week)

1Where evaluation fits

  1. Needfinding (establish requirements)
  1. Design alternatives
  1. Prototyping
  1. Evaluation
Evaluation

Evaluation identifies how usable the interface we designed is. It's a measure of usability 00:03.

  • Evaluation is the core of user-centred design 00:26. If you don't know whether the user is satisfied, there's no point doing user-centred design.
  • User feedback tells you the user's level of satisfaction, the areas to improve, and the features they like most. It also guides good practices for the rest of development.
  • It's feedback on what we have designed, not what we planned 01:23.

2Match the evaluation to the prototype

The design is usually shown to users as a prototype, and the prototype serves different functions at different stages 01:38:

  • Early in design, its functions are basic, to find out what's really needed.
  • Later, it's evaluated for the functions it offers.
Rule

The method of evaluation should match the prototype, based on its functions and stage 02:28. Functions, stage and type of prototype are all interlinked.

3Evaluation across the design life cycle

Stage Type of feedback Example questions and measures
Early 03:18 Qualitative, abstract; user preferences Do users like this colour theme or concept? Should it be prominent or in the background? Is it readable and understandable?
Middle 04:32 Usability Does it minimise your workload? Is it intuitive? Is it easy to learn? (If it isn't easy to learn, people won't use it.)
Late 05:37 Quantitative: numbers, frequencies, performance indicators How long does it take to fill and submit this form? Has the number of jobs an employee completes per day gone up? Have sales volumes increased?
Evaluation is a journey 08:13

You can evaluate early, in the middle and towards the end, and you iterate again and again until you're sure the interface gives users what they really want. The evaluation type is closely coupled with where you are in the design process.

🧪 Try it — Which evaluation fits?

4Three types of evaluation

Type With users? What it does Example
Qualitative Yes Collect qualitative feedback; identify the quality aspects behind preferences. No metric measures "Which of these patterns do you like most?"
Empirical 10:36 Yes Evaluate quantitatively, using controlled experiments, then compare results Measure task time with the old vs new interface
Predictive 11:21 No Use a model of human input and output that virtually represents a person; the evaluator mimics the user Model-based task analysis
Do qualitative first

Before an empirical evaluation, first do a qualitative evaluation and collect qualitative feedback 11:04.

Why predictive is rare

It's slow and expensive: you spend time evaluating without users, pretending to be the user, when you could have asked real users.

What UCD needs

User-centred design needs real feedback, not prediction. So qualitative and empirical evaluations are common; predictive ones are not.

5Four key terms

Term Meaning Lecture example
Reliability 13:50 Whether an assessment is consistent over time: the same result if you do it again The distance between two cities via the Southern Expressway: given the route, you always get the same answer
Validity 15:07 How accurately an assessment measures reality A distance table with a printing mistake (150 km instead of about 100 km) gives the same wrong answer every time: reliable but not valid
Generalisability 19:14 How far what we learned in our evaluation applies to a broader audience Lessons from a few test users that hold for many more people
Precision 19:50 How specific an assessment is 106 km (to the km) · 106.4 km (to 100 m) · 106.43 km (to 10 m) · 106.434 km (to 1 m)
Reliable but not valid: the office clerk

A clerk answers visitors' questions about when they can meet the CEO by reading from a schedule. Ask the same question twice and you get the same answer, so the clerk is reliable. But if the schedule itself is out of date, every answer is wrong, so the answers are not valid 15:36.

Reliability ≠ validity

An assessment can be 100% reliable and still completely inaccurate. Always check both.

🧪 Try it — Precision levels
Exam focus
  • Explain why evaluation is the core of UCD.
  • Describe how feedback changes from qualitative → usability → quantitative across the stages, with an example of each.
  • Compare qualitative, empirical and predictive evaluation: who's involved, what's measured, pros and cons.
  • Define reliability, validity, generalisability and precision, and give a reliable-but-not-valid example.

6Quick revision sheet

Concept Remember it as
Evaluation Measure of usability; the core of UCD
Feedback on What we designed, not what we planned
Match Evaluation method ↔ prototype's functions and stage
Early Qualitative: preferences, colour, readability
Middle Usability: workload, intuitive, easy to learn
Late Quantitative: task time, jobs per day, sales
Journey Evaluate early, middle and late; iterate
Qualitative Users; preferences; no metrics
Empirical Users; quantitative; controlled experiments; do qualitative first
Predictive No users; model of the human; slow, expensive, rare in UCD
Reliability Consistent over time
Validity Accurate to reality (reliable ≠ valid)
Generalisability Applies to a broader audience
Precision How specific: 106 vs 106.4 vs 106.43 km

?Self-check quiz

16 questions on this week. Answer them all, then press Check my answers. Your best and latest scores are saved and shown on the subject page.

← PreviousNeedfinding — Understanding Users, Biases & Requirement-Gathering Methods