User Experience Evaluation Methods in Academic

Topics: Usability, Evaluation methods, Qualitative research Pages: 15 (3562 words) Published: October 17, 2012
User Experience Evaluation Methods in Academic and
Industrial Contexts
Virpi Roto1,2, Marianna Obrist3, Kaisa Väänänen-Vainio-Mattila1,2 1


Tampere University of Technology, Human-Centered Technology, Korkeakoulunkatu 6, 33720 Tampere, Finland. [virpi.roto, kaisa.vaananen-vainio-mattila]

Nokia Research Center, P.O.Box 407, 00045 Nokia Group, Finland.

ICT&S Center, University of Salzburg, Sigmund-Haffner-Gasse 18, 5020 Salzburg, Austria.


In this paper, we investigate 30 user experience (UX)
evaluation methods that were collected during a special
interest group session at the CHI2009 Conference. We
present a categorization of the collected UX evaluation
methods and discuss the range of methods from both
academic and industrial perspectives.

User experience, Evaluation methods


As particular industry sectors mature, usability and technical reliability of products is taken for granted and users start to look for products that provide engaging user experience
(UX). Although the term user experience originated from
industry and is a widely used term also in academia, the tools for managing UX in product development are still
inadequate. UX evaluation methods play a key role in
ensuring that product development is going to the right
Many methods exist for doing traditional usability
evaluations, but user experience (UX) evaluation differs
clearly from usability evaluation. Whereas usability
emphasizes effectiveness and efficiency [5], UX includes
hedonic characteristics in addition to the pragmatic ones
[6], and is thus subjective [3,12]. Therefore, UX cannot be
evaluated with stopwatches or logging. The objective
measures such as task execution time and the number of
clicks or errors are not valid measures for UX, but we need
to understand how the user feels about the system. User’s
motivation and expectations affect the experience more
than in traditional usability [13].
User experience is also very context-dependent [12], so the
experience with the same design in different circumstances
is often very different. This means that UX evaluation
should not be conducted just by observing user’s task
completion in a laboratory test, but it needs to take into
account a broader set of factors.

In order to gain more insights on globally used UX
evaluation methods, we organized a Special Interest Group
(SIG) session on “User Experience Evaluation – Do you
know which Method to use?” at the CHI’09 conference
[16]. The objective of the CHI’09 SIG was to gather
evaluation methods that provide information on how users
feel about using a designed system, in addition to the
efficiency and effectiveness of using the system. This is a
common requirement for all UX evaluation methods, but
various kinds of methods need to be designed for different
cases. Specific, various methods are often needed for
academic and industrial contexts and a toolkit of methods
would help in finding the proper method for each case. In
this paper, we present and discuss the results from the SIG
session in particular with the goal to answer a research
question “What are the recently used and known UX
evaluation methods in industry and academia?”

The data on UX evaluation methods were collected during
a 1.5 hour SIG session at CHI’09. The audience included
about 35 conference participants, equally representing
academia and industry. The goal of the session was to
collect UX evaluation methods used or known in industry
and academia. In the opening plenary, we explained what
we mean by UX evaluation, emphasizing the
hedonic/emotional nature of UX in addition to the primarily
pragmatic nature of usability. We also highlighted the
temporal aspects related to UX. We presented three
examples of UX evaluation methods with our pre-defined
method template:...

References: 1. Amaldi, P., Gill, S., G., Fields, B., Wong, W., (Eds.),
ceedings.doc (accessed: 26.06. 2009).
(2008). Instant Card Technique: How and Why to apply
in User-Centered Design
3. Desmet, P.M.A. (2002). Designing Emotions. Doctoral
dissertation, Technical University of Delft.
interactive systems (formerly known as 13407).
6. Hassenzahl, M. 2004. The interplay of beauty,
goodness, and usability in interactive products
7. Hazlett, R., L., (2006). Measuring emotional valence
during interactive experiences: boys at video game play,
8. Fields, B., Amaldi, P., Wong, W., Gill, S., (2007).
Methods in Product Development workshop. Uppsala,
Sweden, August 25, 2009.
Conference on Human Factors in Computing Systems
(Florence, Italy, April 05 - 10, 2008)
Continue Reading

Please join StudyMode to read the full document

You May Also Find These Documents Helpful

  • Evaluation of Differential Research Methods Essay
  • Heuristic Evaluation of User Interfaces Essay
  • Evaluation of Experience Essay
  • Personality Methods and Evaluation Essay
  • User Sampling Methods Essay
  • Evaluation Research Methods Essay
  • My Academic Experience Essay
  • Job Evaluation Method Essay

Become a StudyMode Member

Sign Up - It's Free