User Experience Evaluation Methods in Academic

Only available on StudyMode
  • Download(s) : 93
  • Published : October 17, 2012
Open Document
Text Preview
User Experience Evaluation Methods in Academic and
Industrial Contexts
Virpi Roto1,2, Marianna Obrist3, Kaisa Väänänen-Vainio-Mattila1,2 1


Tampere University of Technology, Human-Centered Technology, Korkeakoulunkatu 6, 33720 Tampere, Finland. [virpi.roto, kaisa.vaananen-vainio-mattila]

Nokia Research Center, P.O.Box 407, 00045 Nokia Group, Finland.

ICT&S Center, University of Salzburg, Sigmund-Haffner-Gasse 18, 5020 Salzburg, Austria.


In this paper, we investigate 30 user experience (UX)
evaluation methods that were collected during a special
interest group session at the CHI2009 Conference. We
present a categorization of the collected UX evaluation
methods and discuss the range of methods from both
academic and industrial perspectives.

User experience, Evaluation methods


As particular industry sectors mature, usability and technical reliability of products is taken for granted and users start to look for products that provide engaging user experience
(UX). Although the term user experience originated from
industry and is a widely used term also in academia, the tools for managing UX in product development are still
inadequate. UX evaluation methods play a key role in
ensuring that product development is going to the right
Many methods exist for doing traditional usability
evaluations, but user experience (UX) evaluation differs
clearly from usability evaluation. Whereas usability
emphasizes effectiveness and efficiency [5], UX includes
hedonic characteristics in addition to the pragmatic ones
[6], and is thus subjective [3,12]. Therefore, UX cannot be
evaluated with stopwatches or logging. The objective
measures such as task execution time and the number of
clicks or errors are not valid measures for UX, but we need
to understand how the user feels about the system. User’s
motivation and expectations affect the experience more
than in traditional usability [13].
User experience is also very context-dependent [12], so the
experience with the same design in different circumstances
is often very different. This means that UX evaluation
should not be conducted just by observing user’s task
completion in a laboratory test, but it needs to take into
account a broader set of factors.

In order to gain more insights on globally used UX
evaluation methods, we organized a Special Interest Group
(SIG) session on “User Experience Evaluation – Do you
know which Method to use?” at the CHI’09 conference
[16]. The objective of the CHI’09 SIG was to gather
evaluation methods that provide information on how users
feel about using a designed system, in addition to the
efficiency and effectiveness of using the system. This is a
common requirement for all UX evaluation methods, but
various kinds of methods need to be designed for different
cases. Specific, various methods are often needed for
academic and industrial contexts and a toolkit of methods
would help in finding the proper method for each case. In
this paper, we present and discuss the results from the SIG
session in particular with the goal to answer a research
question “What are the recently used and known UX
evaluation methods in industry and academia?”

The data on UX evaluation methods were collected during
a 1.5 hour SIG session at CHI’09. The audience included
about 35 conference participants, equally representing
academia and industry. The goal of the session was to
collect UX evaluation methods used or known in industry
and academia. In the opening plenary, we explained what
we mean by UX evaluation, emphasizing the
hedonic/emotional nature of UX in addition to the primarily
pragmatic nature of usability. We also highlighted the
temporal aspects related to UX. We presented three
examples of UX evaluation methods with our pre-defined
method template:...
tracking img