EURASIP Journal on Image and Video Processing
Volume 2008, Article ID 824726, 30 pages
A Review and Comparison ofMeasures for
Automatic Video Surveillance Systems
Axel Baumann, Marco Boltz, Julia Ebling, Matthias Koenig, Hartmut S. Loos, Marcel Merkel, Wolfgang Niem, Jan KarlWarzelhan, and Jie Yu
Corporate Research, Robert Bosch GmbH, D-70049 Stuttgart, Germany Correspondence should be addressed to Julia Ebling, firstname.lastname@example.org Received 30 October 2007; Revised 28 February 2008; Accepted 12 June 2008 Recommended by Andrea Cavallaro
Today’s video surveillance systems are increasingly equipped with video content analysis for a great variety of applications. However, reliability and robustness of video content analysis algorithms remain an issue. They have to be measured against ground truth data in order to quantify the performance and advancements of new algorithms. Therefore, a variety of measures have been proposed in the literature, but there has neither been a systematic overview nor an evaluation of measures for specific video analysis tasks yet. This paper provides a systematic review of measures and compares their effectiveness for specific aspects, such as segmentation, tracking, and event detection. Focus is drawn on details like normalization issues, robustness, and representativeness. A software framework is introduced for continuously evaluating and documenting the performance of video surveillance systems. Based on many years of experience, a new set of representative measures is proposed as a fundamental part of an evaluation framework.
Copyright © 2008 Axel Baumann et al. This is an open access article distributed under the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited. 1. INTRODUCTION
The installation of videosurveillance systems is driven by the need to protect privateproperties, and by crime prevention,
detection, and prosecution, particularly for terrorism in
public places. However, the effectiveness of surveillance
systems is still disputed . One effect which is thereby often mentioned is that of crime dislocation. Another problem is
that the rate of crime detection using surveillance systems is not known. However, they have become increasingly useful
in the analysis and prosecution of known crimes.
Surveillance systems operate 24 hours a day, 7 days a
week. Due to the large number of cameras which have to
be monitored at large sites, for example, industrial plants, airports, and shopping areas, the amount of information to
be processedmakes surveillance a tedious job for the security personnel . Furthermore, since most of the time video
streams show ordinary behavior, the operator may become
inattentive, resulting in missing events.
In the last few years, a large number of automatic realtime
video surveillance systems have been proposed in the
literature  as well as developed and sold by companies.
The idea is to automatically analyze video streams and alert operators of potentially relevant security events. However,
the robustness of these algorithms as well as their performance is difficult to judge. When algorithms produce too
many errors, they will be ignored by the operator, or even
distract the operator from important events.
During the last few years, several performance evaluation
projects for video surveillance systems have been undertaken [3–9], each with different intentions. CAVIAR  addresses city center surveillance and retail applications. VACE 
has a wide spectrum including the processing of meeting
videos and broadcasting news. PETS workshops  focus
on advanced algorithms and evaluation tasks like multiple
object detection and event recognition. CLEAR  deals with people tracking and identification as well as pose estimation and face...