CHALLENGES IN BENCHMARKING OF COMPUTER SYSTEMS
Dept of CSE R&D, East Point College of Engineering & Technology Bangalore, India firstname.lastname@example.org
Abstract Benchmarking plays a critical role in evaluating the performance of systems that are ready for operation. However, with so many benchmarks available, and with absence of standardization, choosing the right benchmark is always not an easy task. Further interpretation of benchmarking result requires statistical analyses. The paper discusses the benchmarks available for different types of workloads and touches upon the challenges facing the performance analysts interested in benchmarking. Key words Computer Systems Performance Evaluation, Benchmarking, Synthetic Benchmarks, Kernel Benchmarks, Application Benchmarks. 1. Introduction Scientific evaluation of the performance of computer systems is needed during : Vendor evaluation, and selecting the hardware and software configuration of any computer system prior to procurement Improvement of performance, reduction of cost etc. Of any computer system System Design when different implementation alternatives are being evaluated.
Modeling using Queuing Networks, Petri Nets  are used when designing a new system and for improving the performance of an existing system (when the system is not completely built). Measuring the performance by running some standard programs referred to as Benchmark Programs are required for comparison of systems that are ready for installation. With the evolution of different types of system architecture such as Cloud Computing, it has become more difficult to compare the performance of various computer systems simply by looking at their specifications. Tests are needed for comparison of different systems to ascertain the suitability of that system for executing the workload of a particular installation. In the area of Computer Systems Performance Evaluation, Benchmarking  is the act of running a computer program, a set of programs, or other operations, in order to assess the relative performance of the system or a subsystem. The steps involved in benchmarking are: Identification of the system whose performance is to be evaluated Determination of the representative workload planned to run on the system Finding the standard benchmark whose behavior is close to the representative workload
Running the benchmark and measuring the performance metrics Making decision after analyzing the results
Benchmarks are designed to mimic a particular type of workload on a system and involves running a set of well known programs which are • • written in a high level language making it portable across different machines representative of some kind of application: systems programming, numerical programming, commercial programming etc.) • representative of workload of computer systems Benchmarking can be the most useful technique for comparison of the performance of various computers, particularly when making a procurement decision, only if it is representative of the workload in a particular installation. While standard benchmarks written in high level languages are available, ensuring that a benchmark is representative of the workload in a particular organization is the real challenge. The challenges lie in choosing the right benchmark as the workload of a computer installation may involve jobs of various types and with divergent resource requirements. Due to these, in some cases decisions of vendor selection, of procurement of hardware and software and its suitability of matching the requirements may not be taken in the most scientific way. Benchmarking is not easy and often involves statistical analysis in order to arrive at correct decision and useful conclusion. Interpretation of benchmarking data is also difficult. Jain in  has highlighted the common mistakes in Benchmarking. A partial list of common challenges in Benchmarking are: Vendors tend to...
Please join StudyMode to read the full document