Performance evaluation is a critical stage of software- and hardware-system development that every computer engineer and scientist should master. Although complex – requiring skills in mathematics, measurement techniques and simulation – performance evaluation is primarily an art; indeed, the most difficult stage in a performance analysis is defining the approach: once you know what to do, it is less difficult to define a plan of attack with your familiar software tools. We present a set of topics, which we believe should be part of every engineer's intellectual toolkit. This includes the statistical exploitation of numerical results in an efficient and ethical way, for example: how to summarize variability or fairness; what transient removal in a simulation is; and how to make predictions from a time series. We also present well-known performance patterns, which helps to quickly bring the engineer to the main issues. For queuing theory, we focus on a subset of very useful results, such as operational laws. A highlight of the book is the development of Palm calculus, also called ìthe importance of the viewpoint,î which is central to queuing theory. Indeed, this topic has so many applications to simulation and to system analysis in general that it is a very good time investment. This book began as a set of lecture notes for a course given at EPFL.
Table of Contents:
Methodology
What is Performance Evaluation ?
Factors
Evaluation Methods
The Scientific Method
Performance Patterns
Summarizing Performance Data, Confidence Intervals
Summarized Performance Data
Confidence Intervals
The Independence Assumption
Prediction Interval
Which Summarization To Use?
Other Aspects of Confidence/Prediction Intervals
Proofs
Model Fitting
Model Fitting Criteria
Linear Regression
Linear Regression with Norm Minimization
Choosing a Distribution
Heavy Tail
Proofs
Tests
The Neyman Pearson Framework
Likelihood Ratio Tests
ANOVA
Asymptotic Results
Other Tests
Proofs
Forecasting
What is Forecasting ?
Linear Regression
The Overfitting Problem
Differencing the Data
Fitting Differenced Data to an ARMA Model
Sparse ARMA and ARIMA Models
Proofs
Discrete Event Simulation
What is a Simulation?
Simulation Tehniques
Computing the Accuracy of Stochastic Simulations
Monte Carlo Simulation
Random Number Generators
How to Sample from a Distribution
Importance Sampling
Proofs
Palm Calculus, or the Importance of the Viewpoint
An Informal Introduction
Palm Calculus
Other Useful Palm Calculus Results
Simulation Defined as Stochastic Recurrence
Application to Markov Chain Models and the PASTA Property
Appendix: Quick Review of Markov Chains
Proofs
Review Questions
Queuing Theory for Those Who Cannot Wait
Deterministic Analysis
Operational Laws For Queuing Systems
Classical Results for a Single Queue
Definitions for Queuing Networks
The Product-Form Theorem
Computational Aspects
What This Tells Us
Mathematical Details About Product-Form Queuing Networks
Case Study
Proofs
Each chapter concludes with a Review that includes review questions.
About the Author :
Jean-Yves Le Boudec is professor at EPFL and fellow of the IEEE. He graduated from Ecole Normale Superieure de Saint-Cloud, Paris, and received his doctorate in 1984 from the University of Rennes, France. Before joining the EPFL in 1994, he had worked at Bell Northern Research, Ottawa, Canada; and at the IBM Zurich Research Laboratory where he was manager of the Customer Premises Network Department. His interests are in the performance and architecture of communication systems. He is recipient of numerous awards, including the 2008 IEEE Communications Society William R. Bennett Prize in the field of Communications Networking.
Review :
… written by a scientist successful in performance evaluation, it is based on his experience and provides many ideas not only to laymen entering the field, but also to practitioners looking for inspiration. The work can be read systematically as a textbook on how to model and test the derived hypotheses on the basis of simulations. Also, separate parts can be studied, as the chapters are self-contained. … the book can be successfully used either for self-study or as a supplementary book for a lecture. I believe that different types of readers will like it: practicing engineers and researchers dealing with new solutions, as well as graduate students starting their adventures in the jungle of performance evaluation.
—Piotr Cholda, in IEEE Communications Magazine, October 2011
