Camera Image Quality Benchmarking: (The Wiley-IS&T Series in Imaging Science and Technology)

The essential guide to the entire process behind performing a complete characterization and benchmarking of cameras through image quality analysis Camera Image Quality Benchmarking contains the basic information and approaches for the use of subjectively correlated image quality metrics and outlines a framework for camera benchmarking.  The authors show how to quantitatively compare image quality of cameras used for consumer photography. This book helps to fill a void in the literature by detailing the types of objective and subjective metrics that are fundamental to benchmarking still and video imaging devices. Specifically, the book provides an explanation of individual image quality attributes and how they manifest themselves to camera components and explores the key photographic still and video image quality metrics. The text also includes illustrative examples of benchmarking methods so that the practitioner can design a methodology appropriate to the photographic usage in consideration. The authors outline the various techniques used to correlate the measurement results from the objective methods with subjective results. The text also contains a detailed description on how to set up an image quality characterization lab, with examples where the methodological benchmarking approach described has been implemented successfully. This vital resource: Explains in detail the entire process behind performing a complete characterization and benchmarking of cameras through image quality analysis Provides best practice measurement protocols and methodologies, so readers can develop and define their own camera benchmarking system to industry standards Includes many photographic images and diagrammatical illustrations to clearly convey image quality concepts Champions benchmarking approaches that value the importance of perceptually correlated image quality metrics  Written for image scientists, engineers, or managers involved in image quality and evaluating camera performance, Camera Image Quality Benchmarking combines knowledge from many different engineering fields, correlating objective (perception-independent) image quality with subjective (perception-dependent) image quality metrics. 

Table of Contents:
About the Authors xv Series Preface xvii Preface xix List of Abbreviations xxiii About the CompanionWebsite xxvii 1 Introduction 1 1.1 Image Content and Image Quality 2 1.1.1 Color 3 1.1.2 Shape 8 1.1.3 Texture 10 1.1.4 Depth 11 1.1.5 Luminance Range 12 1.1.6 Motion 15 1.2 Benchmarking 18 1.3 Book Content 22 Summary of this Chapter 24 References 25 2 Defining Image Quality 27 2.1 What is Image Quality? 27 2.2 Image Quality Attributes 29 2.3 Subjective and Objective Image Quality Assessment 31 Summary of this Chapter 32 References 33 3 Image Quality Attributes 35 3.1 Global Attributes 35 3.1.1 Exposure, Tonal Reproduction, and Flare 35 3.1.2 Color 39 3.1.3 Geometrical Artifacts 40 3.1.3.1 Perspective Distortion 40 3.1.3.2 Optical Distortion 42 3.1.3.3 Other Geometrical Artifacts 42 3.1.4 Nonuniformities 43 3.1.4.1 Luminance Shading 45 3.1.4.2 Color Shading 45 3.2 Local Attributes 45 3.2.1 Sharpness and Resolution 45 3.2.2 Noise 49 3.2.3 Texture Rendition 50 3.2.4 Color Fringing 50 3.2.5 Image Defects 51 3.2.6 Artifacts 51 3.2.6.1 Aliasing and Demosaicing Artifacts 52 3.2.6.2 Still Image Compression Artifacts 53 3.2.6.3 Flicker 53 3.2.6.4 HDR Processing Artifacts 55 3.2.6.5 Lens Ghosting 55 3.3 Video Quality Attributes 56 3.3.1 Frame Rate 56 3.3.2 Exposure and White Balance Responsiveness and Consistency 58 3.3.3 Focus Adaption 58 3.3.4 Audio-Visual Synchronization 58 3.3.5 Video Compression Artifacts 59 3.3.6 Temporal Noise 60 3.3.7 Fixed Pattern Noise 60 3.3.8 Mosquito Noise 60 Summary of this Chapter 60 References 61 4 The Camera 63 4.1 The Pinhole Camera 63 4.2 Lens 64 4.2.1 Aberrations 64 4.2.1.1 Third-Order Aberrations 65 4.2.1.2 Chromatic Aberrations 66 4.2.2 Optical Parameters 67 4.2.3 Relative Illumination 69 4.2.4 Depth of Field 70 4.2.5 Diffraction 71 4.2.6 Stray Light 73 4.2.7 Image Quality Attributes Related to the Lens 74 4.3 Image Sensor 75 4.3.1 CCD Image Sensors 75 4.3.2 CMOS Image Sensors 77 4.3.3 Color Imaging 81 4.3.4 Image Sensor Performance 82 4.3.5 CCD versus CMOS 89 4.3.6 Image Quality Attributes Related to the Image Sensor 90 4.4 Image Signal Processor 91 4.4.1 Image Processing 91 4.4.2 Image Compression 98 4.4.2.1 Chroma Subsampling 98 4.4.2.2 Transform Coding 98 4.4.2.3 Coefficient Quantization 99 4.4.2.4 Coefficient Compression 100 4.4.3 Control Algorithms 101 4.4.4 Image Quality Attributes Related to the ISP 101 4.5 Illumination 102 4.5.1 LED Flash 103 4.5.2 Xenon Flash 103 4.6 Video Processing 103 4.6.1 Video Stabilization 103 4.6.1.1 Global Motion Models 104 4.6.1.2 Global Motion Estimation 105 4.6.1.3 Global Motion Compensation 106 4.6.2 Video Compression 107 4.6.2.1 Computation of Residuals 107 4.6.2.2 Video Compression Standards and Codecs 109 4.6.2.3 Some Significant Video Compression Standards 110 4.6.2.4 A Note On Video Stream Structure 111 4.7 System Considerations 111 Summary of this Chapter 112 References 113 5 Subjective Image Quality Assessment—Theory and Practice 117 5.1 Psychophysics 118 5.2 Measurement Scales 120 5.3 PsychophysicalMethodologies 122 5.3.1 Rank Order 123 5.3.2 Category Scaling 123 5.3.3 Acceptability Scaling 124 5.3.4 Anchored Scaling 125 5.3.5 Forced-Choice Comparison 125 5.3.6 Magnitude Estimation 125 5.3.7 Methodology Comparison 126 5.4 Cross-Modal Psychophysics 126 5.4.1 Example Research 127 5.4.2 Image Quality-Related Demonstration 128 5.5 Thurstonian Scaling 129 5.6 Quality Ruler 131 5.6.1 Ruler Generation 134 5.6.2 Quality Ruler Insights 135 5.6.2.1 Lab Cross-Comparisons 135 5.6.2.2 SQS2 JND Validation 136 5.6.2.3 Quality Ruler Standard Deviation Trends 139 5.6.2.4 Observer Impact 141 5.6.3 Perspective from Academia 142 5.6.4 Practical Example 144 5.6.5 Quality Ruler Applications to Image Quality Benchmarking 147 5.7 Subjective Video Quality 148 5.7.1 Terminology 149 5.7.2 Observer Selection 149 5.7.3 Viewing Setup 150 5.7.4 Video Display and Playback 151 5.7.5 Clip Selection 152 5.7.6 Presentation Protocols 154 5.7.7 Assessment Methods 156 5.7.8 Interpreting Results 158 5.7.9 ITU Recommendations 159 5.7.9.1 The Double-Stimulus Impairment Scale Method 160 5.7.9.2 The Double-Stimulus Continuous Quality Scale Method 160 5.7.9.3 The Simultaneous Double-Stimulus for Continuous Evaluation Method 160 5.7.9.4 The Absolute Category Rating Method 161 5.7.9.5 The Single Stimulus Continuous Quality Evaluation Method 161 5.7.9.6 The Subjective Assessment of Multimedia Video Quality Method 161 5.7.9.7 ITU Methodology Comparison 162 5.7.10 Other Sources 162 Summary of this Chapter 162 References 163 6 Objective Image Quality Assessment—Theory and Practice 167 6.1 Exposure and Tone 168 6.1.1 Exposure Index and ISO Sensitivity 168 6.1.2 Optoelectronic Conversion Function 169 6.1.3 Practical Considerations 170 6.2 Dynamic Range 170 6.3 Color 171 6.3.1 Light Sources 171 6.3.2 Scene 174 6.3.3 Observer 176 6.3.4 Basic Color Metrics 178 6.3.5 RGB Color Spaces 180 6.3.6 Practical Considerations 181 6.4 Shading 181 6.4.1 Practical Considerations 182 6.5 Geometric Distortion 182 6.5.1 Practical Considerations 184 6.6 Stray Light 184 6.6.1 Practical Considerations 185 6.7 Sharpness and Resolution 185 6.7.1 The Modulation Transfer Function 186 6.7.2 The Contrast Transfer Function 191 6.7.3 Geometry in Optical Systems and the MTF 193 6.7.4 Sampling and Aliasing 194 6.7.5 System MTF 195 6.7.6 Measuring the MTF 198 6.7.7 Edge SFR 198 6.7.8 Sine Modulated Siemens Star SFR 201 6.7.9 Comparing Edge SFR and Sine Modulated Siemens SFR 203 6.7.10 Practical Considerations 204 6.8 Texture Blur 204 6.8.1 Chart Construction 206 6.8.2 Practical Considerations 206 6.8.3 AlternativeMethods 207 6.9 Noise 207 6.9.1 Noise and Color 207 6.9.2 Spatial Frequency Dependence 209 6.9.3 Signal to Noise Measurements in Nonlinear Systems and Noise Component Analysis 211 6.9.4 Practical Considerations 212 6.10 Color Fringing 213 6.11 Image Defects 214 6.12 Video Quality Metrics 214 6.12.1 Frame Rate and Frame Rate Consistency 215 6.12.2 Frame Exposure Time and Consistency 215 6.12.3 Auto White Balance Consistency 216 6.12.4 Autofocusing Time and Stability 216 6.12.5 Video Stabilization Performance 217 6.12.6 Audio-Video Synchronization 218 6.13 Related International Standards 218 Summary of this Chapter 221 References 221 7 Perceptually Correlated Image Quality Metrics 227 7.1 Aspects of Human Vision 227 7.1.1 Physiological Processes 227 7.2 HVS Modeling 232 7.3 Viewing Conditions 232 7.4 Spatial Image Quality Metrics 234 7.4.1 Sharpness 235 7.4.1.1 Edge Acutance 235 7.4.1.2 Mapping Acutance to JND Values 237 7.4.1.3 Other Perceptual Sharpness Metrics 239 7.4.2 Texture Blur 239 7.4.3 Visual Noise 240 7.5 Color 244 7.5.1 Chromatic Adaptation Transformations 244 7.5.2 Color Appearance Models 245 7.5.3 Color and Spatial Content—Image Appearance Models 247 7.5.4 Image Quality Benchmarking and Color 249 7.6 Other Metrics 251 7.7 Combination of Metrics 252 7.8 Full-Reference Digital Video Quality Metrics 252 7.8.1 PSNR 253 7.8.2 Structural Similarity (SSIM) 256 7.8.3 VQM 260 7.8.4 VDP 262 7.8.4.1 Further Considerations 263 7.8.5 Discussion 265 Summary of this Chapter 267 References 267 8 Measurement Protocols—Building Up a Lab 273 8.1 Still Objective Measurements 273 8.1.1 Lab Needs 274 8.1.1.1 Lab Space 274 8.1.1.2 Lighting 275 8.1.1.3 Light Booths 278 8.1.1.4 Transmissive Light Sources 279 8.1.1.5 Additional Lighting Options 280 8.1.1.6 Light Measurement Devices 281 8.1.2 Charts 282 8.1.2.1 Printing Technologies for Reflective Charts 282 8.1.2.2 Technologies for Transmissive Charts 286 8.1.2.3 Inhouse Printing 286 8.1.2.4 Chart Alignment and Framing 287 8.1.3 Camera Settings 289 8.1.4 Supplemental Equipment 289 8.1.4.1 RealWorld Objects 290 8.2 Video Objective Measurements 293 8.2.0.2 Visual Timer 293 8.2.0.3 Motion 294 8.3 Still Subjective Measurements 297 8.3.1 Lab Needs 297 8.3.2 Stimuli 298 8.3.2.1 Stimuli Generation 298 8.3.2.2 Stimuli Presentation 301 8.3.3 Observer Needs 302 8.3.3.1 Observer Selection and Screening 302 8.3.3.2 Experimental Design and Duration 303 8.4 Video Subjective Measurements 304 Summary of this Chapter 305 References 305 9 The Camera Benchmarking Process 309 9.1 Objective Metrics for Benchmarking 309 9.2 Subjective Methods for Benchmarking 311 9.2.1 Photospace 312 9.2.2 Use Cases 313 9.2.3 Observer Impact 314 9.3 Methods of Combining Metrics 315 9.3.1 Weighted Combinations 316 9.3.2 Minkowski Summation 316 9.4 Benchmarking Systems 317 9.4.1 GSMArena 317 9.4.2 FNAC 318 9.4.3 VCX 318 9.4.4 Skype Video Capture Specification 319 9.4.5 VIQET 320 9.4.6 DxOMark 321 9.4.7 IEEE P1858 323 9.5 Example Benchmark Results 324 9.5.1 VIQET 324 9.5.2 IEEE CPIQ 325 9.5.2.1 CPIQ Objective Metrics 327 9.5.2.2 CPIQ Quality Loss Predictions from Objective Metrics 337 9.5.3 DxOMark Mobile 338 9.5.4 Real-World Images 339 9.5.5 High-End DSLR Objective Metrics 339 9.6 Benchmarking Validation 345 Summary of this Chapter 348 References 349 10 Summary and Conclusions 353 References 357 Index 359

About the Author :
JONATHAN B. PHILLIPS, is a Staff Image Scientist at Google, USA. He is a United States delegate to the technical committee ISO/TC 42 Photography and a major contributor to the IEEE Camera Phone Image Quality (CPIQ) initiative. HENRIK ELIASSON, PHD, is an image sensor and image analysis specialist at Eclipse Optics, Sweden. He is a senior member of SPIE.