Biologically Inspired Computer Vision: Fundamentals and Applications

As the state-of-the-art imaging technologies became more and more advanced, yielding scientific data at unprecedented detail and volume, the need to process and interpret all the data has made image processing and computer vision increasingly important. Sources of data that have to be routinely dealt with today's applications include video transmission, wireless communication, automatic fingerprint processing, massive databanks, non-weary and accurate automatic airport screening, robust night vision, just to name a few. Multidisciplinary inputs from other disciplines such as physics, computational neuroscience, cognitive science, mathematics, and biology will have a fundamental impact in the progress of imaging and vision sciences. One of the advantages of the study of biological organisms is to devise very different type of computational paradigms by implementing a neural network with a high degree of local connectivity. This is a comprehensive and rigorous reference in the area of biologically motivated vision sensors. The study of biologically visual systems can be considered as a two way avenue. On the one hand, biological organisms can provide a source of inspiration for new computational efficient and robust vision models and on the other hand machine vision approaches can provide new insights for understanding biological visual systems. Along the different chapters, this book covers a wide range of topics from fundamental to more specialized topics, including visual analysis based on a computational level, hardware implementation, and the design of new more advanced vision sensors. The last two sections of the book provide an overview of a few representative applications and current state of the art of the research in this area. This makes it a valuable book for graduate, Master, PhD students and also researchers in the field.

Table of Contents:
List of Contributors XV Foreword XIX Part I Fundamentals 1 1 Introduction 3 Gabriel Cristóbal, Laurent U. Perrinet, and Matthias S. Keil 1.1 Why ShouldWe Be Inspired by Biology? 4 1.2 Organization of Chapters in the Book 6 1.3 Conclusion 9 Acknowledgments 9 References 9 2 Bioinspired Vision Sensing 11 Christoph Posch 2.1 Introduction 11 2.1.1 Neuromorphic Engineering 12 2.1.2 Implementing Neuromorphic Systems 13 2.2 Fundamentals and Motivation: Bioinspired Artificial Vision 13 2.2.1 Limitations in Vision Engineering 14 2.2.2 The Human Retina from an Engineering Viewpoint 14 2.2.3 Modeling the Retina in Silicon 17 2.3 From Biological Models to Practical Vision Devices 18 2.3.1 TheWiring Problem 18 2.3.2 Where and What 20 2.3.3 Temporal Contrast: The DVS 21 2.3.4 Event-Driven Time-Domain Imaging: The ATIS 22 2.4 Conclusions and Outlook 25 References 26 3 Retinal Processing: From Biology to Models and Applications 29 David Alleysson and Nathalie Guyader 3.1 Introduction 29 3.2 Anatomy and Physiology of the Retina 30 3.2.1 Overview of the Retina 30 3.2.2 Photoreceptors 31 3.2.3 Outer and Inner Plexiform Layers (OPL and IPL) 33 3.2.4 Summary 34 3.3 Models of Vision 34 3.3.1 Overview of the Retina Models 34 3.3.2 Biological Models 35 3.3.2.1 Cellular and Molecular Models 35 3.3.2.2 Network Models 36 3.3.2.3 Parallel and Descriptive Models 38 3.3.3 Information Models 39 3.3.4 Geometry Models 40 3.4 Application to Digital Photography 42 3.4.1 Color Demosaicing 43 3.4.2 Color Constancy – Chromatic Adaptation – White Balance – Tone Mapping 44 3.5 Conclusion 45 References 46 4 Modeling Natural Image Statistics 53 Holly E. Gerhard, Lucas Theis, and Matthias Bethge 4.1 Introduction 53 4.2 Why Model Natural Images? 53 4.3 Natural Image Models 55 4.3.1 Model Evaluation 65 4.4 Computer Vision Applications 69 4.5 Biological Adaptations to Natural Images 71 4.6 Conclusions 75 References 76 5 Perceptual Psychophysics 81 C. Alejandro Parraga 5.1 Introduction 81 5.1.1 What Is Psychophysics andWhy DoWe Need It? 81 5.2 Laboratory Methods 82 5.2.1 Accuracy and Precision 83 5.2.2 Error Propagation 84 5.3 PsychophysicalThreshold Measurement 85 5.3.1 Weber’s Law 85 5.3.2 Sensitivity Functions 86 5.4 Classic Psychophysics:Theory and Methods 86 5.4.1 Theory 87 5.4.2 Method of Constant Stimuli 88 5.4.3 Method of Limits 90 5.4.3.1 Forced-Choice Methods 92 5.4.4 Method of Adjustments 93 5.4.5 Estimating Psychometric Function Parameters 94 5.5 Signal Detection Theory 94 5.5.1 Signal and Noise 94 5.5.2 The Receiver Operating Characteristic 96 5.6 Psychophysical Scaling Methods 98 5.6.1 Discrimination Scales 99 5.6.2 Rating Scales 100 5.6.2.1 Equipartition Scales 100 5.6.2.2 Paired Comparison Scales 101 5.7 Conclusions 105 References 106 Part II Sensing 109 6 Bioinspired Optical Imaging 111 Mukul Sarkar 6.1 Visual Perception 111 6.1.1 Natural Single-Aperture and Multiple-Aperture Eyes 112 6.1.1.1 Human Eyes 113 6.1.1.2 Compound Eyes 114 6.1.1.3 Resolution 114 6.1.1.4 Visual Acuity 115 6.1.1.5 Depth Perception 115 6.1.1.6 Color Vision 116 6.1.1.7 Speed of Imaging and Motion Detection 117 6.1.1.8 Polarization Vision 117 6.2 Polarization Vision - Object Differentiation/Recognition 119 6.2.1 Polarization of Light 121 6.2.2 Polarization Imaging 125 6.2.2.1 Detection of Transparent and Opaque Objects 126 6.2.2.2 Shape Detection Using Polarized Light 131 6.3 High-Speed Motion Detection 133 6.3.1 Temporal Differentiation 135 6.4 Conclusion 138 References 139 7 Biomimetic Vision Systems 143 Reinhard Voelkel 7.1 Introduction 143 7.2 Scaling Laws in Optics 144 7.2.1 Optical Properties of Imaging Lens Systems 144 7.2.2 Space Bandwidth Product (SW) 146 7.3 The Evolution of Vision Systems 147 7.3.1 Single-Aperture Eyes 148 7.3.2 Compound Eyes 149 7.3.3 The Array Optics Concept 150 7.3.4 Jumping Spiders: Perfect Eyes for Small Animals 151 7.3.5 Nocturnal Spiders: Night Vision 153 7.4 Manufacturing of Optics for Miniaturized Vision Systems 154 7.4.1 Optics Industry 154 7.4.2 Planar Array Optics for Stereoscopic Vision 155 7.4.3 The Lack of a Suitable Fabrication Technology Hinders Innovation 155 7.4.4 Semiconductor Industry PromotesWafer-Based Manufacturing 156 7.4.5 Image Sensors 156 7.4.6 Wafer-Based Manufacturing of Optics 158 7.4.7 Manufacturing of Microlens Arrays onWafer Level 159 7.4.8 Diffractive Optical Elements and Subwavelength Structures for Antireflection Coatings 160 7.4.9 Microlens Imprint and Replication Processes 162 7.4.10 Wafer-Level Stacking or Packaging (WLP) 162 7.4.11 Wafer-Level Camera (WLC) 163 7.5 Examples for Biomimetic Compound Vision Systems 164 7.5.1 Ultraflat Cameras 165 7.5.2 Biologically Inspired Vision Systems for Smartphone Cameras 166 7.5.3 PiCam Cluster Camera 167 7.5.4 Panoramic Motion Camera for Flying Robots 169 7.5.5 Conclusion 170 References 172 8 Plenoptic Cameras 175 Fernando Pérez Nava, Alejandro Pérez Nava, Manuel Rodríguez Valido, and Eduardo Magdaleno Castellò 8.1 Introduction 175 8.2 Light Field Representation of the Plenoptic Function 177 8.2.1 The Plenoptic Function 177 8.2.2 Light Field Parameterizations 178 8.2.3 Light Field Reparameterization 179 8.2.4 Image Formation 180 8.2.5 Light Field Sampling 180 8.2.5.1 Pinhole and Thin Lens Camera 180 8.2.5.2 Multiple Devices 181 8.2.5.3 Temporal Multiplexing 181 8.2.5.4 Frequency Multiplexing 182 8.2.5.5 Spatial Multiplexing 182 8.2.5.6 Simulation 182 8.2.6 Light Field Sampling Analysis 183 8.2.7 Light Field Visualization 183 8.3 The Plenoptic Camera 185 8.4 Applications of the Plenoptic Camera 188 8.4.1 Refocusing 188 8.4.2 Perspective Shift 190 8.4.3 Depth Estimation 190 8.4.4 Extended Depth of Field Images 192 8.4.5 Superresolution 192 8.5 Generalizations of the Plenoptic Camera 193 8.6 High-Performance Computing with Plenoptic Cameras 195 8.7 Conclusions 196 References 197 Part III Modelling 201 9 Probabilistic Inference and Bayesian Priors in Visual Perception 203 Grigorios Sotiropoulos and Peggy Seriès 9.1 Introduction 203 9.2 Perception as Bayesian Inference 204 9.2.1 Deciding on a Single Percept 205 9.3 Perceptual Priors 207 9.3.1 Types of Prior Expectations 207 9.3.2 Impact of Expectations 208 9.3.3 The Slow Speed Prior 210 9.3.4 Expectations and Environmental Statistics 212 9.4 Outstanding Questions 214 9.4.1 Are Long-Term Priors Plastic? 214 9.4.2 How Specific are Priors? 215 9.4.3 Inference in Biological and Computer Vision 215 9.4.4 Conclusions 217 References 219 10 From NeuronalModels to Neuronal Dynamics and Image Processing 221 Matthias S. Keil 10.1 Introduction 221 10.2 The Membrane Equation as a Neuron Model 222 10.2.1 Synaptic Inputs 224 10.2.2 Firing Spikes 227 10.3 Application 1: A Dynamical Retinal Model 230 10.4 Application 2: Texture Segregation 234 10.5 Application 3: Detection of Collision Threats 236 10.6 Conclusions 239 Acknowledgments 240 References 240 11 ComputationalModels of Visual Attention and Applications 245 Olivier LeMeur and Matei Mancas 11.1 Introduction 245 11.2 Models of Visual Attention 246 11.2.1 Taxonomy 246 11.3 A Closer Look at Cognitive Models 250 11.3.1 Itti et al.’s Model [3] 250 11.3.2 Le Meur et al.’s Model [8] 251 11.3.2.1 Motivations 251 11.3.2.2 Global Architecture 252 11.3.3 Limitations 255 11.4 Applications 256 11.4.1 Saliency-Based Applications: A Brief Review 256 11.4.2 Predicting Memorability of Pictures 257 11.4.2.1 Memorability Definition 257 11.4.2.2 Memorability and Eye Movement 257 11.4.2.3 Computational Models 259 11.4.3 Quality Metric 260 11.4.3.1 Introduction 260 11.4.3.2 Eye Movement During a Quality Task 260 11.4.3.3 Saliency-Based Quality Metrics 261 11.5 Conclusion 262 References 262 12 Visual Motion Processing and Human Tracking Behavior 267 Anna Montagnini, Laurent U. Perrinet, and Guillaume S. Masson 12.1 Introduction 267 12.2 Pursuit Initiation: Facing Uncertainties 269 12.2.1 Where Is the Noise? Motion-Tracking Precision and Accuracy 269 12.2.2 Where Is the Target Really Going? 270 12.2.3 Human Smooth Pursuit as Dynamic Readout of the Neural Solution to the Aperture Problem 272 12.3 Predicting Future and On-Going Target Motion 273 12.3.1 Anticipatory Smooth Tracking 273 12.3.2 If You Don’t See It, You Can Still Predict (and Track) It 274 12.4 Dynamic Integration of Retinal and Extra-Retinal Motion Information: Computational Models 276 12.4.1 A Bayesian Approach for Open-Loop Motion Tracking 276 12.4.2 Bayesian (or Kalman-Filtering) Approach for Smooth Pursuit: Hierarchical Models 278 12.4.3 A Bayesian Approach for Smooth Pursuit: Dealing with Delays 279 12.5 Reacting, Inferring, Predicting: A NeuralWorkspace 282 12.6 Conclusion 286 12.6.1 Interest for Computer Vision 287 Acknowledgments 288 References 288 13 Cortical Networks of Visual Recognition 295 Christian Thériault, Nicolas Thome, and Matthieu Cord 13.1 Introduction 295 13.2 Global Organization of the Visual Cortex 296 13.3 Local Operations: Receptive Fields 297 13.4 Local Operations in V1 298 13.5 Multilayer Models 301 13.6 A Basic IntroductoryModel 302 13.7 Idealized Mathematical Model of V1: Fiber Bundle 307 13.8 Horizontal Connections and the Association Field 311 13.9 Feedback and Attentional Mechanisms 312 13.10 Temporal Considerations, Transformations and Invariance 312 13.11 Conclusion 314 References 315 14 Sparse Models for Computer Vision 319 Laurent U. Perrinet 14.1 Motivation 319 14.1.1 Efficiency and Sparseness in Biological Representations of Natural Images 319 14.1.2 Sparseness Induces Neural Organization 320 14.1.3 Outline: Sparse Models for Computer Vision 322 14.2 What Is Sparseness? Application to Image Patches 323 14.2.1 Definitions of Sparseness 323 14.2.2 Learning to Be Sparse:The SparseNet Algorithm 325 14.2.3 Results: Efficiency of Different Learning Strategies 326 14.3 SparseLets: A Multiscale, Sparse, Biologically Inspired Representation of Natural Images 328 14.3.1 Motivation: Architecture of the Primary Visual Cortex 328 14.3.2 The SparseLets Framework 330 14.3.3 Efficiency of the SparseLets Framework 333 14.4 SparseEdges: Introducing Prior Information 336 14.4.1 Using the Prior in First-Order Statistics of Edges 336 14.4.2 Using the Prior Statistics of Edge Co-Occurrences 338 14.5 Conclusion 341 Acknowledgments 341 References 342 15 Biologically Inspired Keypoints 347 Alexandre Alahi, Georges Goetz, and Emmanuel D’Angelo 15.1 Introduction 347 15.2 Definitions 349 15.3 What Does the Frond-End of the Visual System Tell Us? 350 15.3.1 The Retina 350 15.3.2 From Photoreceptors to Pixels 350 15.3.3 Visual Compression 351 15.3.4 Retinal Sampling Pattern 351 15.3.5 Scale-Space Representation 352 15.3.6 Difference of Gaussians as a Model for RGC-Receptive Fields 353 15.3.7 A Linear Nonlinear Model 354 15.3.8 Gabor-Like Filters 354 15.4 Bioplausible Keypoint Extraction 355 15.4.1 Scale-Invariant Feature Transform 355 15.4.2 Speeded-Up Robust Features 356 15.4.3 Center Surround Extrema 356 15.4.4 Features from Accelerated Segment Test 357 15.5 Biologically Inspired Keypoint Representation 357 15.5.1 Motivations 357 15.5.2 Dense Gabor-Like Descriptors 358 15.5.2.1 Scale-Invariant Feature Transform Descriptor 358 15.5.2.2 Speeded-Up Robust Features Descriptor 359 15.5.3 Sparse Gaussian Kernels 360 15.5.3.1 Local Binary Descriptors 360 15.5.3.2 Fast Retina Keypoint Descriptor 360 15.5.3.3 Fast Retina Keypoint Saccadic Matching 362 15.5.4 Fast Retina Keypoint versus Other Local Binary Descriptors 363 15.6 Qualitative Analysis: Visualizing Keypoint Information 363 15.6.1 Motivations 363 15.6.2 Binary Feature Reconstruction: From Bits to Image Patches 364 15.6.2.1 Feature Inversion as an Inverse Problem 364 15.6.2.2 Interest of the Retinal Descriptor 367 15.6.3 From Feature Visualization to Crowd-Sourced Object Recognition 368 15.7 Conclusions 370 References 371 Part IV Applications 375 16 Nightvision Based on a Biological Model 377 Magnus Oskarsson, Henrik Malm, and EricWarrant 16.1 Introduction 377 16.1.1 RelatedWork 378 16.2 Why Is Vision Difficult in Dim Light? 380 16.3 Why Is Digital Imaging Difficult in Dim Light? 382 16.4 Solving the Problem of Imaging in Dim Light 383 16.4.1 Enhancing the Image 385 16.4.1.1 Visual Image Enhancement in the Retina 385 16.4.1.2 Digital Image Enhancement 387 16.4.2 Filtering the Image 388 16.4.2.1 Spatial and Temporal Summation in Higher Visual Processing 388 16.4.2.2 Structure Tensor Filtering of Digital Images 391 16.5 Implementation and Evaluation of the Night-Vision Algorithm 393 16.5.1 Adaptation of Parameters to Noise Levels 394 16.5.2 Parallelization and Computational Aspects 395 16.5.3 Considerations for Color 396 16.5.4 Experimental Results 397 16.6 Conclusions 399 Acknowledgment 400 References 401 17 Bioinspired Motion Detection Based on an FPGA Platform 405 Tim Köhler 17.1 Introduction 405 17.2 A Motion Detection Module for Robotics and Biology 406 17.3 Insect Motion Detection Models 407 17.4 Overview of Robotic Implementations of Bioinspired Motion Detection 412 17.4.1 Field-Programmable Gate Arrays (FPGA) 413 17.5 An FPGA-Based Implementation 414 17.5.1 FPGA-Camera-Module 414 17.5.2 A Configurable Array of EMDs 416 17.6 Experimental Results 419 17.7 Discussion 421 17.8 Conclusion 422 Acknowledgments 423 References 423 18 Visual Navigation in a ClutteredWorld 425 N. Andrew Browning and Florian Raudies 18.1 Introduction 425 18.2 Cues from Optic Flow: Visually Guided Navigation 426 18.3 Estimation of Self-Motion: Knowing Where You Are Going 429 18.4 Object Detection: Understanding What Is in YourWay 434 18.5 Estimation of TTC: Time Constraints from the Expansion Rate 439 18.6 Steering Control: The Importance of Representation 442 18.7 Conclusions 444 Acknowledgments 445 References 445 Index 447

About the Author :
Prof. Gabriel Cristobal is currently a Research Scientist at the Instituto de Optica, Spanish Council for Scientific Research (CSIC). His current research interests are joint representations, vision modelling and multidimensional signal processing. Prof. Cristobal is a Senior Member of the IEEE, member of the Optical Society of America (OSA), EURASIP Spanish liaison officer for the period 2009-2010 and member of the ISO/IEC JTC1/SC29/WG1 (JPEG2000). He is co-editor of the book "Optical and Digital Image Processing" by G. Cristobal, P. Schelkens and H. Thienpont, Wiley VCH, 2011. Prof. Matthias Keil is currently Ramon and Cajal researcher in the Basic Psychology Department of the University of Barcelona (Spain). He received his PhD degree from the University of Ulm (Germany) for proposing a novel architecture for early visual information processing in the human brain. His research interests are centered on computational neuroscience and diffusion-based image processing. Examples of former and current research lines include computational modeling of brightness and lightness perception, tone mapping, time to contact perception, modeling of insect vision, and biologically motivated collision avoidance systems. Dr. Laurent Perrinet is researcher in Computational Neuroscience at the "Institut de Neurosciences de la Timone" at Aix-Marseille Universite, France. His research is focused on bridging the complex dynamics of realistic models of large-scale models of spiking neurons with functional models of low-level vision.