Pixels & Paintings: Foundations of Computer-assisted Connoisseurship

PIXELS & PAINTINGS “The discussion is firmly grounded in established art historical practices, such as close visual analysis and an understanding of artists’ working methods, and real-world examples demonstrate how computer-assisted techniques can complement traditional approaches.” —Dr. Emilie Gordenker, Director of the Van Gogh Museum The pioneering presentation of computer-based image analysis of fine art, forging a dialog between art scholars and the computer vision community In recent years, sophisticated computer vision, graphics, and artificial intelligence algorithms have proven to be increasingly powerful tools in the study of fine art. These methods—some adapted from forensic digital photography and others developed specifically for art—empower a growing number of computer-savvy art scholars, conservators, and historians to answer longstanding questions as well as provide new approaches to the interpretation of art. Pixels & Paintings provides the first and authoritative overview of the broad range of these methods, which extend from image processing of palette, marks, brush strokes, and shapes up through analysis of objects, poses, style, composition, to the computation of simple interpretations of artworks. This book stresses that computer methods for art analysis must always incorporate the cultural contexts appropriate to the art studies at hand—a blend of humanistic and scientific expertise. Describes powerful computer image analysis methods and their application to problems in the history and interpretation of fine art Discusses some of the art historical lessons and revelations provided by the use of these methods Clarifies the assumptions and applicability of methods and the role of cultural contexts in their use Shows how computation can be used to analyze tens of thousands of artworks to reveal trends and anomalies that could not be found by traditional non-computer methods Pixels & Paintings is essential reading for computer image analysts and graphics specialists, conservators, historians, students, psychologists and the general public interested in the study and appreciation of art.

Table of Contents:
List of Figures xxi List of Tables xlv List of Algorithms xlvii Preface xlix Lorenzo Lotto lviii Giovanni Morelli and the birth of "scientific" connoisseurship lix Overview lxi Intended audience lxii Prerequisites lxiii Acknowledgements lxiv 1 Digital imaging 1 1.1 Introduction 1 1.2 Electromagnetic radiation and light 4 1.3 Interaction of electromagnetic radiation with art materials 7 1.4 Cameras and scanners 9 1.4.1 Cameras 10 1.4.2 Flatbed scanners 11 1.5 Parameters for image acquisition in the visible 12 Billy Pappas 13 1.5.1 Spatial resolution 15 1.5.2 Bit depth 16 1.5.3 Dynamic range and contrast 17 1.6 Reading digital images of art on–screen 18 1.6.1 Reading a digital image of Leonardo's La Bella Principessa 22 Leonardo da Vinci 22 1.7 Infrared photography and reflectography 25 1.8 Ultraviolet imaging 26 1.9 Multispectral and hyperspectral imaging 27 1.9.1 Hyperspectral imaging of the Archimedes Palimpsest 30 1.10 X-radiographic imaging 32 1.11 Fluorescence imaging 35 1.12 Capture of three–dimensional surfaces of art 37 1.12.1 Raking illumination 38 1.12.2 Reflectance transformation imaging (RTI) 40 1.12.3 Stereographic imaging 42 1.13 Optical coherence tomography (OCT) 43 1.14 Raman spectroscopic imaging and X-ray fluorescence imaging 45 1.14.1 Raman spectroscopic imaging (RSI) 45 1.14.2 X-ray fluorescence imaging (XRF) 46 1.15 Summary 47 1.16 Bibliographical remarks 49 2 Image processing 53 2.1 Introduction 53 2.2 Pixel–based image processing 57 2.3 Region–based image processing 61 2.3.1 Linear image processing 62 2.3.2 Nonlinear region–based image processing 63 2.3.3 Color quantization 64 2.3.4 Edge and line detection 69 2.3.5 Dilation and erosion 71 2.3.6 Skeletonization 72 2.4 Inpainting 72 2.5 Feature extraction 74 2.5.1 Keypoint extraction 75 2.5.2 Craquelure and crazing analysis 78 2.5.3 Computational tests for counterproofing by Jan van der Heyden 81 Jan van der Heyden 83 2.6 Segmentation 86 2.6.1 Deep nets for image segmentation 88 2.7 Geometric transformations 95 2.8 Chamfer transform and Chamfer distance 101 2.8.1 Tests for copying of Jan van Eyck's portraits of Niccolò Albergati 103 2.9 Discrete Fourier and wavelet transforms 111 2.9.1 Discrete Fourier transform (DFT) 111 2.9.2 Canvas support weave analysis 114 2.9.3 Discrete wavelet transform (DWT) 116 2.10 Compositing and integrating art images 118 2.10.1 Image compositing 118 2.10.2 Superresolution 119 2.11 Image separation 123 2.12 Summary 123 2.13 Bibliographical remarks 125 3 Color analysis 129 3.1 Introduction 129 3.2 Visible–light spectra and color appearance 132 3.3 Overview of human color vision 133 3.3.1 Properties of color descriptions 134 3.3.2 Opponent color processing and unique hues 137 3.3.3 Humanist descriptions of color 138 3.3.4 Spatial aspects of color perception 139 Josef Albers 140 3.3.5 Color and lightness constancy and brightness perception 141 3.3.6 Quantitative descriptions and additive color mixing 141 3.3.7 Representing artists' palettes 145 3.4 Physics of color in art materials 147 3.4.1 Pigments and color appearance 147 3.5 Representing color arising from mixing paints 151 3.5.1 Identifying pigments in artworks based on spectra 152 3.6 Digital rejuvenation of pigment colors 154 3.6.1 Digital rejuvenation of faded artworks 157 Georges Seurat 158 3.7 Digital cleaning of paintings 160 3.8 Summary 164 3.9 Bibliographical remarks 165 4 Brush stroke and mark analysis 171 4.1 Introduction 171 Cy Twombly 173 4.2 Analysis of printed lines and marks 175 Katsushika Hokusai 178 4.3 Inferring tools from marks 182 Sheila Waters 184 4.3.1 Analysis of brush strokes 185 4.3.2 Segmenting and isolating brush strokes computationally 187 4.3.3 Extracting opaque marks in multiple layers 189 Vincent Willem van Gogh 193 4.3.4 Visual evidence of authorship of Pollock's drip paintings 194 Jackson Pollock 195 4.3.5 Extracting layers of translucent brush strokes 195 4.4 Characterizing the shapes of strokes and marks 203 4.5 Global methods for inferring sequences of marks in paintings 206 4.6 Summary 208 4.7 Bibliographical remarks 208 5 Perspective and geometric analysis 211 5.1 Introduction 211 5.2 Projective geometry 214 5.2.1 The mathematics of projection 216 5.2.2 One–point, two–point, and three–point perspectives 222 5.2.3 Parallel or orthographic perspective in Asian art 223 5.3 Estimating the center of projection 224 5.3.1 Foreshortening and size comparisons of depicted objects 230 Piero della Francesca 231 5.3.2 Cross–ratio analysis 232 5.3.3 Estimating the center of projection from object sizes 234 5.4 Estimating geometric accuracy in artworks 235 5.4.1 Hans Memling's Flower Still-Life 235 Hans Memling 237 5.4.2 The carpet in Lorenzo Lotto's Husband and Wife 238 5.4.3 The chandelier in the Arnolfini Portrait 238 Jan van Eyck 243 5.4.4 Warping Andrea Mantegna's Lamentation of Christ to make consistent perspective 251 5.4.5 Dewarping the murals in Sennedjem's Tomb 252 5.4.6 Warping de Chirico's Ariadne to make consistent perspective 255 Giorgio de Chirico 256 5.4.7 Robert Campin and workshop's Mérode Altarpiece 257 Robert Campin 258 5.5 Slant anamorphic art 260 Ed Ruscha (Edward Joseph Ruscha IV) 260 5.5.1 Hans Holbein's The Ambassadors 263 Hans Holbein 263 5.6 Inferring depth from projected images 264 5.6.1 Computing a three–dimensional model from one perspective image 265 Masaccio 266 5.6.2 Computing a three–dimensional model from two perspective images 267 5.7 Summary 271 5.8 Bibliographical remarks 272 6 Optical analysis 275 6.1 Introduction 275 6.2 Reflection and refraction 277 6.3 Plane mirrors 278 6.3.1 Virtual image formation by plane mirrors 279 6.3.2 Depictions of plane mirrors in art 281 6.3.3 Diego Velázquez’s Las Meninas 283 Diego Velázquez 284 6.4 Convex spherical mirrors 288 6.4.1 Virtual image formation by convex spherical mirrors 290 6.4.2 Jan van Eyck’s Portrait of Giovanni Arnolfini and his Wife 292 6.4.3 Claude glass 297 6.4.4 Parmigianino’s Self–Portrait in a Convex Mirror 298 Parmigianino (Girolamo Francesco Maria Mazzola) 298 6.4.5 Hans Memling's Virgin and Child and Maarten van Nieuwenhove 304 6.4.6 Dewarping images in generalized cylindrical mirrors 308 6.5 Conical and cylindrical mirrors and anamorphic art 312 6.5.1 Conical mirror anamorphic art 313 6.5.2 Cylindrical mirror anamorphic art 317 6.6 Concave spherical mirrors 318 6.6.1 Virtual image formation by concave mirrors 320 6.6.2 Real image formation by concave mirrors 322 6.7 Converging lenses 323 6.7.1 Virtual image formation by converging lenses 325 6.7.2 Real image formation by convex lenses 327 6.8 Camera lucida and camera obscura 328 6.8.1 Camera lucida 328 6.8.2 Camera obscura 331 6.8.3 Depth of field, depth of focus, and blur spots 333 6.9 Optical projections and the creation of art 336 6.9.1 Jan van Eyck's Portrait of Giovanni Arnolfini and his wife 337 6.9.2 Caravaggio's Supper at Emmaus 342 6.9.3 Lorenzo Lotto's Husband and Wife 345 6.9.4 Johannes Vermeer's Lady at the Virginals with a Gentleman 349 Johannes Vermeer 349 6.9.5 Canaletto's Piazza San Marco 363 Canaletto (Giovanni Antonio Canal) 364 6.9.6 Photorealists 364 Philip Barlow 366 6.10 Refraction and nonimaging optics in art 366 6.10.1 Leonardo's Salvator Mundi 366 6.11 Summary 371 6.12 Bibliographical remarks 372 7 Lighting analysis 377 7.1 Introduction 377 7.2 Basic shadows 381 7.2.1 General classes of lighting analysis methods 383 7.3 Cast–shadow analysis 383 7.3.1 Illumination from two or more point-sources 388 7.3.2 Cast–shadow analysis under geometric constraints 388 7.4 Lighting information from highlights 389 7.4.1 Illumination direction from highlights on simple estimated shapes 393 7.5 The optics of diffuse reflections 394 7.6 Inferring illumination from plane surfaces 396 Georges de la Tour 398 7.7 Interreflection 400 7.8 Occluding–contour algorithms 401 7.8.1 Single–point occluding–contour algorithm 403 7.8.2 General occluding–contour algorithm 405 Caravaggio (Michelangelo Merisi da Caravaggio) 407 7.8.3 Lightfield occluding–contour algorithm 408 Garth Herrick 409 7.8.4 Theory of the lightfield occluding–contour algorithm 410 7.8.5 Application of the lightfield occluding–contour algorithm 415 7.9 Computer graphics for the analysis of lighting 418 7.9.1 Georges de la Tour's Christ in the Carpenter's Studio (model) 419 7.9.2 Johannes Vermeer's Girl with a Pearl Earring 421 7.9.3 René Magritte's The Menaced Assassin 422 7.9.4 Bidirectional reflectance distribution functions (BRDFs) 424 7.9.5 Caravaggio's The Calling of St. Matthew 425 7.10 Shape–from–shading algorithms 426 7.10.1 Shape–from–shading by deep neural networks 429 7.10.2 Shape–from–shading for estimating both illumination and depth 430 7.11 Integrating lighting estimates 433 7.11.1 Integrating one–dimensional lighting estimates 433 7.11.2 Integrating two–dimensional lighting estimates 436 7.12 Lighting analysis for dating depicted scenes 439 7.13 Summary 442 7.14 Bibliographical remarks 444 8 Object analysis 449 8.1 Introduction 449 8.2 Image–based object classification 452 8.2.1 Feature–based object recognition 452 8.3 Feature–based analysis of faces and bodies 454 8.3.1 Feature–based analysis of body pose 464 8.3.2 Feature–based analysis of head poses 466 8.4 Deep neural network–based object recognition 468 Jacques-Louis David 472 8.4.1 Transfer training 472 8.5 Summary 474 8.6 Bibliographical remarks 475 9 Style and composition analysis 477 9.1 Introduction 477 9.2 Automatic classification of style 480 9.3 Compositional balance 482 9.3.1 Computational balance of actors 485 9.4 Geometric properties of composition 486 9.4.1 Design in Piet Mondrian's Neoplastic paintings 487 Piet Mondrian 487 9.5 Analysis of trends and similarities in artistic style 497 9.5.1 Trends in landscape compositions 498 9.5.2 Large–scale trends in the development of style 502 9.5.3 Graph representations of stylistic similarities 503 9.6 Style transfer 505 9.6.1 Style transfer by deep networks 505 9.6.2 Rejuvenating tapestries 506 9.6.3 Coloration of black–and–white photographs of artworks 507 9.6.4 Style transfer for visualizing underdrawings 509 9.7 Recovering Rembrandt's complete The Night Watch 513 Rembrandt 514 9.8 Computational generation of images for art analysis 516 9.8.1 Computational recovery of lost artworks 518 9.9 Summary 521 9.10 Bibliographical remarks 522 10 Semantic analysis 525 10.1 Introduction 525 Jacques-Louis David 528 10.2 Semantics and visual art 534 10.2.1 Natural language processing and knowledge representation 536 10.3 Meaning through associations 538 10.3.1 Signifiers and signifieds 538 10.4 Semantics of color 544 10.5 Identifying saints by their attributes 546 Andrea del Verrocchio 549 10.6 Learning associations between signifiers and signifieds 550 Harmen Steenwijck 551 10.7 Meaning through artistic style 554 10.7.1 Context in the creation of meaning 556 10.8 Automatic image captioning and question answering 557 10.8.1 Image captioning 557 10.8.2 Automatic answering of questions about artworks 559 10.9 Meaning through shape relations and associations 563 Rogier van der Weyden 563 10.9.1 Recognizing meaning–bearing stories 565 Albrecht Dürer 567 10.10 Summary 568 10.11 Bibliographical remarks 569 Appendix 573 A Symbols, acronyms, and mathematical notation 573 A.1 Mathematical notation, definitions, and operations 573 A.2 Solving simultaneous linear equations 578 A.3 Lagrange optimization 579 A.4 Basis functions 580 A.5 Discrete Fourier analysis and synthesis 580 A.6 Discrete wavelet transform 582 A.7 Spherical harmonics 582 B Probability 584 B.1 Accuracy, precision, and recall 585 B.2 Conditional probability 585 B.3 The definition of information 586 B.4 Hidden Markov models (HMMs) 586 C Bayes' theorem and reasoning about uncertainty 588 C.1 Statistical independence 588 C.2 Maximum likelihood estimation 589 C.3 Bias and variance 591 C.4 Intersection over Union metric 592 D Deep neural networks 593 E Ray tracing and image formation in mirrors and lenses 596 E.1 Converging lenses 596 E.2 Diverging lenses 599 E.3 Mirrors 600 E.4 The focal length and radius of curvature of a spherical mirror 602 E.5 Spherical versus parabolic mirrors 603 F Resources 604 Epilog 607 Glossary 609 Bibliography 615 Figure credits 673 Timeline of artists 682 Index of artists 683 Index 687 About the book 713

About the Author :
Dr. David G. Stork is a graduate of MIT and the University of Maryland and studied art history at Wellesley College. He is an Adjunct Professor at Stanford University. Dr. Stork holds 64 U.S. patents and has published over 220 peer-reviewed scholarly works in machine learning, pattern recognition, computational optics, and image understanding of art. His many books include Seeing the Light, Pattern Classification Second Edition, and HAL’s Legacy. He is a Fellow of IEEE, OSA, SPIE, IS&T, IAPR, IARIA, and AAIA, and a 2023 Leonardo@ Djerassi Fellow.