Vision Instruction and Action: (Artificial Intelligence)

"Vision, Instruction and Action" describes a sophisticated integrated system called Sonja that takes instruction, can interpret its environment visually, and can play games (in this case the video game, Amazon) on its own. Sonja integrates advances in intermediate visual processing, interactive activity, and natural language pragmatics. In demonstrating that such systems, rare in artificial intelligence, are possible, David Chapman shows how discoveries in visual psychophysics can be incorporated into AI, how complex activity can result from participation rather than plan following, and how physical contest can be used to interpret indexical instructions. Sonja is able to play a competent beginner's game of Amazon autonomously and at the same time can also make flexible use of human instructions in knowing how to kill off monsters, pick up and use tools, and find its way in a dungeon maze. It extends the author's previous work in developing a new theory of activity by addressing linguistic issues and providing a better understanding of the architecture underlying activity, incorporating many technical improvements. Sonja also models several pragmatic issues in computational linguistics, focusing on external reference and including linguistic repair processing, and the use of temporal and spatial expressions. It connects language use with more detailed and realistic theories of vision and activity. In the field of vision research, Sonja provides an implementation of a unified visual architecture, demonstrating that this architecture can support a serious theory of activity. It demonstrates the first instance that various visual mechanisms previously proposed on psychophysical, neurophysiological, and speculative computational grounds can be made useful by connecting them with a natural task domain.

Table of Contents:
Part 1 Introduction: what Sonja does; contributions - contributions to active theory, contributions to computational linguistics, contributions to vision research; guide to the book. Part 2 The concrete-situated approach: routineness; situatedness; interactivity; dynamics; routines; improvisation; deictic representation. Part 3 Hardware and architecture: essential connectionism - essential connectinist commitments and constraints, advantages of essential connectionist hardware, alternative implementations of essential connectionism; the central system - modularity, the three-phase central system architecture; architectures for vision - the purpose of vision, insect vision, visual routines, implications for the central system, visual routines implement deictic representations. Part 4 Amazon: criteria for a domain; Amazon; modifications; realism. Part 5 Instruction use: video game instruction use; reference - theories, grounding reference in activity, indexicality, reflexivity; giving Sonja instructions; how Sonja uses instructions - instructions reorient attention, instructions manage autonomous routines, clearing instruction buffers, some kinds of instructions Sonja can use, generating replies; related work - instruction taking, reference. Part 6 the peripheral systems: visual attention; visual search - psychophysics of visual search, an architecture for visual search, extensions, related computational work, open questions; intermediate objects; criteria on visual operators; the specific operators - visual attention and search, tracking, distances, directions and angles, marker, line and ray manipulation, activation, gaps and edges, blanking; other visual systems; Sonja's other peripheral systems. Part 7 The central system architecture: MACNET; the arbitration macrology - proposers, conditions and overridings, defaults, hierarchical arbitration, discrimination, arbitration compilation, arbitration in practice; debugging facilities; related work. Part 8 Sonja: scenario; global implementation issues; registering the Amazon and its aspects; simple registrations; registering scrolls; combat; navigation. Part 9 Analysis: evaluation; the concrete-situated approach; the architecture; instruction use - specific issues in reference and reflexivity; vision; the system as a whole. Appendices: skill acquisition - some existing skill acquisition methods; WOMBAT; the phenomenology and dynamics of skill acquisition. (Part contents)