Design and Analysis of Efficient Learning Algorithms: (ACM Doctoral Dissertation Award)

Approaches to building machines that can learn from experience abound - from connectionist learning algorithms and genetic algorithms to statistical mechanics and a learning system based on Piaget's theories of early childhood development. This monograph describes results derived from the mathematically oriented framework of computational learning theory. Focusing on the design of efficient learning algorithms and their performance, it develops a sound, theoretical foundation for studying and understanding machine learning. Since many results concern the fundamental problem of learning a concept from examples, Schapire begins with a brief introduction to the Valiant model, which has generated much of the research on this problem. Four self-contained chapters then consider different aspects of machine learning. Their contributions include a general technique for dramatically improving the error rate of a "weak" learning algorithm that can also be used to improve the space efficiency of many known learning algorithms; a detailed exploration of a powerful statistical method for efficiently inferring the structure of certain kinds of Boolean formulas from random examples of the formula's input-output behaviour; the extension of a standard model of concept learning to accommodate concepts that exhibit uncertain or probabilistic behaviour; (including a variety of tools and techniques for designing efficient learning algorithms in such a probabilistic setting); and a description of algorithms that can be used by a robot to infer the "structure" of its environment through experimentation.

Table of Contents:
Part 1 The strength of weak learnability: the equivalence of strong and weak learnability; improving Learn's time and sample complexity; variations on the learning model; general complexity bounds for PAC learning; conclusions and open problems. Part 2 Statistical methods for inference of read-once formulas: exact identification of read-once majority formulas; exact identification of read-once positive NAND formulas; handling random misclassification noise; learning unbounded-depth formulas; learning probabilistic read-once formulas; conclusion and open problems. Part 3 Efficient distribution-free learning of probabilistic concepts: the learning model; efficient algorithms - the direct approach; hypothesis testing and expected loss; uniform convergence methods; a lower bound on sample size; Occam's Razor for genenral loss functions; conclusions and open problems. Part 4 Inference of finite automata using homing sequences: two representations of finite automata; homing sequences; a state-based algorithm for general automata; a diversity-based algorithm for general automata; a state-based algorithm for permutation automata; a diversity-based algorithm for permulation automata; experimental results; conclusions and open questions.