Introduction to Stochastic Processes

This concise, informal introduction to stochastic processes evolving with time was designed to meet the needs of graduate students not only in mathematics and statistics, but in the many fields in which the concepts presented are important, including computer science, economics, business, biological science, psychology, and engineering. With emphasis on fundamental mathematical ideas rather than proofs or detailed applications, the treatment introduces the following topics: Markov chains, with focus on the relationship between the convergence to equilibrium and the size of the eigenvalues of the stochastic matrix Infinite state space, including the ideas of transience, null recurrence and positive recurrence The three main types of continual time Markov chains and optimal stopping of Markov chains Martingales, including conditional expectation, the optional sampling theorem, and the martingale convergence theorem Renewal process and reversible Markov chains Brownian motion, both multidimensional and one-dimensional Introduction to Stochastic Processes is ideal for a first course in stochastic processes without measure theory, requiring only a calculus-based undergraduate probability course and a course in linear algebra.

Table of Contents:
PRELIMINARIES Introduction Linear Differential Equations Linear Difference Equations FINITE MARKOV CHAINS Definitions and Examples Long-Range Behavior and Invariant Probability Classification of States Return Times Transient States Examples COUNTABLE MARKOV CHAINS Introduction Recurrence and Transience Positive Recurrence and Null Recurrence Branching Process CONTINUOUS-TIME MARKOV CHAINS Poisson Process Finite State Space Birth-and-Death Processes General Case OPTIMAL STOPPING Optimal Stopping of Markov Chains Optimal Stopping with Cost Optimal Stopping with Discounting MARTINGALES Conditional Expectation Definition and Examples Optional Sampling theorem Uniform Integrability Martingale Convergence Theorem RENEWAL PROCESSES Introduction Renewal Equation Discrete Renewal Processes M/G/1 and B/M/1 Queues REVERSIBLE MARKOV CHAINS Reversible Processes Convergence to Equilibrium Markov Chain Algorithms A Criterion for Recurrence BROWNIAN MOTION Introduction Markov Property Zero Set of Brownian Motion Brownian Motion in Several Dimensions Recurrence and Transience Fractal Nature of Brownian Motion Brownian Motion with Drift STOCHASTIC INTEGRATION Integration with Respect to Random walk Integration with Respect to Brownian Motion Ito's Formula Simulation INDEX