Nonlinear System Identification: NARMAX Methods in the Time, Frequency, and Spatio-Temporal Domains

Nonlinear System Identification: NARMAX Methods in the Time, Frequency, and Spatio-Temporal Domains describes a comprehensive framework for the identification and analysis of nonlinear dynamic systems in the time, frequency, and spatio-temporal domains. This book is written with an emphasis on making the algorithms accessible so that they can be applied and used in practice. Includes coverage of: The NARMAX (nonlinear autoregressive moving average with exogenous inputs) model The orthogonal least squares algorithm that allows models to be built term by term where the error reduction ratio reveals the percentage contribution of each model term Statistical and qualitative model validation methods that can be applied to any model class Generalised frequency response functions which provide significant insight into nonlinear behaviours A completely new class of filters that can move, split, spread, and focus energy The response spectrum map and the study of sub harmonic and severely nonlinear systems Algorithms that can track rapid time variation in both linear and nonlinear systems The important class of spatio-temporal systems that evolve over both space and time Many case study examples from modelling space weather, through identification of a model of the visual processing system of fruit flies, to tracking causality in EEG data are all included to demonstrate how easily the methods can be applied in practice and to show the insight that the algorithms reveal even for complex systems NARMAX algorithms provide a fundamentally different approach to nonlinear system identification and signal processing for nonlinear systems. NARMAX methods provide models that are transparent, which can easily be analysed, and which can be used to solve real problems. This book is intended for graduates, postgraduates and researchers in the sciences and engineering, and also for users from other fields who have collected data and who wish to identify models to help to understand the dynamics of their systems.

Table of Contents:
Preface xv 1 Introduction 1 1.1 Introduction to System Identification 1 1.1.1 System Models and Simulation 1 1.1.2 Systems and Signals 3 1.1.3 System Identification 3 1.2 Linear System Identification 3 1.3 Nonlinear System Identification 5 1.4 NARMAX Methods 7 1.5 The NARMAX Philosophy 8 1.6 What is System Identification For? 9 1.7 Frequency Response of Nonlinear Systems 11 1.8 Continuous-Time, Severely Nonlinear, and Time-Varying Models and Systems 12 1.9 Spatio-temporal Systems 13 1.10 Using Nonlinear System Identification in Practice and Case Study Examples 13 References 14 2 Models for Linear and Nonlinear Systems 17 2.1 Introduction 17 2.2 Linear Models 18 2.2.1 Autoregressive Moving Average with Exogenous Input Model 18 2.2.2 Parameter Estimation for Linear Models 20 2.3 Piecewise Linear Models 22 2.3.1 Spatial Piecewise Linear Models 23 2.3.2 Models with Signal-Dependent Parameters 26 2.3.3 Remarks on Piecewise Linear Models 29 2.4 Volterra Series Models 30 2.5 Block-Structured Models 31 2.5.1 Parallel Cascade Models 32 2.5.2 Feedback Block-Structured Models 32 2.6 NARMAX Models 33 2.6.1 Polynomial NARMAX Model 35 2.6.2 Rational NARMAX Model 37 2.6.3 The Extended Model Set Representation 39 2.7 Generalised Additive Models 40 2.8 Neural Networks 41 2.8.1 Multi-layer Networks 41 2.8.2 Single-Layer Networks 42 2.9 Wavelet Models 45 2.9.1 Dynamic Wavelet Models 46 2.10 State-Space Models 48 2.11 Extensions to the MIMO Case 49 2.12 Noise Modelling 49 2.12.1 Noise-Free 50 2.12.2 Additive Random Noise 50 2.12.3 Additive Coloured Noise 50 2.12.4 General Noise 51 2.13 Spatio-temporal Models 52 References 53 3 Model Structure Detection and Parameter Estimation 61 3.1 Introduction 61 3.2 The Orthogonal Least Squares Estimator and the Error Reduction Ratio 64 3.2.1 Linear-in-the-Parameters Representation 64 3.2.2 The Matrix Form of the Linear-in-the-Parameters Representation 65 3.2.3 The Basic OLS Estimator 65 3.2.4 The Matrix Formulation of the OLS Estimator 67 3.2.5 The Error Reduction Ratio 68 3.2.6 An Illustrative Example of the Basic OLS Estimator 69 3.3 The Forward Regression OLS Algorithm 70 3.3.1 Forward Regression with OLS 72 3.3.2 An Illustrative Example of Forward Regression with OLS 77 3.3.3 The OLS Estimation Engine and Identification Procedure 78 3.4 Term and Variable Selection 79 3.5 OLS and Sum of Error Reduction Ratios 80 3.5.1 Sum of Error Reduction Ratios 82 3.5.2 The Variance of the s -Step-Ahead Prediction Error 82 3.5.3 The Final Prediction Error 83 3.5.4 The Variable Selection Algorithm 83 3.6 Noise Model Identification 84 3.6.1 The Noise Model 84 3.6.2 A Simulation Example with Noise Modelling 87 3.7 An Example of Variable and Term Selection for a Real Data Set 87 3.8 ERR is Not Affected by Noise 94 3.9 Common Structured Models to Accommodate Different Parameters 95 3.10 Model Parameters as a Function of Another Variable 98 3.10.1 System Internal and External Parameters 98 3.10.2 Parameter-Dependent Model Structure 98 3.10.3 Modelling Auxetic Foams – An Example of External Parameter-Dependent Model Identification 99 3.11 OLS and Model Reduction 100 3.12 Recursive Versions of OLS 102 References 102 4 Feature Selection and Ranking 105 4.1 Introduction 105 4.2 Feature Selection and Feature Extraction 106 4.3 Principal Components Analysis 107 4.4 A Forward Orthogonal Search Algorithm 108 4.4.1 The Basic Idea of the FOS-MOD Algorithm 108 4.4.2 Feature Detection and Ranking 109 4.4.3 Monitoring the Search Procedure 111 4.4.4 Illustrative Examples 112 4.5 A Basis Ranking Algorithm Based on PCA 113 4.5.1 Principal Component-Derived Multiple Regression 113 4.5.2 PCA-Based MFROLS Algorithms 114 4.5.3 An Illustrative Example 115 References 117 5 Model Validation 119 5.1 Introduction 119 5.2 Detection of Nonlinearity 121 5.3 Estimation and Test Data Sets 123 5.4 Model Predictions 124 5.4.1 One-Step-Ahead Prediction 124 5.4.2 Model Predicted Output 126 5.5 Statistical Validation 127 5.5.1 Correlation Tests for Input–Output Models 128 5.5.2 Correlation Tests for Time Series Models 132 5.5.3 Correlation Tests for MIMO Models 133 5.5.4 Output-Based Tests 134 5.6 Term Clustering 135 5.7 Qualitative Validation of Nonlinear Dynamic Models 137 5.7.1 Poincaré Sections 139 5.7.2 Bifurcation Diagrams 139 5.7.3 Cell Maps 140 5.7.4 Qualitative Validation in Nonlinear System Identification 140 References 145 6 The Identification and Analysis of Nonlinear Systems in the Frequency Domain 149 6.1 Introduction 149 6.2 Generalised Frequency Response Functions 151 6.2.1 The Volterra Series Representation of Nonlinear Systems 153 6.2.2 Generalised Frequency Response Functions 156 6.2.3 The Relationship Between GFRFs and Output Response of Nonlinear Systems 157 6.2.4 Interpretation of the Composition of the Output Frequency Response of Nonlinear Systems 162 6.2.5 Estimation and Computation of GFRFs 165 6.2.6 The Analysis of Nonlinear Systems Using GFRFs 176 6.3 Output Frequencies of Nonlinear Systems 184 6.3.1 Output Frequencies of Nonlinear Systems under Multi-tone Inputs 185 6.3.2 Output Frequencies of Nonlinear Systems for General Inputs 187 6.4 Nonlinear Output Frequency Response Functions 191 6.4.1 Definition and Properties of NOFRFs 192 6.4.2 Evaluation of NOFRFs 195 6.4.3 Damage Detection Using NARMAX Modelling and NOFRF-Based Analysis 196 6.5 Output Frequency Response Function of Nonlinear Systems 202 6.5.1 Definition of the OFRF 203 6.5.2 Determination of the OFRF 203 6.5.3 Application of the OFRF to Analysis of Nonlinear Damping for Vibration Control 207 References 213 7 Design of Nonlinear Systems in the Frequency Domain – Energy Transfer Filters and Nonlinear Damping 217 7.1 Introduction 217 7.2 Energy Transfer Filters 218 7.2.1 The Time and Frequency Domain Representation of the NARX Model with Input Nonlinearity 220 7.2.2 Energy Transfer Filter Designs 222 7.3 Energy Focus Filters 240 7.3.1 Output Frequencies of Nonlinear Systems with Input Signal Energy Located in Two Separate Frequency Intervals 241 7.3.2 The Energy Focus Filter Design Procedure and an Example 245 7.4 OFRF-Based Approach for the Design of Nonlinear Systems in the Frequency Domain 249 7.4.1 OFRF-Based Design of Nonlinear Systems in the Frequency Domain 249 7.4.2 Design of Nonlinear Damping in the Frequency Domain for Vibration Isolation: An Experimental Study 251 References 259 8 Neural Networks for Nonlinear System Identification 261 8.1 Introduction 261 8.2 The Multi-layered Perceptron 263 8.3 Radial Basis Function Networks 264 8.3.1 Training Schemes for RBF Networks 266 8.3.2 Fixed Kernel Centres with a Single Width 266 8.3.3 Limitation of RBF Networks with a Single Kernel Width 268 8.3.4 Fixed Kernel Centres and Multiple Kernel Widths 269 8.4 Wavelet Networks 270 8.4.1 Wavelet Decompositions 271 8.4.2 Wavelet Networks 272 8.4.3 Limitations of Fixed Grid Wavelet Networks 273 8.4.4 A New Class of Wavelet Networks 274 8.5 Multi-resolution Wavelet Models and Networks 277 8.5.1 Multi-resolution Wavelet Decompositions 277 8.5.2 Multi-resolution Wavelet Models and Networks 280 8.5.3 An Illustrative Example 282 References 284 9 Severely Nonlinear Systems 289 9.1 Introduction 289 9.2 Wavelet NARMAX Models 291 9.2.1 Nonlinear System Identification Using Wavelet Multi-resolution NARMAX Models 292 9.2.2 A Strategy for Identifying Nonlinear Systems 299 9.3 Systems that Exhibit Sub-harmonics and Chaos 301 9.3.1 Limitations of the Volterra Series Representation 301 9.3.2 Time Domain Analysis 302 9.4 The Response Spectrum Map 305 9.4.1 Introduction 305 9.4.2 Examples of the Response Spectrum Map 306 9.5 A Modelling Framework for Sub-harmonic and Severely Nonlinear Systems 313 9.5.1 Input Signal Decomposition 314 9.5.2 MISO NARX Modelling in the Time Domain 317 9.6 Frequency Response Functions for Sub-harmonic Systems 320 9.6.1 MISO Frequency Domain Volterra Representation 320 9.6.2 Generating the GFRFs from the MISO Model 322 9.7 Analysis of Sub-harmonic Systems and the Cascade to Chaos 326 9.7.1 Frequency Domain Response Synthesis 326 9.7.2 An Example of Frequency Domain Analysis for Sub-harmonic Systems 332 References 334 10 Identification of Continuous-Time Nonlinear Models 337 10.1 Introduction 337 10.2 The Kernel Invariance Method 338 10.2.1 Definitions 338 10.2.2 Reconstructing the Linear Model Terms 342 10.2.3 Reconstructing the Quadratic Model Terms 346 10.2.4 Model Structure Determination 348 10.3 Using the GFRFs to Reconstruct Nonlinear Integro-differential Equation Models Without Differentiation 352 10.3.1 Introduction 352 10.3.2 Reconstructing the Linear Model Terms 355 10.3.3 Reconstructing the Quadratic Model Terms 358 10.3.4 Reconstructing the Higher-Order Model Terms 361 10.3.5 A Real Application 364 References 367 11 Time-Varying and Nonlinear System Identification 371 11.1 Introduction 371 11.2 Adaptive Parameter Estimation Algorithms 372 11.2.1 The Kalman Filter Algorithm 372 11.2.2 The RLS and LMS Algorithms 375 11.2.3 Some Practical Considerations for the KF, RLS, and LMS Algorithms 376 11.3 Tracking Rapid Parameter Variations Using Wavelets 376 11.3.1 A General Form of TV-ARX Models Using Wavelets 376 11.3.2 A Multi-wavelet Approach for Time-Varying Parameter Estimation 377 11.4 Time-Dependent Spectral Characterisation 378 11.4.1 The Definition of a Time-Dependent Spectral Function 378 11.5 Nonlinear Time-Varying Model Estimation 380 11.6 Mapping and Tracking in the Frequency Domain 381 11.6.1 Time-Varying Frequency Response Functions 381 11.6.2 First and Second-Order TV-GFRFs 382 11.7 A Sliding Window Approach 388 References 389 12 Identification of Cellular Automata and N -State Models of Spatio-temporal Systems 391 12.1 Introduction 391 12.2 Cellular Automata 393 12.2.1 History of Cellular Automata 393 12.2.2 Discrete Lattice 393 12.2.3 Neighbourhood 394 12.2.4 Transition Rules 396 12.2.5 Simulation Examples of Cellular Automata 399 12.3 Identification of Cellular Automata 402 12.3.1 Introduction and Review 402 12.3.2 Polynomial Representation 403 12.3.3 Neighbourhood Detection and Rule Identification 405 12.4 N -State Systems 414 12.4.1 Introduction to Excitable Media Systems 414 12.4.2 Simulation of Excitable Media 415 12.4.3 Identification of Excitable Media Using a CA Model 419 12.4.4 General N-State Systems 424 References 427 13 Identification of Coupled Map Lattice and Partial Differential Equations of Spatio-temporal Systems 431 13.1 Introduction 431 13.2 Spatio-temporal Patterns and Continuous-State Models 432 13.2.1 Stem Cell Colonies 433 13.2.2 The Belousov–Zhabotinsky Reaction 434 13.2.3 Oxygenation in Brain 434 13.2.4 Growth Patterns 435 13.2.5 A Simulated Example Showing Spatio-temporal Chaos from CML Models 435 13.3 Identification of Coupled Map Lattice Models 437 13.3.1 Deterministic CML Models 437 13.3.2 The Identification of Stochastic CML Models 454 13.4 Identification of Partial Differential Equation Models 458 13.4.1 Model Structure 458 13.4.2 Time Discretisation 459 13.4.3 Nonlinear Function Approximation 459 13.5 Nonlinear Frequency Response Functions for Spatio-temporal Systems 466 13.5.1 A One-Dimensional Example 467 13.5.2 Higher-Order Frequency Response Functions 468 References 471 14 Case Studies 473 14.1 Introduction 473 14.2 Practical System Identification 474 14.3 Characterisation of Robot Behaviour 478 14.3.1 Door Traversal 478 14.3.2 Route Learning 482 14.4 System Identification for Space Weather and the Magnetosphere 484 14.5 Detecting and Tracking Iceberg Calving in Greenland 493 14.5.1 Causality Detection 494 14.5.2 Results 495 14.6 Detecting and Tracking Time-Varying Causality for EEG Data 498 14.6.1 Data Acquisition 499 14.6.2 Causality Detection 500 14.6.3 Detecting Linearity and Nonlinearity 504 14.7 The Identification and Analysis of Fly Photoreceptors 505 14.7.1 Identification of the Fly Photoreceptor 506 14.7.2 Model-Based System Analysis in the Time and Frequency Domain 507 14.8 Real-Time Diffuse Optical Tomography Using RBF Reduced-Order Models of the Propagation of Light for Monitoring Brain Haemodynamics 514 14.8.1 Diffuse Optical Imaging 515 14.8.2 In-vivo Real-Time 3-D Brain Imaging Using Reduced-Order Forward Models 517 14.9 Identification of Hysteresis Effects in Metal Rubber Damping Devices 522 14.9.1 Dynamic Modelling of Metal Rubber Damping Devices 523 14.9.2 Model Identification of a Metal Rubber Specimen 526 14.10 Identification of the Belousov–Zhabotinsky Reaction 528 14.10.1 Data Acquisition 529 14.10.2 Model Identification 530 14.11 Dynamic Modelling of Synthetic Bioparts 534 14.11.1 The Biopart and the Experiments 535 14.11.2 NARMAX Model of the Synthetic Biopart 536 14.12 Forecasting High Tides in the Venice Lagoon 539 14.12.1 Time Series Forecasting Problem 540 14.12.2 Water-Level Modelling and High-Tide Forecasting 541 References 543 Index 549

About the Author :
Stephen A. Billings is Professor of Signal Processing and Complex Systems, and Director of the Signal Processing and Complex Systems Research Group, in the Department of Automatic Control and Systems Engineering at the University of Sheffield, He is counted as "highly cited" by the ISI Web of Knowledge with 250 publications to his name.