Cybernetical Intelligence: Engineering Cybernetics with Machine Intelligence

CYBERNETICAL INTELLIGENCE Highly comprehensive, detailed, and up-to-date overview of artificial intelligence and cybernetics, with practical examples and supplementary learning resources Cybernetical Intelligence: Engineering Cybernetics with Machine Intelligence is a comprehensive guide to the field of cybernetics and neural networks, as well as the mathematical foundations of these technologies. The book provides a detailed explanation of various types of neural networks, including feedforward networks, recurrent neural networks, and convolutional neural networks as well as their applications to different real-world problems. This groundbreaking book presents a pioneering exploration of machine learning within the framework of cybernetics. It marks a significant milestone in the field’s history, as it is the first book to describe the development of machine learning from a cybernetics perspective. The introduction of the concept of “Cybernetical Intelligence” and the generation of new terminology within this context propel new lines of thought in the historical development of artificial intelligence. With its profound implications and contributions, this book holds immense importance and is poised to become a definitive resource for scholars and researchers in this field of study. Each chapter is specifically designed to introduce the theory with several examples. This comprehensive book includes exercise questions at the end of each chapter, providing readers with valuable opportunities to apply and strengthen their understanding of cybernetical intelligence. To further support the learning journey, solutions to these questions are readily accessible on the book’s companion site. Additionally, the companion site offers programming practice exercises and assignments, enabling readers to delve deeper into the practical aspects of the subject matter. Cybernetical Intelligence includes information on: The history and development of cybernetics and its influence on the development of neural networks Developments and innovations in artificial intelligence and machine learning, such as deep reinforcement learning, generative adversarial networks, and transfer learning Mathematical foundations of artificial intelligence and cybernetics, including linear algebra, calculus, and probability theory Ethical implications of artificial intelligence and cybernetics as well as responsible and transparent development and deployment of AI systems Presenting a highly detailed and comprehensive overview of the field, with modern developments thoroughly discussed, Cybernetical Intelligence is an essential textbook that helps students make connections with real-life engineering problems by providing both theory and practice, along with a myriad of helpful learning aids.

Table of Contents:
Preface xv About the Author xix About the Companion Website xxi 1 Artificial Intelligence and Cybernetical Learning 1 1.1 Artificial Intelligence Initiative 1 1.2 Intelligent Automation Initiative 4 1.2.1 Benefits of IAI 5 1.3 Artificial Intelligence Versus Intelligent Automation 5 1.3.1 Process Discovery 6 1.3.2 Optimization 7 1.3.3 Analytics and Insight 8 1.4 The Fourth Industrial Revolution and Artificial Intelligence 9 1.4.1 Artificial Narrow Intelligence 10 1.4.2 Artificial General Intelligence 12 1.4.3 Artificial Super Intelligence 13 1.5 Pattern Analysis and Cognitive Learning 14 1.5.1 Machine Learning 15 1.5.1.1 Parametric Algorithms 16 1.5.1.2 Nonparametric Algorithms 17 1.5.2 Deep Learning 20 1.5.2.1 Convolutional Neural Networks in Advancing Artificial Intelligence 21 1.5.2.2 Future Advancement in Deep Learning 22 1.5.3 Cybernetical Learning 23 1.6 Cybernetical Artificial Intelligence 24 1.6.1 Artificial Intelligence Control Theory 24 1.6.2 Information Theory 26 1.6.3 Cybernetic Systems 27 1.7 Cybernetical Intelligence Definition 28 1.8 The Future of Cybernetical Intelligence 30 Summary 32 Exercise Questions 32 Further Reading 33 2 Cybernetical Intelligent Control 35 2.1 Control Theory and Feedback Control Systems 35 2.2 Maxwell’s Analysis of Governors 37 2.3 Harold Black 39 2.4 Nyquist and Bode 40 2.5 Stafford Beer 42 2.5.1 Cybernetic Control 42 2.5.2 Viable Systems Model 42 2.5.3 Cybernetics Models of Management 43 2.6 James Lovelock 43 2.6.1 Cybernetic Approach to Ecosystems 43 2.6.2 Gaia Hypothesis 44 2.7 Macy Conference 44 2.8 McCulloch–Pitts 45 2.9 John von Neumann 47 2.9.1 Discussions on Self-Replicating Machines 47 2.9.2 Discussions on Machine Learning 48 Summary 48 Exercise Questions 49 Further Reading 50 3 The Basics of Perceptron 51 3.1 The Analogy of Biological and Artificial Neurons 51 3.1.1 Biological Neurons and Neurodynamics 52 3.1.2 The Structure of Neural Network 53 3.1.3 Encoding and Decoding 56 3.2 Perception and Multilayer Perceptron 57 3.2.1 Back Propagation Neural Network 59 3.2.2 Derivative Equations for Backpropagation 59 3.3 Activation Function 61 3.3.1 Sigmoid Activation Function 61 3.3.2 Hyperbolic Tangent Activation Function 62 3.3.3 Rectified Linear Unit Activation Function 62 3.3.4 Linear Activation Function 64 Summary 65 Exercise Questions 67 Further Reading 67 4 The Structure of Neural Network 69 4.1 Layers in Neural Network 69 4.1.1 Input Layer 69 4.1.2 Hidden Layer 70 4.1.3 Neurons 70 4.1.4 Weights and Biases 71 4.1.5 Forward Propagation 72 4.1.6 Backpropagation 72 4.2 Perceptron and Multilayer Perceptron 73 4.3 Recurrent Neural Network 75 4.3.1 Long Short-Term Memory 76 4.4 Markov Neural Networks 77 4.4.1 State Transition Function 77 4.4.2 Observation Function 78 4.4.3 Policy Function 78 4.4.4 Loss Function 78 4.5 Generative Adversarial Network 78 Summary 79 Exercise Questions 80 Further Reading 81 5 Backpropagation Neural Network 83 5.1 Backpropagation Neural Network 83 5.1.1 Forward Propagation 85 5.2 Gradient Descent 85 5.2.1 Loss Function 85 5.2.2 Parameters in Gradient Descent 88 5.2.3 Gradient in Gradient Descent 88 5.2.4 Learning Rate in Gradient Descent 89 5.2.5 Update Rule in Gradient Descent 89 5.3 Stopping Criteria 89 5.3.1 Convergence and Stopping Criteria 90 5.3.2 Local Minimum and Global Minimum 91 5.4 Resampling Methods 91 5.4.1 Cross-Validation 93 5.4.2 Bootstrapping 93 5.4.3 Monte Carlo Cross-Validation 94 5.5 Optimizers in Neural Network 94 5.5.1 Stochastic Gradient Descent 94 5.5.2 Root Mean Square Propagation 96 5.5.3 Adaptive Moment Estimation 96 5.5.4 AdaMax 97 5.5.5 Momentum Optimization 97 Summary 97 Exercise Questions 99 Further Reading 100 6 Application of Neural Network in Learning and Recognition 101 6.1 Applying Backpropagation to Shape Recognition 101 6.2 Softmax Regression 105 6.3 K-Binary Classifier 107 6.4 Relational Learning via Neural Network 108 6.4.1 Graph Neural Network 109 6.4.2 Graph Convolutional Network 111 6.5 Cybernetics Using Neural Network 112 6.6 Structure of Neural Network for Image Processing 115 6.7 Transformer Networks 116 6.8 Attention Mechanisms 116 6.9 Graph Neural Networks 117 6.10 Transfer Learning 118 6.11 Generalization of Neural Networks 119 6.12 Performance Measures 120 6.12.1 Confusion Matrix 120 6.12.2 Receiver Operating Characteristic 121 6.12.3 Area Under the ROC Curve 122 Summary 123 Exercise Questions 123 Further Reading 124 7 Competitive Learning and Self-Organizing Map 125 7.1 Principal of Competitive Learning 125 7.1.1 Step 1: Normalized Input Vector 128 7.1.2 Step 2: Find the Winning Neuron 128 7.1.3 Step 3: Adjust the Network Weight Vector and Output Results 129 7.2 Basic Structure of Self-Organizing Map 129 7.2.1 Properties Self-Organizing Map 130 7.3 Self-Organizing Mapping Neural Network Algorithm 131 7.3.1 Step 1: Initialize Parameter 132 7.3.2 Step 2: Select Inputs and Determine Winning Nodes 132 7.3.3 Step 3: Affect Neighboring Neurons 132 7.3.4 Step 4: Adjust Weights 133 7.3.5 Step 5: Judging the End Condition 133 7.4 Growing Self-Organizing Map 133 7.5 Time Adaptive Self-Organizing Map 136 7.5.1 TASOM-Based Algorithms for Real Applications 138 7.6 Oriented and Scalable Map 139 7.7 Generative Topographic Map 141 Summary 145 Exercise Questions 146 Further Reading 147 8 Support Vector Machine 149 8.1 The Definition of Data Clustering 149 8.2 Support Vector and Margin 152 8.3 Kernel Function 155 8.3.1 Linear Kernel 155 8.3.2 Polynomial Kernel 156 8.3.3 Radial Basis Function 157 8.3.4 Laplace Kernel 159 8.3.5 Sigmoid Kernel 159 8.4 Linear and Nonlinear Support Vector Machine 160 8.5 Hard Margin and Soft Margin in Support Vector Machine 164 8.6 I/O of Support Vector Machine 167 8.6.1 Training Data 167 8.6.2 Feature Matrix and Label Vector 168 8.7 Hyperparameters of Support Vector Machine 169 8.7.1 The C Hyperparameter 169 8.7.2 Kernel Coefficient 169 8.7.3 Class Weights 170 8.7.4 Convergence Criteria 170 8.7.5 Regularization 171 8.8 Application of Support Vector Machine 171 8.8.1 Classification 171 8.8.2 Regression 173 8.8.3 Image Classification 173 8.8.4 Text Classification 174 Summary 174 Exercise Questions 175 Further Reading 176 9 Bio-Inspired Cybernetical Intelligence 177 9.1 Genetic Algorithm 178 9.2 Ant Colony Optimization 181 9.3 Bees Algorithm 184 9.4 Artificial Bee Colony Algorithm 186 9.5 Cuckoo Search 189 9.6 Particle Swarm Optimization 193 9.7 Bacterial Foraging Optimization 196 9.8 Gray Wolf Optimizer 197 9.9 Firefly Algorithm 199 Summary 200 Exercise Questions 201 Further Reading 202 10 Life-Inspired Machine Intelligence and Cybernetics 203 10.1 Multi-Agent AI Systems 203 10.1.1 Game Theory 205 10.1.2 Distributed Multi-Agent Systems 206 10.1.3 Multi-Agent Reinforcement Learning 207 10.1.4 Evolutionary Computation and Multi-Agent Systems 209 10.2 Cellular Automata 211 10.3 Discrete Element Method 212 10.3.1 Particle-Based Simulation of Biological Cells and Tissues 214 10.3.2 Simulation of Microbial Communities and Their Interactions 215 10.3.3 Discrete Element Method-Based Modeling of Biological Fluids and Soft Materials 216 10.4 Smoothed Particle Hydrodynamics 218 10.4.1 SPH-Based Simulations of Biomimetic Fluid Dynamic 219 10.4.2 SPH-Based Simulations of Bio-Inspired Engineering Applications 220 Summary 221 Exercise Questions 222 Further Reading 223 11 Revisiting Cybernetics and Relation to Cybernetical Intelligence 225 11.1 The Concept and Development of Cybernetics 225 11.1.1 Attributes of Control Concepts 225 11.1.2 Research Objects and Characteristics of Cybernetics 226 11.1.3 Development of Cybernetical Intelligence 227 11.2 The Fundamental Ideas of Cybernetics 227 11.2.1 System Idea 227 11.2.2 Information Idea 229 11.2.3 Behavioral Idea 230 11.2.4 Cybernetical Intelligence Neural Network 231 11.3 Cybernetic Expansion into Other Fields of Research 234 11.3.1 Social Cybernetics 234 11.3.2 Internal Control-Related Theories 237 11.3.3 Software Control Theory 237 11.3.4 Perceptual Cybernetics 238 11.4 Practical Application of Cybernetics 240 11.4.1 Research on the Control Mechanism of Neural Networks 240 11.4.2 Balance Between Internal Control and Management Power Relations 240 11.4.3 Software Markov Adaptive Testing Strategy 242 11.4.4 Task Analysis Model 244 Summary 245 Exercise Questions 246 Further Reading 247 12 Turing Machine 249 12.1 Behavior of a Turing Machine 250 12.1.1 Computing with Turing Machines 251 12.2 Basic Operations of a Turing Machine 252 12.2.1 Reading and Writing to the Tape 253 12.2.2 Moving the Tape Head 254 12.2.3 Changing States 254 12.3 Interchangeability of Program and Behavior 255 12.4 Computability Theory 256 12.4.1 Complexity Theory 257 12.5 Automata Theory 258 12.6 Philosophical Issues Related to Turing Machines 259 12.7 Human and Machine Computations 260 12.8 Historical Models of Computability 261 12.9 Recursive Functions 262 12.10 Turing Machine and Intelligent Control 263 Summary 264 Exercise Questions 265 Further Reading 265 13 Entropy Concepts in Machine Intelligence 267 13.1 Relative Entropy of Distributions 268 13.2 Relative Entropy and Mutual Information 268 13.3 Entropy in Performance Evaluation 269 13.4 Cross-Entropy Softmax 271 13.5 Calculating Cross-Entropy 272 13.6 Cross-Entropy as a Loss Function 273 13.7 Cross-Entropy and Log Loss 274 13.8 Application of Entropy in Intelligent Control 275 13.8.1 Entropy-Based Control 275 13.8.2 Fuzzy Entropy 276 13.8.3 Entropy-Based Control Strategies 277 13.8.4 Entropy-Based Decision-Making 278 Summary 279 Exercise Questions 279 Further Reading 280 14 Sampling Methods in Cybernetical Intelligence 283 14.1 Introduction to Sampling Methods 283 14.2 Basic Sampling Algorithms 284 14.2.1 Importance of Sampling Methods in Machine Intelligence 286 14.3 Machine Learning Sampling Methods 287 14.3.1 Random Oversampling 288 14.3.2 Random Undersampling 290 14.3.3 Synthetic Minority Oversampling Technique 290 14.3.4 Adaptive Synthetic Sampling 292 14.4 Advantages and Disadvantages of Machine Learning Sampling Methods 293 14.5 Advanced Sampling Methods in Cybernetical Intelligence 294 14.5.1 Ensemble Sampling Method 295 14.5.2 Active Learning 297 14.5.3 Bayesian Optimization in Sampling 299 14.6 Applications of Sampling Methods in Cybernetical Intelligence 302 14.6.1 Image Processing and Computer Vision 302 14.6.2 Natural Language Processing 304 14.6.3 Robotics and Autonomous Systems 307 14.7 Challenges and Future Directions 308 14.8 Challenges and Limitations of Sampling Methods 309 14.9 Emerging Trends and Innovations in Sampling Methods 309 Summary 310 Exercise Questions 311 Further Reading 312 15 Dynamic System Control 313 15.1 Linear Systems 314 15.2 Nonlinear System 316 15.3 Stability Theory 318 15.4 Observability and Identification 320 15.5 Controllability and Stabilizability 321 15.6 Optimal Control 323 15.7 Linear Quadratic Regulator Theory 324 15.8 Time-Optimal Control 326 15.9 Stochastic Systems with Applications 328 15.9.1 Stochastic System in Control Systems 329 15.9.2 Stochastic System in Robotics and Automation 329 15.9.3 Stochastic System in Neural Networks 330 Summary 331 Exercise Questions 331 Further Reading 332 16 Deep Learning 333 16.1 Neural Network Models in Deep Learning 335 16.2 Methods of Deep Learning 336 16.2.1 Convolutional Neural Networks 337 16.2.2 Recurrent Neural Networks 340 16.2.3 Generative Adversarial Networks 342 16.2.4 Deep Learning Based Image Segmentation Models 345 16.2.5 Variational Auto Encoders 348 16.2.6 Transformer Models 350 16.2.7 Attention-Based Models 352 16.2.8 Meta-Learning Models 354 16.2.9 Capsule Networks 357 16.3 Deep Learning Frameworks 358 16.4 Applications of Deep Learning 359 16.4.1 Object Detection 360 16.4.2 Intelligent Power Systems 361 16.4.3 Intelligent Control 362 Summary 362 Exercise Questions 363 References 364 Further Reading 365 17 Neural Architecture Search 367 17.1 Neural Architecture Search and Neural Network 369 17.2 Reinforcement Learning-Based Neural Architecture Search 371 17.3 Evolutionary Algorithms-Based Neural Architecture Search 374 17.4 Bayesian Optimization-Based Neural Architecture Search 376 17.5 Gradient-Based Neural Architecture Search 378 17.6 One-shot Neural Architecture Search 379 17.7 Meta-Learning-Based Neural Architecture Search 381 17.8 Neural Architecture Search for Specific Domains 383 17.8.1 Cybernetical Intelligent Systems: Neural Architecture Search in Real-World 384 17.8.2 Neural Architecture Search for Specific Cybernetical Control Tasks 385 17.8.3 Neural Architecture Search for Cybernetical Intelligent Systems in Real-World 386 17.8.4 Neural Architecture Search for Adaptive Cybernetical Intelligent Systems 388 17.9 Comparison of Different Neural Architecture Search Approaches 389 Summary 391 Exercise Questions 391 Further Reading 392 Final Notes on Cybernetical Intelligence 393 Index 399

About the Author :
Prof. Dr. Kelvin K. L. Wong, is a distinguished expert in medical image processing and computational science, earning his Ph.D. from The University of Adelaide. With a strong academic background from Nanyang Technological University and The University of Sydney, he has been at the forefront of merging the fields of cybernetics and artificial intelligence (AI). He is renowned for coining the term “Cybernetical Intelligence” and is the inventor and founder of Deep Red AI.