Systems Performance

Systems Performance, Second Edition, covers concepts, strategy, tools, and tuning for operating systems and applications, using Linux-based operating systems as the primary example. A deep understanding of these tools and techniques is critical for developers today. Implementing the strategies described in this thoroughly revised and updated edition can lead to a better end-user experience and lower costs, especially for cloud computing environments that charge by the OS instance. Systems performance expert and best-selling author Brendan Gregg summarizes relevant operating system, hardware, and application theory to quickly get professionals up to speed even if they have never analyzed performance before. Gregg then provides in-depth explanations of the latest tools and techniques, including extended BPF, and shows how to get the most out of cloud, web, and large-scale enterprise systems. Key topics covered include Hardware, kernel, and application internals, and how they perform Methodologies for rapid performance analysis of complex systems Optimizing CPU, memory, file system, disk, and networking usage Sophisticated profiling and tracing with perf, Ftrace, and BPF (BCC and bpftrace) Performance challenges associated with cloud computing hypervisors Benchmarking more effectively Featuring up-to-date coverage of Linux operating systems and environments, Systems Performance, Second Edition, also addresses issues that apply to any computer system. The book will be a go-to reference for many years to come and, like the first edition, required reading at leading tech companies.   Register your book for convenient access to downloads, updates, and/or corrections as they become available. See inside book for details.

Table of Contents:
Preface xxix Acknowledgments xxxv About the Author xxxvii Chapter 1: Introduction 1 1.1 Systems Performance 1 1.2 Roles 2 1.3 Activities 3 1.4 Perspectives 4 1.5 Performance Is Challenging 5 1.6 Latency 6 1.7 Observability 7 1.8 Experimentation 13 1.9 Cloud Computing 14 1.10 Methodologies 15 1.11 Case Studies 16 1.12 References 19 Chapter 2: Methodologies 21 2.1 Terminology 22 2.2 Models 23 2.3 Concepts 24 2.4 Perspectives 37 2.5 Methodology 40 2.6 Modeling 62 2.7 Capacity Planning 69 2.8 Statistics 73 2.9 Monitoring 77 2.10 Visualizations 79 2.11 Exercises 85 2.12 References 86 Chapter 3: Operating Systems 89 3.1 Terminology 90 3.2 Background 91 3.3 Kernels 111 3.4 Linux 114 3.5 Other Topics 122 3.6 Kernel Comparisons 124 3.7 Exercises 124 3.8 References 125 Chapter 4: Observability Tools 129 4.1 Tool Coverage 130 4.2 Tool Types 133 4.3 Observability Sources 138 4.4 sar 160 4.5 Tracing Tools 166 4.6 Observing Observability 167 4.7 Exercises 168 4.8 References 168 Chapter 5: Applications 171 5.1 Application Basics 172 5.2 Application Performance Techniques 176 5.3 Programming Languages 182 5.4 Methodology 186 5.5 Observability Tools 199 5.6 Gotchas 213 5.7 Exercises 216 5.8 References 217 Chapter 6: CPUs 219 6.1 Terminology 220 6.2 Models 221 6.3 Concepts 223 6.4 Architecture 229 6.5 Methodology 244 6.6 Observability Tools 254 6.7 Visualizations 288 6.8 Experimentation 293 6.9 Tuning 294 6.10 Exercises 299 6.11 References 300 Chapter 7: Memory 303 7.1 Terminology 304 7.2 Concepts 305 7.3 Architecture 311 7.4 Methodology 323 7.5 Observability Tools 328 7.6 Tuning 350 7.7 Exercises 354 7.8 References 355 Chapter 8: File Systems 359 8.1 Terminology 360 8.2 Models 361 8.3 Concepts 362 8.4 Architecture 372 8.5 Methodology 383 8.6 Observability Tools 391 8.7 Experimentation 411 8.8 Tuning 414 8.9 Exercises 419 8.10 References 420 Chapter 9: Disks 423 9.1 Terminology 424 9.2 Models 425 9.3 Concepts 427 9.4 Architecture 435 9.5 Methodology 449 9.6 Observability Tools 458 9.7 Visualizations 487 9.8 Experimentation 490 9.9 Tuning 493 9.10 Exercises 495 9.11 References 496 Chapter 10: Network 499 10.1 Terminology 500 10.2 Models 501 10.3 Concepts 503 10.4 Architecture 509 10.5 Methodology 524 10.6 Observability Tools 533 10.7 Experimentation 562 10.8 Tuning 567 10.9 Exercises 574 10.10 References 575 Chapter 11: Cloud Computing 579 11.1 Background 580 11.2 Hardware Virtualization 587 11.3 OS Virtualization 605 11.4 Lightweight Virtualization 630 11.5 Other Types 634 11.6 Comparisons 634 11.7 Exercises 636 11.8 References 637 Chapter 12: Benchmarking 641 12.1 Background 642 12.2 Benchmarking Types 651 12.3 Methodology 656 12.4 Benchmark Questions 667 12.5 Exercises 668 12.6 References 669 Chapter 13: perf 671 13.1 Subcommands Overview 672 13.2 One-Liners 674 13.3 perf Events 679 13.4 Hardware Events 681 13.5 Software Events 683 13.6 Tracepoint Events 684 13.7 Probe Events 685 13.8 perf stat 691 13.9 perf record 694 13.10 perf report 696 13.11 perf script 698 13.12 perf trace 701 13.13 Other Commands 702 13.14 perf Documentation 703 13.15 References 703 Chapter 14: Ftrace 705 14.1 Capabilities Overview 706 14.2 tracefs (/sys) 708 14.3 Ftrace Function Profiler 711 14.4 Ftrace Function Tracing 713 14.5 Tracepoints 717 14.6 kprobes 719 14.7 uprobes 722 14.8 Ftrace function_graph 724 14.9 Ftrace hwlat 726 14.10 Ftrace Hist Triggers 727 14.11 trace-cmd 734 14.12 perf ftrace 741 14.13 perf-tools 741 14.14 Ftrace Documentation 748 14.15 References 749 Chapter 15: BPF 751 15.1 BCC 753 15.2 bpftrace 761 15.3 References 782 Chapter 16: Case Study 783 16.1 An Unexplained Win 783 16.2 Additional Information 792 16.3 References 793 Appendix A: USE Method: Linux 795 Appendix B: sar Summary 801 Appendix C: bpftrace One-Liners 803 Appendix D: Solutions to Selected Exercises 809 Appendix E: Systems Performance Who's Who 811 Glossary 815 Index 825