Data Engineering Design Patterns: Scalable data engineering for efficient data systems and workflows (English Edition)

Data engineering has gained even more relevance than before, and data engineering patterns are key to the successful implementation of data engineering projects. This book enables a data engineer to not only become familiar with data engineering patterns but also understand their application in real world use cases. This book presents a comprehensive collection of data engineering patterns, each illustrated with relevant enterprise use cases to highlight their value and simplicity. It showcases both open-source and cloud technologies, guiding readers in building data systems for on-premise and cloud environments. The book covers patterns for data ingestion, transformation, storage, and serving, while also offering insights into performance engineering for data pipelines. Once we understand fundamental data engineering patterns, we then shift focus to patterns that help us build high-performance low latency data systems. We cover data caching, partitioning, replication, and how to select the technology stack for building out the patterns in this book. By the end of the book, readers will have a deep understanding of various data engineering use cases and will be able to map the appropriate patterns to address them. They will also be equipped to choose the right technical stack for implementing these patterns, enabling them to create robust and efficient data systems in a secure and a cost-effective manner. WHAT YOU WILL LEARN ● Key data engineering patterns. ● Data ingestion and processing patterns. ● Modern architectures like Lambda. ● Explore time-tested data patterns of ETL and ELT. ● Modern data systems like data lake and medallion architectures. ● Domain-specific patterns and also on data orchestration, observability, and security. ● Overcoming performance challenges in building complex data systems. WHO THIS BOOK IS FOR This book is designed for data engineers with beginner to intermediate experience in building enterprise-grade data systems. ETL developers transitioning into data engineering roles will also find this book valuable for understanding essential data engineering patterns. The code snippets provided throughout the book are written in Python or Scala, so a basic understanding of either language will help readers more easily grasp the concepts presented. TABLE OF CONTENTS 1. Understanding Data Engineering 2. Data Engineering Patterns, Terminologies, and Technical Stack 3. Batch Ingestion and Processing 4. Real-time Ingestion and Processing 5. Micro-batching 6. Lambda Architecture 7. ETL and ELT 8. Data Fundamentals 9. Databases and Transactional Data 10. Data Warehouse and Data Analytics 11. Data Lake and Medallion Architecture 12. Data Replication and Partitioning 13. Hot Versus Cold Data Storage 14. Data Caching and Low Latency Serving 15. Data Search Patterns 16. Domain Specific Patterns 17. Data Security Patterns 18. Data Observability and Monitoring Patterns 19. Idempotency and Deduplication Patterns 20. Data Orchestration Patterns 21. Common Performance Pitfalls 22. Technology and Infrastructure Selection 23. Recap and Next Steps