About the Book
Build efficient data lakes that can scale to virtually unlimited size using AWS Glue
Key Features
Book DescriptionOrganizations these days have gravitated toward services such as AWS Glue that undertake undifferentiated heavy lifting and provide serverless Spark, enabling you to create and manage data lakes in a serverless fashion. This guide shows you how AWS Glue can be used to solve real-world problems along with helping you learn about data processing, data integration, and building data lakes.
Beginning with AWS Glue basics, this book teaches you how to perform various aspects of data analysis such as ad hoc queries, data visualization, and real-time analysis using this service. It also provides a walk-through of CI/CD for AWS Glue and how to shift left on quality using automated regression tests. You’ll find out how data security aspects such as access control, encryption, auditing, and networking are implemented, as well as getting to grips with useful techniques such as picking the right file format, compression, partitioning, and bucketing. As you advance, you’ll discover AWS Glue features such as crawlers, Lake Formation, governed tables, lineage, DataBrew, Glue Studio, and custom connectors. The concluding chapters help you to understand various performance tuning, troubleshooting, and monitoring options.
By the end of this AWS book, you’ll be able to create, manage, troubleshoot, and deploy ETL pipelines using AWS Glue.What you will learn
Apply various AWS Glue features to manage and create data lakes
Use Glue DataBrew and Glue Studio for data preparation
Optimize data layout in cloud storage to accelerate analytics workloads
Manage metadata including database, table, and schema definitions
Secure your data during access control, encryption, auditing, and networking
Monitor AWS Glue jobs to detect delays and loss of data
Integrate Spark ML and SageMaker with AWS Glue to create machine learning models
Who this book is for ETL developers, data engineers, and data analysts
Table of Contents:
Table of Contents- Data Management – Introduction and Concepts
- Introduction to Important AWS Glue Features
- Data Ingestion
- Data Preparation
- Data Layouts
- Data Management
- Metadata Management
- Data Security
- Data Sharing
- Data Pipeline Management
- Monitoring
- Tuning, Debugging, and Troubleshooting
- Data Analysis
- Machine Learning Integration
- Architecting Data Lakes for Real-World Scenarios and Edge Cases
About the Author :
Vishal Pathak is a Data Lab Solutions Architect at AWS. Vishal works with customers on their use cases, architects solutions to solve their business problems, and helps them build scalable prototypes. Prior to his journey in AWS, Vishal helped customers implement business intelligence, data warehouse, and data lake projects in the US and Australia. Subramanya Vajiraya is a Senior Cloud Engineer at AWS Sydney specializing in AWS Glue. He obtained his Bachelor of Engineering degree in Information Science & Engineering from NMAM Institute of Technology, Nitte, KA, India, in 2015 and his Master of Information Technology degree in Internetworking from the University of New South Wales, Sydney, Australia, in 2017. He is passionate about helping customers solve challenging technical issues related to their ETL workloads and implement scalable data integration and analytics pipelines on AWS. Noritaka Sekiyama is an experienced big data engineer working at a data and AI company. He is responsible for building scalable data platforms with unified governance in the cloud. He is passionate about software engineering, cloud computing, big data technologies, distributed systems, data platforms, system monitoring, and automation. Tomohiro Tanaka is a big data specialist with deep, hands-on expertise in data infrastructure. His expertise covers large-scale migrations, performance tuning, and production troubleshooting, with a focus on Apache Spark and Apache Iceberg. He contributes to the Apache Iceberg open-source project and speaks at community events and conferences to help teams adopt Apache Iceberg in practice. Albert Quiroga is a Senior Solutions Architect at Amazon, where he creates solutions and architectural designs for one of the largest data lakes in the world. Prior to that, he spent four years working at AWS, where he specialized in big data technologies such as Amazon EMR, Amazon Athena, AWS Glue, and Amazon SageMaker. His 11 years of experience in the industry have empowered him to work with several Fortune 500 companies to overcome large-scale data and analytics challenges, and he has helped launch and develop features for several AWS services. Ishan Gaur has more than 17 years of IT experience in software development, data engineering, and cloud architecture, building distributed systems and highly scalable data processing pipelines using Apache Spark, Scala, and various AWS data services, such as AWS Glue, Amazon SageMaker Unified Studio, and Amazon EMR. He currently works at AWS as a Principal Cloud Engineer, where he is focused on AI/ML operations and proactive cloud optimization. He works with AWS enterprise customers to design resilient data pipelines, automate incident response, troubleshoot large-scale distributed data platforms, and adopt GenAI-powered services and operational tools. He is passionate about turning reactive support patterns into proactive, self-healing architectures.
