Kukreja Data Engineering with Apache Spark, Delta Lake, and Lakehouse

Create scalable pipelines that ingest, curate, and aggregate complex data in a timely and secure way
1. Auflage 2021
ISBN: 978-1-80107-432-2
Verlag: De Gruyter
Format: EPUB
Kopierschutz: 0 - No protection

E-Book, Englisch, 480 Seiten

No detailed description available for "Data Engineering with Apache Spark, Delta Lake, and Lakehouse".

Autoren/Hrsg.

Kukreja, Manoj

Weitere Mitwirkende

Zburivsky, Danil

Fachgebiete

• Mathematik | Informatik EDV | Informatik Daten / Datenbanken Automatische Datenerfassung, Datenanalyse
• Mathematik | Informatik EDV | Informatik Daten / Datenbanken Datenbankdesign & Datenbanktheorie
• Mathematik | Informatik EDV | Informatik Informatik Mensch-Maschine-Interaktion Informationsarchitektur
• Mathematik | Informatik EDV | Informatik Programmierung | Softwareentwicklung Programmier- und Skriptsprachen

Inhaltsverzeichnis

Table of Contents - The Story of Data Engineering and Analytics
- Discovering Storage and Compute Data Lake Architectures
- Data Engineering on Microsoft Azure
- Understanding Data Pipelines
- Data Collection Stage - The Bronze Layer
- Understanding Delta Lake
- Data Curation Stage - The Silver Layer
- Data Aggregation Stage - The Gold Layer
- Deploying and Monitoring Pipelines in Production
- Solving Data Engineering Challenges
- Infrastructure Provisioning
- Continuous Integration and Deployment (CI/CD) of Data Pipelines

Über Autor(innen)

Kukreja Manoj:
Manoj Kukreja is a Principal Architect at Northbay Solutions who specializes in creating complex Data Lakes and Data Analytics Pipelines for large-scale organizations such as banks, insurance companies, universities, and US/Canadian government agencies. Previously, he worked for Pythian, a large managed service provider where he was leading the MySQL and MongoDB DBA group and supporting large-scale data infrastructure for enterprises across the globe. With over 25 years of IT experience, he has delivered Data Lake solutions using all major cloud providers including AWS, Azure, GCP, and Alibaba Cloud. On weekends, he trains groups of aspiring Data Engineers and Data Scientists on Hadoop, Spark, Kafka and Data Analytics on AWS and Azure Cloud.