Sign-In
Cart
Data Engineering with Azure Databricks: Design, build, and optimize scalable data pipelines and analytics solutions with Azure Databricks
Paperback
ISBN13: 9781806106370
Publisher: Packt Publishing
Published: Apr 30 2026
Pages: 412
Weight: 1.55
Height: 0.84 Width: 7.50 Depth: 9.25
Language: English
Master end-to-end data engineering on Azure Databricks. From data ingestion and Delta Lake to CI/CD and real-time streaming, build secure, scalable, and performant data solutions with Spark, Unity Catalog, and ML tools.
Key Features:
- Build scalable data pipelines using Apache Spark and Delta Lake
- Automate workflows and manage data governance with Unity Catalog
- Learn real-time processing and structured streaming with practical use cases
- Implement CI/CD, DevOps, and security for production-ready data solutions
- Explore Databricks-native ML, AutoML, and Generative AI integration
Book Description:
Data Engineering with Azure Databricks is your essential guide to building scalable, secure, and high-performing data pipelines using the powerful Databricks platform on Azure. Designed for data engineers, architects, and developers, this book demystifies the complexities of Spark-based workloads, Delta Lake, Unity Catalog, and real-time data processing.
Beginning with the foundational role of Azure Databricks in modern data engineering, you'll explore how to set up robust environments, manage data ingestion with Auto Loader, optimize Spark performance, and orchestrate complex workflows using tools like Azure Data Factory and Airflow.
The book offers deep dives into structured streaming, Delta Live Tables, and Delta Lake's ACID features for data reliability and schema evolution. You'll also learn how to manage security, compliance, and access controls using Unity Catalog, and gain insights into managing CI/CD pipelines with Azure DevOps and Terraform.
With a special focus on machine learning and generative AI, the final chapters guide you in automating model workflows, leveraging MLflow, and fine-tuning large language models on Databricks. Whether you're building a modern data lakehouse or operationalizing analytics at scale, this book provides the tools and insights you need.
What You Will Learn:
- Set up a full-featured Azure Databricks environment
- Implement batch and streaming ingestion using Auto Loader
- Optimize Spark jobs with partitioning and caching
- Build real-time pipelines with structured streaming and DLT
- Manage data governance using Unity Catalog
- Orchestrate production workflows with jobs and ADF
- Apply CI/CD best practices with Azure DevOps and Git
- Secure data with RBAC, encryption, and compliance standards
- Use MLflow and Feature Store for ML pipelines
- Build generative AI applications in Databricks
Who this book is for:
This book is for data engineers, solution architects, cloud professionals, and software engineers seeking to build robust and scalable data pipelines using Azure Databricks. Whether you're migrating legacy systems, implementing a modern lakehouse architecture, or optimizing data workflows for performance, this guide will help you leverage the full power of Databricks on Azure. A basic understanding of Python, Spark, and cloud infrastructure is recommended.
Table of Contents
- The role of Azure Databricks in modern data engineering
- Setting up an end-to-end Azure Databricks environment
- Data ingestion strategies for Azure Databricks
- Deep dive into Apache Spark on Azure Databricks
- Streaming architectures with structured streaming
- Working with Delta Lake: ACID transactions & schema evolution
- Automating data pipelines with Delta Live Tables (DLT)
- Orchestrating data workflows: from notebooks to production
- CI/CD and DevOps for Azure Databricks
- Optimizing query performance and cost management
- Security, compliance, and data governance
- Machine learning, AutoML, and generative AI in Databricks
Also from
Foshin, Dmitry
Jumpstart Snowflake: A Step-By-Step Guide to Modern Cloud Analytics
Foshin, Dmitry
Strok, Donna
Anoshin, Dmitry
Paperback
Azure Data Factory Cookbook - Second Edition: A data engineer's guide to building and managing ETL and ELT pipelines with data integration
Foshin, Dmitry
Anoshin, Dmitry
Chernyshova, Tonya
Paperback
Azure Data Factory Cookbook: Build and manage ETL and ELT pipelines with Microsoft Azure's serverless data integration service
Anoshin, Dmitry
Foshin, Dmitry
Storchak, Roman
Paperback
Also in
Databases
Designing Data-Intensive Applications: The Big Ideas Behind Reliable, Scalable, and Maintainable Systems
Kleppmann, Martin
Riccomini, Chris
Paperback
Fundamentals of Data Engineering: Plan and Build Robust Data Systems
Housley, Matt
Reis, Joe
Paperback
SQL QuickStart Guide: The Simplified Beginner's Guide to Managing, Analyzing, and Manipulating Data With SQL
Shields, Walter
Hardcover
Observability Engineering: Achieving Production Excellence
Miranda, George
Majors, Charity
Fong-Jones, Liz
Paperback
The Data Warehouse Toolkit: The Definitive Guide to Dimensional Modeling
Ross, Margy
Kimball, Ralph
Paperback
Introduction to Statistics: An Intuitive Guide for Analyzing Data and Unlocking Discoveries
Frost, Jim
Paperback
Learning Tableau 2025 - Sixth Edition: Leverage Tableau's newest features to revolutionize your data storytelling with AI-enhanced insights
Milligan, Joshua N.
Paperback
Ai-Assisted Statistics for Data Scientists: 50+ Essential Concepts Using R and Python
Bruce, Peter
Bruce, Andrew
Gedeck, Peter
Paperback
The Data Storyteller's Handbook: How to create business impact using data storytelling
Greenbrook, Kat
Paperback
Database Systems: Introduction to Databases and Data Warehouses, Edition 2.0
Jukic, Nenad
Vrbsky, Susan
Sharma, Abhishek
Paperback
Microsoft Power Bi Data Analyst Associate Study Guide: Prepare for the Pl-300 Exam and Use Power Bi Successfully
Turley, Paul
Paperback
SQL for Data Analysis: Advanced Techniques for Transforming Data Into Insights
Tanimura, Cathy
Paperback
High Performance Spark: Best Practices for Scaling and Optimizing Apache Spark
Polak, Adi
Warren, Rachel
Karau, Holden
Paperback
Data Engineering for Cybersecurity: Build Secure Data Pipelines with Free and Open-Source Tools
Bonifield, James
Paperback
Data Engineering Design Patterns: Recipes for Solving the Most Common Data Engineering Problems
Konieczny, Bartosz
Paperback
SQL Server 2025 Unveiled: The Ai-Ready Enterprise Database with Microsoft Fabric Integration
Ward, Bob
Paperback
Mathletics: How Gamblers, Managers, and Fans Use Mathematics in Sports, Second Edition
Winston, Wayne L.
Nestler, Scott
Pelechrinis, Konstantinos
Paperback
Becoming a Data Head: How to Think, Speak, and Understand Data Science, Statistics, and Machine Learning
Gutman, Alex J.
Goldmeier, Jordan
Paperback
SQL QuickStart Guide: The Simplified Beginner's Guide to Managing, Analyzing, and Manipulating Data With SQL
Shields, Walter
Paperback
Databricks Certified Data Engineer Associate Study Guide: In-Depth Guidance and Practice
Alhussein, Derar
Paperback
Observability Engineering: Achieving Production Excellence
Majors, Charity
Fong-Jones, Liz
Miranda, George
Paperback
Product Operations: How successful companies build better products at scale
Tilles, Denise
Perri, Melissa
Paperback
Turning Data into Wisdom: How We Can Collaborate with Data to Change Ourselves, Our Organizations, and Even the World
Hanegan, Kevin
Paperback
Kafka: The Definitive Guide: Real-Time Data and Stream Processing at Scale
Sivaram, Rajini
Shapira, Gwen
Palino, Todd
Paperback
PR Technology, Data and Insights: Igniting a Positive Return on Your Communications Investment
Weiner, Mark
Paperback
Foundations of Deep Reinforcement Learning: Theory and Practice in Python
Graesser, Laura
Keng, Wah Loon
Paperback
High Performance Python: Practical Performant Programming for Humans
Gorelick, Micha
Ozsvald, Ian
Paperback
Head First Algorithms and Data Structures: A Learner's Guide to the Foundations of Computing
Freeman, Eric
Robson, Elisabeth
Paperback
Data Contracts: Developing Production-Grade Pipelines at Scale
Freeman, Mark
Sanderson, Chad
Schmidt, B. E.
Paperback
Databricks Data Intelligence Platform: Unlocking the Genai Revolution
Gupta, Nikhil
Yip, Jason
Paperback
Data Analytics Made Easy: Analyze and present data to make informed decisions without writing any code
Mauro, Andrea de
Paperback
ML and Generative AI in the Data Lakehouse: Building and Deploying AI Applications at Scale
Haelen, Bennie
Paperback
Data Governance with Unity Catalog on Databricks: Implement Data and AI Governance with Databricks Data Intelligence Platform
Sreekumar, Kiran
Subbarao, Karthik
Paperback
SQL Cookbook: Query Solutions and Techniques for All SQL Users
Graaf, Robert de
Molinaro, Anthony
Paperback
Numerical Python: Scientific Computing and Data Science Applications with Numpy, Scipy and Matplotlib
Johansson, Robert
Paperback
Data Analytics & Visualization All-In-One for Dummies
McFedries, Paul
Massaron, Luca
Hyman, Jack A.
Paperback
The Data Center as a Computer: Designing Warehouse-Scale Machines
Barroso, Luiz André
Hölzle, Urs
Ranganathan, Parthasarathy
Hardcover
Hands-On Data Analysis with Pandas: Efficiently perform data collection, wrangling, analysis, and visualization using Python
Molin, Stefanie
Paperback
Advanced Snowflake: Processing Data, Developing Applications, and Deploying ML Models at Scale
Ullah, Muhammad Fasih
Paperback
Non-Invasive Data Governance: The Path of Least Resistance and Greatest Success
Seiner, Robert
Paperback
Kafka for Architects: Event-Driven Architecture, Logs, Microservices, Real-Time Event Processing
Gorshkova, Katya
Paperback
SQL Server 2025 Query Performance Tuning: Troubleshoot and Optimize Query Performance
Fritchey, Grant
Paperback
Ai-Ready Data Blueprints: From Raw Data to Ai-Driven Innovation
Pham, Kien
Sopirala, Srikanth
Shukla, Navnit
Paperback
Apache Iceberg: The Definitive Guide: Data Lakehouse Functionality, Performance, and Scalability on the Data Lake
Shiran, Tomer
Hughes, Jason
Merced, Alex
Paperback
Learning Statistics with Jamovi: A Tutorial for Beginners in Statistical Analysis
Foxcroft, David
Navarro, Danielle
Paperback
Agile Data Warehouse Design: Collaborative Dimensional Modeling, from Whiteboard to Star Schema
Corr, Lawrence
Stagnitto, Jim
Paperback
Real-Time Intelligence with Microsoft Fabric: Empowering Data-Driven Decisions in the Era of AI
Brattã S. Johan Ludvig
Geisler, Frank
Paperback
iPhone 11 Pro Max Guide: Learn Step-By-Step How To Use Your iPhone Pro Max Step-By-Step
Meyer, Albrecht
Paperback
Fusion Strategy: How Real-Time Data and AI Will Power the Industrial Future
Venkatraman, Venkat
Govindarajan, Vijay
Hardcover
Learning Ransomware Response & Recovery: Stopping Ransomware One Restore at a Time
Saylor, Michael
Preston, W. Curtis
Paperback
Data Pipelines with Apache Airflow, Second Edition: Orchestration for Data and AI
Geusebroek, Kris
Ruiter, Julian de
Cabral, Ismael
Paperback
Data and Reality: A Timeless Perspective on Perceiving and Managing Information in Our Imprecise World, 3rd Edition
Kent, William
Paperback
Building Data Integration Solutions: Unifying Data for Enhanced Decision Making
Borthen, Jay
Paperback
Apache Hudi: The Definitive Guide: Building Robust, Open, and High-Performing Data Lakehouses
Xu, Shiyan
Wason, Prashant
Saktheeswaran, Bhavani Sudha
Paperback
How to Interpret Data: Using Data to Improve Your Influence and Decision-Making
Kelly, Nicholas
Paperback
Apache Polaris: The Definitive Guide: Enriching Apache Iceberg Data Lakehouses with an Open Source Catalog
Shiran, Tomer
Madson, Andrew
Merced, Alex
Paperback
Data Governance Change Management: From Drift to Direction-Why Every Data Leader Needs an ANCHOR
Agrawal, Arvind
Agrawal, Aakriti
Paperback
Delta Lake: The Definitive Guide: Modern Data Lakehouse Architectures with Data Lakes
Haines, Scott
Lee, Denny
Wentling, Tristen
Paperback
Mastering PostgreSQL 17 - Sixth Edition: Elevate your database skills with advanced deployment, optimization, and security strategies
Schönig, Hans-Jürgen
Paperback
Practical Natural Language Processing: A Comprehensive Guide to Building Real-World Nlp Systems
Gupta, Anuj
Vajjala, Sowmya
Majumder, Bodhisattwa
Paperback
Football Analytics with Python & R: Learning Data Science Through the Lens of Sports
Eager, Eric A.
Erickson, Richard a.
Paperback
PostgreSQL 16 Administration Cookbook: Solve real-world Database Administration challenges with 180+ practical recipes and best practices
Mejías, Boriss
Angelakos, Jimmy
Ciolli, Gianni
Paperback
Data Modeling with Microsoft Power BI: Self-Service and Enterprise Data Warehouse with Power BI
Ehrenmueller-Jensen, Markus
Paperback
Everybody Lies: Big Data, New Data, and What the Internet Can Tell Us about Who We Really Are
Stephens-Davidowitz, Seth
Paperback
The Elements of Statistical Learning: Data Mining, Inference, and Prediction, Second Edition
Tibshirani, Robert
Friedman, Jerome
Hastie, Trevor
Hardcover
Unlocking Dbt: Design and Deploy Transformations in Your Cloud Data Warehouse
Cyr, Cameron
Dorsey, Dustin
Paperback
The Data Product Playbook: Designing and Delivering Data Products that Power Decisions, Analytics, and AI
Koenders, Willem
Paperback
Big Data in Der Mobilität: Akteure, Geschäftsmodelle Und Nutzenpotenziale Für Die Welt Von Morgen
Müller-Peters, Horst
Gatzert, Nadine
Knorre, Susanne
Paperback
Analytics Engineering with SQL and Dbt: Building Meaningful Data Models at Scale
Russa, Hélder
Machado, Rui
Paperback
Learn PostgreSQL - Second Edition: Use, manage and build secure and scalable databases with PostgreSQL 16
Pirozzi, Enrico
Ferrari, Luca
Paperback
SQL for Data Analytics - Fourth Edition: Analyze data effectively, uncover insights and master advanced SQL for real-world applications
Li, Haibin
Goldwasser, Matt
Shan, Jun
Paperback
Natural Language Processing with Pytorch: Build Intelligent Language Applications Using Deep Learning
Rao, Delip
McMahan, Brian
Paperback
A Comprehensive Introduction to Python: From the Basics to Data Analytics: From the Basics to Data Analytics
Masloff, Jackie
Paperback
The Expert's Guide to SQL Server: Tuning, Security, and Cloud Migration
Pasam, Venkata Reddy
Andiappan, Petchikumar
Paperback
Architecting an Apache Iceberg Lakehouse: A Scalable, Open-Source Data Platform
Merced, Alex
Paperback
Statistics Every Programmer Needs: Practical Python Implementations and Quantitative Methods
Sutton, Gary
Paperback
Data Science at the Command Line: Obtain, Scrub, Explore, and Model Data with UNIX Power Tools
Janssens, Jeroen
Paperback
Introduction to Statistics: An Intuitive Guide for Analyzing Data and Unlocking Discoveries
Frost, Jim
Hardcover
Stream Processing with Apache Flink: Fundamentals, Implementation, and Operation of Streaming Applications
Hueske, Fabian
Kalavri, Vasiliki
Paperback
Data Center Critical Facilities Migration from Construction to Operations
K. Tidjani, Yekini
Paperback
Data Strategy: How to Use Data and Artificial Intelligence to Transform Your Business
Marr, Bernard
Paperback
SQL in a Week: A Fast-Track Guide to Writing and Optimizing Queries Like a Pro
Shah, Bhumika
Dubey, Devtosh
Paperback
From Data to Dollars: Getting Started with Data Analytics and AI in Startups
Sidoruk, Piotr
Paperback
Data Governance: The Definitive Guide: People, Processes, and Tools to Operationalize Data Trustworthiness
Eryurek, Evren
Gilad, Uri
Lakshmanan, Valliappa
Paperback
Data Science on the Google Cloud Platform: Implementing End-To-End Real-Time Data Pipelines: From Ingest to Machine Learning
Lakshmanan, Valliappa
Paperback
The Elephant in the Fridge: Guided Steps to Data Vault Success through Building Business-Centered Models
Giles, John
Paperback
Data Engineering with Python: Work with massive datasets to design data models and automate data pipelines using Python
Crickard, Paul
Paperback
SAP S/4hana Financial Accounting Certification Guide: Application Associate Exam
Pougkas, Stefanos
Paperback
PostgreSQL Query Optimization: The Ultimate Guide to Building Efficient Queries
Bailliekova, Anna
Database Expert
Dombrovskaya, Henrietta
Paperback
Mastering PostgreSQL Administration: Internals, Operations, Monitoring, and Oracle Migration Strategies
Samayam, Arun Kumar
Kadambari, Phani
Ravi Kumar, Y. V.
Paperback