Sign-In
Cart
Engineering Big Data Pipelines with Hadoop and Spark: Store, Process, and Analyze Large-Scale Datasets Using Hadoop, Spark, and PySpark
Paperback
Publisher: Independently Published
Published: Aug 14 2026
Pages: 168
Weight: 0.47
Height: 0.42 Width: 6.00 Depth: 9.00
Language: English
Learn how to build practical big data pipelines with Hadoop, Apache Spark, and PySpark.
As datasets grow beyond the practical limits of a single computer, organizations need reliable ways to store, process, clean, transform, and analyze information at scale. Engineering Big Data Pipelines with Hadoop and Spark provides a practical introduction to the tools and workflows used to solve these problems.
This beginner friendly guide takes you from the foundations of big data and distributed computing to complete processing pipelines using Hadoop, Spark, and Python. You will learn not only what the technologies do, but how their individual components fit together in a repeatable data engineering workflow.
Inside this book, you will learn how to:
- Understand big data, distributed computing, batch processing, and streaming
- Understand the Hadoop ecosystem and how its major components work together
- Work with HDFS for distributed storage
- Understand MapReduce and parallel batch processing
- Use YARN for cluster resource management
- Set up Hadoop, Spark, and PySpark on a practical local environment
- Create and work with PySpark applications
- Understand RDDs, transformations, actions, and Spark execution
- Work with Spark DataFrames for large scale data processing
- Query large datasets using Spark SQL
- Read and clean CSV, JSON, and Parquet datasets
- Transform large datasets using PySpark
- Understand partitions, shuffling, caching, persistence, and broadcast joins
- Read Spark execution plans and use the Spark Web UI for performance analysis
- Process continuously arriving data with Structured Streaming
- Build introductory machine learning workflows using Spark MLlib
- Develop classification and regression models with Spark
- Save and reload machine learning pipelines
- Build and monitor a complete end to end big data processing project
The book emphasizes a practical workflow rather than memorizing commands. You will learn to move data through a complete pipeline: store it, inspect it, clean it, transform it, analyze it, optimize the processing job, and save the results.
Practical examples use familiar business data including sales transactions, customer orders, product records, website logs, incoming events, and customer behavior data. The final project combines the major skills developed throughout the book by moving raw transaction data into HDFS, processing it with PySpark, analyzing it with Spark SQL, optimizing the job, and saving the results as Parquet.
You do not need previous experience with Hadoop, Spark, distributed systems, or data engineering. Basic computer skills are enough to begin, while some familiarity with Python and SQL can be helpful as you progress into PySpark and Spark SQL.
Whether you are a student learning data processing, a Python user working with larger datasets, an analyst exploring distributed computing, a software developer moving into data engineering, or a technical professional preparing to work with Hadoop and Spark, this book provides a structured path from fundamentals to practical implementation.
Build the skills to design, process, optimize, and understand large scale data pipelines with Hadoop, Spark, and PySpark.
Also from
Westwood, Nathan
Developing with PX4: A Guide to Customizing and Extending the Open-Source Flight Stack
Westwood, Nathan
Paperback
Home Assistant Control Panels for Beginners: Design Clean Dashboards for Lighting, Security, Climate, and Energy Use
Westwood, Nathan
Paperback
Natural Language Processing with spaCy: Extracting insights and entities from unstructured text data
Westwood, Nathan
Paperback
Ethical AI and Responsible Coding: Learn fairness, bias prevention, and transparent design
Westwood, Nathan
Paperback
Micro-ROS for Microcontrollers: How to Bring Professional Robotics Software to Tiny, Low-Power Computer Chips
Westwood, Nathan
Paperback
Open-Source Automation for Beginners: Design Efficient Workflows with Home Assistant, Node-RED, MQTT, and Custom Components
Westwood, Nathan
Paperback
MQTT Smart Home Messaging for Beginners: Connect Home Assistant, ESPHome, and IoT Devices with Simple Workflows
Westwood, Nathan
Paperback
Generative AI with Stable Diffusion: Creating custom image models and workflows for digital art
Westwood, Nathan
Paperback
PyTorch for Deep Learning: Building and training neural networks for predictive analytics
Westwood, Nathan
Paperback
Drone Coding with Python: A Practical Approach to Automating Aerial Tasks and Workflows
Westwood, Nathan
Paperback
Procedural Programming at Scale: Exploring Advanced Concepts in Structured Code, Memory Management, and Optimization
Westwood, Nathan
Paperback
MAVLink for Developers: Building Secure and Reliable Communication Links for Unmanned Systems
Westwood, Nathan
Paperback
Smart Home Repair Guide for Beginners: Fix Offline Devices, Failed Automations, Weak Networks, and Common Setup Problems
Westwood, Nathan
Paperback
The Smart Home Announcer: Set Up Private, Text-to-Speech Audio Alerts That Tell You When the Laundry Is Done or a Door Opens
Westwood, Nathan
Paperback
Renter-Friendly Smart Home Setup: Affordable Automation Without Rewiring, Drilling, or Permanent Changes
Westwood, Nathan
Paperback
Smart Lighting Projects for Beginners: Build Dynamic Lighting Systems with WLED, ESPHome, and Home Assistant Integration
Westwood, Nathan
Paperback
Private Smart Home Setup for Beginners: Build Reliable Home Assistant Automations Without Depending on the Cloud
Westwood, Nathan
Paperback
Smart Home Systems for Beginners: Build Reliable Solutions for Lighting, Monitoring, Notifications, and Home Automation
Westwood, Nathan
Paperback
Connected Smart Homes for Beginners: Learn How to Integrate Home Assistant, MQTT, and Smart Devices for Reliable Home Management
Westwood, Nathan
Paperback
Deep Learning on Mobile Robots: Running AI on Your Robot to Recognize Objects, Street Signs, and Human Faces in Real Time
Westwood, Nathan
Paperback
LiDAR Data Processing with Python: Turn 3D Laser Scanner Point Clouds into Detailed Spatial Maps for Autonomous Mobile Robots
Westwood, Nathan
Paperback
The Off-Grid Smart Cabin: Automating solar power, water tanks, and lighting for remote homes and RVs without internet access
Westwood, Nathan
Paperback
The Healthy Smart Home: Build Custom Sensors to Monitor Indoor Air Quality and Trigger Your Purifiers and Fans Automatically
Westwood, Nathan
Paperback
Zigbee Device Setup for Beginners: Build a Stable Smart Home Network with Sensors, Switches, and Home Assistant
Westwood, Nathan
Paperback
Smart Home Control for Beginners: Master Device Integration Using Sensors, Switches, Zigbee Networks, and Home Assistant
Westwood, Nathan
Paperback
Connected Devices for Beginners: A Hands-On Guide to Zigbee Networks, MQTT Communication, and Home Assistant Integration
Westwood, Nathan
Paperback
The Kinematics of Robotic Arms: Master the Math Needed to Calculate Exactly Where a Robot Hand Will End Up in 3D Space
Westwood, Nathan
Paperback
TensorFlow.js for Web Developers: Running machine learning models directly in the browser
Westwood, Nathan
Paperback
Simulation in Ros: GAZEBO AND RVIZ: Using simulation tools like Gazebo and RViz to visualize and test robots in ROS
Westwood, Nathan
Paperback
Fine-Tuning Large Language Models: Adapting GPT and Llama architectures for specialized industry tasks
Westwood, Nathan
Paperback
Generative AI with Python: Create text, images, and audio using open-source models
Westwood, Nathan
Paperback
Understanding the MAVLink Protocol: A Guide to Parsing Telemetry and Sending Commands Over the Air
Westwood, Nathan
Paperback
Flight Control with PX4: Understanding the Mathematical Logic Behind Autonomous Stability
Westwood, Nathan
Paperback
Designing Autonomous Systems: A Comprehensive Guide to Robotic Intelligence and Safety
Westwood, Nathan
Paperback
CI/CD Made Simple: Automate builds, testing, and deployment pipelines efficiently
Westwood, Nathan
Paperback
Docker for Beginners: Simplify app deployment with hands-on container projects
Westwood, Nathan
Paperback
AI-Powered Apps with TensorFlow Lite: Deploy models efficiently on mobile and embedded devices
Westwood, Nathan
Paperback
Reinforcement Learning in Robotics: Training autonomous agents to navigate complex physical environments
Westwood, Nathan
Paperback
Advanced Concepts in High-Level Programming: Leveraging Modern Programming Languages to Design Robust and Scalable Applications
Westwood, Nathan
Paperback
Mastering Procedural Programming Paradigms: Advanced Techniques in C, Fortran, and COBOL for Optimizing Sequential Systems
Westwood, Nathan
Paperback
Swarm Robotics Systems: Programming Groups of Robots to Explore a Space and Solve Problems Collectively
Westwood, Nathan
Paperback
Embedded Systems for Robotics: Writing the High-Speed Code That Connects Python Commands to Electrical Signals
Westwood, Nathan
Paperback
Network Services Administration: Managing DNS, DHCP, Web Services, File Services, Remote Access, and Network Infrastructure
Westwood, Nathan
Paperback
Linux Server Management: Practical Methods for Configuring, Securing, Monitoring, and Maintaining Linux Servers
Westwood, Nathan
Paperback
Network Troubleshooting Fundamentals: Methods for Diagnosing Connectivity, DNS, Routing, Wireless, and Performance Problems
Westwood, Nathan
Paperback
The Open Source IoT Workshop: Hands On Projects with ESP32, Raspberry Pi, MQTT, Node RED, LoRaWAN, Zigbee, and Home Assistant
Westwood, Nathan
Paperback
Device Control for Beginners: Manage Sensors, Switches, and Smart Home Technology Through Home Assistant Workflows
Westwood, Nathan
Paperback
ESPHome Device Projects for Beginners: Create Custom Sensors, Relays, and Smart Controls Without Advanced Coding
Westwood, Nathan
Paperback
Open Smart Home Workflows for Beginners: A Beginner-Friendly Guide to Building Reliable Home Assistant and MQTT Automations
Westwood, Nathan
Paperback
Smart Home Energy Tracking for Beginners: Monitor Power Use, Reduce Waste, and Build Home Assistant Dashboards
Westwood, Nathan
Paperback
Advanced PID Tuning: Master the Mathematical Feedback Loops That Keep Robots Moving Smoothly
Westwood, Nathan
Paperback
Single Board Computers for Beginners: Build Projects with Raspberry Pi, Arduino, ESP32, Linux, Sensors, and Connected Devices
Westwood, Nathan
Paperback
Also in
Programming
Interactive Theorem Proving and Program Development: Coq'art: The Calculus of Inductive Constructions
Castéran, Pierre
Bertot, Yves
Hardcover
The Legend of Zelda(tm) Tears of the Kingdom - The Complete Official Guide: Collector's Edition
Piggyback
Hardcover
Python Crash Course, 3rd Edition: A Hands-On, Project-Based Introduction to Programming
Matthes, Eric
Paperback
The Legend of Zelda(tm) Tears of the Kingdom - The Complete Official Guide: Standard Edition
Piggyback
Paperback
Crisis Engineering: Time-Tested Tools for Turning Chaos Into Clarity
Dickerson, Mikey
Nitze, Marina
Weaver, Matthew
Paperback
Accelerate: The Science of Lean Software and DevOps: Building and Scaling High Performing Technology Organizations
Humble, Jez
Kim, Gene
Forsgren Phd, Nicole
Paperback
Embedded Systems with ARM Cortex-M Microcontrollers in Assembly Language and C: Fourth Edition
Zhu, Yifeng
Paperback
Fundamentals of Software Architecture: A Modern Engineering Approach
Richards, Mark
Ford, Neal
Paperback
Make: Electronics: Learning by Discovery: A Hands-On Primer for the New Electronics Enthusiast
Platt, Charles
Paperback
Vibe Coding: Building Production-Grade Software with Genai, Chat, Agents, and Beyond
Kim, Gene
Yegge, Steve
Paperback
AI Projects with Raspberry Pi: High-Performance Artificial Intelligence for Robotics, Security, Home Automation, and Vision
Jepson, Brian
Hattersley, Lucy
Paperback
The Manager's Path: A Guide for Tech Leaders Navigating Growth and Change
Fournier, Camille
Paperback
I Have an App Idea: The Essential Guide to Building an App Without Tech Skills
Spann, Amanda
Paperback
The Return on AI: From Promise to Profit in the Age of Intelligent Business
Mittal, Ashwin
Sawhney, Mohanbir
Paperback
The Devops Handbook, 2nd Edition: How to Create World-Class Agility, Reliability, & Security in Technology Organizations
Humble, Jez
Debois, Patrick
Kim, Gene
Paperback
The Staff Engineer's Path: A Guide for Individual Contributors Navigating Growth and Change
Reilly, Tanya
Paperback
The Developer's Guide to AI: A Field Guide for the Working Developer
Thompson, Danny
Orshalick, Jacob
Reghunadh, Jerry M.
Paperback
Building Applications with AI Agents: Designing and Implementing Multiagent Systems
Albada, Michael
Paperback
Software Architecture: The Hard Parts: Modern Trade-Off Analyses for Distributed Architectures
Sadalage, Pramod
Ford, Neal
Richards, Mark
Paperback
Cracking the Coding Interview: 189 Programming Questions and Solutions
McDowell, Gayle Laakmann
Paperback
Learning Web Design: A Beginner's Guide to Html, Css, Javascript, and Web Images
Robbins, Jennifer
Paperback
Observability Engineering: Achieving Production Excellence
Fong-Jones, Liz
Miranda, George
Majors, Charity
Paperback
Forecasting: Principles and Practice, the Pythonic Way
Hyndman, Rob J.
Athanasopoulos, George
Paperback
The New AI Cold War: Liberty vs. Tyranny in the Age of Machine Empires
Maginnis, Ltc Robert L.
Paperback
Linux Basics for Hackers, 2nd Edition: Getting Started with Networking, Scripting, and Security in Kali
Occupytheweb
Paperback
Building Safer Technology: A Field Guide to Failing Well
Kissner, Lea
Zunger, Yonatan
Coles, Neil
Paperback
The Pragmatic Programmer: Your Journey to Mastery, 20th Anniversary Edition
Hunt, Andrew
Thomas, David
Hardcover
A Human's Guide to Machine Intelligence: How Algorithms Are Shaping Our Lives and How We Can Stay in Control
Hosanagar, Kartik
Paperback
SQL QuickStart Guide: The Simplified Beginner's Guide to Managing, Analyzing, and Manipulating Data With SQL
Shields, Walter
Hardcover
Grokking Algorithms, Second Edition: An Illustrated Guide for Programmers and Other Curious People
Bhargava, Aditya Y.
Paperback
The Official Raspberry Pi Handbook 2026: Astounding Projects with Raspberry Pi Computers
Makers of Raspberry Pi Official Magazine, The
Paperback
Generative AI Design Patterns: Solutions to Common Challenges When Building Genai Agents and Applications
Hapke, Hannes
Lakshmanan, Valliappa
Paperback
Concrete Mathematics: A Foundation for Computer Science
Knuth, Donald
Graham, Ronald
Patashnik, Oren
Hardcover
Learning Domain-Driven Design: Aligning Software Architecture and Business Strategy
Khononov, Vlad
Paperback
Head First Design Patterns: Building Extensible and Maintainable Object-Oriented Software
Freeman, Eric
Robson, Elisabeth
Paperback
The Software Architect Elevator: Redefining the Architect's Role in the Digital Enterprise
Hohpe, Gregor
Paperback
Arduino Programming for Beginners: A Comprehensive Beginner's Guide to Learn the Realms of Arduino Programming from A-Z
Protosmith, Ada
Paperback
Learning Generative AI Tools for Excel: Speed Up Your Everyday Tasks with Microsoft Excel, Copilot, Chatgpt, and Beyond
Duca, Angelica Lo
Paperback
Building AI-Powered Products: The Essential Guide to AI and Genai Product Management
Nika, Marily
Paperback
Debugging: The 9 Indispensable Rules for Finding Even the Most Elusive Software and Hardware Problems
Agans, David J.
Paperback
Building Resilient Distributed Systems: Patterns and Practices for Stable Software
Newman, Sam
Paperback
Frictionless: 7 Steps to Remove Barriers, Unlock Value, and Outpace Your Competition in the AI Era
Noda, Abi
Forsgren, Nicole
Paperback
Build an AI Agent (from Scratch): Agents That Reason, Plan, and ACT Autonomously
Hur, Jungjun
Song, Younghee
Paperback
AI Agents with MCP: Model Context Protocol for Building Clients, Servers, and End-To-End Agents
Stratis, Kyle
Paperback
Signals & Levers: Systems Thinking Tools to Unblock Software Delivery
Tosi, Joel
Hendrickson, Elisabeth
Paperback
Software Engineering at Google: Lessons Learned from Programming Over Time
Manshreck, Tom
Wright, Hyrum
Winters, Titus
Paperback
Learning Php, MySQL & JavaScript: A Step-By-Step Guide to Creating Dynamic Websites
Nixon, Robin
Paperback
Python Programming for Young Coders: A Hands-On, Project-Based Introduction to Coding for Beginners, Kids, and Teens
Pandey, Anand
Paperback
Living a Jewish Life, Revised and Updated: Jewish Traditions, Customs, and Values for Today's Families
Diamant, Anita
Cooper, Howard
Paperback
Java in a Nutshell: A Desktop Quick Reference
Flanagan, David
Evans, Benjamin J.
Clark, Jason R.
Paperback
Programming Massively Parallel Processors: A Hands-On Approach
El Hajj, Izzat
Hwu, Wen-Mei W.
Kirk, David B.
Paperback
Computer Science from Scratch: Building Interpreters, Art, Emulators and ML in Python
Kopec, David
Paperback
Adobe Photoshop, 2nd Edition: A Complete Course and Compendium of Features
Laskevitch, Stephen
Paperback
Site Reliability Engineering: How Google Runs Production Systems
Beyer, Betsy
Jones, Chris
Leng, Christof
Paperback
Kubernetes in Action, Second Edition: Deploying and Managing Containers and Cloud-Native Applications
Luksa, Marko
Conner, Kevin
Paperback
Electronic Music and Sound Design - Theory and Practice with Max 9 - Volume 1 (Fifth Edition)
Maurizio, Giri
Cipriani, Alessandro
Paperback
Prompt Engineering for Llms: The Art and Science of Building Large Language Model-Based Applications
Berryman, John
Ziegler, Albert
Paperback
Practical AI Security: A Hands-On Guide to Attacking, Defending, and Securing Modern AI Systems
Farlow, Harriet
Paperback