Sign-In
Cart
A/B Testing and Experiment Design: Plan Valid Experiments and Make Better Data-Driven Decisions
Paperback
MarketingGeneral ComputersProbability & Statistics
Publisher: Independently Published
Published: Sep 17 2026
Pages: 168
Weight: 0.47
Height: 0.42 Width: 6.00 Depth: 9.00
Language: English
A statistically significant result does not automatically mean you made the right decision.
A/B testing sounds simple: divide users into groups, introduce a change, compare the results, and choose the better-performing option.
In practice, trustworthy experimentation is far more demanding.
Was the assignment truly random? Did users actually see the treatment? Is the metric measuring real value-or just something easy to count? Was the experiment large enough to detect an effect that matters? Did repeated checking inflate the chance of a false positive? And what happens when the results are statistically significant but practically worthless?
A/B Testing and Experiment Design is a practical guide to planning, running, interpreting, and scaling experiments that produce evidence you can actually trust.
Rather than focusing only on formulas and significance thresholds, this book connects statistical thinking with the real decisions faced by product teams, analysts, marketers, engineers, founders, and data professionals.
Inside, you'll learn how to:
- Turn business questions into clear, testable hypotheses
- Separate correlation from causal evidence
- Choose meaningful primary metrics, guardrails, and diagnostic measures
- Select the right randomization unit for users, accounts, devices, stores, regions, or time periods
- Distinguish assignment from actual treatment exposure
- Detect sample ratio mismatch and other experiment-integrity failures
- Plan sample size, statistical power, minimum detectable effect, and experiment duration
- Interpret confidence intervals, p-values, effect sizes, and practical significance correctly
- Handle noisy revenue metrics, ratio metrics, clustering, and repeated observations
- Avoid common mistakes caused by peeking, multiple comparisons, and post-hoc analysis
- Use variance-reduction techniques such as CUPED, blocking, and covariate adjustment
- Choose between fixed-horizon, sequential, factorial, cluster, switchback, geo, crossover, and holdout designs
- Understand when multi-armed and contextual bandits are more appropriate
- Apply experimentation to product development, growth, marketing, pricing, recommendation systems, machine learning, generative AI, infrastructure, and release engineering
- Build experimentation platforms, metric catalogs, review processes, and organizational learning systems
The book emphasizes that trustworthy experiments require alignment between the decision, causal design, measurement, operations, and statistical inference-not simply a favorable dashboard result.
It also moves beyond basic A/B comparisons into factorial designs, cluster randomization, switchbacks, geo experiments, long-term holdouts, crossover designs, bandits, and quasi-experimental approaches.
Whether you're a product manager, data analyst, growth professional, marketer, UX researcher, software engineer, engineering manager, founder, operations professional, or student, this book provides a practical framework for turning uncertainty into disciplined evidence and better decisions.
Stop asking which variant won. Start asking what the experiment actually proved-and what decision the evidence justifies.
Also from
Halstead, Corin
Camera Vision for Drones: Build Detection, Tracking, and Landing Workflows with Practical Image-Processing Methods
Halstead, Corin
Paperback
The Cloud Compass: An Absolute Beginner's Guide to Understanding Cloud Computing, AWS, Azure, and Google Cloud
Halstead, Corin
Paperback
Drone Software Stack Explained: Understand How Firmware, Sensors, Commands, and Mission Apps Work Together
Halstead, Corin
Paperback
Drone Communication Protocols: Understand telemetry, commands, heartbeats, and link failures before they cause risk.
Halstead, Corin
Paperback
Drone Failsafe Engineering: Design responses for low battery, position drift, link loss, and emergency landing
Halstead, Corin
Paperback
Drone Autonomy with Practical Code: Build safer missions in simulation before testing them in real flight.
Halstead, Corin
Paperback
Network Security from Scratch: A Beginner's Guide to Firewalls, VPNs, and Building a Secure Home and Office Network
Halstead, Corin
Paperback
The Asynchronous Mind: Mastering Promises, Async/Await, and Modern Concurrency in JavaScript
Halstead, Corin
Paperback
The Responsive Revolution: Learn Modern CSS, Flexbox, and Grid to Build Stunning, Mobile-First Websites from Scratch
Halstead, Corin
Paperback
Beyond console.log(): A Practical Guide to JavaScript Functions, Objects, and Arrays for Aspiring Developers
Halstead, Corin
Paperback
DOMination: The Definitive Guide to Manipulating the DOM and Building Interactive Web Pages with Vanilla JavaScript
Halstead, Corin
Paperback
Secure Drone Communication: Protect Telemetry and Command Links from Interference, Spoofing, and Poor Error Handling
Halstead, Corin
Paperback
Aerial Robot Simulation: Test navigation, sensors, and mission logic before flying
Halstead, Corin
Paperback
Functional TypeScript: Applying Functional Concepts within a Strictly Typed Environment
Halstead, Corin
Paperback
Long-Range Sensor Networks: Build private monitoring systems for farms, homes, and remote assets
Halstead, Corin
Paperback
Flight Controller Integration: Connect Mission Code to Sensors, Commands, and Safety Checks
Halstead, Corin
Paperback
Drone Cybersecurity Field Guide: Reduce common risks in signals, firmware, networks, and ground-control workflows
Halstead, Corin
Paperback
UAV Flight Control for Beginners: Learn the software decisions behind stable and reliable autonomous aircraft
Halstead, Corin
Paperback
Practical UAV Algorithms: Improve path planning, telemetry handling, and mission performance without overpromising results
Halstead, Corin
Paperback
Drone Mapping Automation: Plan image capture, organize flight logs, and prepare reliable mapping data
Halstead, Corin
Paperback
Drone Mission Planning: Create Safer Waypoint, Mapping, Inspection, and Recovery Workflows
Halstead, Corin
Paperback
Smart Home Mesh Networks: Improve Device Reliability by Planning Signal Coverage, Routes, and Power Placement
Halstead, Corin
Paperback
REST API Security for Developers: Protect Data in Transit with Clearer Authentication, Encryption, and Error Handling
Halstead, Corin
Paperback
AI for Drone Decisions: Use lightweight models carefully for perception, planning, and safer autonomy
Halstead, Corin
Paperback
Drone SLAM and Mapping: Build safer mapping workflows with sensors, logs, and simulation tests
Halstead, Corin
Paperback
Drone Scripting Fundamentals: Read telemetry, control missions, and handle flight errors with confidence
Halstead, Corin
Paperback
Edge Intelligence for Aerial Robots: Run small models near the aircraft while respecting power and latency limits
Halstead, Corin
Paperback
Home IoT Projects: Connect sensors, messages, dashboards, and alerts in practical weekend builds
Halstead, Corin
Paperback
Robot Middleware for Drones: Connect sensors, maps, control logic, and flight software in a clear architecture
Halstead, Corin
Paperback
Modern Aerial Robotics: Build a safer drone software stack from simulation to field testing
Halstead, Corin
Paperback
Drone Navigation Systems: Plan movement, manage coordinates, and reduce errors in autonomous flight
Halstead, Corin
Paperback
AWS for the Absolute Beginner: Your Step-by-Step, Jargon-Free Introduction to the Amazon Web Services Ecosystem
Halstead, Corin
Paperback
Lua Game Project Lab for Beginners: Build Small Games, Mods, and Interactive Systems
Halstead, Corin
Paperback
Rust Programming Blueprint for Beginners: Learn Memory-Safe Coding Through Practical Projects
Halstead, Corin
Paperback
Lua Scripting Blueprint for Beginners: Learn Variables, Functions, Tables, and Game Logic
Halstead, Corin
Paperback
Visual Automation Workflows: Connect Home Devices, Messages, Sensors, and Alerts with Simple Drag-and-Build Logic
Halstead, Corin
Paperback
Incident Response Strategies: A Guide to Managing and Recovering from Security Breaches
Halstead, Corin
Paperback
Aerial Photography Basics: Composition and Settings for Capturing Stunning Drone Images
Halstead, Corin
Paperback
JavaScript from Scratch: The Ultimate Beginner's Guide to Learning Modern JavaScript (ES6+) Through Real-World Projects
Halstead, Corin
Paperback
The Hacker's Mindset: An Introduction to Ethical Hacking and Penetration Testing for Absolute Beginners
Halstead, Corin
Paperback
The Azure Advantage: A Practical Guide for Developers to Build, Deploy, and Scale Applications on Microsoft Azure
Halstead, Corin
Paperback
Beyond JavaScript: A JavaScript Developer's Fast-Track Guide to Adopting and Mastering TypeScript
Halstead, Corin
Paperback
The Type-Safe Code: Mastering TypeScript from the Ground Up, from Basic Types to Advanced Generics and Decorators
Halstead, Corin
Paperback
TypeScript and React: A Project-Based Guide to Building Robust, Scalable, and Maintainable Front-End Applications
Halstead, Corin
Paperback
Smart Home Interoperability: Make Mixed Devices Work Together with Clearer Planning, Naming, and Automation Rules
Halstead, Corin
Paperback
Autonomous Drone Projects: Build complete workflows for sensing, planning, testing, and controlled flight
Halstead, Corin
Paperback
Companion Computer Drones: Add onboard logic, telemetry checks, and mission apps without losing safety control
Halstead, Corin
Paperback
Machine Learning for Beginners: A Practical Introduction to Training Predictive Models
Halstead, Corin
Paperback
Applied Machine Learning: Solving Real-World Problems with Regression and Classification
Halstead, Corin
Paperback
Diagnosing Production Software: Use Logs, Metrics, Traces, and Runtime Context to Find Failures Faster
Halstead, Corin
Paperback
Controlled Feature Rollouts: Use Feature Flags, Staged Releases, Guardrails, and Fast Rollbacks to Ship Changes Safely
Halstead, Corin
Paperback
Computer Networking for Developers: Understand TCP/IP, DNS, HTTP, Routing, and Real World Network Failures
Halstead, Corin
Paperback
Computer Memory, Caches, and Performance: Understand How Data Movement Shapes Program Speed
Halstead, Corin
Paperback
Computer Architecture Through Code: Understand CPUs, Caches, Memory, Instructions, and Performance
Halstead, Corin
Paperback
Algorithmic Problem-Solving Patterns: Choose Data Structures, Analyze Complexity, and Build Efficient Solutions
Halstead, Corin
Paperback
Defold Game Development Operations Manual for Beginners: Build Lightweight 2D Games with Lua
Halstead, Corin
Paperback
Causal Inference for Data Analysts: Move Beyond Correlation and Measure What Actually Drives Outcomes
Halstead, Corin
Paperback
Also in
Probability & Statistics
R for Data Science: Import, Tidy, Transform, Visualize, and Model Data
Grolemund, Garrett
Wickham, Hadley
Paperback
AP Statistics Premium, 2027: 9 Practice Tests + Comprehensive Review + Online Practice
Sternstein, Martin, PH. D.
Paperback
R for Data Science: Import, Tidy, Transform, Visualize, and Model Data
Cetinkaya-Rundel, Mine
Wickham, Hadley
Grolemund, Garrett
Paperback
Graph Paper Composition Notebook: Quad Ruled 5x5, Grid Paper for Students in Math and Science
Wizo, Math
Paperback
Stat 208 Statistical Thinking: A Book for Stat 208 at Virginia Commonwealth University
Durfee, Becky
Street IV, W. Scott
Paperback
Forecasting: Principles and Practice, the Pythonic Way
Athanasopoulos, George
Hyndman, Rob J.
Paperback
The Art of Uncertainty: How to Navigate Chance, Ignorance, Risk and Luck
Spiegelhalter, David
Paperback
Mastering Claude AI: Practical Journey from First Prompts to Pro with Claude AI
Dickey, Ryan
Paperback
Introduction to Statistics: An Intuitive Guide for Analyzing Data and Unlocking Discoveries
Frost, Jim
Paperback
An Introduction to Statistical Learning: With Applications in R
Witten, Daniela
Hastie, Trevor
James, Gareth
Paperback
Ai-Assisted Statistics for Data Scientists: 50+ Essential Concepts Using R and Python
Bruce, Andrew
Gedeck, Peter
Bruce, Peter
Paperback
How the World Really Works: The Science Behind How We Got Here and Where We're Going
Smil, Vaclav
Paperback
The Art of Uncertainty: How to Navigate Chance, Ignorance, Risk and Luck
Spiegelhalter, David
Hardcover
Scale: The Universal Laws of Life, Growth, and Death in Organisms, Cities, and Companies
West, Geoffrey
Paperback
The Random Universe: How Models and Probability Help Us Make Sense of the Cosmos
Jaffe, Andrew H.
Hardcover
Schaum's Outline of Probability and Statistics, 4th Edition: 897 Solved Problems + 20 Videos
Spiegel, Murray R.
Schiller, John J.
Srinivasan, R. Alu
Paperback
Advanced Statistics in Research: Reading, Understanding, and Writing Up Data Analysis Results
Hatcher, Larry
Paperback
The Only Math Book You'll Ever Need, Revised Edition: Hundreds of Easy Solutions and Shortcuts for Mastering Everyday Numbers
Heller, Barbara R.
Kogelman, Stanley
Paperback
Mathematical Foundations of Quantum Computing: A Scaffolding Approach
Cheng, Ran
Lee, Peter
Yu, James
Hardcover
Schaum's Outline of Probability, Random Variables, and Random Processes, Fourth Edition
Hsu, Hwei P.
Paperback
Mathletics: How Gamblers, Managers, and Fans Use Mathematics in Sports, Second Edition
Winston, Wayne L.
Nestler, Scott
Pelechrinis, Konstantinos
Paperback
SQL Server 2025 Unveiled: The Ai-Ready Enterprise Database with Microsoft Fabric Integration
Ward, Bob
Paperback
Building Large Language Models from Scratch: Design, Train, and Deploy Llms with Pytorch
Grigorov, Dilyan
Paperback
How the World Really Works: The Science Behind How We Got Here and Where We're Going
Smil, Vaclav
Hardcover
Regression Modeling Strategies: With Applications to Linear Models, Logistic and Ordinal Regression, and Survival Analysis
Harrell Jr, Frank E.
Paperback
Damned Lies and Statistics: Untangling Numbers from the Media, Politicians, and Activists
Best, Joel
Hardcover
Challenging Mathematical Problems with Elementary Solutions, Vol. I: Volume 1
Yaglom, I. M.
Yaglom, A. M.
Paperback
Databricks Data Intelligence Platform: Unlocking the Genai Revolution
Gupta, Nikhil
Yip, Jason
Paperback
Quantum Computing and Quantum Machine Learning for Engineers and Developers
Rosas-Bustos, Jose
Fraser, Roydon Andrew
Van Griensven Thé, Jesse
Hardcover
The End of Average: Unlocking Our Potential by Embracing What Makes Us Different
Rose, Todd
Paperback
The Master Algorithm: How the Quest for the Ultimate Learning Machine Will Remake Our World
Domingos, Pedro
Paperback
Probably Overthinking It: How to Use Data to Answer Questions, Avoid Statistical Traps, and Make Better Decisions
Downey, Allen B.
Paperback
Making ChatGPT Work for You: Getting the Most Out of Generative AI as a Non-Techie
Evelyn, Lydia
Paperback
Large Language Models: A Deep Dive: Bridging Theory and Practice
Somers, Garrett
Kamath, Uday
Keenan, Kevin
Hardcover
Probably the Best Book on Statistics Ever Written: How to Beat the Odds and Make Better Decisions
Shapira, Haim
Hardcover
Scorecasting: The Hidden Influences Behind How Sports Are Played and Games Are Won
Moskowitz, Tobias
Wertheim, L. Jon
Paperback
Guide to Methods for Students of Political Science: Property, Proof, and Dispute in Catalonia Around the Year 1000
Van Evera, Stephen
Paperback
Learning Statistics with Jamovi: A Tutorial for Beginners in Statistical Analysis
Foxcroft, David
Navarro, Danielle
Paperback
Time Series Analysis and Its Applications: With R Examples
Stoffer, David S.
Shumway, Robert H.
Hardcover
Probably Overthinking It: How to Use Data to Answer Questions, Avoid Statistical Traps, and Make Better Decisions
Downey, Allen B.
Hardcover
AI for Robotics: Toward Embodied and General Intelligence in the Physical World
Imran, Alishba
Gopalakrishnan, Keerthana
Paperback
Machine Learning Q and AI: 30 Essential Questions and Answers on Machine Learning and AI
Raschka, Sebastian
Paperback
An Introduction to Statistical Learning: With Applications in Python
Witten, Daniela
Hastie, Trevor
James, Gareth
Hardcover
A Modern Introduction to Probability and Statistics: Understanding Why and How
Lopuhaä, H. P.
Dekking, F. M.
Kraaikamp, C.
Hardcover
System Design with AI Interview Guide: Designing Scalable, Agentic, and Defensible Systems
Bhardwaj, Rohit
Paperback
Observability for Large Language Models: Site Reliability and Chaos Engineering for AI at Scale
Sharma, Ankush
Paperback
The Theory That Would Not Die: How Bayes' Rule Cracked the Enigma Code, Hunted Down Russian Submarines, and Emerged Triumphant from Two Centuries of C
McGrayne, Sharon Bertsch
Paperback
High-Dimensional Probability: An Introduction with Applications in Data Science
Vershynin, Roman
Hardcover
Challenging Mathematical Problems with Elementary Solutions, Vol. II: Volume 2
Yaglom, I. M.
Yaglom, A. M.
Paperback
Modern Time Series Forecasting with Python - Second Edition: Industry-ready machine learning and deep learning time series analysis with PyTorch and p
Tackes, Jeffrey
Joseph, Manu
Paperback
The Elements of Statistical Learning: Data Mining, Inference, and Prediction, Second Edition
Tibshirani, Robert
Friedman, Jerome
Hastie, Trevor
Hardcover
Mathematical Foundations of Quantum Computing: A Scaffolding Approach
Lee, Peter
Yu, James
Cheng, Ran
Paperback
Prompt Engineering for Everyone: A Self-Taught, Human-Centered Approach to AI Programming
Tavakoli, Hamid
Paperback
The Design Inference: Eliminating Chance through Small Probabilities
Dembski, William A.
Ewert, Winston
Hardcover
The ESRI Guide to GIS Analysis, Volume 2: Spatial Measurements and Statistics
Mitchell, Andy
Griffin, Lauren Scott
Paperback
Introduction to Statistics: An Intuitive Guide for Analyzing Data and Unlocking Discoveries
Frost, Jim
Hardcover
Building Generative AI Agents: Using Langgraph, Autogen, and Crewai
Deshmukh, Gaurav
Taulli, Tom
Paperback
R in Action, Third Edition: Data Analysis and Graphics with R and Tidyverse
Kabacoff, Robert I.
Paperback
Large Language Model Recipes: A Hands-On Guide to Fine-Tuning, Optimization, Deployment, and Real-World Applications
Bolla, Bharath Kumar
Subbaiah, Kalpa
Kaata, Sashi Kiran
Paperback
Telling Time Workbook: Practice Reading and Draw the Hand on the Clocks One Hour Half Hour 15 5 1 Minutes
Winny, Renzo
Paperback
ACCUPLACER Math Prep: ACCUPLACER Math Test Study Guide with Two Practice Tests [Includes Detailed Answer Explanations]
Tpb Publishing
Paperback
Unlocking Dbt: Design and Deploy Transformations in Your Cloud Data Warehouse
Dorsey, Dustin
Cyr, Cameron
Paperback
Deep Learning in Computational Mechanics: An Introductory Course
Weeger, Oliver
Jokeit, Moritz
Herrmann, Leon
Hardcover
From Data to Dollars: Getting Started with Data Analytics and AI in Startups
Sidoruk, Piotr
Paperback
Databricks Data Intelligence Platform: Powering the Agentic Era
Gupta, Nikhil
Wojtyczka, Marcin
Yip, Jason
Paperback
The Practical Guide to Large Language Models: Hands-On AI Applications with Hugging Face Transformers
Gridin, Ivan
Paperback
Bayesian Analysis with Python - Third Edition: A practical guide to probabilistic modeling
Martin, Osvaldo
Paperback
Large Language Models: From Theory to Production
Vanegas, Esteban
Noguer I. Alonso, Miquel
Pacheco Aznar, David
Paperback
Guesstimation: Solving the World's Problems on the Back of a Cocktail Napkin
Adam, John a.
Weinstein, Lawrence
Paperback
Practical Generative Ai: From Concept to Deployment: Building and Deploying Ethical AI-Powered Solutions
McKeone, James
Singh, Pramod
Paperback
An Introduction to Statistical Learning: With Applications in Python
Witten, Daniela
Hastie, Trevor
James, Gareth
Paperback