flowchart TD
Q["Marketing question"] --> D{"What is the<br/>object of interest?"}
D -->|"Accurate Y at new inputs"| P["PREDICTION<br/>(black-box f-hat)"]
D -->|"A functional of f:<br/>effect, elasticity"| I["INFERENCE<br/>(structure of f)"]
P --> PV["Validate by:<br/>out-of-sample loss,<br/>cross-validation"]
I --> IV["Validate by:<br/>identification,<br/>standard errors, design"]
PV --> PU["Use: scoring, ranking,<br/>forecasting, matching"]
IV --> IU["Use: pricing, budget<br/>allocation, policy"]
PU -.->|"Do NOT read feature<br/>importance as causal"| IU
67 Artificial Intelligence and Machine Learning in Marketing
Machine learning is the study of algorithms that improve their performance on a task as they are exposed to more data, rather than by being explicitly programmed with the rules of that task. Artificial intelligence (AI) is the broader project of building systems that perform tasks we associate with human cognition; contemporary AI in marketing is, almost entirely, applied machine learning at scale. The distinction matters less than what the two share: a willingness to let flexible, high-capacity function approximators discover structure in data that a human analyst could neither specify in advance nor write down as equations. That willingness is also the source of every pitfall in this chapter.
Marketing is an unusually hospitable host for these methods, for three reasons. First, the field is awash in the kind of data—clickstreams, transactions, images, reviews, search queries—on which modern learning algorithms thrive (Wedel and Kannan 2016; Martin, Borah, and Palmatier 2017). Second, many marketing tasks are natively predictive: which customer will churn, which creative will earn the click, which product to surface next. Prediction is exactly what supervised learning does well. Third, the economic stakes of small improvements are large, because marketing decisions are made billions of times a day across recommendation, bidding, and targeting systems, so a one-percent lift compounds into real money (Varian 2016; Gao, Wang, and Yu 2024).
This chapter has two jobs. The first is to give a working, formal command of the methods—supervised and unsupervised learning, recommender systems, deep learning, and large language models—at a level that lets the reader state each method’s estimator, its assumptions, and the conditions under which it fails. The second is to install a discipline that the hype around AI actively erodes: the distinction between prediction and inference, and with it a sober catalogue of the ways machine learning quietly breaks in deployment—leakage, distribution drift, and unfairness. A model that predicts well in a notebook and harms the business in production is the modal failure, not the exception, and most of those failures are conceptual rather than computational. By the end, the reader should be able to map a marketing problem onto the right learning paradigm, build and validate a model that does not lie to them, and recognize when a predictive tool is being asked, illegitimately, to answer a causal question.
We assume familiarity with the regression and choice-modeling machinery developed earlier in the book (Chapter 37 for measurement; the causal-inference and marketing-mix chapters for identification), and we connect to them rather than re-deriving them.
67.1 Prediction Versus Inference: The Organizing Distinction
The single most consequential idea in this chapter is also the one most often skipped. Consider a generic supervised relationship between an outcome \(Y\) and features \(\mathbf{X}\), \[ Y = f(\mathbf{X}) + \varepsilon, \qquad \mathbb{E}[\varepsilon \mid \mathbf{X}] = 0, \tag{67.1}\] with \(\hat f\) an estimate learned from data. There are two fundamentally different things one can want from \(\hat f\), and conflating them is the root of most misuse of machine learning in marketing.
In a prediction problem the object of interest is \(\hat f(\mathbf{x})\) itself: we want accurate values of \(Y\) at new inputs and treat \(\hat f\) as a black box. In an inference problem the object of interest is some functional of \(f\)—a coefficient, an elasticity, a treatment effect—and \(\hat f\) is a means to learn how \(\mathbf{X}\) relates to \(Y\), not merely to forecast \(Y\).
The two goals reward different choices. Prediction tolerates—indeed often prefers—biased, uninterpretable, highly flexible estimators, because the only scorecard is out-of-sample loss, \(\mathbb{E}[L(Y, \hat f(\mathbf{X}))]\) on data the model has never seen. The bias–variance trade-off is the governing law: expected squared prediction error decomposes as \[ \mathbb{E}\!\left[(Y - \hat f(\mathbf{x}_0))^2\right] = \underbrace{\big(\mathbb{E}[\hat f(\mathbf{x}_0)] - f(\mathbf{x}_0)\big)^2}_{\text{bias}^2} + \underbrace{\operatorname{Var}\!\big(\hat f(\mathbf{x}_0)\big)}_{\text{variance}} + \underbrace{\sigma^2_\varepsilon}_{\text{irreducible}}, \tag{67.2}\] and a method that accepts some bias to cut variance can win. Inference, by contrast, demands an estimator whose sampling distribution we understand—unbiasedness or a known bias, valid standard errors, an identification argument linking the estimand to features of the data-generating process. A random forest can have lower test error than a linear regression while being useless for the question “what is the effect of a $1 price cut,” because flexible learners trade interpretable, consistent parameters for predictive accuracy (Varian 2016).
The practical hazard is that a predictive model’s coefficients—or its feature-importance scores—look like answers to inference questions and are routinely read as such. They are not. A churn model may load heavily on “number of support tickets,” but acting on that association by suppressing support tickets would be disastrous; the feature predicts churn because both are caused by underlying dissatisfaction. Predictive importance is not causal importance, and no amount of test-set accuracy converts one into the other. When the marketing question is “what will happen,” supervised learning is the right tool; when it is “what should we do,” the model must be embedded in a design that identifies a causal effect—a randomized experiment, an instrument, or one of the causal-machine-learning estimators we reach at the end of the chapter. Figure 67.1 fixes the fork in the road.
67.2 Supervised Learning
In supervised learning the training data are labeled pairs \(\{(\mathbf{x}_i, y_i)\}_{i=1}^n\), and the goal is to learn a mapping \(\hat f : \mathcal{X} \to \mathcal{Y}\) that generalizes to unlabeled inputs. When \(Y\) is continuous the task is regression; when \(Y\) is categorical it is classification. Almost every workhorse marketing model—propensity to buy, churn, response, lifetime-value, lead scoring, ad click-through—is a supervised classifier or regressor.
67.2.1 The learning problem and regularization
Formally, learning chooses \(\hat f\) to minimize regularized empirical risk, \[ \hat f = \arg\min_{f \in \mathcal{F}} \; \frac{1}{n}\sum_{i=1}^{n} L\big(y_i, f(\mathbf{x}_i)\big) + \lambda\, \Omega(f), \tag{67.3}\] where \(L\) is a loss function (squared error for regression, log-loss/cross-entropy for classification), \(\mathcal{F}\) is the hypothesis class (linear functions, trees, neural networks), \(\Omega(f)\) is a complexity penalty, and \(\lambda \ge 0\) tunes the trade-off. Minimizing training loss alone (\(\lambda = 0\), \(\mathcal{F}\) rich) yields overfitting: \(\hat f\) memorizes noise and generalizes poorly, the high-variance failure in Equation 67.2. The penalty \(\Omega\) buys generalization by shrinking the effective complexity of \(\hat f\). For linear models the two canonical choices are the \(\ell_2\) (ridge) penalty \(\Omega(\boldsymbol\beta)=\|\boldsymbol\beta\|_2^2\), which shrinks coefficients smoothly, and the \(\ell_1\) (lasso) penalty \(\Omega(\boldsymbol\beta)=\|\boldsymbol\beta\|_1\), which sets some coefficients exactly to zero and thereby performs variable selection—valuable when the feature space is wide, as it almost always is with behavioral data (Varian 2016).
The assumptions behind Equation 67.3 are easy to state and easy to violate. Empirical risk minimization is consistent for the risk-minimizing \(f\) only if the training data are drawn from the same distribution as the deployment data (the identically-distributed assumption) and if observations are exchangeable in the way the validation scheme assumes (commonly independence). Both assumptions fail routinely in marketing: deployment data drift away from training data over time (Section 67.10.2), and observations are correlated within customers, sessions, and time, which—if ignored—makes naïve cross-validation report accuracy the model will never achieve in production.
67.2.2 Tree ensembles: the marketing workhorse
For tabular marketing data—mixed numeric and categorical features, nonlinearities, interactions, missingness—ensembles of decision trees are, empirically, the default high-performer. A single regression tree partitions \(\mathcal{X}\) into rectangular regions \(R_1,\dots,R_M\) and predicts the within-region mean, \(\hat f(\mathbf{x})=\sum_m c_m \mathbf{1}\{\mathbf{x}\in R_m\}\); it is interpretable but high-variance. Two ensemble strategies tame the variance. Random forests average many trees grown on bootstrap samples with randomly restricted split candidates, reducing variance through decorrelation. Gradient-boosted trees instead fit trees sequentially, each new tree \(h_t\) targeting the gradient of the loss left by the running ensemble, \[ \hat f_t(\mathbf{x}) = \hat f_{t-1}(\mathbf{x}) + \nu\, h_t(\mathbf{x}), \qquad h_t \approx -\,\frac{\partial L}{\partial \hat f_{t-1}}, \tag{67.4}\] with learning rate \(\nu \in (0,1]\). Boosting reduces bias and variance jointly and typically tops leaderboards on tabular data, at the cost of more careful tuning to avoid overfitting (the number of trees becomes a regularization parameter set by validation). The worked example below builds a churn classifier and—critically—shows how to validate it honestly. The two workhorses of that toolkit are Breiman (2001), which averages decorrelated trees to cut variance, and Friedman (2001), which fits them sequentially to the residual and so cuts bias instead.
Code
set.seed(48)
# --- Simulate a customer-churn dataset with a known structure ----------------
n <- 4000
tenure <- rpois(n, lambda = 18) # months as customer
monthly_spend <- round(rgamma(n, shape = 2, scale = 25), 2) # $ per month
support_calls <- rpois(n, lambda = 1 + 0.05 * (40 - pmin(tenure, 40)))
discount_user <- rbinom(n, 1, 0.35)
# True churn propensity: short tenure, low spend, many support calls raise risk.
lin <- -1.0 - 0.06 * tenure - 0.015 * monthly_spend +
0.45 * support_calls + 0.30 * discount_user
prob_churn <- plogis(lin)
churn <- rbinom(n, 1, prob_churn)
dat <- data.frame(churn = factor(churn, labels = c("stay", "leave")),
tenure, monthly_spend, support_calls,
discount_user = factor(discount_user))
# --- Honest train/test split -------------------------------------------------
idx <- sample(seq_len(n), size = floor(0.7 * n))
train <- dat[idx, ]
test <- dat[-idx, ]
# --- Gradient-boosted trees (gbm); fall back to logistic if gbm absent -------
has_gbm <- requireNamespace("gbm", quietly = TRUE)
if (has_gbm) {
fit <- gbm::gbm(I(as.integer(churn) - 1) ~ tenure + monthly_spend +
support_calls + discount_user,
data = train, distribution = "bernoulli",
n.trees = 600, interaction.depth = 3,
shrinkage = 0.03, bag.fraction = 0.7, verbose = FALSE)
best <- gbm::gbm.perf(fit, plot.it = FALSE, method = "OOB")
p_hat <- gbm::predict.gbm(fit, test, n.trees = best, type = "response")
} else {
fit <- glm(churn ~ tenure + monthly_spend + support_calls + discount_user,
data = train, family = binomial())
p_hat <- predict(fit, test, type = "response")
}
# --- Out-of-sample evaluation: AUC and a calibration check -------------------
auc <- function(score, label) { # rank-based AUC, no extra packages
pos <- score[label == "leave"]; neg <- score[label == "stay"]
mean(outer(pos, neg, ">") + 0.5 * outer(pos, neg, "=="))
}
cat("Test AUC:", round(auc(p_hat, test$churn), 3), "\n")
#> Test AUC: 0.707
# Calibration: do predicted probabilities match realized churn rates?
bins <- cut(p_hat, breaks = quantile(p_hat, 0:5/5), include.lowest = TRUE)
calib <- aggregate(as.integer(test$churn) - 1 ~ bins, FUN = mean)
calib$predicted <- tapply(p_hat, bins, mean)
names(calib) <- c("bin", "observed_churn", "mean_predicted")
calib
#> bin observed_churn mean_predicted
#> 1 [0.0764,0.109] 0.07083333 0.09603519
#> 2 (0.109,0.131] 0.13333333 0.11983961
#> 3 (0.131,0.163] 0.12033195 0.14376007
#> 4 (0.163,0.237] 0.16317992 0.19626848
#> 5 (0.237,0.67] 0.42083333 0.35999483The example reports two diagnostics, not one. Discrimination (AUC) measures whether the model ranks churners above non-churners; calibration measures whether a predicted 30% churn probability corresponds to a 30% realized rate. A model can discriminate well yet be badly calibrated, and marketing decisions that multiply predicted probabilities by margins—expected-value targeting—need calibration, not just ranking (Neumann, Tucker, and Whitfield 2019). Reporting only AUC is a common and costly omission.
67.2.3 Classification thresholds and the cost of errors
A classifier outputs a score \(\hat p(\mathbf{x}) = \widehat{\Pr}(Y=1\mid\mathbf{x})\); turning it into an action requires a threshold \(\tau\) such that we treat customers with \(\hat p > \tau\). The optimal threshold is not \(0.5\)—it depends on the asymmetric costs of false positives and false negatives. If contacting a non-churner costs \(c_{\text{FP}}\) and failing to retain a churner costs \(c_{\text{FN}}\), the expected-cost-minimizing rule acts when \(\hat p / (1-\hat p) > c_{\text{FP}} / c_{\text{FN}}\). This is the point at which the predictive model meets the decision problem, and it is where the marketing economics re-enter: the ROC and precision–recall curves exist precisely because the right operating point is a business choice, not a statistical default.
67.3 Unsupervised Learning
In unsupervised learning the data are unlabeled, \(\{\mathbf{x}_i\}_{i=1}^n\), and the goal is to discover latent structure—groups, dimensions, topics—without a target variable to supervise the search. The two dominant marketing uses are segmentation (clustering customers or products) and dimension reduction (compressing high-dimensional behavior into interpretable factors). Because there is no label, there is no test-set accuracy to adjudicate “correctness”; validation is intrinsically harder and more judgmental, which is both the method’s flexibility and its danger.
67.3.1 Clustering and the segmentation problem
The canonical objective is \(k\)-means, which partitions observations into \(K\) clusters to minimize within-cluster squared distance, \[ \min_{\{S_k\}}\; \sum_{k=1}^{K}\sum_{\mathbf{x}_i \in S_k} \big\|\mathbf{x}_i - \boldsymbol\mu_k\big\|_2^2, \qquad \boldsymbol\mu_k = \frac{1}{|S_k|}\sum_{\mathbf{x}_i \in S_k}\mathbf{x}_i. \tag{67.5}\] This connects directly to the a priori versus post hoc segmentation distinction of Section 34.4: post hoc segmentation is precisely the application of a clustering algorithm to behavioral or attitudinal data to discover segments rather than impose them. The estimator (Lloyd’s algorithm) alternates assigning points to the nearest centroid and recomputing centroids; it converges to a local optimum, so results depend on initialization and on the (analyst-chosen) number of clusters \(K\). Three assumptions break identification of a “true” segmentation, and all three are routinely violated in practice: \(k\)-means presumes roughly spherical, equal-variance clusters (Euclidean distance encodes this); it is not scale-invariant, so features must be standardized or the largest-variance feature dominates; and \(K\) is not learned but assumed. Model-based clustering via finite mixtures replaces the hard geometry with a probabilistic generative model and lets information criteria choose \(K\), at the cost of distributional assumptions. The deeper caution is that a clustering algorithm always returns clusters, whether or not the population is actually clustered; the burden is on the analyst to show the segments are stable, managerially distinguishable, and reproducible out of sample, not merely that the algorithm ran.
Code
set.seed(48)
# Three latent customer segments differing in recency, frequency, monetary value
make_seg <- function(n, r, f, m) data.frame(
recency = pmax(1, round(rnorm(n, r, 8))),
frequency = pmax(1, round(rnorm(n, f, 3))),
monetary = pmax(5, round(rnorm(n, m, 40)))
)
rfm <- rbind(make_seg(300, 10, 14, 220), # champions
make_seg(300, 45, 4, 60), # at-risk
make_seg(300, 25, 8, 130)) # mainstream
# Standardize before clustering: k-means is NOT scale-invariant
rfm_z <- scale(rfm)
km <- kmeans(rfm_z, centers = 3, nstart = 25)
# Profile the recovered segments on the original (interpretable) scale
prof <- aggregate(rfm, by = list(segment = km$cluster), FUN = function(x) round(mean(x), 1))
prof$size <- as.integer(table(km$cluster))
prof
#> segment recency frequency monetary size
#> 1 1 44.6 4.3 59.0 300
#> 2 2 24.8 8.0 132.6 295
#> 3 3 11.0 14.2 218.6 30567.3.2 Dimension reduction
When behavior is high-dimensional—thousands of SKUs, pages, or features—dimension reduction finds a low-dimensional representation that preserves the information that matters. Principal component analysis (PCA) projects \(\mathbf{X}\) onto the orthogonal directions of maximal variance, the leading eigenvectors of the covariance matrix; the first few components often capture interpretable axes of behavior (e.g., overall intensity, then category mix). Non-negative matrix factorization and, for text, topic models such as latent Dirichlet allocation generalize the idea to parts-based and probabilistic decompositions (Tirunillai and Tellis 2014; Büschken and Allenby 2016). The methods double as a feature-engineering step for supervised models and as a listening tool: factorizing the term–document matrix of online reviews recovers the latent dimensions of quality consumers actually discuss, a structure managers cannot specify in advance (Tirunillai and Tellis 2014; Netzer et al. 2008).
67.4 Recommender Systems
Recommender systems are the most economically consequential deployment of machine learning in marketing: they choose which of millions of items to surface to each user, and on platforms from retail to streaming they drive a large share of demand. Formally, a recommender estimates a utility or preference score \(\hat r_{ui}\) for each user \(u\) and item \(i\), then ranks items by that score. The data are a sparse user–item matrix \(\mathbf{R}\), mostly missing, with observed entries being ratings, clicks, or purchases.
Two paradigms, with a hybrid, organize the field. Content-based filtering recommends items similar to those a user has liked, using item features; it handles new items but cannot discover tastes outside a user’s history. Collaborative filtering ignores item content and exploits the wisdom of the crowd—users who agreed in the past will agree in the future—and is the more powerful approach when interaction data are dense. The dominant collaborative formulation is matrix factorization, which embeds users and items in a shared latent space of dimension \(K\) and models preference as an inner product, \[ \hat r_{ui} = \mathbf{p}_u^{\top}\mathbf{q}_i + b_u + b_i + \mu, \qquad \min_{\mathbf{P},\mathbf{Q},\mathbf{b}} \sum_{(u,i)\in\mathcal{K}} \big(r_{ui} - \hat r_{ui}\big)^2 + \lambda\big(\|\mathbf{p}_u\|^2 + \|\mathbf{q}_i\|^2 + b_u^2 + b_i^2\big), \tag{67.6}\] where \(\mathbf{p}_u, \mathbf{q}_i \in \mathbb{R}^K\) are the learned user and item factors, \(b_u, b_i, \mu\) are bias terms, and the sum runs only over observed entries \(\mathcal{K}\). The \(K\) latent dimensions are discovered, not specified, and often correspond to interpretable axes of taste. The regularizer is essential because \(\mathbf{R}\) is extremely sparse. The standard remedy is to factor the sparse matrix into low-rank user and item embeddings, the approach that won the Netflix Prize and remains the baseline against which newer recommenders are measured (Koren, Bell, and Volinsky 2009).
Three structural problems define the research frontier and the deployment risk. The cold-start problem—no data for new users or items—forces a fallback to content features or popularity until interaction data accrue. Feedback loops are subtler and more dangerous: a recommender trained on logged interactions learns from data its own past recommendations generated, so popular items get recommended, become more popular, and crowd out the long tail, narrowing exposure in ways that can entrench rather than reveal preferences (Zheng et al. 2023). And recommendations that maximize predicted clicks need not maximize incremental value: an item the user would have bought anyway earns the recommender credit it did not create, a confound between prediction and causal lift that only experimentation resolves. The example below factorizes a small implicit-feedback matrix.
Code
set.seed(48)
# --- Simulate implicit feedback from latent tastes (rank-2 truth) ------------
n_users <- 200; n_items <- 60; K_true <- 2
P_true <- matrix(rnorm(n_users * K_true), n_users)
Q_true <- matrix(rnorm(n_items * K_true), n_items)
logits <- P_true %*% t(Q_true)
R <- matrix(rbinom(n_users * n_items, 1, plogis(logits)), n_users) # 1 = engaged
# Hide 15% of entries as a test set (missing-at-random for illustration)
mask <- matrix(runif(length(R)) > 0.15, n_users) # TRUE = observed in training
Rtr <- R; Rtr[!mask] <- NA
# --- Matrix factorization by regularized alternating least squares -----------
K <- 2; lambda <- 0.1; iters <- 30
P <- matrix(rnorm(n_users * K, sd = 0.1), n_users)
Q <- matrix(rnorm(n_items * K, sd = 0.1), n_items)
solve_factor <- function(fixed, target_row, obs) { # ridge solve per row
F <- fixed[obs, , drop = FALSE]; y <- target_row[obs]
solve(t(F) %*% F + lambda * diag(ncol(fixed)), t(F) %*% y)
}
for (it in seq_len(iters)) {
for (u in seq_len(n_users)) { o <- which(!is.na(Rtr[u, ])); if (length(o)) P[u, ] <- solve_factor(Q, Rtr[u, ], o) }
for (i in seq_len(n_items)) { o <- which(!is.na(Rtr[, i])); if (length(o)) Q[i, ] <- solve_factor(P, Rtr[, i], o) }
}
R_hat <- P %*% t(Q)
# Evaluate ranking quality on held-out entries via AUC
test_idx <- which(!mask)
auc_rank <- {
s <- R_hat[test_idx]; y <- R[test_idx]
pos <- s[y == 1]; neg <- s[y == 0]
mean(outer(pos, neg, ">") + 0.5 * outer(pos, neg, "=="))
}
cat("Held-out ranking AUC:", round(auc_rank, 3), "\n")
#> Held-out ranking AUC: 0.6567.4.1 Bundle Recommendation at Scale
Matrix factorization scores single items. A distinct and commercially large problem is recommending bundles—sets of products to buy together—which turns on a relationship factorization ignores: whether two products are complements (a phone and a case, bought together) or substitutes (two phones, bought instead of each other). Raw co-occurrence cannot tell them apart, because substitutes co-occur too—shoppers view and compare them—so a naive “frequently seen together” recommender fills bundles with substitutes, which no one buys as a pair.
Kumar, Eckles, and Aral (2026) solve this at scale with dense product embeddings. Products are embedded from behavioral co-occurrence—purchases and clickstream consideration sets—so that products used in similar ways land near one another, exactly as words do in a language model. The geometry then supplies the two signals a bundle needs: substitutability is proximity in the consideration-set space (interchangeable products are considered together), while complementarity is co-purchase association net of substitutability (products bought together that are not merely alternatives). Bundling a focal product with its complements while screening out its substitutes, and then using a field experiment to learn the recommendation policy offline—mapping bundle features to purchase likelihood and optimizing over them (Chapter 59)—their policy generalizes across the retailer’s assortment and raises bundle purchases by an estimated 35% over the incumbent.
The replication below runs the whole pipeline in miniature. It simulates purchase baskets (complements) and consideration sets (substitutes), learns product embeddings by the standard PPMI-plus-SVD route—the closed-form cousin of word2vec—separates complementarity from substitutability, and shows a bundle policy built on that separation beating a naive co-occurrence recommender; it then tunes the substitute penalty by offline policy evaluation, the paper’s optimization step in miniature.
Code
set.seed(48)
# --- Catalog with latent complement/substitute structure ---------------------
n_prod <- 120; n_cat <- 12
cat_id <- rep(seq_len(n_cat), length.out = n_prod) # each product's category
role <- matrix(rnorm(n_prod * 3), n_prod, 3) # latent "goes-with" role
# --- Baskets: complements co-occur (cross-category), with some substitute
# contamination (people sometimes buy similar items too) -------------------
complement_of <- function(i) {
cross <- which(cat_id != cat_id[i])
aff <- as.numeric(role[cross, ] %*% role[i, ])
cross[sample.int(length(cross), 1, prob = {p <- exp(2 * (aff - max(aff))); p / sum(p)})]
}
substitute_of <- function(i) {
same <- setdiff(which(cat_id == cat_id[i]), i)
if (!length(same)) i else same[sample.int(length(same), 1)]
}
make_basket <- function() {
items <- i <- sample.int(n_prod, 1)
for (r in seq_len(rbinom(1, 3, 0.6))) items <- c(items, complement_of(i))
if (runif(1) < 0.25) items <- c(items, substitute_of(i)) # contamination
unique(items)
}
make_view <- function() { # consideration set: substitutes
pool <- which(cat_id == sample.int(n_cat, 1))
if (length(pool) < 2) pool else sample(pool, min(length(pool), 2 + rbinom(1, 3, 0.5)))
}
baskets <- replicate(6000, make_basket(), simplify = FALSE)
views <- replicate(6000, make_view(), simplify = FALSE)
# --- Co-occurrence -> PPMI -> dense embeddings (SVD, the word2vec cousin) -----
co_matrix <- function(sets, n) {
M <- matrix(0, n, n)
for (s in sets) if (length(s) > 1) M[s, s] <- M[s, s] + 1
diag(M) <- 0; M
}
ppmi <- function(M) {
tot <- sum(M); if (tot == 0) return(M)
P <- log((M / tot + 1e-12) / (outer(rowSums(M), colSums(M)) / tot^2 + 1e-12))
P[P < 0 | !is.finite(P)] <- 0; P
}
embed <- function(M, k = 16) { sv <- svd(ppmi(M)); sv$u[, 1:k] %*% diag(sqrt(sv$d[1:k])) }
cosine <- function(E) { En <- E / sqrt(rowSums(E^2) + 1e-12); En %*% t(En) }
Cpur <- co_matrix(baskets, n_prod) # co-purchase (complementarity signal)
Cview <- co_matrix(views, n_prod) # co-view (substitutability signal)
Craw <- Cpur + Cview # what a naive co-occurrence recommender sees
Comp <- ppmi(Cpur) # complementarity = co-purchase association
Sub <- cosine(embed(Cview)) # substitutability = co-view embedding similarity
# --- Policies: naive co-occurrence vs. complement-minus-substitute -----------
naive <- function(i, k = 3) { s <- Craw[i, ]; s[i] <- -Inf; order(s, decreasing = TRUE)[1:k] }
policy_w <- function(w) function(i, k = 3) {
s <- Comp[i, ] * (1 - w * pmax(Sub[i, ], 0)); s[i] <- -Inf
order(s, decreasing = TRUE)[1:k]
}
# --- Ground-truth attach & offline policy evaluation -------------------------
true_attach <- function(i, j) plogis(1.5 * sum(role[i, ] * role[j, ]) -
1.2 * (cat_id[i] == cat_id[j]))
eval_policy <- function(policy, k = 3)
mean(vapply(seq_len(n_prod),
function(i) mean(vapply(policy(i, k), function(j) true_attach(i, j), 0)), 0))
# --- Offline policy learning: tune the substitute penalty w ------------------
ws <- seq(0, 1.5, by = 0.25)
vals <- vapply(ws, function(w) eval_policy(policy_w(w)), 0)
w_star <- ws[which.max(vals)]
base <- eval_policy(naive)
cat(sprintf("Naive co-occurrence attach : %.3f\n", base))
#> Naive co-occurrence attach : 0.428
cat(sprintf("Embedding policy, no penalty (w=0) : %.3f\n", vals[1]))
#> Embedding policy, no penalty (w=0) : 0.813
cat(sprintf("Optimized policy w* = %.2f : %.3f\n", w_star, max(vals)))
#> Optimized policy w* = 0.50 : 0.931
cat(sprintf("Lift of optimized policy over naive : %+.0f%%\n", 100 * (max(vals) - base) / base))
#> Lift of optimized policy over naive : +118%The naive recommender, ranking by raw co-occurrence, fills bundles with the very substitutes that pollute that signal; switching to a purchase-based complementarity score and then penalizing embedded substitutes lifts expected attach sharply. The magnitude in this stylized market runs well above the field experiment’s 35%—the simulation makes the naive baseline deliberately poor—but the direction and, more to the point, the reason are the paper’s: the value is in separating complements from substitutes, which the embedding geometry does almost for free, and in learning the policy from an experiment rather than assuming it.
67.5 Deep Learning
Deep learning refers to neural networks with many layers of learned, nonlinear transformations. A feedforward network composes affine maps and nonlinearities, \[ \hat f(\mathbf{x}) = \sigma_L\!\big(\mathbf{W}_L\,\sigma_{L-1}(\cdots \sigma_1(\mathbf{W}_1\mathbf{x}+\mathbf{b}_1)\cdots)+\mathbf{b}_L\big), \tag{67.7}\] where each \(\mathbf{W}_\ell\) is a learned weight matrix, \(\mathbf{b}_\ell\) a bias, and \(\sigma_\ell\) an elementwise nonlinearity (commonly the rectified linear unit, \(\sigma(z)=\max(0,z)\)). The parameters are fit by gradient descent on the loss in Equation 67.3, with gradients computed by backpropagation—the chain rule applied layer by layer—and the data scanned in mini-batches (stochastic gradient descent). Depth matters because composition lets the network build features from features: early layers learn simple patterns, later layers compose them into abstractions, so the network learns its own representation rather than relying on hand-engineered features. The general case for learned rather than hand-engineered representations is set out in LeCun, Bengio, and Hinton (2015).
The marketing payoff is largest where the data are unstructured—precisely the domains where hand-engineering features is hopeless. Convolutional networks read images, enabling brand-perception measurement directly from consumer-generated photos at a scale and speed no survey could match: Liu, Dzyabura, and Mizik (2020) train a multi-label convolutional network to detect perceptual brand attributes in user images, recovering survey-consistent perceptions in near real time, and image content systematically shapes engagement (Li and Xie 2019). Recurrent and, later, attention-based architectures read sequences—text, clickstreams, purchase histories—turning the unstructured trace of customer behavior into predictive features (Martin, Borah, and Palmatier 2017). The price of this expressive power is steep: deep models are data-hungry, computationally expensive, prone to overfitting without heavy regularization (dropout, early stopping, weight decay), and—most relevant to this chapter—they are opaque, which makes them excellent predictors and poor instruments of inference. The temptation to read a neural network’s learned representations as explanations is the deep-learning incarnation of the prediction–inference confusion.
For tabular marketing data of modest size, it bears emphasizing that deep learning usually does not beat gradient-boosted trees; the deep-learning advantage is specific to large, unstructured, high-signal data. Choosing a neural network for a 50-feature churn table is a common and avoidable error.
67.5.1 Representations as Measurement Instruments
The warning just issued—that learned representations make poor instruments of inference—needs a companion, because a growing and distinctly different line of work uses the representation as the measurement itself, and does so legitimately. The distinction is worth stating precisely. Reading a network’s internal activations as an explanation of why it predicted what it did is the error. Constructing an embedding space whose geometry operationalizes a construct the researcher has defined in advance, then validating that geometry against outcomes, is measurement, and it is subject to the ordinary rules of construct validity (Chapter 37) rather than to the interpretability critique.
Two recent studies show the pattern and its payoff. Sozuer, Netzer, and Krstovski (2026) embed recipe ingredients from co-occurrence across more than 57,000 recipes and read two prespecified quantities out of the resulting geometry—the internal coherence among an idea’s components and the uniqueness of those components for the category—then show that coherence raises popularity, trial, and post-trial evaluation, while unique ingredients lower trial but raise ratings among those who try. The construct is creativity; the embedding is the instrument; and the validation is that the measure predicts outcomes it was not fit to. Sikdar, Chakraborty, and Dogonadze (2026) fuse structured listing attributes, images, and text to price novice artwork on Etsy, and pair the price model with a time-to-sale hazard model so that the output is a decision rather than a coefficient (Section 54.5.1).
Three features recur and are worth naming as a template. The construct is defined before the model is fit, so the embedding is asked a specific question rather than mined for whatever it will yield. The measure is validated against downstream outcomes that were not part of its construction. And both studies end not in a table of coefficients but in a tool—a generative recipe assistant, a price recommender—that hands a decision-maker a concrete action. That last move is what most distinguishes this line from conventional predictive modeling, and it is where the prediction–inference distinction becomes genuinely productive rather than merely cautionary: the goal was never inference, so the opacity objection loses its force, while the construct-validity obligation remains fully in place.
The same pattern carries a standing risk. Because the representation is estimated, any regression that uses it as a covariate faces the generated-regressor problem (Section 47.5), and because the artifacts being embedded are chosen by firms and creators, the features are correlated with unobserved sophistication. Neither study can escape this, and both are appropriately associational in their language.
67.6 Large Language Models
Large language models (LLMs) are deep neural networks—almost always transformers, built on the self-attention mechanism—trained on internet-scale text to predict the next token, and then adapted to follow instructions. Their relevance to marketing is twofold. As measurement instruments, they convert the field’s vast unstructured text—reviews, social posts, support transcripts, open-ended survey responses—into structured variables: sentiment, topics, stance, entities, and the latent dimensions of consumer voice that earlier text methods recovered more laboriously (Netzer, Lattin, and Srinivasan 2008; Büschken and Allenby 2016). As generators, they produce marketing content—copy, product descriptions, personalized email, synthetic chat—at near-zero marginal cost, reshaping the economics of content production across the funnel (Appel et al. 2020).
The transformer’s core operation is attention, which lets each token’s representation be a weighted average of the others, with weights computed from learned query, key, and value projections, \[ \operatorname{Attention}(\mathbf{Q},\mathbf{K},\mathbf{V}) = \operatorname{softmax}\!\left(\frac{\mathbf{Q}\mathbf{K}^{\top}}{\sqrt{d_k}}\right)\mathbf{V}, \tag{67.8}\] where \(\mathbf{Q},\mathbf{K},\mathbf{V}\) are linear projections of the input sequence and \(d_k\) is the key dimension. This mechanism, stacked and scaled, is what lets the model condition each word on the entire context. The architecture that made this practical at scale is the transformer (Vaswani et al. 2017), which replaces recurrence with attention and so lets every position attend to every other in one step.
Three properties demand discipline when LLMs are used as research instruments. First, hallucination: an LLM optimizes for plausible continuations, not truth, and will fabricate confidently; outputs used as data must be validated against ground truth, ideally with a human-labeled audit sample and an inter-rater reliability check (Chapter 37). Second, non-determinism and version drift: the same prompt can yield different outputs, and the underlying model changes under the analyst’s feet, so a measurement pipeline built on a hosted LLM is not automatically reproducible—prompts, model versions, and decoding parameters must be logged like any other instrument. Third, contamination and circularity: an LLM trained on the open web may have seen the very reviews or constructs under study, so “predicting” them is not out-of-sample, and using an LLM both to generate and to evaluate content risks a closed loop that measures the model’s preferences rather than consumers’. Used as instruments, LLMs are powerful but require the same validity scaffolding as any measure; used as generators, they require the brand-safety, factuality, and fairness controls developed in the rest of this chapter.
One instrument use has grown prominent enough to name on its own: the LLM as a synthetic respondent. Prompted to adopt a demographic or attitudinal persona, an LLM will answer survey and choice questions as that person plausibly might—silicon sampling (Argyle et al. 2023)—which dangles the prospect of cheap, instant, fatigue-free subjects. The evidence is mixed by design and by domain: LLMs recover some individual-level preferences and reproduce aggregate patterns (Goli and Singh 2024), yet their synthetic distributions are too narrow, shift with the wording of a prompt, and drift over time, so they are unreliable for the variance-dependent inference that survey data exist to support (Bisbee et al. 2024). The productive posture is therefore augmentation rather than replacement—spending a little human data to debias a lot of LLM data (Wang, Zhang, and Zhang 2026)—which Section 38.4 develops for survey respondents and Section 39.2.4 for conjoint.
Both roles cast the LLM as a tool the firm wields. A third relevance is now emerging that the firm does not control at all: the LLM as a distribution channel. When a chat assistant answers a shopping question and links out to a retailer, it refers traffic the way a search engine or an affiliate does—a channel that did not exist before ChatGPT began emitting organic outbound links in August 2024. Kaiser and Schulze (2026) measure it across 973 e-commerce sites—more than 50,000 LLM-referred transactions against 164 million from traditional channels—and find that organic LLM traffic still converts below every traditional channel except paid social (about 13% under organic search, 86% under affiliate) and earns less per session, though the gap narrows in complex product categories and is closing over time.
The closing-over-time claim is where the paper’s method rewards a look, because “how fast is a brand-new channel maturing?” is easy to answer badly. Rather than extrapolate a straight line, the authors fit a months-since-launch trend under competing functional forms—linear against Gompertz and logistic S-curves that saturate—so a nascent channel’s trajectory cannot be projected off to implausible values. Its data sparsity is handled not with anything exotic but with overdispersed quasibinomial models for rate outcomes, linear models weighted by each cell’s session count for dollar outcomes, and weekly aggregation to blunt zero inflation. The tools are standard—the same diffusion-curve logic marketing has long used for new-product growth (Section 26.3), here turned on a distribution channel; the novelty is the channel, not the estimator. The two estimation ideas the paper leans on—overdispersed rate regression and saturating trend curves—are general enough to deserve their own treatment. The first belongs here, with inference; the second sits with diffusion in Section 26.3.
Referral is only the first step of that third role, and the step after it changes the unit of exchange rather than the channel mix. An assistant that transacts—searching, comparing, negotiating, and buying on a customer’s instruction, against a firm that has fielded an agent of its own—is no longer a channel between two human parties but a party in its own right. Pattabhiramaiah et al. (2026) call the resulting triad B2A2C and argue that it defeats both of the theories a marketer would reach for: relationship marketing, which assumes direct firm–customer interaction, and principal–agent theory, which assumes one agent to align. Their CARMA framework—calibration, accountability, reciprocity, mediation, adaptivity—is a conceptual agenda rather than evidence, and Section 5.9 develops what it implies for trust and loyalty as constructs. The point worth carrying into a modeling chapter is narrower: in an agent-mediated market the entity generating the click, the query, and the conversion is a model with its own prompt, its own retrieval, and its own version history. Every measurement caution in this section—prompt sensitivity, silent version drift, narrow synthetic distributions—then applies to the demand side of the data, not merely to the instrument the analyst chose.
67.7 Modeling Rates with Overdispersion: The Quasibinomial GLM
The channel comparison above is an inference question in the exact sense of Section 32.7’s fork: not “predict whether this session converts” but “does channel \(A\) convert at a different rate than channel \(B\), and how sure are we?” The object of interest is a coefficient and its standard error, so the estimator must have a sampling distribution we trust. Aggregate conversion data look deceptively easy to model this way—\(y_i\) conversions out of \(n_i\) sessions is the textbook binomial—but the textbook variance is almost always wrong for marketing rates, and taking it at face value manufactures precision that is not there.
Write the workhorse logistic GLM for a proportion. With \(y_i\) conversions in \(n_i\) sessions and covariates \(\mathbf{x}_i\) (channel, website, device, month), \[ y_i \sim \text{Binomial}(n_i, \mu_i), \qquad \operatorname{logit}(\mu_i) = \mathbf{x}_i^{\top}\boldsymbol\beta, \qquad \operatorname{Var}(y_i) = n_i\,\mu_i(1-\mu_i). \tag{67.9}\] The last equality is not an assumption one gets to make lightly: it says the only source of variation in the count is the coin-flip randomness of \(n_i\) independent sessions sharing a single rate \(\mu_i\). Real channel-week-website cells violate it routinely. Sessions within a cell are correlated (a viral post, a promotion, a bot wave); the rate \(\mu_i\) itself wanders with unmodeled heterogeneity across sites and weeks; a sparse new channel like oLLM piles many near-empty cells beside a few large ones. Each mechanism inflates the variance of \(y_i\) beyond the binomial floor—it is overdispersed—and the inflation is exactly what a naive logistic regression assumes away.
The quasibinomial model repairs the inference without pretending to a full likelihood. Following the quasi-likelihood idea (Wedderburn 1974; McCullagh and Nelder 1989), one specifies only the mean and a variance proportional to the binomial one, \[ \operatorname{Var}(y_i) = \phi\, n_i\,\mu_i(1-\mu_i), \qquad \phi \ge 1, \tag{67.10}\] where \(\phi\) is a dispersion parameter to be estimated rather than fixed at \(1\). This is precisely the variance Kaiser and Schulze (2026) write as \(\operatorname{Var}(y^{\text{txn}}_{it}) = \theta\, n^{\text{sess}}_{it}\, p_{it}(1-p_{it})\): their \(\theta\) is this \(\phi\), and their weekly aggregation and session weighting are complementary devices aimed at the same variance inflation.
Two facts make the quasibinomial attractive. First, because the estimating equations for \(\boldsymbol\beta\), \[ \sum_i \frac{y_i - n_i\mu_i}{\phi\, n_i \mu_i(1-\mu_i)}\, \frac{\partial \mu_i}{\partial \boldsymbol\beta} = \mathbf{0}, \tag{67.11}\] carry \(\phi\) as a constant multiplier, it cancels: the point estimates \(\hat{\boldsymbol\beta}\) are identical to the ordinary binomial fit. Overdispersion does not bias the coefficients; it corrupts their standard errors. Second, \(\phi\) is estimated cheaply from the Pearson residuals, \[ \hat\phi = \frac{1}{N-p}\sum_i \frac{(y_i - n_i\hat\mu_i)^2}{n_i\hat\mu_i(1-\hat\mu_i)}, \tag{67.12}\] and the corrected covariance is just the binomial one scaled, \(\widehat{\operatorname{Var}}_{\text{quasi}}(\hat{\boldsymbol\beta}) = \hat\phi\,\widehat{\operatorname{Var}}_{\text{binom}}(\hat{\boldsymbol\beta})\), so every standard error grows by \(\sqrt{\hat\phi}\) and every naive \(t\)-statistic shrinks by the same factor. A cell that reports \(\hat\phi = 9\) was quoting standard errors three times too small. The simulation below plants overdispersion deliberately and recovers it.
Code
set.seed(2024)
# Weekly conversions for two channels across many websites. Overdispersion is built
# in: each website carries its OWN latent conversion propensity (a Beta draw), so the
# aggregate counts vary far more than a single common Binomial(n, p) would permit.
sim_channel <- function(mean_p, rho, n_site = 500, mean_sess = 300) {
kappa <- (1 - rho) / rho # Beta concentration; rho = dispersion
p_site <- rbeta(n_site, mean_p * kappa, (1 - mean_p) * kappa)
n_sess <- rpois(n_site, mean_sess) + 1L
data.frame(conv = rbinom(n_site, n_sess, p_site), sess = n_sess)
}
d <- rbind(
transform(sim_channel(0.020, rho = 0.10), channel = "oLLM"),
transform(sim_channel(0.023, rho = 0.10), channel = "organic_search")
)
d$channel <- relevel(factor(d$channel), ref = "oLLM")
m_bin <- glm(cbind(conv, sess - conv) ~ channel, family = binomial, data = d)
m_qbin <- glm(cbind(conv, sess - conv) ~ channel, family = quasibinomial, data = d)
phi <- summary(m_qbin)$dispersion # estimated phi-hat
compare <- data.frame(
term = names(coef(m_bin)),
estimate = round(coef(m_bin), 4), # identical across the two fits
se_binom = round(sqrt(diag(vcov(m_bin))), 4),
se_quasi = round(sqrt(diag(vcov(m_qbin))), 4)
)
compare$se_ratio <- round(compare$se_quasi / compare$se_binom, 2)
cat(sprintf("Estimated dispersion phi-hat = %.1f (sqrt = %.2f)\n", phi, sqrt(phi)))
#> Estimated dispersion phi-hat = 32.3 (sqrt = 5.68)
print(compare, row.names = FALSE)
#> term estimate se_binom se_quasi se_ratio
#> (Intercept) -3.8115 0.0177 0.1006 5.68
#> channelorganic_search 0.0467 0.0248 0.1407 5.67The coefficients are the same to the last digit; the quasibinomial standard errors are larger by very close to \(\sqrt{\hat\phi}\), exactly as Equation 67.12 predicts. The practical consequence is a matter of what survives: the organic-search-versus-oLLM contrast that clears significance under the binomial’s optimistic errors may not clear it once the dispersion is admitted, which is why Kaiser and Schulze (2026) report their channel gaps against dispersion-corrected errors rather than the raw binomial ones.
Quasibinomial and its neighbors
The quasibinomial is the lightest of several overdispersion fixes, and the right choice depends on the question. A beta-binomial model puts an explicit distribution on the per-cell rate and yields a genuine likelihood (hence AIC/BIC and likelihood-ratio tests), at the cost of a distributional assumption the quasibinomial avoids. A generalized linear mixed model with a website random effect models the heterogeneity structurally and returns shrinkage estimates per site, but it is heavier and can be finicky with thousands of sparse cells. Cluster-robust (sandwich) standard errors correct inference without touching the variance function and are the natural move when the correlation is within known clusters. All four buy the same thing—honest uncertainty for a rate—so the quasibinomial’s appeal is that it delivers it with one extra estimated scalar and no new distributional commitment. Reach past it only when you need the likelihood machinery or the per-site effects it does not provide.
Note the tidy division of labor with the prediction–inference fork of Section 32.7. A pure predictor of “will this session convert” is indifferent to \(\phi\): overdispersion leaves the fitted probabilities \(\hat\mu_i\) untouched, so out-of-sample loss is unchanged. It is only when the coefficient itself is the deliverable—a channel comparison, an elasticity, a lift—that the dispersion parameter becomes load-bearing. Overdispersion is an inference bug, not a prediction bug, which is why it is invisible to a leaderboard and fatal to a claim.
67.8 Disclosure and the Producer Side of Generative AI
Return to the LLM’s generator role, and to the policy instrument that has grown up around it. Regulators increasingly require that generative-AI involvement in creative work be disclosed—a label attached to the artifact, or a content credential travelling with it—on the theory that audiences deserve to know what they are looking at. The research question this invites is an evaluation question: does a label change how an image, an ad, or a piece of copy is judged? Nearly all of the evidence answers that question, and it holds the artifact fixed while randomizing the label.
Jussupow et al. (2026) show that this design misses most of the effect, because the label reaches the producer first. In two nested mixed-methods experiments in which participants collaborate with a text-to-image tool under different disclosure regimes, creators who merely anticipate that their AI use will be disclosed change how they work: the majority withdraw from the creative process, handing generation over to the model rather than iterating with it. The mechanism the authors theorize, from Goffman’s account of impression management, is not embarrassment about using AI but a threat to validation as a creative self—creators expect that the label will keep the audience from recognizing their own creative agency, so the agency stops being worth investing. The artifacts that result predominantly reflect computational rather than human creativity, and audiences perceive and downgrade them accordingly, label or no label. The authors name this the indirect disclosure effect, and its policy sting is precise: a transparency rule intended to protect human creative agency can erode it before any audience ever sees the disclosure.
Two lessons generalize past the creative setting. The first is a research-design lesson with the structure of a general-equilibrium critique: when an intervention is announced in advance, the artifact under evaluation is itself an outcome of the intervention, so an experiment that randomizes labels across fixed stimuli estimates a partial effect and can get the sign of the policy wrong. Anything that alters what producers anticipate—AI labels, provenance metadata, review-verification badges, algorithmic transparency requirements (Chapter 25)—requires a design that lets production respond. The second is a measurement lesson: the operative variable is a second-order belief, what the creator expects the audience to infer, which is neither the creator’s own attitude toward AI nor the audience’s actual reaction, and which must be measured as its own construct (Chapter 37) rather than assumed from either endpoint. Firms deploying generative tools internally face the same mechanism in miniature: an attribution policy that marks AI-assisted work will shift not only how that work is received but how much of themselves employees put into it.
The same producer-side logic scales up from the individual creator to the platform that hosts them. Huang, Fu, and Ghose (2026) study two opposite-signed policy shocks on Chinese visual-arts platforms—one platform launching its own AI image generator, another prohibiting AI-generated artwork—and find creator activity falling in the first case and rising in the second. As with disclosure, the operative variable is an expectation rather than a feature: creators read the policy as a statement about AI’s role, about the platform’s commitment to human work, and about the competition their work will face, with the largest pullback among popular, multi-homing, and AI-averse creators. Put beside Jussupow et al. (2026), the pair delimits the producer-side response to generative AI at two levels—what a creator does within a task when a label is coming, and whether the creator shows up at all when a platform declares a stance. Neither effect is visible to a design that holds the artifact, or the creator population, fixed; the platform-governance reading of the same evidence is developed in Section 68.6.
67.8.1 Replication: what a label-only design cannot see
The claim that the standard design is biased is worth making precise rather than asserting, because the bias has a definite structure and a measurable size. Write the regime as \(D \in \{0,1\}\)—whether the creator knows in advance that the artifact will carry a disclosure label. The regime reaches the audience’s rating \(Y\) through two routes. The direct route is the label itself, seen at evaluation time. The indirect route runs through production: \(D\) raises the creator’s second-order belief \(B\) that the audience will not recognize their agency, \(B\) triggers withdrawal from the collaboration, withdrawal lowers the human creative content \(H\) of the artifact, and \(H\) is what the audience actually rewards.
flowchart LR
D["Anticipated disclosure\n(regime announced to creator)"]
B["Second-order belief:\n'they will not see my agency'"]
W["Withdrawal from\nthe collaboration"]
H["Human creative content\nof the artifact"]
Y["Audience evaluation"]
L["Disclosure label\nshown at evaluation"]
D --> B --> W --> H --> Y
D --> L
L -. "direct path (all a label-only design sees)" .-> Y
The simulation below instantiates that chain with a modest direct label penalty and a withdrawal response calibrated to the paper’s qualitative finding—a minority of creators hand generation over to the model when nothing will be disclosed, a majority when disclosure is anticipated. It then runs the two designs side by side on the same population: the conventional experiment, which draws artifacts from the no-disclosure regime and randomizes only the label; and the regime comparison, which lets production respond. A third contrast strips the label from the audience entirely, to check whether the artifact is already worse on its own.
Code
set.seed(951)
n <- 3000
# --- Producer stage: creators randomized to ANTICIPATED disclosure ------------
disc <- rbinom(n, 1, 0.5) # will my AI use be labeled?
# Second-order belief: "the audience will not recognize my creative agency."
belief <- 1.0 * disc + rnorm(n, 0, 1)
# Withdrawal: accept the model's output instead of iterating with it.
withdraw <- rbinom(n, 1, plogis(-0.7 + 1.2 * belief))
# Creative involvement, and the human creativity it puts into the artifact.
iters <- pmax(0, rnorm(n, 6 - 3 * withdraw - 0.8 * belief, 1.5))
human <- 0.35 * iters + rnorm(n, 0, 1) # human-creativity content
# --- Audience stage: rating responds to the artifact AND to the label --------
b_human <- 0.55 # audiences reward human creative content
b_label <- -0.25 # direct penalty from seeing the label
rate <- function(h, lab) 3.2 + b_human * h + b_label * lab + rnorm(length(h), 0, 1)
# (a) The standard label-only design: hold artifacts fixed at what creators make
# when no disclosure is anticipated, then randomize the label.
pool <- human[disc == 0]
lab_r <- rbinom(length(pool), 1, 0.5)
d_lab <- data.frame(y = rate(pool, lab_r), lab = lab_r)
est_labelonly <- coef(lm(y ~ lab, data = d_lab))["lab"]
# (b) The policy-relevant contrast: artifacts are produced under the regime and
# then carry its label.
y_regime <- rate(human, disc)
est_total <- coef(lm(y_regime ~ disc))["disc"]
# (c) Blind the audience to the label: is the artifact itself already worse?
y_blind <- rate(human, rep(0, n))
est_blind <- coef(lm(y_blind ~ disc))["disc"]
# --- Decomposition: how much of the total runs through production? -----------
decomp <- function(idx) {
h <- human[idx]; d <- disc[idx]
fit <- lm(rate(h, d) ~ h + d)
dh <- mean(h[d == 1]) - mean(h[d == 0]) # regime -> artifact
c(indirect = unname(dh * coef(fit)["h"]), # artifact -> rating
direct = unname(coef(fit)["d"]))
}
boot <- replicate(400, decomp(sample.int(n, n, replace = TRUE)))
ci <- apply(boot, 1, quantile, c(0.025, 0.975))
cat(sprintf("Withdrawal rate, no disclosure anticipated : %.0f%%\n",
100 * mean(withdraw[disc == 0])))
#> Withdrawal rate, no disclosure anticipated : 34%
cat(sprintf("Withdrawal rate, disclosure anticipated : %.0f%%\n",
100 * mean(withdraw[disc == 1])))
#> Withdrawal rate, disclosure anticipated : 61%
cat(sprintf("Creative iterations, control vs. disclosure : %.2f vs %.2f\n\n",
mean(iters[disc == 0]), mean(iters[disc == 1])))
#> Creative iterations, control vs. disclosure : 5.02 vs 3.47
cat(sprintf("Label-only design (artifacts held fixed) : %+.3f\n", est_labelonly))
#> Label-only design (artifacts held fixed) : -0.306
cat(sprintf("Regime effect on rating (label + response) : %+.3f\n", est_total))
#> Regime effect on rating (label + response) : -0.599
cat(sprintf("Regime effect with the label hidden : %+.3f\n\n", est_blind))
#> Regime effect with the label hidden : -0.310
cat(sprintf(" producer-side (indirect) path : %+.3f [%+.3f, %+.3f]\n",
mean(boot["indirect", ]), ci[1, "indirect"], ci[2, "indirect"]))
#> producer-side (indirect) path : -0.308 [-0.358, -0.250]
cat(sprintf(" audience-side (direct) path : %+.3f [%+.3f, %+.3f]\n",
mean(boot["direct", ]), ci[1, "direct"], ci[2, "direct"]))
#> audience-side (direct) path : -0.253 [-0.322, -0.182]
cat(sprintf(" share of the total a label-only design cannot see : %.0f%%\n",
100 * mean(boot["indirect", ]) / est_total))
#> share of the total a label-only design cannot see : 51%The label-only experiment is not wrong; it is answering a different question, correctly. It recovers the direct penalty—what a label does to the perception of a fixed artifact—and misses roughly half of the effect of the policy, because the other half was realized upstream when creators, anticipating the label, stopped putting themselves into the work. The blinded contrast is the diagnostic that makes this visible: artifacts produced under an anticipated-disclosure regime rate lower even when the audience never sees a label, which cannot happen if the label operates only at the point of evaluation.
The structure generalizes to any intervention whose existence is known to the party generating the outcome, which is most of marketing regulation and a great deal of platform policy. Verified-review badges, provenance metadata, algorithmic-transparency notices, and internal attribution rules all share the shape: announce the rule and the stimulus itself becomes endogenous. The design implication is that the randomization must be assigned at the level of the regime, before production, not at the level of the artifact after it—and that the second-order belief, what the producer expects the audience to infer, has to be measured as its own construct (Chapter 37) rather than backed out from either side’s behavior.
The producer-side response is not always a cost. Where an anticipated label makes creators withdraw, a well-designed first interaction makes users lean in: Jiang et al. (2026) show on a generative-AI co-creation platform that revealing part of the co-created output before the registration wall raises both the user’s sense that their input mattered and their curiosity about what remains, and so raises registration and return visits (Section 68.4). The same mechanism—what the human expects the interaction to say about their own contribution—runs in both cases; only the sign of the design changes.
67.9 Machine Learning Across the Marketing Funnel
The methods above are not siloed by funnel stage; the same supervised, unsupervised, and generative tools recur, retargeted at different objectives. Organizing them by the customer journey clarifies what is being predicted at each step and where the prediction–inference distinction bites. Table 67.1 maps the terrain, and the recurring lesson is that most funnel applications are predictive scoring problems, while the decisions they feed—how much to spend, what to charge, whom to target—are causal questions that prediction alone cannot answer.
| Funnel stage | Representative task | Learning paradigm | Pred. vs. inf. | Anchor |
|---|---|---|---|---|
| Awareness | Audience look-alike modeling; media-mix forecasting | Supervised classification; sequence models | Prediction (lift is causal) | Wedel and Kannan (2016) |
| Consideration | Search/social listening; brand-perception mining | Unsupervised; LLM/text | Prediction (measurement) | Netzer et al. (2008); Liu, Dzyabura, and Mizik (2020) |
| Conversion | Propensity-to-buy; dynamic creative; recommendation | Supervised; recommender | Prediction; targeting is causal | Neumann, Tucker, and Whitfield (2019) |
| Retention | Churn prediction; next-best-action | Supervised classification | Prediction; action is causal | Neumann, Tucker, and Whitfield (2019) |
| Advocacy | Review/UGC analysis; influencer identification | Unsupervised; LLM/text | Prediction (measurement) | Tirunillai and Tellis (2014) |
The “prediction (lift is causal)” and “action is causal” entries flag the same trap in different clothes. A look-alike model predicts who resembles a converter, but whether advertising to them causes incremental conversion is a question only a holdout experiment answers. A churn model predicts who will leave, but the right retention action depends on who will leave because of, or despite, the intervention—an uplift, not a prediction, problem. Ascarza (2018) makes the point with unusual force: in two field experiments, targeting the customers a churn model ranked as highest-risk was no better than random, because risk of leaving and responsiveness to an intervention are different quantities and the industry standard optimizes the wrong one. The funnel view is useful precisely because it keeps surfacing the boundary the chapter is built around.
67.10 Pitfalls: How Machine Learning Quietly Breaks
A model that scores well offline and fails in production is the rule, not the exception, and the failures cluster into three families: leakage, which inflates offline performance with information that will not exist at decision time; drift, which erodes performance as the world moves away from the training distribution; and unfairness, which encodes and amplifies inequities the data inherited. None of the three is a coding bug; all three are violations of the assumptions behind Equation 67.3. Figure 67.3 situates them in the model lifecycle.
flowchart LR A["Data collection"] --> B["Feature engineering<br/>& training"] B --> C["Offline validation"] C --> D["Deployment &<br/>decisions"] D --> E["Monitoring"] E -.->|"new data"| A B -. "LEAKAGE<br/>(future/target info<br/>leaks into features)" .-> C D -. "DRIFT<br/>(world moves away<br/>from training dist.)" .-> E D -. "FAIRNESS<br/>(disparate harm at<br/>the decision boundary)" .-> A
67.10.1 Data leakage
Leakage occurs when information unavailable at prediction time contaminates the training data, so the model “cheats” offline and collapses in production. The signature is an offline metric that is too good to be true. Leakage takes several forms, each common in marketing pipelines. Target leakage includes a feature that is a proxy for, or a consequence of, the outcome—predicting purchase using “added-to-cart in the same session,” which is nearly the outcome itself. Temporal leakage uses information from after the prediction timestamp—computing a customer’s average order value over a window that includes the very order being predicted. Preprocessing leakage is the most insidious because it survives a careful feature audit: when scaling, imputation, feature selection, or target encoding is fit on the full dataset before the train/test split, statistics from the test set bleed into training, and cross-validation reports an accuracy the deployed model cannot reach.
The discipline that prevents leakage is to fit every data-dependent transformation inside the cross-validation fold, on training data only, and to respect time: when data are temporally ordered—as marketing data almost always are—validation must use a forward-chaining split that trains on the past and tests on the future, never a random split that lets the model peek ahead. The example contrasts the two and quantifies the illusion.
Code
set.seed(48)
# Time-ordered data where the signal is non-stationary (drifts over time)
N <- 1500
time <- seq_len(N)
beta_t <- 1.5 - 1.0 * (time / N) # the effect of x decays over time
x <- rnorm(N)
y <- as.integer(plogis(-0.2 + beta_t * x + rnorm(N, 0, 0.5)) > 0.5)
df <- data.frame(time, x, y)
auc <- function(score, label) {
pos <- score[label == 1]; neg <- score[label == 0]
if (!length(pos) || !length(neg)) return(NA_real_)
mean(outer(pos, neg, ">") + 0.5 * outer(pos, neg, "=="))
}
# (a) RANDOM split: leaks future into training on non-stationary data
ridx <- sample(N, 0.7 * N)
m_rand <- glm(y ~ x, df[ridx, ], family = binomial())
auc_rand <- auc(predict(m_rand, df[-ridx, ], type = "response"), df[-ridx, ]$y)
# (b) FORWARD-CHAINING split: train on past, test on future (honest)
cut <- floor(0.7 * N)
m_fwd <- glm(y ~ x, df[df$time <= cut, ], family = binomial())
auc_fwd <- auc(predict(m_fwd, df[df$time > cut, ], type = "response"),
df[df$time > cut, ]$y)
cat("AUC, random split (optimistic): ", round(auc_rand, 3), "\n")
#> AUC, random split (optimistic): 0.903
cat("AUC, forward-chaining (honest): ", round(auc_fwd, 3), "\n")
#> AUC, forward-chaining (honest): 0.865The random-split AUC overstates the performance the model will actually deliver on future data, because random splitting on a non-stationary process lets the model borrow the future to predict the past. The gap between the two numbers is a direct measure of self-deception, and it is invisible to anyone who validates with a random split—the default in most tutorials.
67.10.2 Distribution drift
Even a leakage-free model decays, because the assumption that deployment data share the training distribution expires. Drift comes in two flavors that demand different responses. Covariate shift changes the input distribution while the relationship is stable, \(p_{\text{test}}(\mathbf{x}) \neq p_{\text{train}}(\mathbf{x})\) but \(p(y\mid\mathbf{x})\) unchanged—a new acquisition channel brings customers unlike the training population. Concept drift changes the relationship itself, \(p_{\text{test}}(y\mid\mathbf{x}) \neq p_{\text{train}}(y\mid\mathbf{x})\)—a recession, a competitor’s launch, or a pandemic rewrites how features map to behavior, exactly the non-stationarity simulated above. Marketing is a near-worst case for drift because the environment is adversarial and reflexive: competitors react, consumer tastes move, and—uniquely—the model’s own actions change the data it next sees, so a targeting model trained on one policy’s data is evaluated under another. The defenses are monitoring (track input distributions and live performance against a holdout, and alarm on divergence), scheduled or triggered retraining, and—where decisions feed back into data—maintaining randomized holdouts so the model never fully determines its own training distribution.
67.10.3 Fairness and algorithmic bias
Machine-learning models inherit and can amplify the biases latent in their training data, with legal and ethical force when marketing decisions touch credit, housing, employment, insurance, or protected groups. The mechanism is not malice but statistics: if historical data reflect discrimination, a model that faithfully predicts the historical target reproduces the discrimination, and omitting a protected attribute does not fix it, because correlated proxies (postal code, device, browsing history) reconstruct the attribute—a phenomenon known as redundant encoding.
Fairness must therefore be defined, measured, and traded off explicitly, and a basic impossibility result disciplines expectations: several intuitive fairness criteria—equal false-positive and false-negative rates across groups (equalized odds) versus equal calibration across groups versus statistical parity in selection rates—cannot in general all hold simultaneously unless base rates are equal or the classifier is perfect. There is no single “fair” model; there is a choice among incompatible fairness criteria, and that choice is normative, not technical. The worked example audits a scorer for disparate impact—whether selection rates differ across groups—and shows that proxy features carry bias even when the protected attribute is excluded. The impossibility results are worth confronting directly: except in degenerate cases, calibration within groups and equal false-positive and false-negative rates across groups cannot all hold at once, so a fairness criterion is a choice among incompatible goods rather than a box to tick (Kleinberg et al. 2017).
Code
set.seed(48)
n <- 6000
group <- rbinom(n, 1, 0.5) # protected attribute A in {0,1}
# A proxy correlated with the protected group (e.g., neighborhood), NOT the label cause
proxy <- rnorm(n, mean = 0.8 * group)
quality <- rnorm(n) # the legitimately relevant signal
# True outcome depends ONLY on quality (group is irrelevant to merit)...
y <- rbinom(n, 1, plogis(1.2 * quality))
df <- data.frame(y, quality, proxy, group)
# Model that EXCLUDES the protected attribute but KEEPS the correlated proxy
fit <- glm(y ~ quality + proxy, df, family = binomial())
df$score <- predict(fit, type = "response")
tau <- quantile(df$score, 0.7) # select top 30%
df$selected <- as.integer(df$score > tau)
# Disparate impact: ratio of selection rates across groups (4/5ths rule -> >= 0.8)
sel_rate <- tapply(df$selected, df$group, mean)
di_ratio <- min(sel_rate) / max(sel_rate)
cat("Selection rate by group:", round(sel_rate, 3), "\n")
#> Selection rate by group: 0.31 0.291
cat("Disparate-impact ratio: ", round(di_ratio, 3),
if (di_ratio < 0.8) " (FAILS 4/5ths rule)" else " (passes)", "\n")
#> Disparate-impact ratio: 0.938 (passes)The disparate-impact ratio falls below the conventional four-fifths threshold even though the protected attribute never entered the model and plays no role in the true outcome—the proxy alone manufactures the gap. The lesson generalizes: fairness is a property of a model-in-context that must be measured on outcomes, not assumed from the feature list, and remediation (reweighting, constrained optimization, post-hoc threshold adjustment by group) is an explicit, auditable design choice with its own trade-offs against accuracy and against other fairness criteria.
67.10.3.1 When the fairness policy needs the attribute it is not allowed to demand
The audit above assumes the protected attribute is known to the analyst even when it is excluded from the model. In recommendation settings it frequently is not. A firm that wants to commit to accuracy parity—consistent recommendation quality across demographic groups—generally has to rely on consumers disclosing their identity voluntarily, and that turns the fairness policy into a game rather than a constraint.
Smith, Shulman, and Kim (2026) model it as one. Digital platforms recommending video, music, and news deliver systematically worse recommendations to some consumer groups, and a parity commitment is the natural remedy; but consumers choose whether to disclose, the firm chooses what to recommend, and the firm also chooses how much to invest in the AI’s learning. Their game-theoretic model puts all three in one system, and it is worth being clear about why they cannot be separated. Disclosure is what makes parity enforceable, and it is also what determines the training data the firm’s learning investment operates on, so a consumer’s disclosure decision is partly a decision about how good the firm’s model will become for people like them—while the firm’s parity commitment changes the payoff to disclosing in the first place. The equilibrium characterization of that system is the paper’s contribution and is left to it here.
The methodological point for this chapter is that the impossibility results above are stated for a given dataset with group membership observed. Once group membership is elicited rather than observed, the feasible set is endogenous to the policy, and a parity rule that looks costless in an audit can be undermined or amplified by who chooses to be counted. Any marketing team designing an accuracy-parity commitment should treat the disclosure margin as a first-order design variable rather than a data-collection detail.
67.11 Deploying Marketing AI: The Operations Interface
Read the three pitfalls again and a pattern emerges: none of them is fixed by a better estimator. Leakage is a failure of pipeline discipline, drift is a failure of monitoring and refresh cadence, and unfairness is a failure of specification and audit. They are operations problems attached to a statistical object, and the literature that studies them systematically is operations management rather than machine learning. Dai and Swaminathan (2026) organize that literature into a triadic framework whose three pillars translate cleanly onto marketing.
The first pillar, AI for operations, asks where AI improves an operational process—in their vocabulary the design–procure–produce–deliver chain, in marketing the campaign chain of targeting, creative generation, bidding, and service delivery. This is the pillar the chapter has occupied so far, and its discipline is the one already insisted upon: performance claims are empirical and context-dependent, so AI may improve a task, leave it unchanged, or introduce new frictions, and the burden is on measurement rather than assumption. The second pillar, operations for AI, inverts the lens: the AI system is itself something produced and operated, so the tools built for factories—workflow design, capacity management, statistical process control, lot-sizing, continuous improvement—apply to the model lifecycle. The third pillar, human–AI interaction, holds that the value realized is mediated by whether the humans in the loop trust, override, or rubber-stamp the system, which makes interface design, incentives, and supervisory capacity part of the estimator’s performance. Figure 67.4 situates the three and their dependencies.
flowchart TB A["AI FOR MARKETING<br/>Targeting, creative generation,<br/>bidding, recommendation,<br/>service delivery"] B["MARKETING OPS FOR AI<br/>Data pipelines, retraining cadence,<br/>drift monitoring, audit,<br/>governance"] C["HUMAN-AI INTERACTION<br/>Trust calibration, override policy,<br/>supervisory capacity,<br/>accountability"] A -->|"generates lifecycle<br/>demands"| B B -->|"sustains deployable<br/>models"| A A -->|"output must be<br/>acted on"| C C -->|"adoption, overrides,<br/>and labeled feedback"| B B -->|"explanations, alerts,<br/>and thresholds"| C
67.11.1 The model pipeline as a production line
The productive analogy is exact enough to borrow results, not merely vocabulary. Data are inventory, subject to quality inspection at receipt; training is production, with each pass a lot whose size trades off setup against holding cost; deployment is service delivery under a latency service-level agreement; and monitoring with retraining is continuous improvement. The claim carrying the most weight for a marketing analytics team is the one about cadence, because it is where practice most often defaults to habit (“we retrain quarterly”) rather than to a decision.
Formalize the lot-sizing analogy that Dai and Swaminathan (2026) draw. Let \(K\) denote the fixed cost of one refresh cycle—compute, validation, review, and the risk of a bad deployment—and let staleness damage accrue at rate \(c_d\) per period, so that a model \(a\) periods old costs \(c_d a\) in foregone margin relative to a freshly fitted one. Over a refresh cycle of length \(T\), average cost per period is \[ C(T) \;=\; \frac{K}{T} \;+\; \frac{c_d\,(T-1)}{2}, \tag{67.13}\] whose minimizer is the economic-order-quantity square root, \[ T^\* \;=\; \sqrt{\frac{2K}{c_d}}. \tag{67.14}\]
Two consequences follow that are not obvious from the machine-learning side. First, optimal cadence is flat near the optimum and scales with the square root of the cost ratio: halving the cost of a retraining cycle shortens the optimal interval by only about 30%, so the case for automating the pipeline rests on reliability and on the option to respond to shocks, not on the arithmetic of cadence alone. Second, the whole calculation presumes a constant drift rate, which marketing violates—drift arrives in regime breaks (a competitor’s launch, a platform policy change, a seasonal discontinuity) rather than as a steady trickle. Under lumpy drift a fixed schedule is dominated by a triggered policy that retrains on a monitoring signal, which is the operational content of the monitoring advice in Section 67.10.2.
Code
set.seed(4826)
weeks <- 520
K <- 40 # fixed cost of one refresh cycle (compute + validation + review)
# Weekly margin lost to staleness. Most weeks the world moves slowly; ~4% of weeks
# bring a regime break (competitor launch, platform policy change, seasonal shift)
# that pushes the deployed model away from the world far faster.
regime_break <- rbinom(weeks, 1, 0.04)
drift <- ifelse(regime_break == 1, rgamma(weeks, 2, 0.4), rgamma(weeks, 2, 4))
# A model of age a costs the accumulated drift since its last refresh.
cost_scheduled <- function(T) {
age <- (seq_len(weeks) - 1) %% T # weeks since last refresh
dmg <- ave(drift, cumsum(age == 0), FUN = cumsum)
(sum(dmg) + K * ceiling(weeks / T)) / weeks
}
grid <- 2:52
c_sched <- vapply(grid, cost_scheduled, numeric(1))
T_best <- grid[which.min(c_sched)]
T_eoq <- sqrt(2 * K / mean(drift)) # Eq. for T* at the average drift rate
# Triggered policy: refresh when a noisily monitored staleness measure crosses h.
cost_triggered <- function(h) {
acc <- 0; dmg <- 0; n <- 0
for (t in seq_len(weeks)) {
acc <- acc + drift[t]
dmg <- dmg + acc
if (acc + rnorm(1, 0, 0.3) > h) { acc <- 0; n <- n + 1 } # monitor is imperfect
}
(dmg + K * n) / weeks
}
h_grid <- seq(1, 40, by = 1)
c_trig <- vapply(h_grid, cost_triggered, numeric(1))
cat("EOQ cadence at average drift: ", round(T_eoq, 1), "weeks\n")
#> EOQ cadence at average drift: 10.3 weeks
cat("Best fixed schedule: every", T_best, "weeks,",
"cost", round(min(c_sched), 2), "\n")
#> Best fixed schedule: every 11 weeks, cost 8.02
cat("Best triggered policy: threshold", h_grid[which.min(c_trig)], ",",
"cost", round(min(c_trig), 2), "\n")
#> Best triggered policy: threshold 6 , cost 7.09
cat("Cost reduction from monitoring:",
paste0(round(100 * (min(c_sched) - min(c_trig)) / min(c_sched), 1), "%\n"))
#> Cost reduction from monitoring: 11.7%The square-root formula lands close to the best fixed schedule, which is the point of carrying it: it gives an order-of-magnitude answer from two quantities a team can estimate—what a refresh costs and how fast holdout performance decays—without a simulation. The triggered policy then buys a further reduction precisely because drift is lumpy: it spends refreshes when the world moves and withholds them when it does not. The managerial reading is that monitoring infrastructure is not overhead attached to the model but a substitute for retraining capacity, and the exchange rate between them is estimable.
The same lens rehabilitates a practice the machine-learning literature treats as hygiene. Statistical process control asks whether a monitored series is in control, and drift detection is that question asked of a model: an alarm is an out-of-control signal, and the specification problem is choosing which characteristics to chart. Accuracy alone is inadequate, because a marketing model can hold its aggregate hit rate while its false-positive rate on a segment, its latency at the tail, or its calibration in a new channel deteriorates. The discipline Dai and Swaminathan (2026) describe as a “Six Sigma for AI”—define the failure modes, measure baseline rates, analyze root causes, improve, and control through ongoing monitoring—is the manufacturing version of the validation habits urged in Section 67.10.2, with the addition that the metric set is chosen before deployment and charted continuously rather than reconstructed after a complaint.
67.11.2 Human supervision is a capacity constraint
The pillar that most often goes unmodeled is the human one, and it has a capacity that can be written down. Define the neglect time \(T_N\) as how long an automated agent runs unattended before its performance degrades, and the interaction time \(T_I\) as how long a human needs to review and resolve one exception. The fan-out \[ F \;=\; \frac{T_N}{T_I} \tag{67.15}\] is the number of agents one operator can effectively supervise (Dai and Swaminathan 2026). Treat the reviewer as a server and exceptions as arrivals and the constraint becomes a queueing one: with arrival rate \(\lambda\), service rate \(\mu = 1/T_I\), and \(k\) reviewers, stability requires \(\rho = \lambda/(k\mu) < 1\), and because waiting times explode as \(\rho \to 1\), practice targets \(\rho \lesssim 0.7\) to leave headroom for alert storms.
Code
# A marketing-operations review desk supervising autonomous agents (campaign
# generation, bid adjustment, creative approval, escalation triage).
T_N <- 6.0 # hours an agent runs unattended before quality degrades
T_I <- 0.4 # hours to review and resolve one exception
fan_out <- T_N / T_I # agents one reviewer can hold (Eq. for F)
lambda <- 1 / T_N # exceptions per agent-hour
mu <- 1 / T_I # reviews per reviewer-hour
k <- 2 # reviewers on shift
agents <- c(10, 15, 20, 25, 30)
rho <- agents * lambda / (k * mu)
cat("Fan-out per reviewer:", fan_out, "agents\n")
#> Fan-out per reviewer: 15 agents
data.frame(
agents,
rho = round(rho, 2),
stable = ifelse(rho < 1, "yes", "NO"),
headroom = ifelse(rho <= 0.7, "buffered", "at risk")
)
#> agents rho stable headroom
#> 1 10 0.33 yes buffered
#> 2 15 0.50 yes buffered
#> 3 20 0.67 yes buffered
#> 4 25 0.83 yes at risk
#> 5 30 1.00 NO at riskThe table makes the design levers concrete. A desk of two reviewers is stable up to thirty agents only in the knife-edge sense; it is robust to roughly twenty, and the way to supervise more is not to hire linearly but to raise \(T_N\) (better agents, narrower autonomy, batched or escalated alerts so that only \(m\) concurrent anomalies interrupt) or to cut \(T_I\) (pre-filled context, one-click rollback, interfaces that do not impose extraneous cognitive load).
Getting this wrong produces the two failure modes that bracket the human–AI literature. Overload the desk and review degrades to rubber-stamping—automation bias, the tendency to accept algorithmic recommendations without the scrutiny that justified deploying a human reviewer at all. Under-deliver on reliability and the opposite appears: algorithm aversion, the documented tendency to abandon a statistically superior algorithm after seeing it err once, even when the human alternative errs more often (Dietvorst, Simmons, and Massey 2015). Both are miscalibrations of trust rather than preferences about automation, which is why the remedies are operational: calibrated exposure through parallel human-and-model pilots, explanations that expose the drivers of a recommendation rather than post hoc rationalizations, and interpretable models where the stakes make opacity untenable. The design objective is that reliance should track the model’s actual conditional accuracy—high where it is reliable, low where it is not—and neither uniform deference nor uniform suspicion achieves that.
67.11.3 Supervision assumes an independent supervisor
Equation 67.15 counts reviewers. It says nothing about whether a reviewer who is looking is still exercising independent judgment, and that assumption turns out to be the fragile one.
Lakhiwal et al. (2026) study algorithmically evaluated asynchronous video interviews—a hiring process in which an AI scores recorded candidate responses—and manipulate how transparent the system is about what the algorithm does. The two sides of the interaction move in opposite directions. For the people being evaluated, transparency helped: it reduced stress and reduced impression management, and interview performance improved. For the human evaluators, transparency did something else. When AI-generated scores were visible, evaluators anchored on them and discounted their own judgment—even though the scores could not, in fact, discriminate honest from deceptive candidates.
That pairing is the finding worth carrying. Transparency is normally argued for as a remedy for automation’s opacity, and on the subject side it behaves like one. On the supervisor side, showing the score is not neutral disclosure: it is an anchor, and it substitutes for the judgment the supervisor was retained to supply. A review desk staffed to a comfortable \(\rho\) can therefore still be a rubber stamp, and the audit that would detect it has to compare reviewer decisions against ground truth rather than against agreement rates with the model—agreement is exactly what anchoring produces. Where the human is the control on the model, the interface should surface the evidence rather than the verdict, and the model’s output should arrive after the human’s, not before it.
67.11.4 The flywheel makes today’s policy tomorrow’s training data
One feedback loop deserves separate statement because it changes the objective function. Adoption of a model generates data, the data improve the model, and the improvement drives further adoption—the AI flywheel whose contracting and pricing implications Gurkan and Véricourt (2022) formalize. For marketing the implication is sharper than for most settings, because the firm’s action is the sampling design: a targeting policy determines who is exposed and therefore whose responses are observed, and a pricing policy determines which price points are ever sampled. A myopically revenue-optimal policy is therefore generically suboptimal as a data-collection policy, and the gap is the exploration term that bandit and adaptive-experimentation methods (Chapter 20) exist to price. This is the same feedback that Section 67.10.2 flags as a source of drift, seen from the design side rather than the diagnostic side, and it supplies the affirmative case for the randomized holdout: the holdout is not a tax on revenue but the mechanism that keeps the training distribution from being fully determined by the model’s own past decisions.
The synthesis Dai and Swaminathan (2026) offer is worth stating plainly, because it cuts against the way AI initiatives are usually justified internally. Stalled pilots and degrading models are typically diagnosed as model-quality problems and answered with better models; the operations reading is that they are usually symptoms of brittle pipelines, unclear ownership, weak monitoring, and unsupervisable autonomy. The complementarity runs both ways—AI expands the set of feasible marketing decisions, and operations discipline determines which of those decisions survive contact with deployment.
67.12 From Prediction to Decision: Causal Machine Learning
The chapter’s organizing distinction has a constructive resolution. The right way to use machine learning’s flexibility for inference is not to read coefficients off a predictive model but to embed flexible learners inside an estimator whose identification comes from design. Double/debiased machine learning uses nonparametric learners to flexibly absorb high-dimensional confounders while preserving valid inference on a low-dimensional causal parameter, via orthogonalization and cross-fitting that immunize the target estimate against the nuisance learner’s regularization bias. Causal forests and related heterogeneous- treatment-effect estimators adapt tree ensembles to estimate how a treatment effect \(\tau(\mathbf{x}) = \mathbb{E}[Y(1) - Y(0)\mid \mathbf{X}=\mathbf{x}]\) varies across customers—the uplift that retention and targeting decisions actually require, as distinct from the level a churn model predicts. These methods, developed in the causal-inference chapters, are the principled bridge from machine learning’s predictive power to the causal questions marketing decisions pose, and they are where the field’s frontier is moving (Varian 2016). Two developments define that frontier. Double/debiased machine learning uses flexible first-stage predictions with cross-fitting and an orthogonalized moment so that regularization bias in the nuisance functions does not contaminate the treatment-effect estimate (Chernozhukov et al. 2018). Causal forests adapt the same trees to estimate heterogeneous effects with valid confidence intervals rather than point predictions (Wager and Athey 2018; Athey and Imbens 2016).
The generative turn adds a version of this problem that the causal-ML toolkit was not built for: the treatment itself is now cheap to manufacture. When a model can write two thousand candidate emails, the constraint moves from creation to selection, and selection requires predicting the effect of copy that has never been deployed. Ellickson et al. (2026) build the response surface for exactly this—embedding past content with a pretrained LLM, estimating how its features causally relate to outcomes in the spirit of the component-effect estimates of Ellickson, Kar, and Reeder (2023), and then screening AI-generated candidates by rejection sampling so that the model refuses to score content lying outside the support of what the firm has actually run. In a 3.3-million-observation email application spanning 34 campaigns, the framework improved both held-out prediction and deployed performance. The design principle generalizes past text: any decision system that scores novel machine-generated artifacts needs an explicit in-support / out-of-support boundary, because the alternative is a confident extrapolation with no error bar attached to its confidence. Section 45.7 develops the framework with a runnable illustration of the screen.
67.13 What Firm-Level Adoption Data Say About AI and Jobs
The most-asked question about this technology—does adopting it cost jobs?—has been hard to answer because adoption itself was unmeasured. The dominant empirical strategy assigns firms or workers an exposure score from occupational task content (Brynjolfsson, Li, and Raymond 2025), which varies across jobs but cannot distinguish a firm that adopted AI from an identical firm that did not.
Kharazian, Simon, and Stevens (2026) attack the measurement problem directly by linking observed corporate card and bill-pay records identifying payments to AI vendors with monthly workforce records for 21,559 US firms. Adoption is dated by first observed AI payment, and intensity is measured by AI spend per employee per month; the preferred design compares adopters against firms in the same intensity tercile that have not yet adopted, using the Callaway–Sant’Anna doubly robust estimator with sector fixed effects (Callaway and Sant’Anna 2021; Sant’Anna and Zhao 2020). Over the 24 months after adoption, low-intensity adopters show no detectable headcount change (\(-0.006\) log points), while high-intensity adopters grow total headcount by 0.097 log points (about 10.2%) and entry-level headcount by 0.113 (about 12.0%), with gains spread across sales, administration, engineering, and customer service and concentrated, so far, in the Information sector (about 13.4%). The event-study path is a learning curve rather than a jump: essentially nothing in the adoption month, 0.071 by month 6, 0.188 by month 12, 0.277 by month 18.
Three things make this worth reading closely, and one makes it worth reading carefully. It replaces a proxy with an observed behavior, which is the highest-leverage move available in most measurement-constrained literatures. It puts the identification burden where it belongs, on when firms adopt rather than on whether they could have, and it reports pre-period diagnostic flags alongside every estimate rather than only for the headline. And it declines the tempting comparison: adopters versus never-adopters yields larger effects (12.6% for high intensity) but fails 8 of 11 pre-period checks, and the authors report it as a completeness exercise rather than as the result. The caution is that this is industry research, not peer-reviewed work, on a sample of firms that use one payments platform and are visible to one workforce-data vendor—selected toward younger, technology-intensive companies—and that the design identifies the effect of adoption timing among eventual adopters, not the effect of AI on the labor market as a whole. Read against Section 15.8, the two studies make the same methodological point from opposite substantive corners: who took an action, and when, is not a nuisance to be differenced away but the variable that determines what the action was worth.
67.14 Key Takeaways
- The prediction–inference distinction (Equation 67.1) is the organizing idea: predictive accuracy and causal interpretation are different goals validated by different criteria, and a model’s internals are not causal estimates no matter how well it predicts.
- Supervised learning minimizes regularized empirical risk (Equation 67.3); for tabular marketing data, gradient-boosted trees are the empirical default, and honest evaluation requires both discrimination and calibration, on a split that respects time.
- Unsupervised learning discovers structure without labels; clustering always returns clusters, so the burden of proof is on the analyst to show segments are stable and managerially meaningful, not merely that the algorithm converged.
- Recommender systems turn sparse user–item data into rankings via matrix factorization (Equation 67.6), but cold-start, feedback loops, and the prediction-versus-incremental-lift gap make naïve click-maximization a trap.
- Deep learning and LLMs excel on unstructured data (images, text, sequences) and are powerful measurement and generation instruments, but their opacity makes them poor instruments of inference and demands explicit validity, reproducibility, and factuality controls.
- The three deployment pitfalls—leakage, drift, and unfairness—are assumption violations, not bugs: validate inside the fold and forward in time, monitor and retrain against a moving world, and measure fairness on outcomes because proxies encode protected attributes even when those attributes are excluded.
- Deployment is an operations problem, not a modeling one: the model pipeline is a production line whose refresh cadence obeys a lot-sizing trade-off (Equation 67.14), whose monitoring is statistical process control, and whose human reviewers have a finite fan-out (Equation 67.15) that bounds how much autonomy can safely be deployed (Dai and Swaminathan 2026).
- The constructive resolution is causal machine learning, which embeds flexible learners inside design-based estimators to recover the uplift that marketing decisions require.
Appel, Gil, Lauren Grewal, Rhonda Hadi, and Andrew T Stephen. 2020. “The Future of Social Media in Marketing.” Journal of the Academy of Marketing Science 48 (1): 79–95.
Argyle, Lisa P., Ethan C. Busby, Nancy Fulda, Joshua R. Gubler, Christopher Rytting, and David Wingate. 2023. “Out of One, Many: Using Language Models to Simulate Human Samples.” Political Analysis 31 (3): 337–51. https://doi.org/10.1017/pan.2023.2.
Ascarza, Eva. 2018. “Retention Futility: Targeting High-Risk Customers Might Be Ineffective.” Journal of Marketing Research 55 (1): 80–98. https://doi.org/10.1509/jmr.16.0163.
Athey, Susan, and Guido Imbens. 2016. “Recursive Partitioning for Heterogeneous Causal Effects.” Proceedings of the National Academy of Sciences 113 (27): 7353–60. https://doi.org/10.1073/pnas.1510489113.
Bisbee, James, Joshua D. Clinton, Cassy Dorff, Brenton Kenkel, and Jennifer M. Larson. 2024. “Synthetic Replacements for Human Survey Data? The Perils of Large Language Models.” Political Analysis 32 (4): 401–16. https://doi.org/10.1017/pan.2024.5.
Breiman, Leo. 2001. “Random Forests.” Machine Learning 45 (1): 5–32. https://doi.org/10.1023/a:1010933404324.
Brynjolfsson, Erik, Danielle Li, and Lindsey Raymond. 2025. “Generative AI at Work.” The Quarterly Journal of Economics 140 (2): 889–942. https://doi.org/10.1093/qje/qjae044.
Büschken, Joachim, and Greg M Allenby. 2016. “Sentence-Based Text Analysis for Customer Reviews.” Marketing Science 35 (6): 953–75.
Callaway, Brantly, and Pedro H. C. Sant’Anna. 2021. “Difference-in-Differences with Multiple Time Periods.” Journal of Econometrics 225 (2): 200–230. https://doi.org/10.1016/j.jeconom.2020.12.001.
Chernozhukov, Victor, Denis Chetverikov, Mert Demirer, Esther Duflo, Christian Hansen, Whitney Newey, and James Robins. 2018. “Double/Debiased Machine Learning for Treatment and Structural Parameters.” The Econometrics Journal 21 (1): C1–68. https://doi.org/10.1111/ectj.12097.
Dai, Tinglong, and Jayashankar M. Swaminathan. 2026. “Artificial Intelligence and Operations: A Foundational Framework of Emerging Research and Practice.” Production and Operations Management 35 (9): 3255–71. https://doi.org/10.1177/10591478251412943.
Dietvorst, Berkeley J., Joseph P. Simmons, and Cade Massey. 2015. “Algorithm Aversion: People Erroneously Avoid Algorithms After Seeing Them Err.” Journal of Experimental Psychology: General 144 (1): 114–26. https://doi.org/10.1037/xge0000033.
Ellickson, Paul B., Wreetabrata Kar, and James C. Reeder. 2023. “Estimating Marketing Component Effects: Double Machine Learning from Targeted Digital Promotions.” Marketing Science 42 (4): 704–28. https://doi.org/10.1287/mksc.2022.1401.
Ellickson, Paul B., Wreetabrata Kar, James C. Reeder, and Guang Zeng. 2026. “Evaluating Novel Unstructured Treatments with Generative AI: A Causal Prediction Framework.” Journal of Marketing Research. https://doi.org/10.1177/00222437261476639.
Friedman, Jerome H. 2001. “Greedy Function Approximation: A Gradient Boosting Machine.” The Annals of Statistics 29 (5). https://doi.org/10.1214/aos/1013203451.
Gao, Janet, Wenyu Wang, and Xiaoyun Yu. 2024. “Big Fish in Small Ponds: Human Capital Migration and the Rise of Boutique Banks.” Management Science.
Goli, Ali, and Amandeep Singh. 2024. “Frontiers: Can Large Language Models Capture Human Preferences?” Marketing Science 43 (4): 709–22. https://doi.org/10.1287/mksc.2023.0306.
Gurkan, Huseyin, and Francis de Véricourt. 2022. “Contracting, Pricing, and Data Collection Under the AI Flywheel Effect.” Management Science 68 (12): 8791–8808. https://doi.org/10.1287/mnsc.2022.4333.
Huang, Hongxian, Runshan Fu, and Anindya Ghose. 2026. “Generative AI, Platform Stances, and Content Creator Behavior.” Information Systems Research. https://doi.org/10.1287/isre.2025.2155.
Jiang, Shenyang, Akshat Lakhiwal, Che-Wei Liu, and Jiang Duan. 2026. “Seeing Less, Engaging More: Rethinking Early User Experience on GenAI Co-Creation Platforms—Findings from a Field Experiment.” Information Systems Research. https://doi.org/10.1287/isre.2024.1200.
Jussupow, Ekaterina, Kevin Bauer, Rebecca Maria Heigl, Benjamin Vogt, and Oliver Hinz. 2026. “The Indirect Disclosure Effect: How Disclosing Generative AI Use Impacts Human Creative Collaboration with AI.” Information Systems Research. https://doi.org/10.1287/isre.2024.0951.
Kaiser, Maximilian, and Christian Schulze. 2026. “Frontiers: ChatGPT Referrals to e-Commerce Websites: How Do LLMs Compare Against Traditional Channels?” Marketing Science 45 (4): 699–715. https://doi.org/10.1287/mksc.2025.0489.
Kharazian, Ara, Lisa Simon, and Ryan Stevens. 2026. “A New Look at AI’s Impact on Jobs: Firm-Level AI Spending and Workforce Adjustment.” Ramp Economics Lab; Revelio Labs. https://ramp.com/data/ai-jobs-impact.
Kleinberg, Jon, Himabindu Lakkaraju, Jure Leskovec, Jens Ludwig, and Sendhil Mullainathan. 2017. “Human Decisions and Machine Predictions.” The Quarterly Journal of Economics 133 (1): 237–93. https://doi.org/10.1093/qje/qjx032.
Koren, Yehuda, Robert Bell, and Chris Volinsky. 2009. “Matrix Factorization Techniques for Recommender Systems.” Computer 42 (8): 30–37. https://doi.org/10.1109/mc.2009.263.
Kumar, Madhav, Dean Eckles, and Sinan Aral. 2026. “Scalable Bundle Recommendations: A Large-Scale Field Experiment.” Management Science. https://doi.org/10.1287/mnsc.2022.01864.
Lakhiwal, Akshat, Che-Wei Liu, Hillol Bala, and Hung-Yue Suen. 2026. “From Opacity to Transparency: User Behavior and Downstream Effects in Algorithmic Evaluation.” Information Systems Research. https://doi.org/10.1287/isre.2023.0579.
LeCun, Yann, Yoshua Bengio, and Geoffrey Hinton. 2015. “Deep Learning.” Nature 521 (7553): 436–44. https://doi.org/10.1038/nature14539.
Li, Yiyi, and Ying Xie. 2019. “Is a Picture Worth a Thousand Words? An Empirical Study of Image Content and Social Media Engagement.” Journal of Marketing Research 57 (1): 1–19. https://doi.org/10.1177/0022243719881113.
Liu, Liu, Daria Dzyabura, and Natalie Mizik. 2020. “Visual Listening In: Extracting Brand Image Portrayed on Social Media.” Marketing Science 39 (4): 669–86. https://doi.org/10.1287/mksc.2020.1226.
Martin, Kelly D, Abhishek Borah, and Robert W Palmatier. 2017. “Data Privacy: Effects on Customer and Firm Performance.” Journal of Marketing 81 (1): 36–58.
McCullagh, Peter, and John A. Nelder. 1989. Generalized Linear Models. 2nd ed. London: Chapman & Hall/CRC.
Netzer, Oded, Olivier Toubia, Eric T. Bradlow, Ely Dahan, Theodoros Evgeniou, Fred M. Feinberg, Eleanor M. Feit, et al. 2008. “Beyond Conjoint Analysis: Advances in Preference Measurement.” Marketing Letters 19 (3-4): 337–54. https://doi.org/10.1007/s11002-008-9046-1.
Neumann, Nico, Catherine E Tucker, and Timothy Whitfield. 2019. “Frontiers: How Effective Is Third-Party Consumer Profiling? Evidence from Field Studies.” Marketing Science 38 (6): 918–26.
Pattabhiramaiah, Adithya, Martin Mende, Huachao Gao, and Shrihari Sridhar. 2026. “Agentic AI and the New Architecture of Marketing Relationships.” Journal of Marketing. https://doi.org/10.1177/00222429261481116.
Sant’Anna, Pedro H. C., and Jun Zhao. 2020. “Doubly Robust Difference-in-Differences Estimators.” Journal of Econometrics 219 (1): 101–22. https://doi.org/10.1016/j.jeconom.2020.06.003.
Sikdar, Sharmistha, Ishita Chakraborty, and Nika Dogonadze. 2026. “Neither a Picasso nor a Leonardo da Vinci: An Examination of Novice Artwork Pricing with Multimodal Data.” Journal of Marketing 90 (5): 125–48. https://doi.org/10.1177/00222429251408346.
Smith, Evelyn O., Jeffrey D. Shulman, and Antino Kim. 2026. “My Fair AI: Accuracy Parity and Strategic Identity Disclosure in AI Content Recommendation.” Marketing Science. https://doi.org/10.1287/mksc.2024.0900.
Sozuer, Sibel, Oded Netzer, and Kriste Krstovski. 2026. “A Recipe for Creativity: An Ingredient-Embedding Approach.” Journal of Marketing 90 (5): 35–54. https://doi.org/10.1177/00222429261464104.
Tirunillai, Seshadri, and Gerard J. Tellis. 2014. “Mining Marketing Meaning from Online Chatter: Strategic Brand Analysis of Big Data Using Latent Dirichlet Allocation.” Journal of Marketing Research 51 (4): 463–79. https://doi.org/10.1509/jmr.12.0106.
Varian, Hal R. 2016. “How to Build an Economic Model in Your Spare Time.” The American Economist 61 (1): 81–90. https://doi.org/10.1177/0569434515627089.
Vaswani, Ashish, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. “Attention Is All You Need.” In Advances in Neural Information Processing Systems 30 (NIPS 2017), 5998–6008. https://doi.org/10.48550/arXiv.1706.03762.
Wager, Stefan, and Susan Athey. 2018. “Estimation and Inference of Heterogeneous Treatment Effects Using Random Forests.” Journal of the American Statistical Association 113 (523): 1228–42. https://doi.org/10.1080/01621459.2017.1319839.
Wang, Mengxin, Dennis J. Zhang, and Heng Zhang. 2026. “Large Language Models for Market Research: A Data-Augmentation Approach.” Marketing Science 45 (4): 728–51. https://doi.org/10.1287/mksc.2025.0009.
Wedderburn, R. W. M. 1974. “Quasi-Likelihood Functions, Generalized Linear Models, and the Gauss–Newton Method.” Biometrika 61 (3): 439–47. https://doi.org/10.1093/biomet/61.3.439.
Wedel, Michel, and P.K. Kannan. 2016. “Marketing Analytics for Data-Rich Environments.” Journal of Marketing 80 (6): 97–121. https://doi.org/10.1509/jm.15.0413.
Zheng, Shuang, Siliang Tong, Hyeokkoo Eric Kwon, Gordon Burtch, and Xianneng Li. 2023. “Recommending What to Search: Sales Volume and Consumption Diversity Effects of a Query Recommender System.” Available at SSRN 4667778.