Empirical marketing research is only as credible as the data behind it, and the data a researcher can obtain increasingly determine which questions are even askable. This chapter is a working guide to acquiring the raw material of marketing science—firm financials, advertising and engagement metrics, household expenditure, and the digital traces of consumer attention—and, just as importantly, to understanding what each source actually measures. Acquisition is never neutral: every dataset embeds a sampling frame, a measurement model, and a set of selection mechanisms that propagate into every downstream estimate. A panel that surveys 6,000 households quarterly, an index that rescales search volume to its own maximum, and an API that silently throttles after 1,400 calls each impose structure on what the analyst can learn.

Controversy: the spreadsheet that moved fiscal policy

Before the acquisition mechanics, one cautionary case about what happens between acquiring data and reporting a result. It comes from economics, it is the most consequential arithmetic error in the modern social sciences, and it was found by a graduate student doing a homework assignment.

2010 — the finding. Reinhart and Rogoff (2010) report that countries with public debt above 90 percent of GDP experience markedly lower average growth. The 90 percent threshold enters political debate almost immediately and is cited in support of austerity programmes in several countries.

2013 — the replication. Herndon, Ash, and Pollin (2013) obtain the original spreadsheet and attempt to reproduce the result. They report three separate problems: selective exclusion of country-year observations, an unconventional weighting of country averages, and a spreadsheet formula error that omitted several countries from the average entirely. Correcting them, the sharp discontinuity at 90 percent does not survive; a weaker negative association remains.

What this case is actually about. Not fraud, and not sophisticated econometrics. The error was in a cell range. It persisted for three years in one of the most heavily cited empirical results of its decade, through peer review and through enormous public scrutiny, because nobody else had the file. It was found the first time someone did.

That is the argument for the practices this chapter and Section 76.7 insist on, stated as a consequence rather than a principle: version-controlled acquisition scripts, a raw layer that is never edited in place, transformations expressed as code rather than as manual steps, and the whole chain posted with the paper. A pipeline built that way makes this class of error findable by you before publication and by others afterwards. A spreadsheet does not.

We organize the material by source, but the through-line is measurement. For each source we state, as precisely as the source allows, the estimand—the population quantity of interest—and the observable the source actually delivers, because the gap between the two is where most empirical mistakes live. Search interest is not search volume; a relative index is not an absolute count; a name-based demographic prediction is a posterior probability, not a fact. Treating these distinctions casually is how plausible pipelines produce biased coefficients.

The chapter is also deliberately reproducible. Connection recipes, query patterns, and rescaling algorithms appear as runnable code so that a reader can move from “this dataset exists” to “I have it in a tidy frame” without reverse engineering undocumented APIs. Where a third-party platform’s own documentation is the authoritative reference—rate limits, OAuth scopes, schema—we point to the mechanism rather than reproduce ephemeral details. Throughout, the goal is the one stated in the style guide: intuition first, then the formal object, then code that runs.

Figure 32.1 situates the sources covered here along the two dimensions that matter most for research design: the unit of observation (firm, household, or individual consumer) and whether the data are behavioral (revealed in actions) or survey-elicited (reported by respondents).

flowchart TB
  subgraph FIRM["Firm-level"]
    WRDS["WRDS / Compustat<br/>financials, R&D, advertising"]
  end
  subgraph HH["Household-level"]
    CE["Consumer Expenditure Survey<br/>elicited expenditure (panel + diary)"]
  end
  subgraph IND["Individual / keyword-level"]
    GT["Google Trends<br/>relative search interest"]
    BD["Baidu Index<br/>search volume"]
    YT["YouTube<br/>views, comments, engagement"]
    NAME["Name-based inference<br/>gender / age / nationality"]
  end
  WRDS -->|"revealed (accounting)"| OUT["Research design:<br/>estimand vs. observable"]
  CE -->|"elicited (survey)"| OUT
  GT -->|"revealed (behavior)"| OUT
  BD -->|"revealed (behavior)"| OUT
  YT -->|"revealed (behavior)"| OUT
  NAME -->|"inferred (model)"| OUT
Figure 32.1: The data sources in this chapter, organized by unit of observation and by whether they capture revealed behavior or elicited reports. Behavioral traces are abundant and timely but noisy proxies for latent constructs; survey instruments measure constructs directly but are costly, lagged, and subject to reporting bias.

32.1 Firm Financial Data: WRDS

Most marketing-finance research (Chapter 25) and a large share of strategy work rest on standardized firm financials. Wharton Research Data Services (WRDS) is the dominant academic gateway to these: it federates Compustat (accounting fundamentals), CRSP (security prices and returns), I/B/E/S (analyst forecasts), and dozens of other libraries behind a single SQL interface. For the marketing researcher the attraction is concrete. Advertising expense (xad) and research-and-development expense (xrd) in Compustat’s fundamentals-annual file (funda) are the canonical proxies for a firm’s marketing and innovation intensity, and they join cleanly to market-based outcomes such as Tobin’s \(q\) or abnormal returns used throughout Chapter 25.

The connection is a standard PostgreSQL session; WRDS exposes its warehouse over the wire, so the same DBI idioms used for any relational database apply. Credentials should never be hard-coded—read them from the environment so that the source file is safe to share.

Code
library(RPostgres)
library(tidyverse)

wrds <- dbConnect(
  Postgres(),
  host    = "wrds-pgdata.wharton.upenn.edu",
  port    = 9737,
  dbname  = "wrds",
  sslmode = "require",
  user    = Sys.getenv("wrds_user"),
  pass    = Sys.getenv("wrds_pass")
)

The warehouse is self-describing through the information_schema, which is the disciplined way to discover what is available rather than guessing table names. Three nested queries answer the three questions a new user has: which libraries (schemas) exist, which tables live in a chosen library, and which columns a chosen table holds.

Code
# Libraries (schemas) available to your subscription
dbGetQuery(wrds, "
  select distinct table_schema
  from information_schema.tables
  order by table_schema")

# Tables inside one library
dbGetQuery(wrds, "
  select distinct table_name
  from information_schema.columns
  where table_schema = 'comp_na_daily_all'
  order by table_name")

# Columns inside one table
dbGetQuery(wrds, "
  select column_name
  from information_schema.columns
  where table_schema = 'comp_na_daily_all'
    and table_name   = 'funda'
  order by column_name")

A representative extraction pulls advertising and R&D for a decade, then enriches each firm-year with its industry classification by joining on the firm identifier gvkey. Pushing the filters (fyear, non-null xad/xrd) into the WHERE clause rather than filtering in R keeps the transfer small; this matters when funda runs to millions of rows.

Code
fin <- dbGetQuery(wrds, "
  select gvkey, fyear, xad, xrd
  from comp_na_daily_all.funda
  where fyear between 2000 and 2010
    and xad is not null
    and xrd is not null") |>
  distinct()

industry <- dbGetQuery(wrds, "
  select gvkey, gind, gsubind, naics, sic
  from comp_na_daily_all.names")

panel <- fin |>
  left_join(industry, by = "gvkey")

head(panel)

Two cautions recur in practice and shape identification downstream. First, xad is reported advertising; under accounting rules many firms expense marketing without breaking it out, so missingness is non-random—it correlates with firm size, industry, and disclosure regime. Treating is null as “no advertising” rather than “not separately disclosed” silently selects the sample on a variable correlated with the outcome, the textbook recipe for selection bias. Second, funda ships in multiple formats (consolidated vs. parent-only, standardized vs. restated) keyed by the indfmt, datafmt, popsrc, and consol flags; the conventional screen for North American industrial firms is indfmt='INDL' and datafmt='STD' and popsrc='D' and consol='C', and omitting it double-counts firm-years. Neither caution is exotic, but both are invisible until an estimate looks wrong.

Note

Beyond WRDS, the Consumer Expenditure Survey (below) and commercial scanner panels (e.g., IRI, Nielsen) cover demand-side behavior that firm financials cannot. The general principle—match the unit of observation to the estimand—is developed in Chapter 38.

32.2 Local Document Stores: MongoDB

Not all marketing data are relational. Social-media posts, scraped reviews, and API responses arrive as nested, schema-flexible JSON, and a document database such as MongoDB stores them without forcing a rectangular schema up front. When ingesting at scale the first operational question is simply where the data physically live, which the running server reports through an administrative command issued in the mongo shell:

db.adminCommand("getCmdLineOpts")

The returned document includes the storage.dbPath, the on-disk location of the data files—useful for capacity planning, backups, and confirming that a large ingest is writing where you expect. The broader point is architectural: document stores trade the relational guarantees of WRDS-style sources (a fixed schema, referential integrity, declarative joins) for flexibility on write. That trade is right for heterogeneous, evolving web data and wrong for the tidy panels demanded by most econometric estimators, which is why a common pipeline lands raw captures in MongoDB and projects a flattened, analysis-ready frame into R or a relational store before modeling.

32.3 Search Interest: Google Trends

Aggregate search behavior is among the most useful behavioral signals in marketing because it is timely, granular, and a leading indicator of attention. The motivating regularity is that public interest and its institutional reflections move together but on different clocks: online search volume and news reporting are strongly correlated, whereas academic publishing lags public interest because of the delay inherent in peer review and publication (Nghiem et al. 2016). Read structurally, news media act as a conductor between the research community and the public, and online search is the most immediate barometer of the public’s side of that exchange.

32.3.2 Reconstructing a Long Daily Series

The granularity rule in Table 32.1 creates a real problem: researchers often need daily resolution over multi-year spans, but a multi-year request returns only weekly (or monthly) data. The fix exploits the co-normalization property of Equation 32.1. Two reconstruction strategies exist, and they differ in how they reconcile the high-frequency and low-frequency series.

The scaling (overlapping) method is the more reliable of the two. It queries short windows at daily resolution, queries the full span at lower resolution to obtain period weights, and rescales the daily pieces so that each month’s daily indices are multiplied by that month’s relative weight. Formally, let \(d_{k,t}\) be the daily index obtained from a short within-window query covering month \(m(t)\), and let \(w_{k,m}\) be the month-\(m\) weight derived from the long-window query (the month’s aggregate index divided by the maximum monthly aggregate). The reconstructed daily series is the product

\[ \hat{v}_{k,t} \;=\; d_{k,t} \times w_{k,m(t)}, \tag{32.2}\]

which restores a common scale across months while preserving within-month daily shape. The concatenation (normalization) method instead stitches one-month daily queries end to end and normalizes them against the weekly series; empirical comparisons favor the scaling method, which is why Equation 32.2 is the recommended estimator.1 This reconstruction underlies published applications that require daily search data over long horizons (Risteski and Davcev 2014), and related approaches to extracting daily volume are documented elsewhere (Y. Liu et al. 2019).

The function below implements Equation 32.2 end to end: it pulls a single aggregated query to derive monthly multipliers mult, loops over months to pull daily data, and joins the two to produce reconstructed daily estimates est_hits. Note the rate-limit budget of roughly 1,400 requests per 4 hours; a long reconstruction must respect it or the loop will start returning empty frames.

Code
library(gtrendsR)
library(tidyverse)
library(lubridate)

get_daily_gtrend <- function(keyword = c("7eleven", "3M"),
                             geo  = "US",
                             from = "2013-01-01",
                             to   = "2013-02-15") {
  # Disallow a 'to' date in the current (incomplete) month
  if (ymd(to) >= floor_date(Sys.Date(), "month")) {
    to <- floor_date(ymd(to), "month") - days(1)
    if (to < from) stop("`to` date in the current month is not allowed")
  }

  # Long query -> monthly multipliers w_{k,m}
  aggregated <- gtrends(keyword = keyword, geo = geo, time = paste(from, to))
  if (is.null(aggregated$interest_over_time)) {
    message("No data in Google Trends for this query.")
    return(invisible(NULL))
  }

  mult_m <- aggregated$interest_over_time |>
    mutate(hits = as.integer(ifelse(hits == "<1", "0", hits))) |>
    group_by(month = floor_date(date, "month"), keyword) |>
    summarise(hits = sum(hits), .groups = "drop") |>
    mutate(ym = format(month, "%Y-%m"), mult = hits / max(hits)) |>
    select(month, ym, keyword, mult)

  # Per-month daily queries -> d_{k,t}
  windows <- tibble(
    s = seq(ymd(from), ymd(to), by = "month"),
    e = seq(ymd(from), ymd(to), by = "month") + months(1) - days(1)
  )

  raw_daily <- map2_dfr(windows$s, windows$e, function(s, e) {
    curr <- gtrends(keyword, geo = geo, time = paste(s, e))
    if (is.null(curr$interest_over_time)) return(tibble())
    curr$interest_over_time
  })

  trend_d <- raw_daily |>
    transmute(date, keyword,
              ym   = format(date, "%Y-%m"),
              hits = as.integer(ifelse(hits == "<1", "0", hits)))

  # Reconstruct: est_hits = d_{k,t} * w_{k,m(t)}   (eq-gtrends-scaling)
  trend_d |>
    left_join(mult_m, by = c("ym", "keyword")) |>
    mutate(est_hits = hits * mult, date = as.Date(date)) |>
    select(date, keyword, est_hits)
}

daily_trend <- get_daily_gtrend(
  keyword = c("7eleven", "3M"), geo = "US",
  from = "2013-01-01", to = "2013-02-01")
head(daily_trend)

32.3.3 Absolute Volume

When the application genuinely requires absolute counts rather than the relative index of Equation 32.1—for example, to size a market rather than track its trajectory—the relative-interest endpoint is the wrong source. Google publishes a separate top-terms dataset through BigQuery that exposes absolute search figures, and that warehouse, not the Trends index, is the appropriate input for level estimates.

32.4 Search Volume in China: Baidu Index

Outside the markets Google serves, the analogous behavioral signal comes from the dominant local search engine. In China that is Baidu, and the Baidu Index plays the role Google Trends plays elsewhere. Because Baidu exposes no comparable public API for bulk extraction, researchers have built crawlers to recover its search-volume series; P. J. Liu et al. (2019) document a Java-based spider for exactly this purpose. The same measurement caveats apply—an index is a rescaled, sampled proxy for latent interest, not a count—so the estimand-versus-observable discipline of Section 32.3 carries over directly.

32.5 Video Engagement: YouTube

Video is now a primary advertising and engagement channel, and YouTube exposes rich behavioral data—views, likes, comments, and watch metrics—at the video, playlist, and channel level. Two access paths exist, and the choice between them is a governance decision rather than a convenience one.

The first path is the public Data API, which serves publicly visible statistics for any video or channel using a simple API key. The second is the Analytics and Reporting APIs, which serve owner-private metrics (impressions, audience retention, revenue) and therefore require OAuth authorization by the channel or content owner that owns the requested data. Table 32.2 contrasts the two; the operative rule is that anything a viewer cannot see requires the owner’s delegated consent.

Table 32.2: Two paths to YouTube data. The boundary is consent: public statistics need only a key, whereas owner-private analytics require OAuth authorization by the data owner.
Dimension Data API (key) Analytics / Reporting (OAuth)
Auth API key OAuth 2.0, owner-delegated
Scope Public statistics Owner-private metrics
Typical fields views, likes, comments impressions, retention, revenue
Who can call anyone with a key the channel/content owner
Use case competitive/market scans first-party performance analytics

32.5.1 OAuth Path via tuber

The tuber package wraps the authenticated endpoints. Authorization is a one-time OAuth handshake with credentials provisioned in the Google Cloud console; thereafter the session token authorizes calls for video statistics, details, captions, search, and comment threads.

Code
library(tuber)

# Provision app_id / app_secret in the Google Cloud console, then:
yt_oauth(app_id = "YOUR_APP_ID", app_secret = "YOUR_APP_SECRET")

get_stats(video_id = "N708P-A45D0")          # public statistics
get_video_details(video_id = "N708P-A45D0")  # snippet metadata
get_captions(video_id = "yJXTXN4xrI8")       # caption track
yt_search(term = "test")                      # search the corpus
get_comment_threads(c(video_id = "N708P-A45D0"))  # top-level comments
get_all_comments(video_id = "a-UQz7fqR3w")    # comments + replies

A common research task is to assemble a panel of every video in a channel. YouTube models a channel’s uploads as a hidden playlist, so the recipe is: read the channel’s relatedPlaylists$uploads id, page through that playlist’s items to collect video ids, then map the statistics call over the ids and bind the results into a frame.

Code
chan <- list_channel_resources(
  filter = c(channel_id = "UCT5Cx1l4IS3wHkJXNyuj4TA"),
  part   = "contentDetails")

uploads_id <- chan$items[[1]]$contentDetails$relatedPlaylists$uploads
vids       <- get_playlist_items(filter = c(playlist_id = uploads_id))
vid_ids    <- as.vector(vids$contentDetails.videoId)

stats_df <- do.call(rbind, lapply(vid_ids, function(id) data.frame(get_stats(id))))
head(stats_df)

32.5.2 Key Path via the Data API

When only public statistics are needed, the raw REST endpoints are lighter than an OAuth session. The pattern is a small set of helper functions that build the request URL, parse the JSON, and return a tidy frame—one helper per resource type (video, playlist, channel).

Code
library(jsonlite)
library(dplyr)

API_key <- Sys.getenv("YOUTUBE_API_KEY")   # never hard-code keys

video_stats <- function(video_id, key) {
  url <- paste0("https://www.googleapis.com/youtube/v3/videos",
                "?part=snippet,statistics&id=", video_id, "&key=", key)
  res <- fromJSON(url)
  data.frame(
    name  = res$items$snippet$channelTitle,
    res$items$statistics,
    title = res$items$snippet$title,
    date  = res$items$snippet$publishedAt
  )
}

channel_stats <- function(channel_id, key) {
  url <- paste0("https://www.googleapis.com/youtube/v3/channels",
                "?part=snippet,contentDetails,statistics&id=", channel_id,
                "&key=", key)
  res <- fromJSON(url)
  data.frame(
    name = res$items$snippet$title,
    res$items$statistics,
    uploads = res$items$contentDetails$relatedPlaylists$uploads
  )
}

These frames feed directly into comparative visualizations—total channel views, or the time path of comments and likes across competing channels—using the ggplot2 idioms already shown for Google Trends. Because the engagement counts (viewCount, commentCount) arrive as character strings, they must be coerced to numeric before plotting.

Warning

A requests-and-BeautifulSoup scraper that pages through search-result HTML can retrieve video ids without an API key, but scraping rendered pages is brittle, violates many terms of service, and breaks whenever the front-end markup changes. Prefer the official endpoints; treat HTML scraping as a last resort for prototyping, never production.

32.6 Household Expenditure: The Consumer Expenditure Survey

When the research question concerns what households buy—budget shares, category demand, the income elasticity of a product class—the canonical U.S. source is the Consumer Expenditure (CE) Survey administered by the Bureau of Labor Statistics. Its sampling design is worth understanding in detail, because the survey’s structure dictates which estimands it can support and at what frequency.

The CE’s unit of observation is the consumer unit, defined as a single person or a group of persons who live together and share responsibility for most major expenditures.

A consumer unit is a single person, or a group of persons who live together and who share responsibilities for most major expenditures.

Within each unit, the reference person is the first individual named when the respondent answers, “Who is responsible for owning or renting this home?”—a convention that fixes the household head deterministically and makes cross-survey linkage well defined.

The survey is not one instrument but two complementary ones, a design choice that follows directly from a measurement trade-off between recall accuracy and coverage. Large, infrequent purchases are remembered well but happen rarely; small, frequent purchases are forgotten quickly but dominate the budget in count. No single instrument measures both well, so the CE splits them:

  • The Interview Survey captures big-ticket and recurring expenditures, plus global estimates for some categories. It is a rotating-panel survey: each consumer unit is interviewed once every three months for four quarters, with roughly 6,000 units in the sample each quarter. The panel structure supports within-unit change over the year, but the four-quarter cap means it cannot follow a household across years.
  • The Diary Survey captures small-ticket, frequently purchased items that recall-based interviewing would miss. It is independent of the interview component and collects roughly 14,000 diaries from participating families each year.

Published tabulations come in several time formats: annual tables run January–December (available since 1984) and, in a fiscal variant, July–June (since 2013); two-year tables span January of year one through December of year two (since 1986). Beyond the standard tables, BLS produces experimental research tabulations—covering, for instance, income detail or generational groups such as millennials and Gen X—available on request. The practical implication for design is that the rotating four-quarter panel is the right frame for within-year, within-household questions, while cross-year dynamics require pooling repeated cross-sections rather than following the same units.

32.7 Inferring Demographics from Names

A recurring need in marketing research is to attach demographic attributes—gender, age, nationality—to records that carry only a name, as when a researcher has a roster of customers, reviewers, or social-media authors but no self-reported demographics. A family of services performs exactly this inference: gender from genderize.io, age from agify.io, and nationality from nationalize.io. Each returns, for a queried name, a predicted attribute together with a confidence.

It is essential to read these outputs as what they are: posterior probabilities from a name-frequency classifier, not ground-truth labels. The underlying logic is a simple Bayesian classifier. Let \(n\) be an observed name and \(g\) a candidate attribute value (say, gender). The services effectively report

\[ \Pr(g \mid n) \;=\; \frac{\Pr(n \mid g)\,\Pr(g)}{\displaystyle\sum_{g'} \Pr(n \mid g')\,\Pr(g')}, \tag{32.3}\]

estimated from large registries of names tagged with the attribute, where \(\Pr(n \mid g)\) is the frequency of name \(n\) among individuals with value \(g\) and \(\Pr(g)\) the base rate of \(g\) in the reference population. Three properties of Equation 32.3 govern responsible use. The prediction inherits the reference population’s composition: a classifier trained predominantly on one region’s registries mispredicts systematically for names common elsewhere, so the base rate \(\Pr(g)\) is not universal. The reported confidence is the posterior itself, so unisex or rare names yield diffuse, low-confidence posteriors that should not be hardened into deterministic labels. And because the inferred attribute is a generated regressor, using it as a covariate in a downstream model imports classification error into that model—the resulting attenuation and the need to propagate the first-stage uncertainty are treated in Chapter 38. Used with those caveats, name-based inference is a cheap way to recover coarse demographic structure; used naively, it manufactures measurement error correlated with name origin.

32.7.1 Scaling the Inference: An Open Firm-Level Diversity Panel

The classifier in Equation 32.3 is usually run on a few thousand names for one study. Wang and Zhang (2026) run it on a population and publish the result. The Organizational Diversity Database covers 466,626 U.S. firms from 2003 to 2023, built by collecting 570 million publicly available LinkedIn profiles—including more than 220 million profile photos—inferring each user’s race and gender from the photo and the name, and aggregating to the firm–occupation–year cell. Each cell reports gender composition, racial composition, and, unusually for a demographic panel, promotion and turnover rates by race and gender. The data are released publicly through a registration portal, with a data dictionary and a methodology document.

For a marketing researcher the appeal is that it supplies a firm-level covariate that was previously available only for the small set of firms filing EEO-1 reports or voluntarily disclosing. Workforce composition can now be joined to the WRDS financials of Section 32.1, to brand and advertising measures, or to the marketing–finance outcomes of Chapter 25, which makes questions such as whether a firm’s diversity claims are matched by its salesforce, or whether composition moves before or after a public commitment, into panel questions rather than case studies.

The cautions are the ones this section has already stated, and aggregation does not retire them. Adding a photo to the name gives the classifier a second signal, but the output is still a posterior, not a label, and a share computed from posteriors is a sum of classification probabilities rather than a headcount. If the classifier’s error rate varies by group—the standard finding for both name-frequency and face-based inference—then the error in a firm’s measured composition is correlated with the composition being measured, which is the one failure mode averaging cannot fix. The promotion and turnover rates are more exposed still, because a rate by race is a ratio of two classified quantities and inherits error from both. A second issue is specific to the source: LinkedIn is a self-selected roster whose coverage varies by occupation, firm size, region, and year, rising steeply across the sample period, so a measured composition is the composition of a firm’s LinkedIn-visible workforce, and a time trend in composition partly tracks a time trend in who joined the platform. Occupation and year fixed effects absorb the common part of that drift; they do not absorb differential coverage growth across groups.

None of this argues against using the database—it argues for using it as Section 32.8 prescribes. Name the estimand (the firm’s actual workforce composition), name the observable (a probabilistic classification of its platform-visible workforce), and carry the gap as a modeling problem: report composition as an interval where the methodology document supplies validation rates, prefer within-firm changes to cross-firm levels, and treat any coefficient on inferred composition as a generated-regressor estimate that needs its first-stage uncertainty propagated (Chapter 38). (The published paper is paywalled; this account rests on its abstract and the public data portal.)

32.8 Pitfalls and Identification

The sources in this chapter differ in unit, frequency, and provenance, but the threats to valid inference share a small number of structural causes. Figure 32.2 maps each common pitfall to the source where it bites hardest and to its remedy.

flowchart LR
  A["Non-random missingness<br/>(unreported xad in Compustat)"] --> A2["Model disclosure;<br/>do not treat NULL as zero"]
  B["Relative, window-dependent index<br/>(Google Trends, Baidu)"] --> B2["Rescale via overlapping<br/>method (eq-gtrends-scaling)"]
  C["Sampling noise in indices<br/>(Trends draws)"] --> C2["Average repeated queries"]
  D["Generated-regressor error<br/>(name-based demographics)"] --> D2["Propagate first-stage<br/>uncertainty"]
  E["Recall vs. coverage trade-off<br/>(CE survey)"] --> E2["Match instrument to estimand<br/>(interview vs. diary)"]
  F["Rate limits / TOS<br/>(APIs, scrapers)"] --> F2["Budget requests; prefer<br/>official endpoints"]
Figure 32.2: Recurring threats to validity across the chapter’s data sources and their remedies. Each threat is a gap between the estimand and the observable that the source actually delivers.

The unifying discipline is the one stated at the outset: name the estimand, name the observable the source delivers, and treat the difference as a modeling problem rather than a nuisance. Reported advertising is not advertising; a relative index is not a volume; a name-based prediction is not a demographic. Each gap is tractable once it is made explicit, and ruinous when it is not.

32.9 Key Takeaways

  • Match the unit of observation to the estimand. WRDS delivers firm-level financials, the CE Survey household expenditure, and search/engagement APIs individual behavioral traces; no source substitutes for another (Figure 32.1).
  • Reported is not measured. Compustat’s xad records disclosed advertising; missingness is non-random and treating NULL as zero selects the sample on a variable correlated with outcomes (Section 32.1).
  • A relative index is not a volume. Google Trends rescales to its window’s maximum (Equation 32.1); long daily series must be reconstructed by the overlapping-scaling method (Equation 32.2), and absolute counts require a different source entirely.
  • Survey design encodes a recall–coverage trade-off. The CE splits big-ticket recall (interview panel) from frequent-purchase coverage (diary), so the right instrument depends on the category (Section 32.6).
  • Name-based demographics are posteriors, not labels. They follow a Bayesian name-frequency classifier (Equation 32.3) whose error is correlated with name origin and must be propagated downstream (Section 32.7).
  • Population-scale inference does not retire the caveat. The Organizational Diversity Database supplies race and gender composition, promotion, and turnover for 466,626 U.S. firms at the firm–occupation–year level, inferred from 570 million LinkedIn profiles and their photos (Wang and Zhang 2026); because classifier error varies by group and LinkedIn coverage varies by occupation and year, the measured quantity is a probabilistic classification of a platform-visible workforce (Section 32.7.1).
  • Respect the platform contract. Owner-private metrics require OAuth, public ones a key (Table 32.2); honor rate limits and prefer official endpoints over brittle scrapers.
Herndon, Thomas, Michael Ash, and Robert Pollin. 2013. “Does High Public Debt Consistently Stifle Economic Growth? A Critique of Reinhart and Rogoff.” Cambridge Journal of Economics 38 (2): 257–79. https://doi.org/10.1093/cje/bet075.
Liu, Peggy J., Kelly L. Haws, Karen Scherr, Joseph P. Redden, James R. Bettman, and Gavan J. Fitzsimons. 2019. “The Primacy of “What” over “How Much”: How Type and Quantity Shape Healthiness Perceptions of Food Portions.” Management Science 65 (7): 3353–81. https://doi.org/10.1287/mnsc.2018.3098.
Liu, Ying, Geng Peng, Lanyi Hu, Jichang Dong, and Qingqing Zhang. 2019. “Using Google Trends and Baidu Index to Analyze the Impacts of Disaster Events on Company Stock Prices.” Industrial Management & Data Systems 120 (2): 350–65. https://doi.org/10.1108/imds-03-2019-0190.
Nghiem, Le T. P., Sarah K. Papworth, Felix K. S. Lim, and Luis R. Carrasco. 2016. “Analysis of the Capacity of Google Trends to Measure Interest in Conservation Topics and the Role of Online News.” Edited by Zhong-Ke Gao. PLOS ONE 11 (3): e0152802. https://doi.org/10.1371/journal.pone.0152802.
Reinhart, Carmen M., and Kenneth S. Rogoff. 2010. “Growth in a Time of Debt.” American Economic Review 100 (2): 573–78. https://doi.org/10.1257/aer.100.2.573.
Risteski, Dimche, and Danco Davcev. 2014. “Can We Use Daily Internet Search Query Data to Improve Predicting Power of EGARCH Models for Financial Time Series Volatility?” International Conference on Computer Science and Information Systems (ICSIS’2014) Oct 17-18, 2014 Dubai (UAE), October. https://doi.org/10.15242/iie.e1014066.
Wang, Simeng, and Letian Zhang. 2026. “Organizational Diversity Database: An Open Data Set on Firm Racial and Gender Composition.” Strategy Science. https://doi.org/10.1287/stsc.2025.0393.

  1. The overlapping-scaling procedure follows the approach popularized by Alex Dyachenko; the concatenation variant mirrors the dailydata routine in the pytrends library. Both predate—and are wrapped by—the function below.↩︎