Quantitative baseline survey lab

This lab uses fictional household records to demonstrate survey weighting, sampling uncertainty, and follow-up analysis. The design assumptions are stated alongside the calculations. All results refer to this teaching population.

Read the sampling guide for the estimands and design assumptions, and the questionnaire guide for measurement design. The JavaScript cells load the synthetic records, calculate estimates, and render the charts in this notebook. For a project survey, use methods that represent its actual design (references).

The synthetic design

The fictional frame has 1,000 households. Sixty households are selected by simple random sampling without replacement in each of three strata. Baseline response is complete. The high-exposure stratum is deliberately oversampled.

The weights expand the sample back to its defined population. Stratum names are fictional classifications; the comparison group has no demonstrated counterfactual role. The variable dictionary and proposed question specifications record the units, universes and limitations of the teaching measures. These specifications are separate from official national survey instruments.

Load the data and calculation functions in Observable Framework:

import {planProportion, weightedMean, stratifiedMean, realValue}
  from "./components/baseline-analysis.js";
const dataset = await FileAttachment("./data/synthetic-baseline.json").json();
const households = dataset.records;

Compare an interview share with a population estimate

The binary variable water_disruption_7d records whether a fictional household had at least one disruption during the specified seven-day period. Its sample mean is 25.00%. After weighting, the estimated population share is 17.33%. The difference arises because households in the smaller high-exposure stratum were sampled more intensively.

The design-based standard error is 2.79 percentage points, with an approximate normal 95% interval from 11.87% to 22.80%. This interval reflects the stated stratified sampling design. It excludes questionnaire error, incorrect classification, frame omissions, and other nonsampling error. Normal intervals can perform poorly for rare proportions or small samples; use an appropriate design-aware method for those settings.

const unweightedWater = households.reduce((sum, d) => sum + d.water_disruption_7d, 0)
  / households.length;
const estimatedWater = stratifiedMean(households, "water_disruption_7d", dataset.metadata.strata);
const waterResults = [
  {measure: "Selected sample share", value: unweightedWater},
  {measure: "Estimated population share", value: estimatedWater.mean}
];
display(Inputs.table(waterResults, {
  format: {value: d => `${(100 * d).toFixed(2)}%`}
}));

The chart uses the same values. The interval applies to the population estimate; the selected-sample share is shown only to explain the weighting difference.

const waterChart = waterResults.map(d => ({...d,
  label: d.measure === "Selected sample share" ? "Sample" : "Population"}));
display(Plot.plot({
  width: Math.min(width, 720),
  height: 250,
  marginLeft: 60,
  marginTop: 30,
  ariaLabel: "Synthetic water disruption: unweighted sample share and weighted population estimate",
  x: {label: null},
  y: {domain: [0, 0.35], label: "Household share", tickFormat: ".0%", grid: true},
  marks: [
    Plot.barY(waterChart, {x: "label", y: "value", fill: "var(--theme-foreground-focus)"}),
    Plot.ruleX([{label: "Population", low: estimatedWater.low, high: estimatedWater.high}],
      {x: "label", y1: "low", y2: "high", stroke: "currentColor", strokeWidth: 2}),
    Plot.text(waterChart, {x: "label",
      y: d => d.label === "Population" ? estimatedWater.high : d.value,
      text: d => `${(100 * d.value).toFixed(2)}%`, dy: -13}),
    Plot.ruleY([0])
  ]
}));

Distinguish household and person welfare measures

The fictional consumption variable is a monthly household aggregate in abstract local currency units. The assumed baseline price index is 100. Mean household per-capita consumption is 218.43 units; the ratio of total represented consumption to total represented people is 219.25 units. These values differ because household size and per-capita consumption vary.

const householdPc = stratifiedMean(households, "consumption_baseline_real_pc", dataset.metadata.strata);
const representedConsumption = households.reduce(
  (sum, d) => sum + d.base_weight * d.consumption_baseline_nominal, 0);
const representedPeople = households.reduce(
  (sum, d) => sum + d.base_weight * d.household_size, 0);
const personConsumption = representedConsumption / representedPeople;
display({
  householdMeanPerCapita: householdPc.mean,
  householdMeanStandardError: householdPc.se,
  populationConsumptionPerPerson: personConsumption
});

The code supplies design-based uncertainty for the household mean. Uncertainty for the person-weighted ratio requires a ratio-variance calculation. Applying an official poverty threshold would also require the corresponding national consumption aggregate, price basis and household definition.

Examine price adjustment and attrition

The synthetic follow-up price index is 110. A nominal amount of 110 therefore represents 100 baseline-price units. Both the index and the assumed stable household membership are teaching inputs.

const paired = households
  .filter(d => d.consumption_followup_nominal !== null)
  .map(d => ({
    ...d,
    real_pc_change: (
      realValue(d.consumption_followup_nominal, d.price_index_followup,
        d.price_index_baseline) - d.consumption_baseline_nominal
    ) / d.household_size
  }));
const pairedChange = weightedMean(paired, "real_pc_change");
display({
  baselineInterviews: households.length,
  reinterviewed: paired.length,
  notReinterviewed: households.length - paired.length,
  descriptiveRealPerCapitaChangeAmongReinterviewed: pairedChange.mean
});

There are 158 reinterviewed households and 22 without follow-up. The descriptive weighted real per-capita change among those reinterviewed is +12.95 units. The generator makes nonresponse more likely among households reporting water disruption. This selective attrition limits the result to the reinterviewed group unless further assumptions and adjustments are justified. Estimating a causal project effect requires a separate evaluation design.

Compare baseline outcomes of retained and lost households and examine attrition by stratum. In an actual survey, plan revisits, tracing, suitable response adjustment, and sensitivity analysis before claiming population restoration. Do not use the complete-response stratifiedMean function on a reduced follow-up sample; its declared sample counts will no longer match.

Explore precision assumptions

The following controls implement the approximate finite-population planning calculation explained in sampling and quantitative analysis. They concern one overall proportion, not subgroup precision or evaluation power.

const desiredMargin = view(Inputs.range([0.02, 0.10], {
  label: "Absolute margin of error", step: 0.01, value: 0.05
}));
const assumedDeff = view(Inputs.range([1, 3], {
  label: "Assumed design effect", step: 0.1, value: 1.5
}));
const assumedResponse = view(Inputs.range([0.60, 1], {
  label: "Anticipated response rate", step: 0.05, value: 0.90
}));

const plannedSample = planProportion({
  N: 1200, p: 0.5, margin: desiredMargin,
  deff: assumedDeff, response: assumedResponse
});
display(plannedSample);

At the default inputs, plan for 390 completed interviews and 434 selected units. If the displayed selection requirement exceeds the eligible frame, revise the precision, response or design assumptions. The calculation flags this condition.

The solid curve shows required completed interviews; the dashed curve includes the anticipated response rate to show selected units. Both respond to the design-effect and response controls. The dots mark the chosen margin of error, and the horizontal line marks the eligible frame of 1,200 households.

const precisionCurve = d3.range(2, 10.01, 0.2).map(marginPoints => ({
  marginPoints,
  ...planProportion({N: 1200, p: 0.5, margin: marginPoints / 100,
    deff: assumedDeff, response: assumedResponse})
}));
const chosenPrecision = [{marginPoints: desiredMargin * 100, ...plannedSample}];
display(Plot.plot({
  width: Math.min(width, 720),
  height: 320,
  marginLeft: 60,
  marginBottom: 45,
  ariaLabel: "Synthetic sample planning: completed interviews and selected units across margins of error",
  x: {label: "Margin of error (percentage points)", domain: [2, 10]},
  y: {label: "Households", grid: true,
    domain: [0, Math.max(1200, ...precisionCurve.map(d => d.invitations))]},
  marks: [
    Plot.ruleY([1200], {stroke: "var(--theme-foreground-muted)", strokeOpacity: 0.65}),
    Plot.lineY(precisionCurve, {x: "marginPoints", y: "completed",
      stroke: "var(--theme-foreground-focus)", strokeWidth: 2}),
    Plot.lineY(precisionCurve, {x: "marginPoints", y: "invitations",
      stroke: "var(--theme-foreground-focus)", strokeWidth: 2, strokeDasharray: "5,3"}),
    Plot.ruleX([desiredMargin * 100], {stroke: "var(--theme-foreground-faint)"}),
    Plot.dot(chosenPrecision, {x: "marginPoints", y: "completed",
      fill: "var(--theme-foreground-focus)", r: 4}),
    Plot.dot(chosenPrecision, {x: "marginPoints", y: "invitations",
      fill: "var(--theme-background)", stroke: "var(--theme-foreground-focus)", r: 4}),
    Plot.text([{marginPoints: 10, households: 1200, label: "Eligible frame: 1,200"}],
      {x: "marginPoints", y: "households", text: "label", textAnchor: "end", dy: -8})
  ]
}));

Use the calculations

Download the synthetic records, numerical summary, variable specifications, and JavaScript calculation functions to adapt these examples. The cells above show how to load the records and calculate each result in Observable Framework. In a source checkout, node src/thematic-issues/baseline-surveys/scripts/verify.mjs checks the numerical results, a hand-calculated two-stratum example, and invalid inputs.

The estimator checks the declared frame strata as well as the sampled rows. Its design-based interval requires complete response and simple random sampling within each stratum. The surrounding text and tables explain the same results as the interactive charts.

To analyze actual records, first define the design, frame populations, selection counts, variable universes, and missingness. Keep confidential information out of the public page. Adapt the estimator to the real design and validate against a survey-aware reference implementation before using estimates in project decisions.