Data processing

Data processing makes evidence traceable from an original observation to an analytical finding. In social impact assessment, that chain can combine household interviews, government statistics, administrative registers, maps, and qualitative accounts. Each source has its own population, period, unit, and limitations. A useful processing plan preserves these distinctions while making justified comparisons possible.

Start the plan alongside the questionnaire and field procedures. Define the intended indicators, tables, identifiers, and quality checks before collection. This makes it possible to discover an unusable denominator or missing join key while the instrument can still be changed.

Preserve originals and define observations

Retain the received file unchanged, with its provider, edition, acquisition date, and reuse conditions. Process working copies through recorded scripts. Keep the questionnaire, translation, codebook, and survey software version with the corresponding data release. Corrections should have a reason and an accountable reviewer.

Give each table an explicit unit: household and round, person and round, parcel, payment, service facility, or interview. Assign stable identifiers and document relationships between tables. A household using three parcels needs linked use records; repeating its income on three parcel rows would inflate an unqualified sum.

The World Bank's DIME data handbook, chapter 5, recommends organising observations into tidy tables and checking completeness, consistency, and distributions. It also addresses duplicate resolution and protection of identifying information. These provide a practical starting point for the processing specification.

Check before correcting

Prepare a review report for every incoming batch. Check unique identifiers, expected records, permitted codes, units, skip patterns, dates, and relationships. Compare submissions with field tracking records. A gap in an upload can otherwise be mistaken for a household that declined an interview.

Keep zero, refusal, unknown, and not applicable as distinct states. Verify apparent outliers against the instrument and field notes before altering them: an unusually large household or income can be real. Record any exclusion, recoding, or imputation, and retain a flag connecting the analytical value to the original response.

For paper records, specify entry verification and resolution of discrepancies. Electronic collection can make checks available sooner, but a compulsory answer or an incorrect skip rule can introduce systematic error. Test the exported tables as well as the screen a fieldworker sees.

Integrate sources deliberately

Before joining datasets, examine the expected relationship and report matched, unmatched, and duplicated keys. Record the boundary edition and crosswalk when connecting district statistics to geometry. Preserve original names alongside harmonised codes so that disputed matches can be reviewed.

Align reference periods, currency and price dates, measurement units, and population definitions. A district poverty estimate provides context for an affected settlement; assigning it to each household does not establish household poverty. The dataset curation guide develops the metadata needed for these decisions.

Qualitative material needs an equivalent audit trail: interview or observation ID, date, context, language, transcription or translation status, and coding decisions. Separate a participant's account from the analyst's interpretation. Investigate disagreement between interviews, surveys, and administrative records rather than automatically treating agreement as confirmation.

Prepare analysis and disclosure editions

Keep sample weights, strata, clusters, and interview outcomes with the analytical file. Weighting and uncertainty calculations follow the documented sampling design. Record the numerator, denominator, eligibility rule, and missing-data treatment for every reported indicator.

Separate contact and identity records from routine analysis, and restrict precise locations and sensitive testimony where disclosure could cause harm. Removing names alone does not establish anonymity. Prepare public aggregates and permitted reader materials as explicit releases, each with its own review, source references, processing recipe, and limitations.

Sources and further reading

Tosch and Berger's SurveyMan research treats survey instruments as programs with logic and potential errors. The original software repository, introduced in this site's earlier discussion, remains a useful historical example of survey testing; adopting it requires a separate assessment of current suitability.

The methods references retain the source and archive trail supporting this discussion.