Update CDC500_states query

[shvngisingh]

Update CDC500_states query (PR #2149 / b/530863116)

Overview

Resolves issue b/530863116. Migrates scripts/us_cdc/cdc500_state/process.py from querying the unpartitioned datcom-store.dc_kg_latest.StatVarObservation table (which required a 3-way Cartesian self-join taking 2 hours 27 minutes / 8,823.68s in production and repeatedly timed out on Cloud Batch) to Spanner-backed BigQuery export tables (TimeSeries and Observation in datcom-store.spanner_dc_graph_prod_DEFAULT). This achieves a ~360x script speedup (4.93s query/download time) and eliminates historical multi-hierarchy double-counting bugs.

---

1. Key Changes

A. BigQuery Query Optimization & Spanner DC Graph Migration (process.py)

Replaces unindexed full-table scans with indexed joins on Observation and TimeSeries pre-filtered by provenance (dc/base/CDC500 for health indicators and dc/base/CensusACS5YearSurvey for population weights).
Eliminates county + city + tract double-counting by restricting aggregation to 13-character city/CDP DCIDs (LENGTH(O.entity1) = 13), with a surgical exception for 11-character Honolulu County (geoId/15003) in <= 2016 and 2017 non-BP/cholesterol indicators.
Maps percentage StatVars to exact Census ACS 5-Year demographic cohort denominators (Count_Person_65OrMoreYears_Female, Count_Person_65OrMoreYears_Male, Count_Person_65OrMoreYears, Count_Person_18To64Years, Count_Person_18OrMoreYears, or Count_Person).
Excludes 4 non-standard multi-year cancer screening indicators (Mammography, CervicalCancerScreening, PapSmearTest, ColorectalCancerScreening) that lack composite Census ACS population denominators. In the legacy query, these indicators were already silently dropped during the inner join with ACS population series; explicitly excluding them in the CTE maintains 100% output parity while eliminating unnecessary processing.
Deduplicates observations deterministically using QUALIFY ROW_NUMBER() OVER (... ORDER BY O.last_update_timestamp DESC, O.facet_id DESC) = 1.
Adds atomic write verification (.tmp write + os.path.getsize > 0 check + os.replace()), project logging before dispatch, and --timeout, --project, and --output_dir flags.

B. Unit Tests & Quality Gates (process_test.py)

Adds 13 hermetic unit tests executing process.QUERY in DuckDB across cohort mappings, Honolulu County 2016/2017/2018 edge cases, deduplication tie-breakers, atomic writes, and error handling (10.00/10 pylint score).
Mock helper fake_to_csv uses *args, **kwargs to eliminate unused argument declarations.

C. Validation & Manifest (validation_config.json, manifest.json, cdc500_state.tmcf)

Adds unit: Percent and scalingFactor: 100 in cdc500_state.tmcf for schema metadata completeness.
Formats SQL_VALIDATOR query (check_statvar_max_dates) into clean single-line JSON, casting MaxDate explicitly to VARCHAR before SUBSTR to ensure compatibility with DuckDB's type inference.
Scoped SQL_VALIDATOR enforces cohort-specific vintage thresholds with non-empty table guard (COUNT(*) OVER () AS total_svs and total_svs > 0):

• 2016: Percent_Person_WithAllTeethLoss (discontinued by CDC in subsequent survey rounds).
• 2020: CorePreventiveServices (follows CDC rotating module cadence).
• 2021: HighBloodPressure | Cholesterol | ChronicKidneyDisease (collected on odd-year BRFSS rotations).
• 2022: All remaining 27 health indicators.
  Enforces percentage bounds [0, 100] and place counts [50, 52] (50 US states + Washington, D.C. = 51 entities).
  Inherits strict default 0% deletion tolerance (threshold: 0 in check_deleted_records_percent), verified with 0 deleted records against baseline.
  Right-sizes manifest resources to cpu: 2, memory: 8, disk: 30 (satisfying mandatory >= 30 GB disk floor) and registers validation_config.json in source_files.

---

2. Production Deployment & Deletion Safety Rationale

unit: Percent and scalingFactor: 100 are retained in cdc500_state.tmcf because CDC 500 indicators represent percentage values (0–100); keeping them ensures semantic accuracy in the Data Commons knowledge graph.
validation_config.json intentionally preserves the default strict 0% threshold (threshold: 0), rather than overriding with a temporary threshold: 100. This avoids weakening automated deletion safety and guarantees that the merged codebase reflects the long-term production contract immediately.
The 100% differ report against pre-migration production data is purely an artifact of series facet identity key changes (_GROUPBY_KEYS including unit and scalingFactor); all 19,314 rows are fully regenerated.
Deployment follows the SOP in scripts/us_cdc/cdc500_state/README.md (lines 125–139): Run 1 will be validated and promoted to establish the updated schema baseline, after which all weekly cron runs will execute against it under strict 0% tolerance.

---

3. Test Environment Validation (datcom-import-test)

Cloud Batch Job ID: cdc500-states-shvngisingh-20260922-155723 (UID: cdc500-states-shvn-aab2fba2-f25b-4a9b0)
Cloud Console Logs: View Logs
Output GCS URI: gs://datcom-import-test/scripts/us_cdc/cdc500_state/CDC500_States/2026_09_22T09_00_31_202686_07_00/ (Pantheon Link)
Execution Performance: Total job runtime: 22s, Python script execution: 4.93s (ImportStatus.STAGING).
Validation Results (validation_output.csv): All 8 validation rules PASSED (19,314 rows across 35 StatVars and 51 state/DC entities; 0 deleted records):

ValidationName	Status	Details	ValidationParams
check_deleted_records_percent	PASSED	{"deleted_records_count": 0, "previous_obs_count": 19314, "percent": 0.0, "threshold": 0}	{"threshold": 0}
check_empty_import	PASSED	{"num_nodes": 19314, "num_rows": 19314}	None
check_missing_refs_count	PASSED	{"missing_refs_count": 0}	{"threshold": 0}
check_lint_error_count	PASSED	{"lint_error_count": 0}	{"threshold": 0}
check_max_value_percentage	PASSED	{"rows_processed": 35, "rows_succeeded": 35, "rows_failed": 0}	{"maximum": 100}
check_min_value_percentage	PASSED	{"rows_processed": 35, "rows_succeeded": 35, "rows_failed": 0}	{"minimum": 0}
check_num_places_state_count	PASSED	{"rows_processed": 35, "rows_succeeded": 35, "rows_failed": 0}	{"minimum": 50, "maximum": 52}
check_statvar_max_dates	PASSED	max_year IS NOT NULL AND max_year >= expected_year AND total_svs > 0 across all 35 StatVars	SQL_VALIDATOR

[gemini-code-assist]

Code Review

This pull request refactors the SQL query in scripts/us_cdc/cdc500_state/process.py by introducing Common Table Expressions (CTEs) and updating the source table references to spanner_dc_graph_prod_DEFAULT. The review identifies an opportunity to improve query performance by removing a redundant join in the svo_count CTE and recommends using SAFE_DIVIDE to handle potential division-by-zero errors during the percentage calculation.

[pulkeet-wq]

Add validation_config.json

[shvngisingh]

Add validation_config.json

Added

[shvngisingh]

/gcbrun