Update CDC500_states query

scripts/us_cdc/cdc500_state/README.md

@@ -5,37 +5,127 @@ Author: Padma Gundapaneni @padma-g
 ## Table of Contents
 1. [About the Dataset](#about-the-dataset)
     1. [Overview](#overview)
+    2. [Data Sources and Tables](#data-sources-and-tables)
+    3. [Aggregation Methodology](#aggregation-methodology)
 2. [About the Import](#about-the-import)
     1. [Artifacts](#artifacts)
     2. [Import Procedure](#import-procedure)
+    3. [Production Rollout / Deployment SOP](#production-rollout--deployment-sop)
+    4. [Troubleshooting](#troubleshooting)
 
 ## About the Dataset
 
 ### Overview
-The state level data is aggragated from city level data coming from CDC500 import. 
+The state-level dataset calculates aggregated health indicator prevalence estimates for US states from the city-level CDC 500 Cities (`CDC500`) project data, weighted by corresponding Census ACS 5-Year population counts.
 
-To get the data for this import run:
-```bash
-$ python3 process.py
-```
+CDC PLACES (Population Level Analysis and Community Estimates) is the official successor program to the original CDC 500 Cities project, expanding health estimates across all US counties, places, and census tracts. In Data Commons, city-level data from both the historical CDC 500 Cities project (2016–2019) and ongoing CDC PLACES releases (2020–present) are ingested under provenance `dc/base/CDC500`.
+
+### Data Sources and Tables
+
+The aggregation script queries Google Cloud BigQuery graph tables in dataset `datcom-store.spanner_dc_graph_prod_snap_latest`:
+
+1. **`TimeSeries`**:
+   - **CDC 500 Series**: Identifies CDC 500 Statistical Variables (`JSON_VALUE(facet.provenance) = 'dc/base/CDC500'` and `variable_measured LIKE 'Percent_%'`) and extracts their measurement methods (`JSON_VALUE(facet.measurementMethod)`). It maps each percentage health metric to its appropriate denominator demographic cohort StatVar (e.g., `Count_Person_18OrMoreYears`, `Count_Person_18To64Years`, `Count_Person_65OrMoreYears`, `Count_Person`, etc.).
+   - **Census ACS 5-Year Series**: Filters and joins population counts from Census ACS 5-Year Survey (`JSON_VALUE(facet.provenance) = 'dc/base/CensusACS5YearSurvey'`).
+
+2. **`Observation`**:
+   - **Health Indicator Percentages**: Fetches city-level percentage values (`value AS percent`), observation dates (`date`), and city geoIds (`entity1 LIKE 'geoId/%'`) for CDC 500 StatVars. To prevent double-counting Oahu/Honolulu County population in years where 13-character Census Designated Places (`geoId/15XXXXX`) are published (2017–2022), `geoId/15003` is explicitly filtered in the `WHERE` clause to include only years `<= '2016'` and `'2017'` county-only indicators (excluding the four 2017 blood pressure and cholesterol indicators that also have 13-character CDP data).
+   - **City Cohort Populations**: Fetches city-level population counts (`value AS population`) for the corresponding demographic cohort StatVars.
+
+### Aggregation Methodology
+
+For each state, indicator StatVar, and observation date:
+- City observations are joined with their corresponding demographic population counts.
+- City geoIds (`geoId/XXXXXXX`) are mapped to state geoIds (`geoId/XX`) using the first 8 characters (including the prefix).
+- State-level prevalence percentages are computed as a population-weighted average:
+
+$$\text{State Percent} = \frac{\sum (\text{City Population} \times \text{City Percent})}{\sum \text{City Population}}$$
+
+The output measurement method is prefixed with `dcAggregate/` (e.g., `dcAggregate/CrudePrevalence`).
+
+*Note on Demographic Cohorts:* Several adult health indicators (such as `Percent_Person_Smoking`, `Percent_Person_Obesity`, `Percent_Person_WithDiabetes`) lack explicit age tokens in their DCID and intentionally fall back to `Count_Person` weighting to preserve bitwise parity with existing Data Commons baseline observations.
+
+#### Excluded Indicators
+
+The following age-bracketed cancer screening indicators are omitted from state-level aggregation:
+- `Percent_Person_50To74Years_Female_ReceivedMammography`
+- `Percent_Person_21To65Years_Female_ReceivedCervicalCancerScreening`
+- `Percent_Person_21To65Years_Female_ReceivedPapSmearTest`
+- `Percent_Person_50To75Years_ReceivedColorectalCancerScreening`
+
+**Rationale**: The Census ACS 5-Year Survey does not publish single composite population StatVars for these non-standard multi-year age brackets (`50To74Years`, `21To65Years`, `50To75Years`). In the legacy query, these indicators were already silently dropped during the inner join with ACS population series. Rather than applying arbitrary proxy weights, these indicators are explicitly excluded in the query CTE, preserving full parity with the legacy pipeline while preventing unnecessary processing.
 
 ## About the Import
 
 ### Artifacts
 
 #### Scripts
-[`process.py`](https://github.com/datacommonsorg/data/blob/master//scripts/us_cdc/cdc500_state/process.py)
+[`process.py`](process.py)
+
+#### Unit Tests
+[`process_test.py`](process_test.py)
+
+#### tMCF Template
+[`cdc500_state.tmcf`](cdc500_state.tmcf)
 
+#### Validation Config
+[`validation_config.json`](validation_config.json)
 
-#### tMCFs
-[`cdc500_state.tmcf`](https://github.com/datacommonsorg/data/blob/master/scripts/us_cdc/cdc500_state/cdc500_state.tmcf)
+#### Manifest
+[`manifest.json`](manifest.json)
 
 ### Import Procedure
 
-#### Data Download and Processing Steps
+#### Prerequisites
+
+Ensure Google Cloud authentication is configured with access to BigQuery dataset `datcom-store.spanner_dc_graph_prod_snap_latest`:
+
+```bash
+$ gcloud auth application-default login
+```
+
+#### Running the Script
 
-To get the data for this import run:
+To run the BigQuery aggregation and write the output CSV to the default output directory (`scripts/us_cdc/cdc500_state/CDC500State_Output/CDC500State_Output.csv` when run from repository root):
 
 ```bash
-$ python3 process.py
+$ python3 scripts/us_cdc/cdc500_state/process.py
 ```
+
+To specify a custom output directory:
+
+```bash
+$ python3 scripts/us_cdc/cdc500_state/process.py --output_dir=/path/to/output
+```
+
+To specify a GCP project and timeout (in seconds) for BigQuery jobs (if not set in ambient environment):
+
+```bash
+$ python3 scripts/us_cdc/cdc500_state/process.py --project=my-gcp-project --timeout=600
+```
+
+#### Running Unit Tests
+
+Run the test suite using Python's `unittest` runner from the repository root:
+
+```bash
+$ python3 -m unittest scripts/us_cdc/cdc500_state/process_test.py
+# or via module path:
+$ python3 -m unittest scripts.us_cdc.cdc500_state.process_test
+```
+
+#### Automation
+
+This import is automated via Data Commons Import Automation and scheduled to run weekly via Cloud Batch every Monday at 01:00 UTC (`cron_schedule: "0 1 * * 1"` in `manifest.json`).
+
+- **Import Type**: Automated (weekly Cloud Batch cron: `0 1 * * 1`)
+- **Production GCS Path**: `gs://datcom-prod-imports/scripts/us_cdc/cdc500_state/CDC500_States/`
+- **Test GCS Path**: `gs://datcom-import-test/scripts/us_cdc/cdc500_state/CDC500_States/`
+
+### Production Rollout / Deployment SOP
+
+Adding `unit: Percent` and `scalingFactor: 100` to `cdc500_state.tmcf` standardizes observation series identity keys across the Knowledge Graph. Manual manipulation of `latest_version.txt` in production GCS is strictly prohibited per `b/548893664` and `CHK-9.3`; baseline promotion is handled strictly through validated automated pipeline executions.
+
+### Troubleshooting
+
+- **`ValueError: BigQuery query returned 0 rows`**: Verify ADC authentication (`gcloud auth application-default login`) and ensure read permissions on `datcom-store.spanner_dc_graph_prod_snap_latest`. Pass `--project=<gcp_project_id>` if running outside the default project.

scripts/us_cdc/cdc500_state/cdc500_state.tmcf

@@ -4,5 +4,7 @@ variableMeasured: C:CDC->statvar
 observationAbout: C:CDC->observation_about
 observationDate: C:CDC->observation_date
 value: C:CDC->percent
+unit: Percent
+scalingFactor: 100
 measurementMethod: C:CDC->measurement_method
-observationPeriod: "P1Y"
+observationPeriod: "P1Y"

scripts/us_cdc/cdc500_state/manifest.json

@@ -17,14 +17,15 @@
                 }
             ],
             "resource_limits": {
-                "cpu": 8,
-                "memory": 64,
-                "disk": 100
+                "cpu": 2,
+                "memory": 8,
+                "disk": 30
             },
             "source_files": [
-                "CDC500State_Output/CDC500State_Output.csv"
+                "validation_config.json"
             ],
-            "cron_schedule": "0 1 * * 1"
+            "cron_schedule": "0 1 * * 1",
+            "validation_config_file": "validation_config.json"
         }
     ]
-}
+}