Add new Commerce EDA USASpending import

[shvngisingh]

[PR #2171] Add Commerce_EDA_USAspending Dataset Import

1. Overview

Adds the automated weekly data import pipeline for Commerce_EDA_USAspending (statvar_imports/commerce_eda_usaspending), ingesting U.S. Economic Development Administration (EDA) federal investment assistance awards by state/territory and program from the official USAspending.gov REST API (https://api.usaspending.gov/api/v2/search/spending_by_award/).

• Buganizer Issue: b/547801183
• Companion Schema & Manifest CL: cl/982670020

2. Key Changes

1. Data Acquisition & Processing (process.py):
   • Dynamically queries USAspending.gov /api/v2/search/spending_by_award/ across federal fiscal years 2012 through get_current_fiscal_year() (accounting for the October 1 federal fiscal year boundary).
   • Filters by awarding agency (Department of Commerce) and sub-agency (Economic Development Administration) across assistance award types (02, 03, 04, 05).
   • Deduplicates award records across pagination and fiscal-year boundaries by Award ID / generated_internal_id.
   • Maps CFDA / Assistance Listing numbers and descriptions to 11 EDA assistance programs plus aggregated Total (Amount_Investment and 11 program-specific Amount_Investment_<Program> StatisticalVariables).
   • Normalizes all 50 U.S. states, DC, and 7 outlying territories to standard geoId/{state_fips} DCIDs.
   • Retains only net-positive state-program-year obligations (with a fail-closed guard if no positive aggregations remain) and writes output atomically via a temporary .tmp file.
2. Schema & Mapping (investment_pvmap.csv, investment_metadata.csv, investment_schema.mcf, manifest.json):
   • Defines PV mappings and statvar_dcid overrides aligned with canonical Google3 StatVar definitions.
   • Defines the 4 new InvestmentProgramEnum instances (DistressedAreaRecompetePilotProgram, RegionalTechnologyAndInnovationHubs, STEMTalentChallenge, ScienceAndResearchParkDevelopmentGrants).
   • Configures weekly cron schedule (30 05 * * 1), import_groups: ["frequent", "infrequent"], node_mcf: ["investment_schema.mcf", "output/*.mcf"], and provenance_url: "https://www.usaspending.gov/".
3. Validation & Unit Tests (validation_config.json, process_test.py, test_data/, README.md):
   • Adds 3 custom validation rules (check_deleted_records_percent at 0.1% threshold, active_programs_freshness verifying all 12 StatVars with total_svs = 12 and cohort-aware MaxDate bounds, and non_negative_investment_amounts).
   • Adds 16 hermetic unit tests in process_test.py and sample fixtures in test_data/ (sample_awards.json, sample_output.csv) covering API pagination, deduplication, retry/failure handling, de-obligation filtering, all-non-positive award guards, dynamic fiscal year boundaries, territory mapping, and rounding.

3. Validation & Test Evidence

• Unit Tests: 16 / 16 PASSED (pytest statvar_imports/commerce_eda_usaspending/process_test.py)
• Import Validation (tools.import_validation.runner): 6 / 6 PASSED (2,695 observations across 58 states/territories and 12 StatisticalVariables, 2,703 total nodes, 0 missing refs, 0 lint errors, 0.0% deleted records).
• Cloud Batch Job (datcom-infosys-dev): commerce-eda-usaspending-shvngisingh-20261009-172155 (commerce-eda-usasp-c45bf705-7412-4cf30, SUCCEEDED)
• Staged Dev GCS Run: gs://datcom-import-test/statvar_imports/commerce_eda_usaspending/Commerce_EDA_USAspending/2026_10_09T10_24_52_712858_07_00

[gemini-code-assist]

Code Review

This pull request introduces a new automated importer for U.S. Economic Development Administration (EDA) investment datasets sourced from USASpending.gov, including preprocessing scripts, configuration files, and documentation. Feedback on the changes suggests improving the robustness of the preprocessing script by raising exceptions on API failures, adding guard checks for date parsing, and removing redundant code that attempts to update a commented-out metadata field. Additionally, it is recommended to explicitly map the 'Total' category in the PV map to properly represent aggregation.