[India_RBI_State_Statistics] Modified download script and handle deletions

[Krishnam24maheshwari]

In this PR we fix import execution failures, data corruption issues, and deletion handling in the India_RBI_State_Statistics import pipeline.

Problems Addressed:

• Unversioned Remote Config Dependency: rbi_download.py had a hard dependency on remote GCS configuration, crashing in environments without GCS access or ADC credentials.
• Download Rejection & Brittle Execution: Requests to rbi.org.in lacked browser User-Agent headers, causing HTTP 403 / throttling rejections; individual table downloads were not isolated; and partial/corrupt files could be left in the working directory.
• Data Corruption & Header Coercion:
  • Blank/empty cells and footnote markers (*, @) caused malformed or literal "nan" string values.
  • Excel year headers parsed by pandas as floats (e.g. 2015.0) failed to match integer string keys ("2015") in PVMaps, dropping observations.
  • Deprecated applymap calls caused runtime errors across different pandas versions.
• Unquoted Globs in run.sh: --input_data=$SCRIPT_PATH/input_files/agriculture/*.xlsx was unquoted, causing Bash to expand the 42 matched files as positional arguments and silently dropping 41 tables (and 7 of 8 tables in price_and_wages).
• Entity Resolution Misses & Erroneous Mapping:
  • NCT of Delhi was misattributed to wikidataId/Q1352 (Chennai) in rbi_places_resolver.csv.
  • Meteorological subdivision "Haryana, Delhi & Chandigarh" was unmapped, causing 6.94% deletions in annual rainfall.
  • Asterisks stripped from places during preprocessing (replace('*', '')) caused "Kolkata" and "Chennai" to miss cache entries in rbi_places_resolver.csv (Kolkata**, Chennai**).

---

Changes Made:

1. rbi_download.py & configs.json:

• Decoupled Config: Migrated table configuration to a local, version-controlled configs.json by default, with automatic fallback if remote GCS paths are unreachable.
• Resilient Connection Pooling: Added pooled requests.Session() with urllib3.util.Retry(4), status forcelists (429, 500, 502, 503, 504), pre-warmed portal cookies, and browser headers (User-Agent, Accept, Accept-Language) to prevent HTTP 403 blocks.
• Atomic Downloads & File Verification: Validated magic bytes (PK\x03\x04 ZIP signature) and used tempfile.TemporaryDirectory with os.replace to prevent partial or corrupted downloads. force_download preserves existing files until the new download is verified.
• Preprocessing & Float Normalization:
  • In clean_cell: Stripped footnote characters (*, @), trimmed whitespace, and converted empty/'nan' strings to true NaN.
  • In safe_to_numeric: Normalized numeric and float-coerced year headers (2015.0 → 2015) so they match integer keys in PVMaps.
• Version Compatibility: Introduced _apply_map() helper supporting both df.map (pandas ≥ 2.1) and df.applymap (pandas < 2.1).

2. run.sh:

• Quoted Input Globs: Enclosed --input_data="$SCRIPT_PATH/input_files/agriculture/*.xlsx" and --input_data="$SCRIPT_PATH/input_files/price_and_wages/*.xlsx" in quotes so stat_var_processor.py expands and processes all 42 agriculture and 8 price/wage tables.
• Counter Tracking: Added COUNTERS_PATH=$SCRIPT_PATH/counters and appended --output_counters to all 18 processing commands.
• Exit Code Propagation: Propagates final exit status with EXIT_STATUS=$?; exit $EXIT_STATUS.

3. Place Mappings & PVMaps:

• Delhi Correction: Updated "NCT of Delhi" to canonical wikidataId/Q1353 in rbi_places_resolver.csv.
• Regional Rainfall Mapping: Mapped "Haryana, Delhi & Chandigarh" and "Haryana, Delhi and Chandigarh" to dcid:HaryanaDelhiChandigarh in environment_pvmap.csv.
• SDG Standardization: Standardized Table 106 SDG composite indicators to IndexScore_SustainableDevelopmentGoal.

4. manifest.json & Dev Verification:

• Two-Stage Dev MCF Verification: Added temporary dev_verification_nodes.mcf under source_files and node_mcf for the dev verification pipeline (datcom-import-test) to eliminate unreleased schema reference errors during dev testing. This file will be removed upon companion schema submission prior to final merge.
• Added counters/*.csv to source_files.

5. Tests & Fixtures:

• Hermetic Unit Tests: 18 unit tests in rbi_download_test.py covering download retries, corrupted payloads, HTTP 404s, and header conversions.
• Test Fixture Regeneration: Regenerated testdata/environment_output.csv with valid observation rows.

6. Validation Config & Deletion Handling:

• Standard 0.1% deletion threshold maintained in validation_config.json.
• In dev testing, the pipeline was verified with a 100% threshold to ensure end-to-end execution reaches ImportStatus.STAGING. In production, the standard 0.1% threshold is enforced; upon the initial baseline reset run, the new output will be promoted as latest_version.txt, after which subsequent weekly runs execute with 0% deletions.
• Legitimate deletions are accounted for by place corrections (dcid:HaryanaDelhiChandigarh, NCT of Delhi), unit conversions (Hectare to MetricTon for fruits), and official RBI retrospective statistical revisions.

Validation Reports (Cloud Batch Test Run):

• Sub-import validation outputs where deletions/updates occured

input0 : https://console.cloud.google.com/storage/browser/_details/datcom-import-test/statvar_imports/database_on_indian_economy/india_rbi_state_statistics/India_RBI_State_Statistics/2026_10_12T05_25_11_249377_07_00/input0/validation/validation_output.csv?project=datcom-infosys-dev

input1 : https://console.cloud.google.com/storage/browser/_details/datcom-import-test/statvar_imports/database_on_indian_economy/india_rbi_state_statistics/India_RBI_State_Statistics/2026_10_12T05_25_11_249377_07_00/input1/validation/validation_output.csv?project=datcom-infosys-dev

input2 : https://console.cloud.google.com/storage/browser/_details/datcom-import-test/statvar_imports/database_on_indian_economy/india_rbi_state_statistics/India_RBI_State_Statistics/2026_10_12T05_25_11_249377_07_00/input2/validation/validation_output.csv?project=datcom-infosys-dev

input3 : https://console.cloud.google.com/storage/browser/_details/datcom-import-test/statvar_imports/database_on_indian_economy/india_rbi_state_statistics/India_RBI_State_Statistics/2026_10_12T05_25_11_249377_07_00/input3/validation/validation_output.csv?project=datcom-infosys-dev

input4 : https://console.cloud.google.com/storage/browser/_details/datcom-import-test/statvar_imports/database_on_indian_economy/india_rbi_state_statistics/India_RBI_State_Statistics/2026_10_12T05_25_11_249377_07_00/input4/validation/validation_output.csv?project=datcom-infosys-dev

input5 : https://console.cloud.google.com/storage/browser/_details/datcom-import-test/statvar_imports/database_on_indian_economy/india_rbi_state_statistics/India_RBI_State_Statistics/2026_10_12T05_25_11_249377_07_00/input5/validation/validation_output.csv?project=datcom-infosys-dev

input8 : https://console.cloud.google.com/storage/browser/_details/datcom-import-test/statvar_imports/database_on_indian_economy/india_rbi_state_statistics/India_RBI_State_Statistics/2026_10_12T05_25_11_249377_07_00/input8/validation/validation_output.csv?project=datcom-infosys-dev

input9 : https://console.cloud.google.com/storage/browser/_details/datcom-import-test/statvar_imports/database_on_indian_economy/india_rbi_state_statistics/India_RBI_State_Statistics/2026_10_12T05_25_11_249377_07_00/input9/validation/validation_output.csv?project=datcom-infosys-dev

[google-cla]

Thanks for your pull request! It looks like this may be your first contribution to a Google open source project. Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA).

View this failed invocation of the CLA check for more information.

For the most up to date status, view the checks section at the bottom of the pull request.

[gemini-code-assist]

Code Review

This pull request updates the RBI state statistics import pipeline by refactoring the preprocessing script to write to a temporary directory, adding unit tests, updating place mappings, and introducing a validation configuration. The reviewer feedback highlights several issues: df.map incorrectly converts NaN values to 'nan' strings; mapping combined regions like 'Haryana, Delhi & Chandigarh' to Haryana's ID (Q1174) is incorrect; the Wikidata ID for NCT of Delhi should be updated to Q9357528 for AA1 level data; and using a broad wildcard pattern for node_mcf in manifest.json is inefficient and should be scoped to specific directories.

[Krishnam24maheshwari]

done

[abhishekjaisw]

LGTM