Dev: Build a CSV parsing utility - Historical data parser

[naman0r]

ℹ️ Issue

Closes #10

historical sheet: https://docs.google.com/spreadsheets/d/1WI8CxTJ7edRajo3JHZX0WmTklvCqRy5x/edit?rtpof=true&sd=true

📝 Description

Adds a parser for the historical records spreadsheet (the All Daily Records tab). The upload lambda can call it to check a file before anything is written to daily_records. It returns either one record per day or a list of every problem in the file, each with the sheet row and column, so staff can fix everything in one pass. The error list matches the shape of imports.errors.

Briefly list the changes made to the code:

1. Added parseHistorical(csv) in apps/backend/upload-lambda/src/parsers/historical.ts. It finds the header row even when rows sit above it (the Drive export has a link and a blank row first), looks up columns by header name, and checks that:

   • Month # and Day are a real calendar day (Feb 29 included), and the Month name agrees with Month #
   • each day appears once
   • every value has its year(s), and blank or "None" values have none ("None" is stored as null for both)
   • years are four digits, wind direction is one of the 16 compass points, and Estimated? is TRUE or FALSE
   • a wind direction or a TRUE Estimated? has a gust to go with it (FALSE is allowed without one, since the sheet fills that column on every row)

   Precip and snow stay strings ("5.30"), matching how pg reads numeric. Source text is not stored, since it repeats the other columns.

2. Added shared types in libs/weather-parsing/src/types.ts: ParseResult<T>, RowError, and HistoricalRecord. HistoricalRecord uses the daily_records column names, so a parsed row plus stationId and importId can go straight into an insert. The daily parser (Dev: Build a CSV parsing utility - Daily hourly-log parser #9) can reuse ParseResult and RowError.

3. Added a small CSV reader in parsers/csv.ts (no new dependency). It handles quoted fields, which year lists like "2014, 1977" need. Dev: Build a CSV parsing utility - Daily hourly-log parser #9 can use it too.

4. Mapped @bho/weather-parsing in tsconfig.base.json. The lambdas list it as a file: dependency, but nothing installs it, so the import didn't resolve before.

✔️ Verification

Unit tests cover the CSV reader and the parser. The parser tests use the synthetic test sheet, exported from Drive as parsers/__fixtures__/historical.csv (366 rows, every field of Jan 1 checked), plus small sheets for each rule above.

yarn install
npx jest apps/backend/upload-lambda   # 20 tests

To run the parser on any exported CSV and print the rows or the errors:

npx ts-node --transpile-only -O '{"module":"commonjs"}' -e "
const { parseHistorical } = require('./apps/backend/upload-lambda/src/parsers/historical');
const r = parseHistorical(require('fs').readFileSync(process.argv[1], 'utf8'));
console.log(r.ok ? r.rows.length + ' rows' : JSON.stringify(r.errors, null, 2));
" path/to/export.csv

No UI changes.

🏕️ (Optional) Future Work / Notes

Open questions for the team (current behavior in parentheses):

• Should a blank value with blank years count as no record, or as an error? (No record, following the schema doc's "empty cell means null".)
• Can a record be a trace (T) for precip or snow? (Rejected.)
• Must an upload cover all 366 days? (Partial files are accepted.)
• Is a blank Estimated? the same as FALSE? (Yes.)

Found along the way:

• This PR breaks the upload lambda's build. The lambda is compiled with tsc, and its rootDir: ./src only allows files under src/. The parser now imports types from libs/weather-parsing/src, so tsc in apps/backend/upload-lambda stops with error TS6059. On main that command passes. CI doesn't build the backend, so the checks stay green.
  • Fix: delete the rootDir line from apps/backend/upload-lambda/tsconfig.json. Compiled files then go to dist/apps/backend/upload-lambda/src/ instead of dist/; nothing reads dist yet. Bundling the lambdas with esbuild in Dev: Empty Upload Lambda #13 would also fix it.
  • nx build upload-lambda already failed on main, for a different reason: it builds the lib first, and the lib had no source files.
• tsc writes dist/**/*.test.js even when it errors, and root jest then runs those copies. Adding dist to jest's ignore paths would stop that.
• The lambdas don't set strictNullChecks, so !result.ok doesn't narrow a ParseResult. Use result.ok === false (noted on the type).
• The root yarn lint:check matches no files under apps/backend/*/src. CI runs nx affected -t lint instead, which has linted upload-lambda since Dev: Empty Upload Lambda #22.

[Rayna-Yu]

Looks good! Nice work, and thanks for flagging the TS6059 break!
After you merge, I'll apply the fixes you outlined and remove rootDir from the upload-lambda tsconfig and ignore dist/ in its Jest config directly on main. (If you have the capacity it would be great if you could do it on this branch, but either way works just lmk!)