Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
13 changes: 13 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -21,6 +21,19 @@ The backend is **Lambda-only**. Four independent Lambda functions live under `ap
- `ingestion-lambda` — EventBridge-scheduled, pulls live conditions from the Davis weather station
- `public-lambda` — serve the public dashboard

### Database

The data lives in PostgreSQL. The schema is shared by all the Lambdas and is defined by [Kysely](https://kysely.dev) migrations in `db/`. For local development, `db/` also provides a disposable Postgres container.

```bash
nx run db:install # one-time install of db/'s dependencies
nx run db:up # start the local Postgres container (needs Docker)
nx run db:migrate # apply migrations
nx run db:make <name> # create a new migration
nx run db:reset # wipe the local database and re-apply everything
```

See [db/README.md](db/README.md) for details, writing migrations, and troubleshooting.

## Running tasks

Expand Down
63 changes: 63 additions & 0 deletions db/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,63 @@
# Local database

A disposable Postgres 16 container and [kysely-ctl](https://github.com/kysely-org/kysely-ctl) for migrations. The schema is shared by all the backend Lambdas.

## Setup

Requires Node 24 (`nvm use` at the repo root) and a running Docker engine with Compose v2.

```bash
cd db
yarn install # install this folder's dependencies
yarn db:up # start Postgres on 127.0.0.1:5432
yarn db:migrate # apply migrations
```

Connection: user `postgres`, password `password`, `127.0.0.1:5432`, database `postgres`.

## Commands

| Command | What it does |
| --------------------- | ----------------------------------------------- |
| `yarn db:up` | Start Postgres |
| `yarn db:down` | Stop it, keep the data |
| `yarn db:migrate` | Apply pending migrations |
| `yarn db:make <name>` | Create `migrations/<timestamp>_<name>.ts` |
| `yarn db:reset` | Delete the data, start fresh, re-run migrations |

The same commands run from the repo root through Nx: `nx run db:install`, `nx run db:up`, `nx run db:migrate`, `nx run db:make <name>`, `nx run db:reset` (prefix with `npx` if `nx` isn't installed globally).

## Adding a migration

The database schema is defined by migrations: the schema is whatever all the migration files produce when run in order (oldest first). To change it, add a new migration file.

Each migration exports an **`up`** function that applies the change: create a table, add a column, add an index. `yarn db:migrate` runs `up` for every migration that hasn't been applied yet. Kysely records the applied ones in a `kysely_migration` table, so each runs once.

We are forward-only: don't write a `down` function (the one that would undo `up`). Leave the generated `down` stub empty (we don't need to support rollback in dev). For local dev we can just drop the database. To undo or fix a change that has been merged, add a new migration.

Run `yarn db:make create_stations`, then fill in `up` in the generated file:

```ts
import type { Kysely } from 'kysely';

export async function up(db: Kysely<any>): Promise<void> {
await db.schema
.createTable('stations')
.addColumn('id', 'serial', (col) => col.primaryKey())
.execute();
}
```

Never edit a migration that has been merged; add a new one. For example, to add a column to `stations` later, create a new migration whose `up` is `alterTable('stations').addColumn(...)`.

## Differs from RDS

Disposable: no backups - `yarn db:reset` deletes everything. This is bound to `127.0.0.1` only.

## Troubleshooting

- **Cannot connect to the Docker daemon:** start Docker Desktop.
- **Port 5432 already in use:** another Postgres is running. Find it with `sudo lsof -nP -iTCP:5432 -sTCP:LISTEN` and stop it.
- **`ECONNREFUSED` right after `db:up`:** Postgres is still starting. Wait a few seconds and rerun.
- **`corrupted migrations … is missing`:** a migration file that was already applied is gone (for example after switching branches). Run `yarn db:reset`.
- **`engine "node" is incompatible`:** run `nvm use`.
12 changes: 12 additions & 0 deletions db/docker-compose.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,12 @@
services:
db:
image: postgres:16-alpine
ports:
- "127.0.0.1:5432:5432"
environment:
POSTGRES_PASSWORD: password
volumes:
- db_data:/var/lib/postgresql/data

volumes:
db_data:
15 changes: 15 additions & 0 deletions db/kysely.config.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,15 @@
import { PostgresDialect } from 'kysely';
import { defineConfig } from 'kysely-ctl';
import { Pool } from 'pg';

export default defineConfig({
dialect: new PostgresDialect({
pool: new Pool({
host: process.env.DB_HOST ?? '127.0.0.1',
port: Number(process.env.DB_PORT ?? 5432),
user: process.env.DB_USER ?? 'postgres',
password: process.env.DB_PASSWORD ?? 'password',
database: process.env.DB_NAME ?? 'postgres',
}),
}),
});
File renamed without changes.
23 changes: 23 additions & 0 deletions db/package.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,23 @@
{
"name": "@bho/db",
"version": "0.0.0",
"private": true,
"type": "module",
"scripts": {
"db:up": "docker compose up -d --wait",
"db:down": "docker compose down",
"db:migrate": "kysely migrate:latest",
"db:make": "kysely migrate:make",
"db:reset": "docker compose down -v && yarn db:up && yarn db:migrate"
},
"dependencies": {
"kysely": "^0.27.0",
"pg": "^8.11.0"
},
"devDependencies": {
"@types/node": "^20.0.0",
"@types/pg": "^8.11.0",
"kysely-ctl": "^0.21.0",
"typescript": "^5.4.0"
}
}
32 changes: 32 additions & 0 deletions db/project.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,32 @@
{
"name": "db",
"$schema": "../node_modules/nx/schemas/project-schema.json",
"projectType": "application",
"targets": {
"install": {
"executor": "nx:run-commands",
"options": { "command": "yarn install --frozen-lockfile", "cwd": "db" }
},
"up": {
"executor": "nx:run-commands",
"options": { "command": "yarn db:up", "cwd": "db" }
},
"down": {
"executor": "nx:run-commands",
"options": { "command": "yarn db:down", "cwd": "db" }
},
"migrate": {
"executor": "nx:run-commands",
"options": { "command": "yarn db:migrate", "cwd": "db" }
},
"make": {
"executor": "nx:run-commands",
"options": { "command": "yarn db:make", "cwd": "db" }
},
"reset": {
"executor": "nx:run-commands",
"options": { "command": "yarn db:reset", "cwd": "db" }
}
},
"tags": []
}
12 changes: 12 additions & 0 deletions db/tsconfig.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,12 @@
{
"compilerOptions": {
"module": "nodenext",
"moduleResolution": "nodenext",
"target": "es2022",
"esModuleInterop": true,
"strict": true,
"skipLibCheck": true,
"noEmit": true
},
"include": ["*.ts", "migrations/**/*.ts"]
}
Loading
Loading