Docs: a live browser demo and a How it works explainer

.gitignore

@@ -15,3 +15,5 @@ CHANGELOG.md
 *.ipynb
 /site
 .direnv
+# Pyodide wheels copied in to test the browser demo locally
+docs/demo/*.whl

docs/browser.md

@@ -0,0 +1,153 @@
+# Run in the browser
+
+xarray-sql runs in [Pyodide](https://pyodide.org), Python compiled to
+WebAssembly. The demo below runs SQL over NOAA GFS forecasts read straight
+from [dynamical.org](https://dynamical.org)'s public Zarr store. xarray-sql
+and DuckDB run in a Web Worker in your browser, so there is no server. Each
+query downloads only the chunks that its filters on `init_time`, `latitude`,
+and `longitude` reach.
+
+The demo needs Chrome or Edge 137 or newer. It takes about 15 seconds to
+start, most of it downloading and loading Pyodide, pyarrow, DuckDB, and
+pandas.
+
+<iframe id="demo" src="../demo/" title="xarray-sql browser demo" loading="lazy"
+  style="width: 100%; height: 1150px; border: 1px solid var(--md-default-fg-color--lightest); border-radius: 8px;"></iframe>
+
+[Open the demo in its own page](demo/){ #demo-link target="_blank" }
+
+<script>
+  // Pass ?wheel=<url> through to the demo, to try a locally built wheel.
+  (() => {
+    const wheel = new URLSearchParams(location.search).get("wheel");
+    if (!wheel) return;
+    for (const [id, attr] of [["demo", "src"], ["demo-link", "href"]]) {
+      const el = document.getElementById(id);
+      const url = new URL(el.getAttribute(attr), location.href);
+      url.searchParams.set("wheel", new URL(wheel, location.href).href);
+      el.setAttribute(attr, url.href);
+    }
+  })();
+</script>
+
+## Install in Pyodide
+
+Releases after 0.5.0 publish a Pyodide wheel to PyPI. The DuckDB engine
+adapter works in Pyodide, including the round-trip back to a Dataset with
+[to_dataset][xarray_sql.to_dataset]. DataFusion does not yet, because its
+Python bindings have no Pyodide build, so [Context][xarray_sql.Context] is
+unavailable there.
+
+Copy [`fetch_store.py`](https://github.com/xqlsystems/xarray-sql/tree/main/docs/demo/fetch_store.py)
+from the demo to open Zarr stores over HTTP in Pyodide, since fsspec's HTTP
+filesystem needs sockets. Then open a window of the dataset, read it into
+memory, and query it:
+
+```python
+import micropip
+await micropip.install(["xarray-sql", "duckdb", "zarr"])
+
+import duckdb
+import xarray as xr
+import xarray_sql as xql
+from fetch_store import FetchStore
+
+store = FetchStore("https://data.dynamical.org/noaa/gfs/forecast/latest.zarr")
+ds = (
+    xr.open_zarr(store, chunks=None)[["temperature_2m"]]
+    .isel(init_time=-1, lead_time=slice(0, 24))
+    .sel(latitude=slice(50, 25), longitude=slice(-100, -75))
+)
+ds = await ds.load_async()
+
+con = duckdb.connect()
+xql.register(con, "gfs", ds)
+hottest = con.sql("""
+    SELECT latitude, longitude, max(temperature_2m) AS t2m_max
+    FROM gfs
+    GROUP BY latitude, longitude
+""")
+out = xql.to_dataset(hottest, template=ds)  # dims latitude, longitude
+```
+
+The window is read into memory first because a DuckDB scan cannot wait on
+the network in Pyodide (see the next section). Pyodide also cannot start
+threads, so xarray-sql loads chunks one at a time there. To query a larger
+store, read it through a cache like the demo's.
+
+## How lazy reads work in the browser
+
+Browsers can only read over the network asynchronously. zarr's synchronous
+API waits on those reads with Pyodide's JavaScript Promise Integration
+(JSPI), but JSPI cannot suspend while DuckDB or pyarrow is calling back into
+Python: their C++ frames sit between the suspension point and Python. So a
+query cannot read chunks in the middle of a scan.
+
+The demo's `ChunkCache` runs each query against a cache of decoded chunks
+instead:
+
+1. Run the query. A chunk that is not cached is recorded as missing and
+   scanned as fill values, so one run records every chunk that survives
+   xarray-sql's chunk pruning.
+2. Fetch the missing chunks concurrently with `Dataset.load_async`, then run
+   the query again.
+3. A run with no missing chunks used only real data, so its result is
+   exact.
+
+Most queries need two runs. The stats line under the editor shows the runs,
+chunks, requests, and bytes for each query.
+
+The demo's files are in
+[`docs/demo/`](https://github.com/xqlsystems/xarray-sql/tree/main/docs/demo):
+
+| File | Role |
+|------|------|
+| `index.html` | The page: example queries, SQL editor, results table, and an interactive map ([deck.gl](https://deck.gl) over a [MapLibre](https://maplibre.org) basemap) for `latitude, longitude, value` results. |
+| `worker.js` | Loads Pyodide, installs xarray-sql, DuckDB, and zarr, and runs queries off the main thread. |
+| `app.py` | Opens the dataset, registers it on DuckDB as `gfs`, and runs queries. |
+| `chunk_cache.py` | Lazy reads that work in the browser, as described above. |
+| `fetch_store.py` | A read-only Zarr store that uses `fetch` with `Range` requests. |
+| `smoke.mjs` | Runs every example query headlessly in Node. |
+
+## Run it locally
+
+Serve `docs/demo` over HTTP and open it in Chrome or Edge:
+
+```shell
+python -m http.server 8000 --directory docs/demo
+```
+
+By default the worker installs xarray-sql from PyPI. To test a local build,
+copy the wheel from `pyodide build` (see "Building for WebAssembly" in the
+[contributing guide](contributing.md)) into `docs/demo` and pass it in the
+URL:
+
+```
+http://localhost:8000/?wheel=./xarray_sql-0.5.0-cp310-abi3-pyemscripten_2026_0_wasm32.whl
+```
+
+On this docs page, `?wheel=` is passed through to the embedded demo, so a
+locally built docs site can use a local wheel too (for example,
+`/browser/?wheel=../demo/xarray_sql-...whl` after copying the wheel into the
+built `demo/` directory).
+
+To run the examples without a browser:
+
+```shell
+cd docs/demo
+npm install --no-save pyodide@314.0.7
+node smoke.mjs path/to/xarray_sql-*-pyemscripten_*.whl
+```
+
+## Known limits
+
+- **`lead_time` filters do not prune chunks.** DuckDB does not push
+  `INTERVAL` predicates down to Arrow scans. The demo registers only the
+  first lead-time chunk (forecast hours 0–104), so every query reads at most
+  that one.
+- **Queries that touch the whole grid are slow.** For example,
+  `SELECT max(init_time) FROM gfs` builds every coordinate row (about 440
+  million) even though it downloads nothing. Use the literal timestamps that
+  the example queries use.
+- **The cache holds 768 MiB of decoded chunks.** A query that needs more
+  fails with a message to narrow its filters.

docs/demo/app.py

@@ -0,0 +1,217 @@
+"""The Python half of the browser demo: open a remote Zarr, query it with SQL.
+
+``worker.js`` loads this module into Pyodide and calls :func:`setup` once
+and :func:`run` per query. Both return JSON strings so the worker can hand
+results to the page without converting Python objects.
+"""
+
+from __future__ import annotations
+
+import json
+import math
+from decimal import Decimal
+from typing import Any
+
+import duckdb
+import numpy as np
+import pandas as pd
+import pyarrow as pa
+import xarray as xr
+from chunk_cache import ChunkCache
+from fetch_store import FetchStore
+
+import xarray_sql as xql
+
+GFS_URL = "https://data.dynamical.org/noaa/gfs/forecast/latest.zarr"
+GFS_VARIABLES = [
+    "temperature_2m",
+    "precipitation_surface",
+    "wind_u_10m",
+    "wind_v_10m",
+]
+# The store's inner chunks; one cached chunk is then one range request.
+GFS_CHUNKS = {
+    "init_time": 1,
+    "lead_time": 105,
+    "latitude": 121,
+    "longitude": 121,
+}
+
+_state: dict[str, Any] = {}
+
+
+def _ts(value: Any) -> str:
+    return str(pd.Timestamp(value))[:19]
+
+
+def _examples(ds: xr.Dataset) -> list[dict[str, str]]:
+    latest = _ts(ds.init_time.values[-1])
+    earliest = _ts(ds.init_time.values[0])
+    return [
+        {
+            "title": "San Francisco: 2 m temperature for the next 3 days",
+            "sql": f"""SELECT init_time + lead_time AS valid_time,
+       round(avg(temperature_2m), 1) AS temperature_c
+FROM gfs
+WHERE init_time = TIMESTAMP '{latest}'
+  AND lead_time <= INTERVAL 72 HOUR
+  AND latitude BETWEEN 37.5 AND 38 AND longitude BETWEEN -122.75 AND -122.25
+GROUP BY ALL ORDER BY valid_time""",
+        },
+        {
+            "title": "Map: Europe 2 m temperature, 24 h ahead",
+            "sql": f"""SELECT latitude, longitude, temperature_2m
+FROM gfs
+WHERE init_time = TIMESTAMP '{latest}'
+  AND lead_time = INTERVAL 24 HOUR
+  AND latitude BETWEEN 35 AND 70 AND longitude BETWEEN -10 AND 40""",
+        },
+        {
+            "title": "Map: contiguous US wind speed, 12 h ahead",
+            "sql": f"""SELECT latitude, longitude,
+       round(sqrt(wind_u_10m ^ 2 + wind_v_10m ^ 2), 1) AS wind_speed
+FROM gfs
+WHERE init_time = TIMESTAMP '{latest}'
+  AND lead_time = INTERVAL 12 HOUR
+  AND latitude BETWEEN 24 AND 50 AND longitude BETWEEN -125 AND -66""",
+        },
+        {
+            "title": "Wettest places in the contiguous US, next 24 h",
+            "sql": f"""SELECT latitude, longitude,
+       round(sum(precipitation_surface) * 3600, 1) AS approx_precip_mm
+FROM gfs
+WHERE init_time = TIMESTAMP '{latest}'
+  AND lead_time BETWEEN INTERVAL 1 HOUR AND INTERVAL 24 HOUR
+  AND latitude BETWEEN 24 AND 50 AND longitude BETWEEN -125 AND -66
+GROUP BY ALL ORDER BY approx_precip_mm DESC LIMIT 10""",
+        },
+        {
+            "title": "How the forecast for San Francisco changed across runs",
+            "sql": f"""SELECT init_time,
+       round(max(temperature_2m), 1) AS max_temperature_c
+FROM gfs
+WHERE init_time BETWEEN TIMESTAMP '{earliest}' AND TIMESTAMP '{latest}'
+  AND init_time + lead_time BETWEEN TIMESTAMP '{latest}' + INTERVAL 24 HOUR
+                                AND TIMESTAMP '{latest}' + INTERVAL 48 HOUR
+  AND lead_time <= INTERVAL 72 HOUR
+  AND latitude BETWEEN 37.5 AND 38 AND longitude BETWEEN -122.75 AND -122.25
+GROUP BY ALL ORDER BY init_time""",
+        },
+    ]
+
+
+async def setup(
+    url: str = GFS_URL,
+    variables: list[str] | None = None,
+    runs: int = 4,
+    max_cache_mib: int = 768,
+) -> str:
+    """Open the dataset and register it on a DuckDB connection as ``gfs``."""
+    store = FetchStore(url)
+    ds = xr.open_zarr(store, chunks=None)
+    # DuckDB does not push INTERVAL predicates down to Arrow scans, so a
+    # lead_time filter cannot prune chunks. Keeping only the first lead-time
+    # chunk (forecast hours 0-104) halves what every query downloads.
+    ds = ds[variables or GFS_VARIABLES].isel(
+        init_time=slice(-runs, None),
+        lead_time=slice(0, GFS_CHUNKS["lead_time"]),
+    )
+    cache = ChunkCache(ds, GFS_CHUNKS, max_bytes=max_cache_mib * 2**20)
+    con = duckdb.connect()
+    con.sql("SET enable_progress_bar = false")
+    xql.register(con, "gfs", cache.dataset(), chunks=GFS_CHUNKS)
+    _state.update(store=store, cache=cache, con=con)
+    return json.dumps(
+        {
+            "url": url,
+            "variables": {
+                name: {
+                    "dims": list(var.dims),
+                    "units": var.attrs.get("units", ""),
+                    "long_name": var.attrs.get("long_name", name),
+                }
+                for name, var in ds.data_vars.items()
+            },
+            "sizes": dict(ds.sizes),
+            "init_times": [_ts(t) for t in ds.init_time.values],
+            "examples": _examples(ds),
+        }
+    )
+
+
+def _jsonable(value: Any) -> Any:
+    """A Python value from Arrow, made safe for JSON (which has no NaN)."""
+    if isinstance(value, float):
+        return value if math.isfinite(value) else None
+    if isinstance(value, Decimal):
+        return float(value)
+    # Exactly tuple: Arrow intervals are named tuples, shown as text below.
+    if isinstance(value, list) or type(value) is tuple:
+        return [_jsonable(v) for v in value]
+    if isinstance(value, dict):
+        return {str(k): _jsonable(v) for k, v in value.items()}
+    if value is None or isinstance(value, (str, int)):
+        return value
+    return str(value)  # Timestamps, intervals, bytes, ...
+
+
+def _grid(table: pa.Table) -> dict[str, Any] | None:
+    """Columns for the page's map: latitude, longitude, and one number."""
+    names = table.column_names
+    if "latitude" not in names or "longitude" not in names:
+        return None
+    values = [
+        n
+        for n in names
+        if n not in ("latitude", "longitude")
+        and (
+            pa.types.is_floating(table[n].type)
+            or pa.types.is_integer(table[n].type)
+            or pa.types.is_decimal(table[n].type)
+        )
+    ]
+    if len(values) != 1 or table.num_rows < 4:
+        return None
+    lat = table["latitude"].to_numpy().astype("float64")
+    lon = table["longitude"].to_numpy().astype("float64")
+    # One pixel per point: with several rows per point (e.g. every lead
+    # time), the table is the honest view.
+    if len(np.unique(np.stack([lat, lon]), axis=1)[0]) != table.num_rows:
+        return None
+    value = table[values[0]].to_numpy().astype("float64")
+    finite = np.isfinite(value)
+    if not finite.any():
+        return None
+    return {
+        "name": values[0],
+        "latitude": lat[finite].tolist(),
+        "longitude": lon[finite].tolist(),
+        "value": value[finite].tolist(),
+    }
+
+
+async def run(sql: str, max_rows: int = 500) -> str:
+    """Run ``sql``, fetching whatever chunks it needs, and return the result."""
+    store, cache, con = _state["store"], _state["cache"], _state["con"]
+    requests, nbytes = store.requests, store.bytes_read
+    table, stats = await cache.query(lambda: con.sql(sql).to_arrow_table())
+    shown = table.slice(0, max_rows)
+    return json.dumps(
+        {
+            "columns": shown.column_names,
+            "rows": _jsonable(
+                list(zip(*(shown[c].to_pylist() for c in shown.column_names)))
+            ),
+            "num_rows": table.num_rows,
+            "grid": _grid(table),
+            "stats": {
+                "runs": stats.rounds,
+                "chunks_fetched": stats.chunks_fetched,
+                "requests": store.requests - requests,
+                "mb_downloaded": round((store.bytes_read - nbytes) / 1e6, 1),
+                "seconds_fetching": round(stats.seconds_fetching, 2),
+                "seconds_querying": round(stats.seconds_querying, 2),
+                "cached_chunks": len(cache),
+            },
+        }
+    )