feat: split raw data storage into per-dataset SQLite files

docs/changes/newsfragments/8219.new

@@ -0,0 +1,20 @@
+Added **split raw data storage** for the DataSet: an opt-in results
+backend selected via the ``dataset.raw_data_backend`` config option
+(set it to ``"sqlite_per_dataset_db"``) that writes raw measurement
+data into individual per-dataset SQLite files while keeping all metadata
+in the main database.  This prevents the main DB file from growing
+excessively large and allows users to have a single main SQLite QCoDeS
+database on their PC as opposed to many that grow infinitely.  When
+selected, no results table is created in the main database at all - it
+holds only metadata - and all public ``DataSet`` APIs continue to work
+unchanged.  Backend-specific settings (such as the per-dataset file
+folder ``raw_data_path``) live under ``dataset.raw_data_backend_config``.
+
+Management helpers included:
+
+- ``update_raw_data_paths()``: update stored paths after raw data files
+  have been moved to a new folder.
+- ``purge_orphaned_datasets()``: find and remove dataset records whose
+  raw data files no longer exist on disk (``dry_run=True`` by default).
+- ``cleanup_datasets()``: remove datasets (DB records and raw data files)
+  by age, sample name, or file size (``dry_run=True`` by default).

docs/dataset/dataset_design.rst

@@ -75,3 +75,51 @@ We note that the dataset currently exclusively supports storing data in an
 SQLite database. This is not an intrinsic limitation of the dataset and
 measurement layer. It is possible that at a future state support for writing
 to a different backend will be added.
+
+.. _sec:design_split_storage:
+
+Split Raw Data Storage
+======================
+
+As the main SQLite database grows with many datasets, managing the database
+file can become inconvenient due to the file size. To address this,
+QCoDeS supports an optional **split raw data storage** mode (see
+:ref:`sec:intro_split_raw_data` for user-facing details).
+
+From a design perspective, this feature adds a pluggable *results backend*
+inside the ``DataSet`` class without changing any public interfaces:
+
+- A ``ResultsBackend`` strategy (in ``qcodes.dataset._results_backend``)
+  encapsulates where and how a dataset's results table is stored. The default
+  ``MainDatabaseResultsBackend`` keeps results in the main database;
+  ``SeparateSqliteFileResultsBackend`` writes them to a per-dataset SQLite file.
+  Backends are registered by their ``backend_name`` (the ``dataset.raw_data_backend``
+  config value); the backend is selected in ``DataSet.__init__`` from that config
+  for new runs, and from the run's recorded state for existing ones. Adding a
+  backend is a matter of registering a new ``ResultsBackend`` subclass and a
+  ``raw_data_backend`` / ``raw_data_backend_config`` entry.
+- A ``_results_conn`` property returns the backend's connection -- the main
+  database connection by default, or a per-dataset raw data connection when
+  results are stored separately.
+- Write paths (``add_results``, ``_BackgroundWriter``) and read paths
+  (``get_parameter_data``, ``DataSetCacheWithDBBackend``, ``number_of_results``,
+  ``__len__``) all go through the backend / ``_results_conn``.
+- The per-dataset SQLite file is a lightweight database containing only the
+  results table and numpy type adapters -- no QCoDeS metadata schema.
+- When raw data storage is enabled, **no results table is created in the main
+  database** at all -- only the run metadata is stored there. This mirrors how
+  ``DataSetInMem`` records runs. A run is identified as a split-storage dataset
+  by the ``raw_data_db_path`` column in the ``runs`` table (recorded at dataset
+  creation), which is used both to reconnect to the raw data file and to
+  distinguish such runs from ``DataSetInMem`` runs (which also have no results
+  table). This column is an internal storage detail and is kept out of the
+  user-facing metadata.
+- Subscriber triggers (used for real-time data callbacks) are created on the
+  results connection. Because the results table only exists once the dataset is
+  started, subscriptions requested before then are deferred and materialised at
+  start time.
+
+The implementation is contained in ``qcodes.dataset._raw_data_storage`` (helper
+functions) and a handful of additions to ``qcodes.dataset.data_set`` (routing
+logic). The ``Measurement`` context manager, ``DataSaver``, and all export
+functions work without modification.

docs/dataset/introduction.rst

@@ -75,3 +75,52 @@ For dataset operations, QCoDeS provides functions for:
 - **Exporting datasets**: :doc:`Exporting data to other file formats <../examples/DataSet/Exporting-data-to-other-file-formats>`
 - **Extracting runs between databases**: :doc:`Extracting runs from one DB file to another <../examples/DataSet/Extracting-runs-from-one-DB-file-to-another>` and :func:`qcodes.dataset.extract_runs_into_db`
 - **Bulk export and metadata-only databases**: :func:`qcodes.dataset.export_datasets_and_create_metadata_db` for creating lightweight metadata-only databases while exporting all data to NetCDF files
+
+.. _sec:intro_split_raw_data:
+
+Split Raw Data Storage
+======================
+
+By default, all measurement data (the results table rows) is stored in the same SQLite database alongside metadata such as experiments, runs, parameters and their dependencies. Over time, the main database file can grow very large, which can slow down operations like browsing experiments and make managing a large database file inconvenient.
+
+QCoDeS supports an optional **split raw data storage** mode in which the actual measurement data for each ``DataSet`` is written to an individual, per-dataset SQLite file while all metadata remains in the main database. Each per-dataset file is named after the dataset's GUID (e.g. ``<guid>.db``) and is stored in a configurable folder.
+
+Which backend stores a dataset's results is selected by the ``dataset.raw_data_backend`` configuration option, with backend-specific settings under ``dataset.raw_data_backend_config`` (see the :doc:`Configuring QCoDeS notebook <../examples/basic_examples/Configuring_QCoDeS>` for how to view and change QCoDeS configuration):
+
+- ``dataset.raw_data_backend`` (string, default ``"sqlite_main_db"``): the results backend. ``"sqlite_main_db"`` keeps results in the main database; ``"sqlite_per_dataset_db"`` enables split storage.
+- ``dataset.raw_data_backend_config.sqlite_per_dataset_db.raw_data_path`` (string, default ``"{db_location}"``): the folder where per-dataset files are created. The ``{db_location}`` placeholder is expanded to a folder derived from the main database path (e.g. ``~/experiments.db`` becomes ``~/experiments_db/``).
+
+When ``"sqlite_per_dataset_db"`` is selected:
+
+- No results table is created in the main database; it stores only the run metadata, keeping it lightweight.
+- All ``INSERT`` and ``SELECT`` operations on results data are transparently routed to the per-dataset file.
+- The path to the per-dataset file is recorded in the ``runs`` table, so ``load_by_id`` and related loading functions automatically reconnect to the correct file.
+- All public ``DataSet`` APIs (``get_parameter_data``, ``to_pandas_dataframe``, ``to_xarray_dataset``, ``cache``, ``export``, etc.) work identically whichever backend is selected.
+
+Example runtime configuration::
+
+    import qcodes as qc
+
+    qc.config.dataset.raw_data_backend = "sqlite_per_dataset_db"
+    qc.config.dataset.raw_data_backend_config.sqlite_per_dataset_db.raw_data_path = "/data/raw_measurements/"
+
+If the per-dataset raw data files are moved to a different folder (e.g. during data migration or archival), the stored paths in the main database will become stale. Use the :func:`~qcodes.dataset.update_raw_data_paths` helper to update them::
+
+    from qcodes.dataset import update_raw_data_paths
+
+    update_raw_data_paths(
+        db_path="/path/to/main_database.db",
+        new_raw_data_folder="/new/location/of/raw_files/"
+    )
+
+This scans all datasets with the ``raw_data_db_path`` internal ``runs`` column set, checks whether the corresponding ``.db`` file exists in the new folder, and updates the stored path accordingly.
+
+Managing Datasets
+-----------------
+
+QCoDeS provides helpers for managing per-dataset raw data files over time:
+
+- :func:`~qcodes.dataset.purge_orphaned_datasets` — remove dataset records whose raw data files no longer exist on disk.
+- :func:`~qcodes.dataset.cleanup_datasets` — remove datasets (DB records and raw data files) by age, sample name, or file size.
+
+For usage examples and more details on database management, see the :doc:`Database notebook <../examples/DataSet/Database>`.

docs/examples/DataSet/Database.ipynb

@@ -167,6 +167,143 @@
     "\n",
     "Moreover, we have also written an [example notebook](Extracting-runs-from-one-DB-file-to-another.ipynb) of transferring `DataSets` between database flies that may serve as a template for more complex data organization protocols."
    ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Split Raw Data Storage\n",
+    "\n",
+    "As the main database grows with many datasets, browsing experiments and loading metadata can become slower. QCoDeS supports an optional **split raw data storage** mode that writes the raw measurement data for each dataset into its own individual SQLite file, while keeping all metadata (experiments, runs, parameters, dependencies) in the main database.\n",
+    "\n",
+    "This keeps the main database lightweight and makes it faster to work with, while still allowing all existing `DataSet` APIs to function identically.\n",
+    "\n",
+    "### Configuration\n",
+    "\n",
+    "Split raw data storage is selected via the `dataset.raw_data_backend` option, with backend-specific settings under `dataset.raw_data_backend_config`:\n",
+    "\n",
+    "- `dataset.raw_data_backend` (string, default `\"sqlite_main_db\"`): the results backend. Set to `\"sqlite_per_dataset_db\"` to enable split storage.\n",
+    "- `dataset.raw_data_backend_config.sqlite_per_dataset_db.raw_data_path` (string, default `\"{db_location}\"`): the folder where per-dataset SQLite files are created. The `{db_location}` placeholder expands to a folder derived from the main database path (e.g. `~/experiments.db` becomes `~/experiments_db/`).\n",
+    "\n",
+    "You can enable it at runtime:"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# Enable split raw data storage\n",
+    "qc.config.dataset.raw_data_backend = \"sqlite_per_dataset_db\"\n",
+    "\n",
+    "# Optionally set a custom path for per-dataset files\n",
+    "qc.config.dataset.raw_data_backend_config.sqlite_per_dataset_db.raw_data_path = (\n",
+    "    \"/data/raw_measurements/\"\n",
+    ")\n",
+    "\n",
+    "# Or use the default which derives from the main DB location:\n",
+    "# qc.config.dataset.raw_data_backend_config.sqlite_per_dataset_db.raw_data_path = \"{db_location}\"\n",
+    "# e.g. ~/experiments.db -> ~/experiments_db/"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "To make split storage the default for future sessions, set the same options through the `qc.config` Python interface shown above and persist your configuration - see [Configuring QCoDeS](../basic_examples/Configuring_QCoDeS.ipynb).\n",
+    "\n",
+    "### How It Works\n",
+    "\n",
+    "When split storage is enabled:\n",
+    "\n",
+    "1. When a measurement starts (`mark_started()`), a per-dataset SQLite file named `<guid>.db` is created in the configured folder.\n",
+    "2. All measurement data (results table rows) is written to this per-dataset file instead of the main database.\n",
+    "3. No results table is created in the main database for such datasets; only the run metadata (experiments, runs, parameters, dependencies) is kept there, keeping the main database small.\n",
+    "4. The path to the per-dataset file is saved in a dedicated internal `raw_data_db_path` column of the `runs` table (not exposed via `DataSet.metadata`), so `load_by_id()` and related functions automatically find and reconnect to the correct file.\n",
+    "5. All `DataSet` methods (`get_parameter_data`, `to_pandas_dataframe`, `to_xarray_dataset`, `cache`, `export`, etc.) work transparently with split storage.\n",
+    "\n",
+    "> **Note:** Datasets created with split storage enabled can always be loaded later, even if the configuration is changed back to the default, as long as the per-dataset files remain at their original paths.\n",
+    "\n",
+    "### Updating Paths After Moving Raw Data Files\n",
+    "\n",
+    "If you move the per-dataset raw data files to a different folder, the paths stored in the main database become stale. Use `update_raw_data_paths` to fix them:"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# from qcodes.dataset import update_raw_data_paths\n",
+    "\n",
+    "# After moving raw data files to a new folder:\n",
+    "# update_raw_data_paths(\n",
+    "#     db_path=\"/path/to/main_database.db\",\n",
+    "#     new_raw_data_folder=\"/new/location/of/raw_files/\"\n",
+    "# )"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "The function scans all datasets that have the internal `raw_data_db_path` column set, checks whether the corresponding `.db` file exists in the new folder, and updates the stored path. Datasets whose files are not found in the new folder are skipped with a warning."
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "### Purging Orphaned Datasets\n",
+    "\n",
+    "After archiving or deleting selected raw data files, some dataset records in the main database may reference files that no longer exist. Use `purge_orphaned_datasets` to clean up those orphaned records:"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# from qcodes.dataset import purge_orphaned_datasets\n",
+    "\n",
+    "# Dry run (default) — see what would be removed:\n",
+    "# result = purge_orphaned_datasets(\"/path/to/main_database.db\")\n",
+    "# print(f\"{len(result.orphaned_datasets)} orphaned datasets found\")\n",
+    "\n",
+    "# Actually remove orphaned records:\n",
+    "# result = purge_orphaned_datasets(\"/path/to/main_database.db\", dry_run=False)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "### Cleaning Up Datasets by Criteria\n",
+    "\n",
+    "To free disk space by removing datasets (both raw data files and DB records) based on age, sample name, or file size, use `cleanup_datasets`. Criteria are combined with AND logic — a dataset must match **all** specified criteria to be selected."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# from qcodes.dataset import cleanup_datasets\n",
+    "\n",
+    "# Remove datasets older than 30 days (dry run first):\n",
+    "# result = cleanup_datasets(\"/path/to/db.db\", older_than_days=30)\n",
+    "# print(f\"{len(result.matching_datasets)} datasets would be removed\")\n",
+    "\n",
+    "# Remove datasets for a specific sample:\n",
+    "# result = cleanup_datasets(\"/path/to/db.db\", sample_name=\"old-sample\", dry_run=False)\n",
+    "\n",
+    "# Remove datasets with raw data files larger than 500 MB:\n",
+    "# result = cleanup_datasets(\"/path/to/db.db\", larger_than_mb=500, dry_run=False)"
+   ]
   }
  ],
  "metadata": {

src/qcodes/configuration/qcodesrc.json

@@ -79,7 +79,13 @@
         "export_chunked_export_of_large_files_enabled": false,
         "export_chunked_threshold": 1000,
         "in_memory_cache": true,
-        "load_from_exported_file": false
+        "load_from_exported_file": false,
+        "raw_data_backend": "sqlite_main_db",
+        "raw_data_backend_config": {
+            "sqlite_per_dataset_db": {
+                "raw_data_path": "{db_location}"
+            }
+        }
     },
     "telemetry":
     {

src/qcodes/configuration/qcodesrc_schema.json

@@ -382,6 +382,35 @@
                     "type": "boolean",
                     "default": true,
                     "description": "Should the data be cached in memory as it is measured. Useful to disable for large datasets to save on memory consumption."
+                },
+                "raw_data_backend": {
+                    "type": "string",
+                    "enum": ["sqlite_main_db", "sqlite_per_dataset_db"],
+                    "default": "sqlite_main_db",
+                    "description": "Which backend stores dataset raw measurement data (the results tables). 'sqlite_main_db' (default) keeps the results in the main QCoDeS database. 'sqlite_per_dataset_db' writes each dataset's results to an individual per-dataset SQLite file while all metadata stays in the main database. Backend-specific settings live under raw_data_backend_config."
+                },
+                "raw_data_backend_config": {
+                    "type": "object",
+                    "default": {
+                        "sqlite_per_dataset_db": {
+                            "raw_data_path": "{db_location}"
+                        }
+                    },
+                    "description": "Backend-specific configuration, keyed by the raw_data_backend name. Each backend reads its own entry; add a new key here (and a new raw_data_backend enum value) when adding a new backend.",
+                    "properties": {
+                        "sqlite_per_dataset_db": {
+                            "type": "object",
+                            "description": "Configuration for the 'sqlite_per_dataset_db' results backend.",
+                            "properties": {
+                                "raw_data_path": {
+                                    "type": "string",
+                                    "default": "{db_location}",
+                                    "description": "Path to the folder where per-dataset raw data SQLite files are stored. {db_location} is a directory in the same folder as the .db file with a matching name, e.g. for ~/experiments.db raw data files will be stored in ~/experiments_db/"
+                                }
+                            },
+                            "required": ["raw_data_path"]
+                        }
+                    }
                 }
             },
             "description": "Settings related to the DataSet and Measurement Context manager",

src/qcodes/dataset/__init__.py

@@ -3,6 +3,11 @@
 and from disk
 """
 
+from ._raw_data_storage import (
+    cleanup_datasets,
+    purge_orphaned_datasets,
+    update_raw_data_paths,
+)
 from .data_set import (
     get_guids_by_run_spec,
     load_by_counter,
@@ -86,6 +91,7 @@
     "ThreadPoolParamsCaller",
     "TogetherSweep",
     "call_params_threaded",
+    "cleanup_datasets",
     "connect",
     "datasaver_builder",
     "do0d",
@@ -121,6 +127,8 @@
     "new_experiment",
     "plot_by_id",
     "plot_dataset",
+    "purge_orphaned_datasets",
     "reset_default_experiment_id",
     "rundescriber_from_json",
+    "update_raw_data_paths",
 ]