Repository navigation
Feature: Bulk Metadata Exports (Export Subsystem Enhancement Part 2) - #12737
Draft
poikilotherm wants to merge 21 commits into
Draft
poikilotherm wants to merge 21 commits into
poikilotherm wants to merge 21 commits into
Conversation
poikilotherm
added
Feature: Metadata
Size: 10
Closed
|
coverage: 25.892% (+0.08%) from 25.809% — poikilotherm:11892-bulk-export-update into IQSS:12686-enhance-export-subsys |
poikilotherm
changed the base branch from
develop
to
12686-enhance-export-subsys
September 23, 2026 08:26
poikilotherm
changed the base branch from
12686-enhance-export-subsys
to
develop
September 23, 2026 08:27
poikilotherm
force-pushed
the
11892-bulk-export-update
branch
from
September 23, 2026 08:45
e60691a to
ba8d09a
Compare
poikilotherm
changed the base branch from
develop
to
12686-enhance-export-subsys
September 23, 2026 08:45
cmbz
added
FY27 Sprint 6
poikilotherm
force-pushed
the
11892-bulk-export-update
branch
6 times, most recently
from
October 1, 2026 18:34
09fa9f8 to
bcf6b9b
Compare
1 of 4 tasks
… less messy in `JsonResponseBuilder`
…ral's snapshot repo IQSS#11892
…an to reusable method IQSS#11892
…s() IQSS#11892 Validates both existence of an exporter for the target format and implementation of a given plugin contract interface. Mostly used to determine if a capability is provided for the given format. For example, if the format's exporter support combining multiple dataset versions into a single output. This commit includes tests for the new method and a small fixture to test without mocking.
…esources Introduced a thread-safe `StreamRegistry` utility for managing `Closeable` resources such as streams. Ensures proper cleanup of resources, even if consumers forget to close them. Includes comprehensive unit tests to verify behavior in various scenarios and edge cases.
… management IQSS#11892 Added `BulkExportPipeline`, a new core provider implementation of the SPI `BulkDatasetContext`, to manage batch processing of dataset versions during bulk exports. Supports efficient handling of streams, ensures resource cleanup via `StreamRegistry`, and provides thread-safe operations. Includes extensive unit tests to validate functionality.
…rtPipelineTest IQSS#11892 Just a styling thing. Nicer to read the different arguments being null tested in one test instead of three. Cannot easily be parameterized due to the resolver being built per execution, but parameter providing method needs to be static.
Implemented `getExportInNewTransaction()` in `ExportServiceBean` to enable dataset version export in a dedicated transaction. Background: in many cases we may need to create either a new transaction because none existed or we want to detach from an existing one, allocating dedicated resources (like a DB connection) for an export operation. This will be especially important for bulk export operations. Added unit tests to validate version lookup, error handling, and export logic. Updated mock setup to integrate `DatasetVersionServiceBean`.
…Bean Added `bulkExport()` to `ExportServiceBean` for efficient handling of bulk dataset version exports. Utilizes lazy context resolution and per-transaction item processing to optimize resource usage. Ensures stream-based export with fail-safe mechanisms, including error isolation for individual items. Includes extensive unit tests for validation.
…ne ConstraintViolationException handling Introduced `violations()` method in `JsonResponseBuilder` to handle structured constraint violations. Refactored `ConstraintViolationExceptionHandler` for better clarity and use of the new utility.
…dling In case an invalid JSON document or a non-bindable JSON document is sent, a ProcessingException wraps any JSON exceptions. Unpacking in the handler to extract a more helpful error message. Unfortunately, JSON-B exceptions are not as granular as Bean Validation exceptions, thus only a simple message can be returned. Also adding JsonbException to the JSON exception handler in case it's ever thrown and not wrapped.
…requests IQSS#11892 Introduced MultiDatasetExportRequest DTO to handle APIs supporting bulk dataset version exports. Defined validation rules for exporter and dataset entries, including symbolic and numeric dataset versions. Added unit tests for request validation scenarios.
Added support for `BulkDatasetExporter` interface in `JSONExporter`. Implemented `exportBulk()` to handle JSON-based bulk dataset exports with stream-based processing and error handling.
…eadability Actual parameters from the query should go first, also making the descriptions be read first.
Added datasets/export POST REST API endpoint to support bulk dataset version exports. Validates exporters, resolves datasets/versions, and processes requests with error aggregation.
…1892 Introduced configurable upper limit for bulk export request size via JVM settings. Added validation to prevent excessive dataset version requests, with a default maximum of 32. Updated validation error messaging for clarity.
…ion #IQSS#11892 Improved the `Schema` documentation for `exporter` and `datasets` fields in `MultiDatasetExportRequest` to add descriptions, examples, and clarify requirements for API consumers.
…#11892 Documented the new bulk dataset export API, its capabilities, HTTP endpoint, error handling, system configuration, and plugin developer updates. Clarified export format support and introduced detailed guidance for end-users and administrators.
…eam handling IQSS#11892 Documented new settings for bulk metadata export request size limits and introduced guidelines for plugin developers on safe stream handling using StreamRegistry. Updated API guide with details for the bulk metadata export endpoint, including usage, error handling, and CORS considerations.
poikilotherm
force-pushed
the
11892-bulk-export-update
branch
from
October 9, 2026 00:02
bcf6b9b to
8394fa6
Compare
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this PR does / why we need it:
Introducing a new API endpoint POST /api/datasets/export accepting a JSON shaped request to export many dataset versions at once.
Which issue(s) this PR closes:
Special notes for your reviewer:
Suggestions on how to test this:
Extensive tests have been added. I'm open for any suggestions what else to test.
Does this PR introduce a user interface change? If mockups are available, please link/include them here:
API users only for now. Might become relevant for Modern/SPA UI
Is there a release notes update needed for this change?:
Will be included
Additional documentation:
None yet