Implement the initial top-level structure of the filewriter process, to prove that we have a design that will work in practice. This should follow DS ADR 10.
Should include:
- Trait definitions for writer modules
- Kafka-listening infrastructure (including pooled architecture)
- HDF5-writing infrastructure
- Including writing fully-static datasets/groups
- Infrastructure for parsing dynamically-provided
nexus_structure (part of run start)
- Including configuration parameters for each writer module
- Logging
- Metrics
- Testing
- Including unit tests for individual writer modules
- Benchmarking
- Include a bench for each writer module
- Include one overall top-level bench for writing a small but representative file in-memory
- Docs
- Error propagation with appropriate context for debugging (likely miette + thiserror)
- Two different writer modules, which may write simple/incomplete data at this stage.
- E.g. simplified IXseblock and simplified NXevent_data
Functionally, the end point of this ticket is that we have a filewriter process that can listen to streams of data from Kafka and write some data from those streams, in a simplified/incomplete way, to a HDF5/NeXus file.
Implement the initial top-level structure of the filewriter process, to prove that we have a design that will work in practice. This should follow DS ADR 10.
Should include:
nexus_structure(part of run start)Functionally, the end point of this ticket is that we have a filewriter process that can listen to streams of data from Kafka and write some data from those streams, in a simplified/incomplete way, to a HDF5/NeXus file.