Skip to content

Filewriter: implement structure #114

Description

@Tom-Willemsen

Implement the initial top-level structure of the filewriter process, to prove that we have a design that will work in practice. This should follow DS ADR 10.

Should include:

  • Trait definitions for writer modules
  • Kafka-listening infrastructure (including pooled architecture)
  • HDF5-writing infrastructure
    • Including writing fully-static datasets/groups
  • Infrastructure for parsing dynamically-provided nexus_structure (part of run start)
    • Including configuration parameters for each writer module
  • Logging
  • Metrics
  • Testing
    • Including unit tests for individual writer modules
  • Benchmarking
    • Include a bench for each writer module
    • Include one overall top-level bench for writing a small but representative file in-memory
  • Docs
  • Error propagation with appropriate context for debugging (likely miette + thiserror)
  • Two different writer modules, which may write simple/incomplete data at this stage.
    • E.g. simplified IXseblock and simplified NXevent_data

Functionally, the end point of this ticket is that we have a filewriter process that can listen to streams of data from Kafka and write some data from those streams, in a simplified/incomplete way, to a HDF5/NeXus file.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

No type

Projects

Milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions