How derived products are stored¶

A derived product is an array that a component computes from a run and keeps next to the data it came from, such as a median over a scan or a filtered copy of every frame. The device and its service write the run's own data; a derived product is the one thing redsun writes, through a Writer. Write a derived product shows every step.

What a writer receives¶

A Writer takes two kinds of input, from two directions. Step through to see them reach it:

What reaches a writer

The camera and its service write the frames. The writer gets the documents and the product from a component.

The component receives the run's documents like any document callback, and computes its product from the frames as they arrive.

It forwards each document to the writer, which learns where the frames went and what shape they have.

The documents can't carry the product, so the component hands the data over too, with write, or with append frame by frame.

The writer puts the product next to the frames, in the store the device chose.

Here is a component that keeps the median of a camera's frames, cut down to the lines that talk to its writer:

from typing import Any

from event_model import DocumentRouter, RunStop

from redsun.writers import Writer


class MedianPresenter(DocumentRouter):
    def __init__(self, name: str) -> None:
        super().__init__()
        self.name = name
        self._writer = Writer()
        self._writer.derive("camera_median", source="camera")

    def __call__(self, name: str, doc: dict[str, Any], validate: bool = False) -> Any:
        result = super().__call__(name, doc, validate)
        self._writer(name, doc)
        return result

    def stop(self, doc: RunStop) -> RunStop:
        self._writer.write("camera_median", self._median)
        return doc

self._median stands for what the component computed while the run went on; a product written frame by frame goes in through append instead. The session never subscribes a writer itself. The component forwards each document after handling it, so the writer sees the run's stop only after the component has written its result.

Where a product is stored¶

A product goes where its source data went, following the device's choice of format and store (ADR 0013). derive("camera_median", source="camera") lays the median out like one camera frame in the camera's store, and two documents of the run fill in the rest:

What derive takes from the run

derive names only the product and its source. Two documents of the run fill in the rest.

The descriptor gives the shape and dtype of the camera's key. derive drops its leading frames-per-event dimension, so the product is laid out like one frame.

The stream_resource gives the store's URI and format. The writer knows plain Zarr and OME-Zarr, and skips the product for any other format, logging it once.

For a product the run says nothing about, declare gives the shape, the type and the store up front instead:

self._writer.declare("mask", shape=(512, 512), dtype="uint8", store=store_uri)

Declaring products up front¶

Both derive and declare belong in the constructor, before the run starts, because the stores can't grow new arrays later. Step through a run to see when each part happens:

A product over one run

derive and declare go in the constructor, before the run starts.

The component forwards the run's first documents. The writer learns each product's layout and store, but opens nothing yet.

The first append or write opens the store's stream with every product known by then. acquire-zarr sizes the arrays once, at this moment, and ome-writers allocates every frame of an image.

A product declared after the stream opened can't join it, so append raises WriterError.

At the run's stop document, the writer closes the stream and writes the metadata: the mapping given to write, and one naming the run, the source, the store and the time.

A product can't join a store whose stream is open. For the same reason, a product that gets a store of its own, beside an OME-Zarr image, is written all at once rather than frame by frame: write opens and finishes that store in one call, and append to it is refused.