Datasets
The layout KonfAI reads and the formats it reads regionally.
The layout
One folder per case, one file per group, under a root named in dataset_filenames:
Dataset/
├── CASE_001/
│ ├── CT.mha
│ └── SEG.mha
└── CASE_002/
├── CT.mha
└── SEG.mhaA DICOM series is a folder (CASE_001/CT/*.dcm), an OME-Zarr store a folder too (CASE_001/CT.ome.zarr/). A folder of files named by case (scans/case_001.nii.gz) is not a dataset: give each case its own folder, symlinks are fine. Cases that do not follow the layout are named by a manifest (./cases.yml:omezarr).
Formats
The format token of dataset_filenames picks the backend that reads and writes the data:
| Backend | Formats | Layout | Region reads | Extra |
|---|---|---|---|---|
SitkFile | mha, mhd, nii, nrrd, gipl, tif (each also .gz), hdr, dcm, png, npy, … | one file per case and group (the default) | MetaImage, NIfTI and NRRD | konfai[itk] |
H5File | h5 | one HDF5 file for the whole dataset | yes | konfai[hdf5] |
OmeZarrFile | omezarr (zarr), omezarr@<level> | one store per case and group | yes, pyramids included | konfai[omezarr] |
DicomFile | dicom | one series folder per case and group | slice by slice | konfai[dicom] |
ItkTransformFile | itktransform | one ITK transform file per case | for displacement fields | konfai[itk], konfai[hdf5] |
konfai[imaging] installs all four storage backends at once. A dataset root in object storage (s3://) is read with konfai[s3]; only a store is read in place, a whole-file format there is copied first.
Regional reads and large images
Dataset patches are read regionally from the stores above when the preprocessing is stream-compatible; a bounded buffer preserves correctness when it is not. That is what lets a training read its patches, a prediction reassemble its outputs and a TRANSFORM process a cohort slab by slab without holding whole volumes in RAM. The guide to images larger than RAM: Working out of core; every format and its flags: Storage formats.