Datasets#

A dataset spec tells modewise where recordings are and how to read them. It is separate from the pipeline, so one pipeline runs on many datasets. It works for BIDS and for other layouts, and for any modality.

root = "/data/ds005237"              # relative paths resolve against this file's folder
filter = { task = ["hammerAP"] }     # optional: keep only these entity values

[[input]]                            # the first input decides which recordings exist
name = "bold"                        # must match a pipeline input
pattern = "sub-{subject}/func/sub-{subject}_task-{task}_run-{run}_bold.h5"
format = "hdf5"
dataset = "timeseries"               # path of the 2D dataset inside the file
layout = "channels_by_time"          # or "time_by_channels"
sidecar = "sub-{subject}/func/sub-{subject}_task-{task}_run-{run}_bold.json"
# sampling_rate_hz = 1.25            # instead of `sidecar`
# labels_file = "channels.txt"       # one channel label per line

[[input]]                            # later inputs must exist for every recording
name = "trials"
pattern = "sub-{subject}/func/sub-{subject}_task-{task}_run-{run}_events.tsv"
format = "events_tsv"
# label_column = "trial_type"        # default

Patterns#

  • Placeholders. {name} stands for an entity value, and the same name must match the same value everywhere in the pattern.

  • Recording IDs are made from the entities, for example subject-01_task-rest_run-01.

  • Labels. Use BIDS-style labels (letters and digits). On case-insensitive disks, the macOS default, sub-A and sub-a are the same folder.

Formats#

format

Gives

Needs

hdf5

series

dataset, plus sidecar or sampling_rate_hz

nifti

series: one channel per atlas label, parcellated at load time (docs/volumes.md)

atlases (paths relative to the spec’s folder); optional standardize; sidecar or sampling_rate_hz, else the header’s TR

events_tsv

events

nothing else

Sampling rate and events#

  • Sidecar: SamplingFrequency (Hz) is read first, otherwise RepetitionTime (s), so the same rule serves EEG, MEG, fNIRS and fMRI.

  • events_tsv follows BIDS. onset and duration are in seconds from the first stored sample. Rows with n/a onset or duration are rejected.

Channels#

A [channels] section says what each channel is, so pipelines can select and summarise regions (docs/regions.md):

[channels]
name = "channels"                                   # pipeline input name (default)
tables = [                                           # in channel order; concatenated
  { path = "atlas/schaefer400_info.txt", format = "schaefer" },
  { path = "atlas/tian_s2.txt", format = "tian" },
]
names_for = ["bold"]                                 # these inputs take the table's names

  • Paths: relative to the dataset spec’s folder, because atlases usually live outside the data.

  • Formats: tsv (covers BIDS dseg.tsv), schaefer, tian and names.

  • Names: inputs in names_for get the table’s names instead of ch0, ch1, …. The number of names must match the channels.

  • Pipeline input: the table becomes one more input (channels = "channels" in a pipeline’s inputs). Pipelines that do not use it are unaffected, because unused dataset inputs are left out.

Participants#

A [participants] section reads subject information for group statistics and machine learning (docs/features.md):

[participants]
entity = "subject"                       # the recording entity tables are keyed by (default)
tables = [
  { path = "participants.tsv", id_pattern = "sub-{subject}" },
  { path = "phenotype/shaps01.tsv", id_column = "subjectkey", id_pattern = "NDAR_{subject}",
    columns = ["shaps_total"], missing = ["999"] },
]

  • Paths: relative to the dataset root, as BIDS places them.

  • Ids: id_column defaults to participant_id. id_pattern says how ids relate to the entity, in the same {entity} syntax as file patterns; by default the id is the entity value.

  • Columns and missing values: columns keeps only the listed columns. missing adds codes meaning “no value”, besides empty cells, n/a and NA.

  • Joins: tables are joined on the subject. A column name that appears in two tables is an error unless one table sets prefix, for example prefix = "shaps.".

Checking a spec#

modewise inspect dataset.toml --load lists the entities found, and loads the first recording to show shapes, sampling rates and event counts.