Dataset templates

Every task tracebloc supports has a dataset template: the exact folder layout the data ingestor expects, a labels or data CSV in the right shape, a ready-to-edit ingest.yaml, and the list of checks the ingestor runs before a single row is stored. Pick your task below, lay your data out the same way, and follow Prepare Data to run the ingest.

Templates by task

Data	Task	Template
Image	Image classification	Layout and config
Image	Object detection	Layout and config
Image	Keypoint detection	Layout and config
Image	Semantic segmentation	Layout and config
Text	Text classification	Layout and config
Text	Token classification	Layout and config
Text	Sentence-pair classification	Layout and config
Text	Masked language modeling	Layout and config
Text	Causal language modeling	Layout and config
Text	Sequence-to-sequence	Layout and config
Text	Embeddings	Layout and config
Tabular	Tabular classification	Layout and config
Tabular	Tabular regression	Layout and config
Time series	Time-series forecasting	Layout and config
Time series	Time-series classification	Layout and config
Time series	Survival analysis (time-to-event)	Layout and config

How the templates fit together

Every template follows the same three steps. The task pages only spell out what differs.

Stage the files on the shared data volume of your secure environment. Inside the ingestor they appear under /data/shared/; how to get them there is covered in Prepare Data.
Write ingest.yaml from the task page. The top of the file is identical for every task; the category field names the task and decides which folders, columns and checks apply.
Run the ingest once per split — once with intent: train and once with intent: test, each with its own table name:

helm install <release-name> tracebloc/ingestor \
  --namespace <namespace> \
  --set-file ingestConfig=./ingest.yaml

The ingestor validates the whole dataset first, then copies files, stores rows and registers only metadata with the platform. Raw data never leaves your infrastructure. If any check fails, nothing is stored.

Where files must live

For tasks with one file per sample (images and text), the ingestor reads every file from a fixed subfolder name next to the others: images/, annotations/, masks/, texts/ or sequences/. It derives the dataset root from the folder you name in ingest.yaml (the parent of images:, texts:, and so on), so keep the subfolders side by side and spelled exactly like that:

/data/shared/<dataset>/
├── labels.csv        # the manifest (any name; point `csv:` at it)
├── images/           # or texts/ or sequences/ — the per-sample files
└── annotations/      # only object detection (masks/ only semantic segmentation)

Tabular and time-series tasks have no per-sample files: the CSV named in csv: is the whole dataset.

The `ingest.yaml` contract

These fields are shared by every task. Task-specific fields (images, texts, schema, target_size, …) are explained on the task pages.

Field	Required	Meaning
`apiVersion`	yes	Always `tracebloc.io/v1`.
`kind`	yes	Always `IngestConfig`.
`category`	yes	The task. One of the 16 identifiers in the table above (for example `image_classification`).
`table`	yes	Name of the dataset table in your secure environment. Letters, digits and underscores only; must start with a letter or underscore. Use a different name for the train and test splits, and a new name for every new version — the ingestor refuses to write into a table folder that already holds data.
`intent`	yes	`train` or `test`.
`csv`	one of `csv` / `json`	Path (inside the ingestor) to the labels CSV or, for tabular tasks, the data CSV. Object detection has neither — see its page.
`json`	one of `csv` / `json`	Path to a JSON manifest (a top-level array of records, or one object) instead of a CSV.
`label`	depends on task	Either the column name (`label: label`) or an object `{column, policy}`. Regression-class tasks require the object form with an explicit `policy` (`bucket` or `passthrough` — see Label policy); self-supervised text tasks must not set it at all. See each task page.
`schema`	tabular, time-series, semantic segmentation	Map of column name to SQL type. See SQL types.
`data_id`	no	How each stored row gets its id. `strategy: content_hash` (default) hashes the row content with a salt that never leaves your secure environment, so a retried run re-uses its rows instead of duplicating them; identical source rows collapse into one. `strategy: uuid` gives every row a fresh id. `strategy: column` with `column: <name>` copies a column of yours — only safe when that column carries no personal data.
`spec.csv_options`	no	`chunk_size` (default 1000), `delimiter` (`,`), `quotechar` (`"`), `escapechar` (`\`). Files are always read as UTF-8.
`spec.file_options`	no	`extension` (one of `.jpg`, `.jpeg`, `.png`, `.txt`, `.text`, `.xml`), `target_size` (`[width, height]`), `min_size` (`[width, height]`, default `[32, 32]`). Per-task defaults are listed on each page.
`columns`	no	Per-column facts the platform cannot infer, used when datasets are combined: `unit` (for example `years`, `USD`) and `ordinal` (category values in order, low to high).
`color_mode`, `bit_depth`	no	Image tasks: `RGB` or `grayscale`; `8` or `16`.
`language`, `normalization`	no	Text tasks: the dataset language (for example `en`) and the text normalization you applied (for example `lowercase`).
`time_unit`, `event_indicator`	no	Survival analysis: `days`, `weeks`, `months` or `years`, and the integer codes for an observed event and a censored case.
`positive_definition`	no	Embeddings: what counts as a positive pair.
`spec.validators`, `spec.sidecars`, `spec.processors`	no	Accepted by the schema but not executed by the current ingestor — it logs a warning and continues with the task defaults. Leave them out.

Rules that apply to every CSV

The file must be valid UTF-8 without NUL bytes. Excel users: save as CSV UTF-8.
Surrounding whitespace in header names is stripped; duplicate header names are rejected.
The label column you configure is matched case- and whitespace-insensitively (Label satisfies label: label).
Tasks with one file per sample need a column named exactly filename (lowercase). Its value may include the file extension or not: cat1.jpeg and cat1 both resolve to images/cat1.jpeg when the configured extension is .jpeg. A value that already ends in .jpeg, .jpg, .png, .xml, .txt or .text (any case) is used as is.
The manifest must contain at least one data row, and at least one referenced file must exist.
String labels have surrounding whitespace stripped before storage. Only a label that is missing from the record is stored as NULL; an empty or whitespace-only label is stored as the empty string "". The NA tokens below apply to schema columns only — in a label column that is not declared in schema, an empty cell is "" and NA or null is a genuine class value.
The id column in the sample CSVs is not stored unless you declare it in schema — and you cannot declare it under that name. These column names are reserved by the ingestor and must not appear in schema: id, created_at, updated_at, status, data_intent, data_id, filename, extension, annotation, ingestor_id. The label column is not declared in schema either — label: names it.
Column names may be at most 64 characters. Any other character is allowed.

Checks every ingest runs

Task-specific checks are listed on each page. These run for every task:

Check	What it rejects
Ingestable records	A CSV with a header but no rows; a per-sample-file task whose manifest lacks an exact `filename` column, or whose referenced files are all missing.
Label diversity	A classification dataset with fewer than 2 distinct label values (after whitespace stripping). Not run for regression-class or self-supervised tasks.
Table name	A `table` outside `^[a-zA-Z_][a-zA-Z0-9_]*$`. Common SQL keywords produce a warning.
Duplicate	A destination folder for this `table` that already contains data. Duplicate `filename` values within the manifest produce a warning.

Label policy for regression-class tasks

Tabular regression, time-series forecasting and survival analysis have a numeric target, and the platform must never receive raw target values. Their label therefore needs the object form with an explicit policy:

label:
  column: price
  policy: bucket

bucket replaces each value with one of 64 stable hash buckets in the metadata sent to the platform only. The rows stored in your secure environment keep the raw value, which is what training reads. A missing target is reported as bucket -1. passthrough sends raw values and is only appropriate when you have cleared that with your compliance owner.

SQL types for `schema`

VARCHAR(n), CHAR(n), TEXT, INT, INTEGER, TINYINT, SMALLINT, MEDIUMINT, BIGINT, FLOAT, DOUBLE, DECIMAL(p,s), NUMERIC(p,s), BOOLEAN, BOOL, DATE, DATETIME, TIMESTAMP, TIME, BLOB, LONGBLOB. Type names are case-insensitive. A near-miss (INTERGER) is rejected with a “did you mean” hint. Every declared column is checked against its type across the whole file: non-numeric values in numeric columns, values over a VARCHAR length, inf, integers outside the 64-bit range and unparseable dates all fail the ingest. In schema columns, empty cells and exactly these tokens are stored as NULL: NA, N/A, n/a, NULL, null, None, none, NaN, nan, <NA>, #N/A. The match is case-sensitive — Null, NONE, NAN or #n/a are ordinary values and fail the type check in a numeric column.

Next steps

Run the ingest: Prepare Data
Define and publish your use case: Define Use Case

Need help?

Email [email protected].

Overview

Create a Use Case

Join a Use Case

Environment Setup

Tools & Help

Templates by task

How the templates fit together

Where files must live

The `ingest.yaml` contract

Rules that apply to every CSV

Checks every ingest runs

Label policy for regression-class tasks

SQL types for `schema`

Next steps

Need help?

​Templates by task

​How the templates fit together

​Where files must live

​The ingest.yaml contract

​Rules that apply to every CSV

​Checks every ingest runs

​Label policy for regression-class tasks

​SQL types for schema

​Next steps

​Need help?

Templates by task

How the templates fit together

Where files must live

The `ingest.yaml` contract

Rules that apply to every CSV

Checks every ingest runs

Label policy for regression-class tasks

SQL types for `schema`

Next steps

Need help?