Causal language modeling

Predict the next word. This task is self-supervised: there is no label. Each sample is one .txt file of raw text in one of two shapes:

Pretraining — the whole file is plain text.
Instruction tuning (SFT) — one line of the form prompt<TAB>completion. Everything before the first tab is the prompt, everything after is the completion.

A dataset may mix both shapes. The ingestor does not enforce either shape; it only checks that files are valid text.

Folder layout

/data/shared/dolly-clm/
├── labels.csv
└── texts/
    ├── clm_0000001.txt      # plain text
    ├── clm_0000002.txt      # plain text
    ├── clm_0000004.txt      # prompt<TAB>completion
    └── ...

The folder is named texts (raw text) and sits next to the manifest CSV.
UTF-8 encoded. Every file must end in .txt. The ingestor also accepts .text, but training opens only <filename>.txt, so a .text dataset ingests and then fails to train.

Manifest CSV

filename
clm_0000001
clm_0000002
clm_0000003

Column	Required	Meaning
`filename`	yes, exactly this name	The text file, with or without extension.
`extension`	no	Present in the shipped sample (`'.txt'`); not read by the ingestor.

The task reads no label column. Setting label: in ingest.yaml is rejected. A label column in the manifest itself is not rejected: the ingest goes on without it and warns: “Column(s) [‘label’] in the manifest are ignored for causal_language_modeling and will not be stored”. The same goes for any other column besides filename and extension. Remove such columns to silence the warning.

Ingest with the CLI

Each split is one folder laid out as above. Ingest the training and the test split separately:

tracebloc data ingest ./dolly-clm-train --task causal_language_modeling --name dolly_clm_train --intent train --holds open_data
tracebloc data ingest ./dolly-clm-test  --task causal_language_modeling --name dolly_clm_test  --intent test --holds open_data

--holds open_data marks the dataset as open data. If your data is about patients, pass --holds patient_data; if it was generated, pass --holds synthetic_data. Leave --holds out on a terminal and the CLI asks; in a script, the dataset is marked as patient data. The mark decides which use case types you can create on the dataset. Do not pass --label-column: the CLI rejects it for this task because it trains on the text itself.

ingest.yaml

apiVersion: tracebloc.io/v1
kind: IngestConfig
category: causal_language_modeling
table: dolly_clm_train
intent: train
csv: /data/shared/dolly-clm/labels.csv
texts: /data/shared/dolly-clm/texts/

Field	Required	Meaning
`csv`	yes	Path to the manifest CSV.
`texts`	yes	The `texts/` folder.
`label`	must not be set	Self-supervised task.
`schema`	no	Extra typed columns only. Never a reserved column name such as `filename` or `extension`.
`spec.file_options.extension`	no	Leave it unset. The default `.txt` is the only extension training reads.
`language`, `normalization`	no	Dataset language and applied normalization.

What the ingestor checks

Check	Rejects
File type	Files under `texts/` with a different extension than configured, or mixed extensions.
Text content	NUL bytes or invalid UTF-8; empty files produce a warning.
Data types	With a `schema`: values that do not match their declared type.

Plus the checks every ingest runs, except label diversity (no label). No tokenizer is needed at ingest; the ingestor records a data-derived text profile (script mix and length distribution, never the text) for the platform’s tokenizer-fit warning.

Sample dataset

The template ships five files: three plain-text passages and two prompt<TAB>completion pairs, with a five-row manifest. The manifest ships as labels_file_sample.csv: rename it to labels.csv before you ingest the folder, because the CLI reads only that name. For the Helm chart, stage the renamed folder so that it matches the paths in the ingest.yaml above.

Score formulas

When you define a use case on this data, you can pick Accuracy (next-token accuracy), Exact Match (the share of completions that are exactly right, counted over completion tokens only; see how CLM scores are computed) or Perplexity.

Next steps

Ingest the data: Prepare Data
Shared rules for every template: Dataset templates

Overview

Create a Use Case

Join a Use Case

Environment Setup

Tools & Help

Releases

Causal language modeling

Folder layout

Manifest CSV

Ingest with the CLI

ingest.yaml

What the ingestor checks

Sample dataset

Score formulas

Next steps

​Folder layout

​Manifest CSV

​Ingest with the CLI

​ingest.yaml

​What the ingestor checks

​Sample dataset

​Score formulas

​Next steps

Folder layout

Manifest CSV

Ingest with the CLI

ingest.yaml

What the ingestor checks

Sample dataset

Score formulas

Next steps