Causal language modeling

Predict the next word. This task is self-supervised: there is no label. Each sample is one .txt file of raw text in one of two shapes:

Pretraining — the whole file is plain text.
Instruction tuning (SFT) — one line of the form prompt<TAB>completion. Everything before the first tab is the prompt, everything after is the completion.

A dataset may mix both shapes. The ingestor does not enforce either shape; it only checks that files are valid text.

Folder layout

/data/shared/dolly-clm/
├── labels.csv
└── texts/
    ├── clm_0000001.txt      # plain text
    ├── clm_0000002.txt      # plain text
    ├── clm_0000004.txt      # prompt<TAB>completion
    └── ...

The folder is named texts (raw text) and sits next to the manifest CSV.
UTF-8 encoded; one extension across the dataset (.txt by default).

Manifest CSV

filename
clm_0000001
clm_0000002
clm_0000003

Column	Required	Meaning
`filename`	yes, exactly this name	The text file, with or without extension.
`extension`	no	Present in the shipped sample (`'.txt'`); not read by the ingestor.

There is no label column. Setting label: in ingest.yaml is rejected.

ingest.yaml

apiVersion: tracebloc.io/v1
kind: IngestConfig
category: causal_language_modeling
table: dolly_clm_train
intent: train
csv: /data/shared/dolly-clm/labels.csv
texts: /data/shared/dolly-clm/texts/

Field	Required	Meaning
`csv`	yes	Path to the manifest CSV.
`texts`	yes	The `texts/` folder.
`label`	must not be set	Self-supervised task.
`schema`	no	Extra typed columns only. Never `filename`.
`spec.file_options.extension`	no	`.txt` (default) or `.text`.
`language`, `normalization`	no	Dataset language and applied normalization.

What the ingestor checks

Check	Rejects
File type	Files under `texts/` with a different extension than configured, or mixed extensions.
Text content	NUL bytes or invalid UTF-8; empty files produce a warning.
Data types	With a `schema`: values that do not match their declared type.

Plus the checks every ingest runs, except label diversity (no label). No tokenizer is needed at ingest; the ingestor records a data-derived text profile (script mix and length distribution, never the text) for the platform’s tokenizer-fit warning.

Sample dataset

The template ships five files: three plain-text passages and two prompt<TAB>completion pairs, with a five-row manifest. The ingest.yaml above ingests it with no overrides.

Next steps

Stage the data and run the ingest: Prepare Data
Shared rules for every template: Dataset templates

Overview

Create a Use Case

Join a Use Case

Environment Setup

Tools & Help

Causal language modeling

Folder layout

Manifest CSV

ingest.yaml

What the ingestor checks

Sample dataset

Next steps

​Folder layout

​Manifest CSV

​ingest.yaml

​What the ingestor checks

​Sample dataset

​Next steps

Folder layout

Manifest CSV

ingest.yaml

What the ingestor checks

Sample dataset

Next steps