> ## Documentation Index
> Fetch the complete documentation index at: https://docs.tracebloc.io/llms.txt
> Use this file to discover all available pages before exploring further.

# Masked language modeling

> Dataset template for masked language modeling: one token sequence per .txt under sequences/, a manifest CSV without labels, ingest.yaml and the checks the data ingestor runs.

Predict masked-out words. This task is **self-supervised**: there is no label — the training side masks tokens on the fly. Each sample is one `.txt` file holding a space-separated token sequence, listed in a manifest CSV.

## Folder layout

```text theme={null}
/data/shared/primekg-mlm/
├── labels.csv
└── sequences/
    ├── seq_0000001.txt
    ├── seq_0000002.txt
    └── ...
```

* The folder is named `sequences` (not `texts`) — the ingestor reserves `sequences/` for pre-tokenized data — and sits next to the manifest CSV.
* One sequence per file, tokens separated by spaces, UTF-8 encoded. Every file must end in `.txt`. The ingestor also accepts `.text`, but training opens only `<filename>.txt`, so a `.text` dataset ingests and then fails to train. Example from the shipped sample: `Lepirudin indication Huntington phenotype_present Chorea associated_with Dystonia`.
* The shipped sample also contains a `tokenizer.json` next to the CSV. The current ingestor neither reads nor copies it; it is part of the sample only.

## Manifest CSV

```csv theme={null}
filename
seq_0000001
seq_0000002
seq_0000003
```

| Column | Required | Meaning |
| - | - | - |
| `filename` | yes, exactly this name | The sequence file, with or without extension. |
| `extension` | no | Present in the shipped sample (`'.txt'`); not read by the ingestor. |

The task reads no label column. Setting `label:` in `ingest.yaml` is rejected. A `label` column in the manifest itself is not rejected: the ingest goes on without it and warns: "Column(s) \['label'] in the manifest are ignored for masked\_language\_modeling and will not be stored". The same goes for any other column besides `filename` and `extension`. Remove such columns to silence the warning.

## Ingest with the CLI

Each split is one folder laid out as above. Ingest the training and the test split separately:

```bash theme={null}
tracebloc data ingest ./primekg-mlm-train --task masked_language_modeling --name primekg_mlm_train --intent train --holds open_data
tracebloc data ingest ./primekg-mlm-test  --task masked_language_modeling --name primekg_mlm_test  --intent test --holds open_data
```

`--holds open_data` marks the dataset as open data. If your data is about patients, pass `--holds patient_data`; if it was generated, pass `--holds synthetic_data`. Leave `--holds` out on a terminal and the CLI asks; in a script, the dataset is marked as patient data. The mark decides which [use case types](https://docs.tracebloc.io/create-use-case/define.md#step-2-data-&-evaluation) you can create on the dataset.

Do not pass `--label-column`: the CLI rejects it for this task because it trains on the text itself.

## ingest.yaml

```yaml theme={null}
apiVersion: tracebloc.io/v1
kind: IngestConfig
category: masked_language_modeling
table: primekg_mlm_train
intent: train
csv: /data/shared/primekg-mlm/labels.csv
sequences: /data/shared/primekg-mlm/sequences/
```

| Field | Required | Meaning |
| - | - | - |
| `csv` | yes | Path to the manifest CSV. |
| `sequences` | yes | The `sequences/` folder. Its parent becomes the dataset root. |
| `label` | **must not be set** | Self-supervised task. |
| `schema` | no | Extra typed columns only. Never a [reserved column name](https://docs.tracebloc.io/create-use-case/templates.md#rules-that-apply-to-every-csv) such as `filename` or `extension`. |
| `spec.file_options.extension` | no | Leave it unset. The default `.txt` is the only extension training reads. |
| `language`, `normalization` | no | Dataset language and applied normalization. |

## What the ingestor checks

| Check | Rejects |
| - | - |
| File type | Files under `sequences/` with a different extension than configured, or mixed extensions. |
| Text content | NUL bytes or invalid UTF-8; empty files produce a warning. |
| Data types | With a `schema`: values that do not match their declared type. |

Plus the [checks every ingest runs](https://docs.tracebloc.io/create-use-case/templates.md#checks-every-ingest-runs), except label diversity (no label). For text tasks the ingestor also records a data-derived text profile — Unicode-script mix and document-length distribution, never the text itself — so the platform can warn when a model's tokenizer is a poor fit for the dataset.

## Sample dataset

The template ships five knowledge-graph random-walk sequences and a five-row manifest. The manifest ships as `labels_file_sample.csv`: rename it to `labels.csv` before you ingest the folder, because the CLI reads only that name. For the Helm chart, stage the renamed folder so that it matches the paths in the `ingest.yaml` above.

## Score formulas

When you [define a use case](https://docs.tracebloc.io/create-use-case/define.md#supported-metrics-per-data-type-and-task) on this data, the platform currently accepts **Masked Token Accuracy** and **Perplexity**.

## Next steps

* Ingest the data: [Prepare Data](https://docs.tracebloc.io/create-use-case/prepare-dataset.md)
* Shared rules for every template: [Dataset templates](https://docs.tracebloc.io/create-use-case/templates.md)


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.