Token classification

Label each word in a sequence — named-entity recognition, part-of-speech tagging. The layout is the same as text classification; the difference is the label: instead of one class it holds the BIO tag sequence for the words in the file.

Folder layout

/data/shared/ner/
├── labels.csv
└── texts/
    ├── sample1.txt
    ├── sample2.txt
    └── ...

The text folder must be named texts and sit next to the labels CSV.
One pre-tokenized sentence per file: words separated by whitespace. The whitespace-separated words are the tokens that get tagged; the model’s own tokenizer handles sub-word splitting at training time.
UTF-8 encoded. Every file must end in .txt. The ingestor also accepts .text, but training opens only <filename>.txt, so a .text dataset ingests and then fails to train.

Labels CSV

filename,label
sample1,B-PER I-PER O O B-ORG
sample2,B-LOC O O
sample3,B-ORG O B-MISC O

With sample1.txt containing John Smith works at Google.

Column	Required	Meaning
`filename`	yes, exactly this name	The text file, with or without extension.
`label`	yes	Space-separated tags, exactly one per word in the file. Each tag is `O`, `B-<TYPE>` or `I-<TYPE>` (IOB2). Any column name works — set it with `label:`.
`extension`	no	Present in the shipped sample; not read by the ingestor.

Do not declare the label column or a reserved column name (such as filename or extension) in schema.

Ingest with the CLI

Each split is one folder laid out as above. Ingest the training and the test split separately:

tracebloc data ingest ./ner-train --task token_classification --name ner_train --intent train --label-column label
tracebloc data ingest ./ner-test  --task token_classification --name ner_test  --intent test  --label-column label

--label-column is required for this task. Set it to the name of the label column in labels.csv. On a terminal, the CLI asks for it if you leave it out. In a script (--no-input, --output-json or no terminal), leaving it out sends an empty label, and the spec fails validation before anything is ingested.

ingest.yaml

apiVersion: tracebloc.io/v1
kind: IngestConfig
category: token_classification
table: ner_conll_train
intent: train
csv: /data/shared/ner/labels.csv
texts: /data/shared/ner/texts/
label: label

Field	Required	Meaning
`csv`	yes	Path to the labels CSV.
`texts`	yes	The `texts/` folder.
`label`	yes	Name of the tag-sequence column.
`schema`	no	Extra typed columns only.
`spec.file_options.extension`	no	Leave it unset. The default `.txt` is the only extension training reads.
`language`, `normalization`	no	Dataset language and applied normalization.

What the ingestor checks

Check	Rejects
File type	Files under `texts/` with a different extension than configured, or mixed extensions.
Text content	NUL bytes or invalid UTF-8; empty files produce a warning.
BIO labels	A missing `filename` or label column; a row whose label cell is empty; a tag that is not `O`, `B-<TYPE>` or `I-<TYPE>`; a tag count that differs from the word count of the referenced file; a referenced file that does not exist. An `I-<TYPE>` that is not preceded by `B-<TYPE>` or `I-<TYPE>` of the same type is reported as a warning (legal in IOB1, malformed in IOB2).
Data types	With a `schema`: values that do not match their declared type.

Plus the checks every ingest runs. No label-diversity check runs for this task: each label is a tag sequence, not a class. The dataset’s classes are the distinct tags across all rows (for example O, B-PER, I-PER).

Sample dataset

The template ships three pre-tokenized sentences with tags spanning PER, ORG, LOC and MISC. The manifest ships as labels_file_sample.csv: rename it to labels.csv before you ingest the folder, because the CLI reads only that name. For the Helm chart, stage the renamed folder so that it matches the paths in the ingest.yaml above.

Score formulas

When you define a use case on this data, the platform currently accepts Accuracy only. The engine also reports precision, recall and F1 for each run, but you cannot pick them as the score formula yet.

Next steps

Ingest the data: Prepare Data
Shared rules for every template: Dataset templates

Overview

Create a Use Case

Join a Use Case

Environment Setup

Tools & Help

Releases

Token classification

Folder layout

Labels CSV

Ingest with the CLI

ingest.yaml

What the ingestor checks

Sample dataset

Score formulas

Next steps

​Folder layout

​Labels CSV

​Ingest with the CLI

​ingest.yaml

​What the ingestor checks

​Sample dataset

​Score formulas

​Next steps

Folder layout

Labels CSV

Ingest with the CLI

ingest.yaml

What the ingestor checks

Sample dataset

Score formulas

Next steps