Token classification

Label each word in a sequence — named-entity recognition, part-of-speech tagging. The layout is the same as text classification; the difference is the label: instead of one class it holds the BIO tag sequence for the words in the file.

Folder layout

/data/shared/ner/
├── labels.csv
└── texts/
    ├── sample1.txt
    ├── sample2.txt
    └── ...

The text folder must be named texts and sit next to the labels CSV.
One pre-tokenized sentence per file: words separated by whitespace. The whitespace-separated words are the tokens that get tagged; the model’s own tokenizer handles sub-word splitting at training time.
UTF-8 encoded; one extension across the dataset (.txt by default).

Labels CSV

filename,label
sample1,B-PER I-PER O O B-ORG
sample2,B-LOC O O
sample3,B-ORG O B-MISC O

With sample1.txt containing John Smith works at Google.

Column	Required	Meaning
`filename`	yes, exactly this name	The text file, with or without extension.
`label`	yes	Space-separated tags, exactly one per word in the file. Each tag is `O`, `B-<TYPE>` or `I-<TYPE>` (IOB2). Any column name works — set it with `label:`.
`extension`	no	Present in the shipped sample; not read by the ingestor.

Do not declare filename or label in schema.

ingest.yaml

apiVersion: tracebloc.io/v1
kind: IngestConfig
category: token_classification
table: ner_conll_train
intent: train
csv: /data/shared/ner/labels.csv
texts: /data/shared/ner/texts/
label: label

Field	Required	Meaning
`csv`	yes	Path to the labels CSV.
`texts`	yes	The `texts/` folder.
`label`	yes	Name of the tag-sequence column.
`schema`	no	Extra typed columns only.
`spec.file_options.extension`	no	`.txt` (default) or `.text`.
`language`, `normalization`	no	Dataset language and applied normalization.

What the ingestor checks

Check	Rejects
File type	Files under `texts/` with a different extension than configured, or mixed extensions.
Text content	NUL bytes or invalid UTF-8; empty files produce a warning.
BIO labels	A missing `filename` or label column; a row whose label cell is empty; a tag that is not `O`, `B-<TYPE>` or `I-<TYPE>`; a tag count that differs from the word count of the referenced file; a referenced file that does not exist. An `I-<TYPE>` that is not preceded by `B-<TYPE>` or `I-<TYPE>` of the same type is reported as a warning (legal in IOB1, malformed in IOB2).
Data types	With a `schema`: values that do not match their declared type.

Plus the checks every ingest runs. The dataset’s classes are the distinct tags, not the distinct tag strings, so a single-class check does not apply here.

Sample dataset

The template ships three pre-tokenized sentences with tags spanning PER, ORG, LOC and MISC. The ingest.yaml above ingests it with no overrides.

Next steps

Stage the data and run the ingest: Prepare Data
Shared rules for every template: Dataset templates

Overview

Create a Use Case

Join a Use Case

Environment Setup

Tools & Help

Token classification

Folder layout

Labels CSV

ingest.yaml

What the ingestor checks

Sample dataset

Next steps

​Folder layout

​Labels CSV

​ingest.yaml

​What the ingestor checks

​Sample dataset

​Next steps

Folder layout

Labels CSV

ingest.yaml

What the ingestor checks

Sample dataset

Next steps