> ## Documentation Index
> Fetch the complete documentation index at: https://docs.tracebloc.io/llms.txt
> Use this file to discover all available pages before exploring further.

# Token classification

> Dataset template for token classification (NER, POS): pre-tokenized .txt files plus BIO tag sequences in the labels CSV, ingest.yaml and the checks the data ingestor runs.

Label each word in a sequence — named-entity recognition, part-of-speech tagging. The layout is the same as [text classification](/create-use-case/templates/text-classification); the difference is the label: instead of one class it holds the **BIO tag sequence** for the words in the file.

## Folder layout

```text theme={null}
/data/shared/ner/
├── labels.csv
└── texts/
    ├── sample1.txt
    ├── sample2.txt
    └── ...
```

* The text folder must be named `texts` and sit next to the labels CSV.
* One pre-tokenized sentence per file: words separated by whitespace. The whitespace-separated words are the tokens that get tagged; the model's own tokenizer handles sub-word splitting at training time.
* UTF-8 encoded; one extension across the dataset (`.txt` by default).

## Labels CSV

```csv theme={null}
filename,label
sample1,B-PER I-PER O O B-ORG
sample2,B-LOC O O
sample3,B-ORG O B-MISC O
```

With `sample1.txt` containing `John Smith works at Google`.

| Column      | Required               | Meaning                                                                                                                                                     |
| ----------- | ---------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `filename`  | yes, exactly this name | The text file, with or without extension.                                                                                                                   |
| `label`     | yes                    | Space-separated tags, **exactly one per word** in the file. Each tag is `O`, `B-<TYPE>` or `I-<TYPE>` (IOB2). Any column name works — set it with `label:`. |
| `extension` | no                     | Present in the shipped sample; not read by the ingestor.                                                                                                    |

Do not declare `filename` or `label` in `schema`.

## ingest.yaml

```yaml theme={null}
apiVersion: tracebloc.io/v1
kind: IngestConfig
category: token_classification
table: ner_conll_train
intent: train
csv: /data/shared/ner/labels.csv
texts: /data/shared/ner/texts/
label: label
```

| Field                         | Required | Meaning                                     |
| ----------------------------- | -------- | ------------------------------------------- |
| `csv`                         | yes      | Path to the labels CSV.                     |
| `texts`                       | yes      | The `texts/` folder.                        |
| `label`                       | yes      | Name of the tag-sequence column.            |
| `schema`                      | no       | Extra typed columns only.                   |
| `spec.file_options.extension` | no       | `.txt` (default) or `.text`.                |
| `language`, `normalization`   | no       | Dataset language and applied normalization. |

## What the ingestor checks

| Check        | Rejects                                                                                                                                                                                                                                                                                                                                                                          |
| ------------ | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| File type    | Files under `texts/` with a different extension than configured, or mixed extensions.                                                                                                                                                                                                                                                                                            |
| Text content | NUL bytes or invalid UTF-8; empty files produce a warning.                                                                                                                                                                                                                                                                                                                       |
| BIO labels   | A missing `filename` or label column; a row whose label cell is empty; a tag that is not `O`, `B-<TYPE>` or `I-<TYPE>`; a tag count that differs from the word count of the referenced file; a referenced file that does not exist. An `I-<TYPE>` that is not preceded by `B-<TYPE>` or `I-<TYPE>` of the same type is reported as a warning (legal in IOB1, malformed in IOB2). |
| Data types   | With a `schema`: values that do not match their declared type.                                                                                                                                                                                                                                                                                                                   |

Plus the [checks every ingest runs](/create-use-case/templates#checks-every-ingest-runs). The dataset's classes are the distinct **tags**, not the distinct tag strings, so a single-class check does not apply here.

## Sample dataset

The template ships three pre-tokenized sentences with tags spanning `PER`, `ORG`, `LOC` and `MISC`. The `ingest.yaml` above ingests it with no overrides.

## Next steps

* Stage the data and run the ingest: [Prepare Data](/create-use-case/prepare-dataset)
* Shared rules for every template: [Dataset templates](/create-use-case/templates)