# Define Use Case
Source: https://docs.tracebloc.io/create-use-case/define
Create, configure, and publish an AI use case on tracebloc in four steps.
This guide walks you through the 4 key steps to create, publish, and manage an AI use case on the tracebloc platform. Make sure you have a [tracebloc client](/environment-setup/setup-guide) running and your data is ingested. Navigate to the [use cases section](https://ai.tracebloc.io/my-use-cases), click on the "+" on the top right corner and simply follow along. Use this documentation for context, clarification and examples when needed.
## Step 1: Initialize and Set Privacy
Objective: Define the basics and visibility of your AI use case.
* Title
* Cover Image (optional): JPG or PNG (max. 25MB)
* Task: Select the task, e.g. "Image Classification", "Object Detection", "Tabular Classification", etc. See the full list of [supported data types and tasks](/create-use-case/prerequisites#supported-data-types-and-tasks). In case your use case is not yet supported, please reach out to us at [support@tracebloc.io](mailto:support@tracebloc.io).
* **Privacy Type:** Choose Public (visible to all users) or Private (invite-only visibility).
Preview your use case tile on the right side of the interface.
***
## Step 2: Data & Evaluation
Objective: Attach datasets and define the benchmarking logic.
* **Training and test metadataset**: A metadataset is a reference to an ingested dataset that stores summary information such as the number of samples and columns. Select the training and test metadatasets that correspond to the datasets you ingested in the [Prepare Data](/create-use-case/prepare-dataset) step.
* **Score**: Define which benchmark to use for evaluation (e.g. Accuracy, F1, etc.). In case your evaluation metric is not yet supported, please reach out to us at [support@tracebloc.io](mailto:support@tracebloc.io).
* **Upload EDA File (Optional)**: Attach a .ipynb EDA file to help participants understand the data context. Explore [template use cases](https://ai.tracebloc.io/explore) for inspiration.
### Supported Metrics per Data Type and Task
The tables below list all evaluation metrics grouped by task type. Each metric uses either **higher is better** or **lower is better** sort order on the leaderboard.
#### Image Classification
| Metric | Description | Sort Order |
| -------------------------------------- | --------------------------------------------------------------------------------------------------------------------------- | ---------------- |
| Accuracy | Proportion of predictions that exactly match the ground truth. Can be misleading on imbalanced datasets. | Higher is better |
| Precision | Measures the proportion of predicted positives that are actually positive. Important when false positives are costly. | Higher is better |
| Recall | Measures the proportion of actual positives correctly identified. Important when missing a positive instance is costly. | Higher is better |
| F1 Score | Balances precision and recall into a single metric. Especially useful for imbalanced datasets. | Higher is better |
| Loss | Quantifies the error between predicted outputs and actual values. A core metric used during training and optimization. | Lower is better |
| Log Loss | Measures how well a model predicts probability estimates for each class. Penalizes overconfident incorrect predictions. | Lower is better |
| AUC-ROC | Measures ability to distinguish between classes across all thresholds, independent of any single decision threshold. | Higher is better |
| AUC-PR | Measures precision-recall balance across different thresholds. Especially useful for highly imbalanced datasets. | Higher is better |
| Top-3 Accuracy | Measures how often the true class label appears among the model's top three predictions. | Higher is better |
| Top-5 Accuracy | Measures how often the true class label appears among the model's top five predictions. | Higher is better |
| Cohen's Kappa | Measures agreement between predicted and ground truth labels while accounting for chance agreement. | Higher is better |
| Matthews Correlation Coefficient (MCC) | Classification quality using all parts of the confusion matrix. Balanced even with imbalanced classes. Ranges from -1 to 1. | Higher is better |
| Quadratic Weighted Kappa (QWK) | Measures agreement between predicted and ground truth labels, penalizing larger disagreements more heavily. | Higher is better |
| Brier Score | Mean squared difference between predicted probabilities and actual outcomes. | Lower is better |
#### Text Classification
| Metric | Description | Sort Order |
| -------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------- | ---------------- |
| Accuracy | Proportion of predictions that exactly match the ground truth. Can be misleading on imbalanced datasets. | Higher is better |
| Precision | Measures the proportion of predicted positives that are actually positive. Important when false positives are costly. | Higher is better |
| Recall | Measures the proportion of actual positives correctly identified. Important when missing a positive instance is costly. | Higher is better |
| F1 Score | Balances precision and recall into a single metric. Especially useful for imbalanced datasets. | Higher is better |
| F1 Weighted | Weights each class's F1 Score by its support (number of true instances). Suitable for imbalanced multi-class classification. | Higher is better |
| Micro F1 | Aggregates true positives, false positives, and false negatives across all classes, treating every prediction equally. | Higher is better |
| Loss | Quantifies the error between predicted outputs and actual values. A core metric used during training and optimization. | Lower is better |
| Log Loss | Measures how well a model predicts probability estimates for each class. Penalizes overconfident incorrect predictions. | Lower is better |
| AUC-ROC | Measures ability to distinguish between classes across all thresholds, independent of any single decision threshold. | Higher is better |
| Hamming Loss | Measures the fraction of labels incorrectly predicted. Commonly used in multi-label classification tasks. | Lower is better |
| Jaccard Score | Measures similarity between predicted and ground truth labels by comparing their intersection to their union. | Higher is better |
| Cohen's Kappa | Measures agreement between predicted and ground truth labels while accounting for chance agreement. | Higher is better |
| Matthews Correlation Coefficient (MCC) | Classification quality using all parts of the confusion matrix. Balanced even with imbalanced classes. Ranges from -1 to 1. | Higher is better |
| Quadratic Weighted Kappa (QWK) | Measures agreement between predicted and ground truth labels, penalizing larger disagreements more heavily. | Higher is better |
| Brier Score | Mean squared difference between predicted probabilities and actual outcomes. | Lower is better |
#### Tabular Classification
| Metric | Description | Sort Order |
| -------------------------------------- | --------------------------------------------------------------------------------------------------------------------------- | ---------------- |
| Accuracy | Proportion of predictions that exactly match the ground truth. Can be misleading on imbalanced datasets. | Higher is better |
| Precision | Measures the proportion of predicted positives that are actually positive. Important when false positives are costly. | Higher is better |
| Recall | Measures the proportion of actual positives correctly identified. Important when missing a positive instance is costly. | Higher is better |
| F1 Score | Balances precision and recall into a single metric. Especially useful for imbalanced datasets. | Higher is better |
| Loss | Quantifies the error between predicted outputs and actual values. A core metric used during training and optimization. | Lower is better |
| Log Loss | Measures how well a model predicts probability estimates for each class. Penalizes overconfident incorrect predictions. | Lower is better |
| AUC | Measures ability to distinguish between positive and negative classes across all classification thresholds. | Higher is better |
| AUC-ROC | Measures ability to distinguish between classes across all thresholds, independent of any single decision threshold. | Higher is better |
| AUC-PR | Measures precision-recall balance across different thresholds. Especially useful for highly imbalanced datasets. | Higher is better |
| Balanced Accuracy | Averages recall across all classes, ensuring each class contributes equally regardless of frequency. | Higher is better |
| Specificity (True Negative Rate) | Measures the proportion of actual negatives correctly identified. Important when false positives must be minimized. | Higher is better |
| NPV (Negative Predictive Value) | Measures the proportion of predicted negatives that are actually negative. Important when confirming absence matters. | Higher is better |
| F-beta Score (beta = 0.5) | Balances precision and recall with more emphasis on precision. Suitable when false positives are more costly. | Higher is better |
| F-beta Score (beta = 2) | Balances precision and recall with more emphasis on recall. Suitable when missing positive instances is more costly. | Higher is better |
| Hamming Loss | Measures the fraction of labels incorrectly predicted. Commonly used in multi-label classification tasks. | Lower is better |
| Jaccard Score | Measures similarity between predicted and ground truth labels by comparing their intersection to their union. | Higher is better |
| Cohen's Kappa | Measures agreement between predicted and ground truth labels while accounting for chance agreement. | Higher is better |
| Matthews Correlation Coefficient (MCC) | Classification quality using all parts of the confusion matrix. Balanced even with imbalanced classes. Ranges from -1 to 1. | Higher is better |
| Quadratic Weighted Kappa (QWK) | Measures agreement between predicted and ground truth labels, penalizing larger disagreements more heavily. | Higher is better |
| Brier Score | Mean squared difference between predicted probabilities and actual outcomes. | Lower is better |
| Gini Coefficient | Measures discriminatory power between positive and negative classes. Closely related to AUC-ROC. | Higher is better |
| Normalized Gini | Scales the Gini Coefficient relative to a perfect model, enabling fair comparison across datasets. Ranges from -1 to 1. | Higher is better |
#### Object Detection
| Metric | Description | Sort Order |
| ----------------------------------- | ---------------------------------------------------------------------------------------------------------------------- | ---------------- |
| Accuracy | Proportion of predictions that exactly match the ground truth. | Higher is better |
| Loss | Quantifies the error between predicted outputs and actual values. A core metric used during training and optimization. | Lower is better |
| Mean Average Precision (mAP) | Evaluates the quality of ranked predictions. Commonly used in object detection and ranking tasks. | Higher is better |
| Mean Average Precision @ IoU 0.50 | Evaluates object detection requiring at least 50% overlap between predicted and ground truth bounding boxes. | Higher is better |
| Mean Average Precision @ IoU 0.75 | Stricter variant requiring at least 75% overlap between predicted and ground truth bounding boxes. | Higher is better |
| mAP per Class | Reports Average Precision individually for each object class. Helps identify which classes the model struggles with. | Higher is better |
| Intersection over Union (IoU) | Measures overlap between predicted and ground truth regions. Used for localization accuracy. | Higher is better |
| GIoU (Generalized IoU) | Extends standard IoU by penalizing non-overlapping predictions using the smallest enclosing box. Ranges from -1 to 1. | Higher is better |
| Mean Average Recall @ 1 Detection | Measures how well the model retrieves ground truth objects when only the single top-scoring detection is allowed. | Higher is better |
| Mean Average Recall @ 10 Detections | Measures retrieval of ground truth objects when up to 10 detections per image are allowed. | Higher is better |
#### Semantic Segmentation
| Metric | Description | Sort Order |
| ----------------------------------- | ---------------------------------------------------------------------------------------------------------------------- | ---------------- |
| Accuracy | Proportion of predictions that exactly match the ground truth. | Higher is better |
| Precision | Measures the proportion of predicted positives that are actually positive. | Higher is better |
| Recall | Measures the proportion of actual positives correctly identified. | Higher is better |
| F1 Score | Balances precision and recall into a single metric. | Higher is better |
| Loss | Quantifies the error between predicted outputs and actual values. | Lower is better |
| Intersection over Union (IoU) | Measures overlap between predicted and ground truth regions. | Higher is better |
| Mean Intersection over Union (mIoU) | Averages IoU across all classes, evaluating how well a model predicts each class region. | Higher is better |
| Frequency Weighted IoU | Weights each class's IoU by its relative frequency in the ground truth, giving more importance to dominant classes. | Higher is better |
| Dice Coefficient | Measures similarity between predicted and ground truth segmentation regions. Especially sensitive to small structures. | Higher is better |
| Pixel Accuracy | Proportion of correctly classified pixels across the entire image. Can be dominated by frequent classes. | Higher is better |
| Mean Pixel Accuracy | Averages pixel accuracy per class, giving equal importance to all classes regardless of frequency. | Higher is better |
| Boundary IoU | Measures how well predicted segmentation boundaries align with ground truth boundaries. Focuses on edge accuracy. | Higher is better |
| Boundary F1 Score | Combines boundary precision and recall to evaluate how accurately predicted boundaries match ground truth edges. | Higher is better |
| Hausdorff Distance | Maximum distance between predicted and ground truth boundaries. Captures the worst-case boundary mismatch. | Lower is better |
| Average Surface Distance (ASD) | Average distance between predicted and ground truth boundary points. Stable measure of overall boundary alignment. | Lower is better |
#### Instance Segmentation
| Metric | Description | Sort Order |
| --------- | -------------------------------------------------------------------------- | ---------------- |
| Accuracy | Proportion of predictions that exactly match the ground truth. | Higher is better |
| Precision | Measures the proportion of predicted positives that are actually positive. | Higher is better |
| Recall | Measures the proportion of actual positives correctly identified. | Higher is better |
| F1 Score | Balances precision and recall into a single metric. | Higher is better |
| Loss | Quantifies the error between predicted outputs and actual values. | Lower is better |
#### Keypoint Detection
| Metric | Description | Sort Order |
| ------------------------------------- | -------------------------------------------------------------------------------------------------------------------- | ---------------- |
| Precision | Measures the proportion of predicted positives that are actually positive. | Higher is better |
| Recall | Measures the proportion of actual positives correctly identified. | Higher is better |
| F1 Score | Balances precision and recall into a single metric. | Higher is better |
| Loss | Quantifies the error between predicted outputs and actual values. | Lower is better |
| Mean Absolute Error (MAE) | Average magnitude of errors between predicted and actual values, expressed in the same units as the target. | Lower is better |
| PCK (Percentage of Correct Keypoints) | Measures how accurately predicted keypoints fall within a specified distance of ground truth. | Higher is better |
| PCK\@0.05 | Keypoints within a normalized distance threshold of 0.05 from ground truth. Strictest variant. | Higher is better |
| PCK\@0.10 | Keypoints within a normalized distance threshold of 0.10 from ground truth. | Higher is better |
| PCK\@0.20 | Keypoints within a normalized distance threshold of 0.20 from ground truth. | Higher is better |
| PCK\@0.30 | Keypoints within a normalized distance threshold of 0.30 from ground truth. | Higher is better |
| PCK\@0.50 | Keypoints within a normalized distance threshold of 0.50 from ground truth. Most lenient variant. | Higher is better |
| Object Keypoint Similarity (OKS) | Measures similarity between predicted and ground truth keypoints, accounting for scale and localization uncertainty. | Higher is better |
| Mean Per Joint Position Error (MPJPE) | Average Euclidean distance between predicted and ground truth joint positions. Standard for pose estimation. | Lower is better |
| Visibility Accuracy | Measures how correctly the model predicts the visibility status of keypoints, independent of spatial localization. | Higher is better |
#### Tabular Regression
| Metric | Description | Sort Order |
| ------------------------------------------- | ------------------------------------------------------------------------------------------------------------------ | ---------------- |
| Accuracy | Proportion of predictions that exactly match the ground truth. | Higher is better |
| Loss | Quantifies the error between predicted outputs and actual values. | Lower is better |
| Mean Absolute Error (MAE) | Average magnitude of errors between predicted and actual values, expressed in the same units as the target. | Lower is better |
| Mean Squared Error (MSE) | Average squared difference between predicted and actual values. Penalizes larger errors more heavily. | Lower is better |
| Root Mean Squared Error (RMSE) | Square root of MSE, expressing prediction error in the same units as the target variable. | Lower is better |
| R² (Coefficient of Determination) | Measures how well a regression model explains variance in the target. A value of 1.0 indicates perfect prediction. | Higher is better |
| Root Mean Squared Logarithmic Error (RMSLE) | Measures error on a logarithmic scale. Useful when target values span several orders of magnitude. | Lower is better |
| Median Absolute Error (Median AE) | Uses the median of absolute errors instead of the mean. Highly robust to outliers. | Lower is better |
| Explained Variance | Measures how well the model captures the variance of the target, independent of systematic bias. Ranges up to 1.0. | Higher is better |
| Mean Bias Error (MBE) | Measures the average bias in predictions. Positive = overestimation, negative = underestimation. | Lower is better |
#### Time Series Forecasting
| Metric | Description | Sort Order |
| ------------------------------------------------ | ---------------------------------------------------------------------------------------------------------------------- | ---------------- |
| Accuracy | Proportion of predictions that exactly match the ground truth. | Higher is better |
| Loss | Quantifies the error between predicted outputs and actual values. | Lower is better |
| Mean Absolute Error (MAE) | Average magnitude of errors between predicted and actual values, expressed in the same units as the target. | Lower is better |
| Mean Squared Error (MSE) | Average squared difference between predicted and actual values. Penalizes larger errors more heavily. | Lower is better |
| Root Mean Squared Error (RMSE) | Square root of MSE, expressing prediction error in the same units as the target variable. | Lower is better |
| R² (Coefficient of Determination) | Measures how well a regression model explains variance in the target. A value of 1.0 indicates perfect prediction. | Higher is better |
| Root Mean Squared Logarithmic Error (RMSLE) | Measures error on a logarithmic scale. Useful when target values span several orders of magnitude. | Lower is better |
| Mean Absolute Percentage Error (MAPE) | Average percentage difference between predicted and actual values. Easy to interpret across different scales. | Lower is better |
| Symmetric Mean Absolute Percentage Error (SMAPE) | Symmetric variant of MAPE that reduces issues when actual values are close to zero. | Lower is better |
| Median Absolute Percentage Error (MdAPE) | Uses the median of absolute percentage errors. More robust to outliers. | Lower is better |
| Theil's U (U2 Statistic) | Measures forecasting accuracy relative to a naive benchmark. Values below 1.0 mean the model outperforms the baseline. | Lower is better |
| Max Error | Captures the single largest absolute difference between predicted and actual values. | Lower is better |
| Direction Accuracy | Measures how often the model correctly predicts the direction of change (up or down) between consecutive values. | Higher is better |
#### Time-to-Event Prediction
| Metric | Description | Sort Order |
| --------------------------- | ------------------------------------------------------------------------------------------------------------------------ | ---------------- |
| F1 Score | Balances precision and recall into a single metric. | Higher is better |
| Concordance Index (C-Index) | Measures how well predicted risk scores agree with the observed ordering of event times. Standard for survival analysis. | Higher is better |
***
## Step 3: Describe Your Use Case
Objective: Describe your use case and objective in detail.
Provide a clear description that helps participants understand the problem, the data context, and the goal. Cover what the data represents, what a good model should achieve, and any domain-specific considerations participants should be aware of. Browse published use cases in the [Explore section](https://ai.tracebloc.io/explore) for examples of well-written descriptions.
***
## Step 4: Review & Submit
Objective: Set collaboration and resource constraints.
Add emails of vendors, colleagues, or researchers. Invitations are sent once the use case is saved or published. For instructions for data scientists about how to join your use case, follow the [join a use case guide](/join-use-case/join-use-case).
### Compute Assignment
Define training budget in PFLOPs.
Example: 10 participants × 200 PFLOPs each = 2,000 PFLOPs
Cost Calculation: 2,000 PFLOPs × €0.025 = €50.00
Always allocate more resources than minimum requirements and monitor resource usage regularly. You can stop or adjust training at any time.
### Final Step: Publish or Save as Draft
Use "Publish" to go live or "Save as Draft" to continue editing later. You can now see your use case in the [use cases section](https://ai.tracebloc.io/my-use-cases).
***
## Next Steps
Once your use case is published, reach out to external vendors, your colleagues or data scientists to train models on your use case.
In the use case view, monitor
* total resource consumption
* daily submits and user activity
* overall leaderboard and submissions
Once models have been submitted, you can [compare them in the leaderboard section](/create-use-case/evaluate-models) of a use case.
***
## Need Help?
* Email us at [support@tracebloc.io](mailto:support@tracebloc.io)
# Evaluate models
Source: https://docs.tracebloc.io/create-use-case/evaluate-models
Compare vendor models side by side on the leaderboard.
Navigate to the Leaderboard section of your use case to compare submitted models, for example the [Breast Cancer Screening Use Case](https://ai.tracebloc.io/explore/ai-breast-cancer-screening-and-image-classification?tab=leaderboard).
## The Leaderboard
Vendors are ranked by the score of their best performing model. Scores are calculated using the evaluation metric you selected when [defining the use case](/create-use-case/define#supported-metrics-per-data-type-and-task) (e.g. Accuracy, F1, mAP).
### Leaderboard Columns
| Column | Description |
| ------------------ | --------------------------------------------------------------------------------- |
| Rank | Position based on the best model score. Lower rank = better performance. |
| Team / Vendor | The participant or team that submitted the model. |
| Score | The model's evaluation result on your test data, using the metric you defined. |
| Model Size | Size of the submitted model in parameters or MB. Useful for comparing efficiency. |
| Energy Consumption | Compute resources consumed during training. Helps assess cost-efficiency. |
| Submissions | Total number of models submitted by the team. |
| Remaining Budget | How much of the allocated compute budget (in PFLOPs) the team has left. |
### Actions You Can Take
From the use case view, you can:
* **Compare models** — Use the leaderboard to identify the best performing and most efficient models side by side.
* **Stop or adjust training** — Reduce or revoke a team's remaining compute budget to control costs or end a round of evaluation.
* **Contact participants** — Reach out to vendors or researchers directly to discuss results, request further iterations, or negotiate next steps.
* **Select a winner** — Once evaluation is complete, choose the best model based on score, model size, and resource efficiency.
***
## Supported Evaluation Metrics
Scores on the leaderboard are calculated using the metric you selected when defining the use case. See the full list of [supported metrics per data type and task](/create-use-case/define#supported-metrics-per-data-type-and-task).
***
## Next Steps
* Browse ready-made examples: [Templates](/create-use-case/templates)
***
## Need Help?
* Email us at [support@tracebloc.io](mailto:support@tracebloc.io)
# Prepare Data
Source: https://docs.tracebloc.io/create-use-case/prepare-dataset
Learn how to prepare and ingest your datasets into tracebloc using containerized data ingestors. Complete guide for CSV, image, and text data with Kubernetes deployment steps.
## Overview
Make your data available to the Kubernetes cluster so it can be used for training and evaluation. Whether your client runs on Azure, AWS, Google Cloud, or a local Minikube setup, the process of ingesting datasets works the same way.
The data ingestor is a lightweight service that bridges your raw data and the cluster's persistent storage. Every supported task has a [dataset template](/create-use-case/templates) — the folder layout, the labels CSV and a ready-to-edit `ingest.yaml` — that you lay your own data out against. By containerizing the ingestion step, the ingestor validates data format and schema, enforces consistency, and transfers the dataset securely into cluster's SQL storage where it becomes accessible to all training and evaluation jobs.
This guide covers:
* Laying your data out to match the dataset template for your task (tabular, images, text, time series)
* Deploying the data ingestor for training and test data using Kubernetes
* Managing datasets through the tracebloc interface
**IMPORTANT** Make sure that the data format and ML task is supported and that data standards are met by reviewing the [docs](/create-use-case/prerequisites). You must run the process twice, once to ingest training and once to ingest testing data.
## Setup options
You can ingest data into your client in two ways:
* **Declarative YAML (recommended, simpler)** — describe your dataset in \~8 lines of `ingest.yaml`, then `helm install`. No Dockerfile, no custom Python script. The official ingestor image runs it for you. Use this for any dataset that fits a supported category.
* **Custom Python script + Kubernetes Job (advanced)** — install the `tracebloc-ingestor` Python package, write a short ingestion script against it, build and push a Docker image, then `kubectl apply` an `ingestor-job.yaml`. Use this when the declarative schema can't express what your data needs — e.g. non-trivial preprocessing, a custom validator, or a `BaseProcessor` subclass.
Start with the declarative method below. Drop down to the custom-script flow only if you need it.
## Declarative YAML (recommended)
Describe your dataset in \~8 lines of YAML, then `helm install`. The official ingestor image (published as `ghcr.io/tracebloc/ingestor`) runs it. No Dockerfile, no Python script.
**Before you run any commands in this section:** if you installed the client via the one-liner (`curl -fsSL https://tracebloc.io/i.sh | bash`), every later `helm upgrade tracebloc/client …` **must** include `--reset-then-reuse-values`, otherwise the upgrade drops the values the installer applied and breaks the workspace:
```bash theme={null}
helm upgrade tracebloc/client -n --reset-then-reuse-values
```
Append `--version ` to pin a specific chart version. This caveat only affects upgrades of the parent `tracebloc/client` chart, not the `helm install tracebloc/ingestor` runs below.
### 1. Add the chart repo (one-time)
```bash theme={null}
helm repo add tracebloc https://tracebloc.github.io/client
helm repo update
```
The `tracebloc/client` parent chart bootstraps the cluster (jobs-manager, MySQL, RBAC). The `tracebloc/ingestor` subchart submits per-dataset ingestion runs against it.
### 2. Stage your data on the cluster's shared PVC
The chart **doesn't transport data into the cluster** — it points at data already accessible to the cluster's shared PVC (`client-pvc` by default, mounted at `/data/shared/` inside the ingestor Pod). Before installing, get your raw files there.
For a single-node workspace (the default install), the PVC is backed by a host directory the installer created at `~/.tracebloc//data/`. Drop your files into a per-dataset subdirectory:
```bash theme={null}
# Host path on the machine where the tracebloc client is installed.
# Pick a per dataset — it becomes the path you reference in ingest.yaml.
mkdir -p ~/.tracebloc//data/
cp -R LOCAL_PATH/images ~/.tracebloc//data//
cp LOCAL_PATH/labels.csv ~/.tracebloc//data//
```
Inside the ingestor Pod those files appear at `/data/shared//...` — that's what you'll put in `ingest.yaml` below.
For multi-node or EKS deployments where the PVC isn't backed by a local host path, use a throwaway `kubectl cp` Pod or a cloud-storage init container instead. See the [client ingestor README](https://github.com/tracebloc/client/blob/develop/ingestor/README.md#stage-your-data-on-the-shared-pvc) for those recipes.
### 3. Write your `ingest.yaml`
The example below is for `image_classification`. **Other tasks require different fields** — e.g. `tabular_classification` has no `images:` and instead needs a typed `schema:` block. Don't copy this one blindly; open the [dataset template for your task](/create-use-case/templates) — one page per task with the folder layout, the CSV columns, a ready-to-edit `ingest.yaml` and the checks the ingestor runs — and edit from there.
```yaml theme={null}
apiVersion: tracebloc.io/v1
kind: IngestConfig
category: image_classification
table: cats_dogs_train
intent: train
csv: /data/shared/cats-dogs/labels.csv
images: /data/shared/cats-dogs/images/
label: label
```
The top-level shape (`apiVersion`, `kind`, `category`, `table`, `intent`, `label`) is the same for every category; the `category` field picks the validator set, file-extension defaults, and column conventions. The data-source fields (`csv:`, `images:`, `schema:`, …) vary per category. The paths are *paths inside the ingestor Pod*, which is the PVC mount you populated in step 2.
### 4. Install once per dataset
The ingestor runs once: validates your data, copies files into the destination directory on the PVC, inserts rows into MySQL, sends metadata to the tracebloc backend, then exits. **Run it twice per dataset** — once with `intent: train`, once with `intent: test` — using distinct `table:` names. The example below shows both releases:
```bash theme={null}
# Train release — points at the ingest.yaml from step 3 (table: cats_dogs_train, intent: train)
helm install cats-dogs-train tracebloc/ingestor \
--namespace \
--set-file ingestConfig=./ingest-train.yaml
# Test release — same shape, with table: cats_dogs_test and intent: test
helm install cats-dogs-test tracebloc/ingestor \
--namespace \
--set-file ingestConfig=./ingest-test.yaml
```
Each `helm install` is a separate release (the first argument is the release name), so the two runs don't collide. The ingestor Pod picks up `CLIENT_ID` / `CLIENT_PASSWORD` automatically from the Kubernetes Secret the parent `tracebloc/client` chart created in `` at install time — you don't pass credentials on the `helm install` command.
**Validation error like `'' is not one of [...]` or `Additional properties are not allowed ( was unexpected)`?** This comes from the cluster's `jobs-manager` validating against its own bundled schema at submit time — the deployed schema is older than the ingestor image you're installing. `helm repo update` won't fix it (that only refreshes the local chart index, not the running server). The fix is on the cluster side: upgrade the parent chart so jobs-manager redeploys with the current schema.
```bash theme={null}
helm upgrade tracebloc/client \
-n --reset-then-reuse-values
```
Then re-run the `helm install` command above.
Full chart docs (data-staging recipe, schema, every category, update model, verification, override knobs) → [client ingestor README](https://github.com/tracebloc/client/blob/develop/ingestor/README.md).
## Custom Python script (advanced)
Use this flow when the declarative schema can't express what your data needs — typically when you have non-trivial preprocessing logic, a custom validator, or a `BaseProcessor` subclass. The sections below — Quick Setup and Detailed Setup — both describe this advanced path.
## Quick Setup
Use this quick setup if you already have an ingestor configured and just want to switch datasets or toggle between training and testing. If you are setting up for the first time, go to the next section for the detailed walkthrough.
### Steps
1. Edit your ingestion script (the one you wrote in [Configure a script](#1-configure-a-script) below)
* Update csv options and data\_path
* Only for tabular data: Update schema
* Set `schema` and `CSVIngestor()`parameters like category, intent, label\_column, etc. to match data type, task and train/test purpose
```python theme={null}
ingestor = CSVIngestor(
...
category=TaskCategory.TABULAR_CLASSIFICATION, # Adjust for your task
csv_options=csv_options, # Defined above
label_column="ColumnName", # Target column
intent=Intent.TRAIN, # TRAIN or TEST
)
```
2. Build and push docker image:
Make sure Docker is running on your system (e.g. by starting Docker Desktop), then execute the following command:
```bash theme={null}
# Build for cloud (multi-arch) and push directly to registry
docker buildx build --platform linux/amd64,linux/arm64 -t /: --push .
```
3. Edit ingestor-job.yaml:
* `metadata.name`: Unique job name (e.g. ingestor-job-train and ingestor-job-test)
* `image`: The tag you built and pushed
* `LABEL_FILE`: Path inside the pod to the labels CSV, under the PVC mount (e.g. `/data/shared/labels.csv`). For tabular data, this is the same file that contains both labels and features.
* `TABLE_NAME`: Unique table name (no spaces, one per dataset). Title is optional
* `SRC_PATH`: Root of the mounted dataset directory inside the pod (`/data/shared`, backed by `~/.tracebloc//data` on the client host)
4. Deploy to Kubernetes
```bash theme={null}
`kubectl apply -f ingestor-job.yaml -n `
```
## Detailed Setup
### 1. Configure a script
This section walks you through the step-by-step setup of a data ingestor. You will install the ingestor package, lay your data out against the dataset template for your task, and write a short ingestion script that matches it. Follow this guide if you are setting up an ingestor for the first time or need full control beyond the quick setup.
### Install the ingestor package
The ingestion library is published on PyPI as `tracebloc-ingestor` (import name `tracebloc_ingestor`). It is the same code the official ingestor image runs, so a script written against it behaves exactly like the declarative path. It needs Python 3.11 or newer:
```bash theme={null}
pip install tracebloc-ingestor
```
**IMPORTANT:** Datasets must be cleaned and preprocessed before ingestion. Participants cannot view, clean or fix raw data, so model performance will only be as good as the data you provide.
### Pick the dataset template for your task
Open the [dataset template](/create-use-case/templates) for your task. It gives you the folder layout and the labels CSV the ingestor expects — lay your data out the same way, then set the matching `category` and `data_format` in your script. The rows below cover the most common tasks; every other task on the templates index works the same way, and its `TaskCategory` constant is the upper-cased `category` identifier from its page (for example `keypoint_detection` → `TaskCategory.KEYPOINT_DETECTION`).
| Data Type | Dataset template | Data Category | Data Format |
| --------- | ---------------------------------------------------------------------------------------- | --------------------------------------- | -------------------- |
| Tabular | [Tabular classification](/create-use-case/templates/tabular-classification) | `TaskCategory.TABULAR_CLASSIFICATION` | `DataFormat.TABULAR` |
| Tabular | [Tabular regression](/create-use-case/templates/tabular-regression) | `TaskCategory.TABULAR_REGRESSION` | `DataFormat.TABULAR` |
| Tabular | [Time-series forecasting](/create-use-case/templates/time-series-forecasting) | `TaskCategory.TIME_SERIES_FORECASTING` | `DataFormat.TABULAR` |
| Tabular | [Survival analysis (time-to-event)](/create-use-case/templates/time-to-event-prediction) | `TaskCategory.TIME_TO_EVENT_PREDICTION` | `DataFormat.TABULAR` |
| Image | [Image classification](/create-use-case/templates/image-classification) | `TaskCategory.IMAGE_CLASSIFICATION` | `DataFormat.IMAGE` |
| Image | [Object detection](/create-use-case/templates/object-detection) | `TaskCategory.OBJECT_DETECTION` | `DataFormat.IMAGE` |
| Text | [Text classification](/create-use-case/templates/text-classification) | `TaskCategory.TEXT_CLASSIFICATION` | `DataFormat.TEXT` |
#### High Level Script Structure
Every ingestion script follows the same structure — save it as `ingestor.py` next to your Dockerfile:
```python theme={null}
import logging
from tracebloc_ingestor import Config, Database, APIClient, CSVIngestor, run_ingestion
from tracebloc_ingestor.utils.logging import setup_logging
from tracebloc_ingestor.utils.constants import TaskCategory, Intent, DataFormat
config = Config()
setup_logging(config)
logger = logging.getLogger(__name__)
def main():
# Initialize components
database = Database(config)
# Initialize API client
api_client = APIClient(config)
# Define csv_options and schema (schema is only needed for tabular data)
csv_options = {...}
schema = {...}
# Initialize ingestor
ingestor = CSVIngestor(...)
# Run and ingest data
run_ingestion(ingestor, config.LABEL_FILE, batch_size=config.BATCH_SIZE, logger=logger)
if __name__ == "__main__":
main()
```
Both Database, APIClient and other values are configured automatically from the environment variables defined in `ingestor_job.yaml`.
* `config.LABEL_FILE`: Path to local csv label file
* `config.BATCH_SIZE`: Batch size used during ingestion
### Customize the script
The structure above is a starting point, but every dataset has its own format and labels. In this step you adapt the script to your data by tuning CSV ingestion options and setting the ingestor parameters (category, label column, intent, data path and schema). The following example shows how to ingest a tabular dataset, but the setup works the same way for image or text data.
#### Needed for Tabular Data: Define Schema
Define the dataset schema as a Python dictionary, mapping each column to its SQL type and constraints. Do not include IDs or the label column into the schema.
```python theme={null}
# Schema definition for tabular data
schema = {
"feature_00": "FLOAT ",
"feature_01": "FLOAT ",
"feature_02": "FLOAT ",
...
}
```
#### Needed for Image Classification Data: Define Image Options
Define image size and file extension.
```python theme={null}
# Image specific options including CSV options
image_options = {
# Image processing options
"target_size": (512, 512), # Define image size. Height = Width
"extension": FileExtension.JPG, # allowed extension for images: jpeg, jpg, png
}
```
#### Needed for Object Detection Data: Define Image Options
Define file extension.
```python theme={null}
# Object detection specific options including CSV options
object_detection_options = {
# Image processing options
"target_size": (448, 448), # Resize images to this fixed dimension. Dimension is not changeable.
"extension": FileExtension.JPG, # allowed extension for images: jpeg, jpg, png
}
```
#### Needed for Text Data: Define File Extension
Define file extensions.
```python theme={null}
text_options = {"extension": FileExtension.TXT} # Allowed text file extensions
```
#### Set CSV ingestion options
Customize parsing, memory handling, and data cleaning with the csv\_options dictionary:
```python theme={null}
csv_options = {
"chunk_size": 1000, # Process rows in batches for efficiency
"delimiter": ",", # Column separator
"quotechar": '"', # Quoted field character
"escapechar": "\\", # Escape character for quotes
"encoding": "utf-8", # File encoding
"on_bad_lines": "warn", # Log malformed rows instead of failing
"skip_blank_lines": True, # Ignore empty rows
"na_values": ["", "NA", "NULL", "None"] # Treat these as missing values
}
```
#### Set Up the Ingestor
Define the Ingestor instance with the required configuration. See the tabular data example below:
```python theme={null}
ingestor = CSVIngestor(
database=database, # From ingestor-job.yaml
api_client=api_client, # From ingestor-job.yaml
table_name=config.TABLE_NAME, # From ingestor-job.yaml
schema=schema, # Defined above, only needed for tabular data
data_format=DataFormat.TABULAR, # Set the data format for the task
category=TaskCategory.TABULAR_CLASSIFICATION, # Adjust for your task
csv_options=csv_options, # Defined above
file_options={"number_of_columns": len(schema)}, # Don´t change
label_column="ColumnName", # Target column
intent=Intent.TRAIN, # TRAIN or TEST
)
```
**Specify:**
* `category`, choose the ML task type (TABULAR\_CLASSIFICATION, IMAGE\_CLASSIFICATION, OBJECT\_DETECTION)
* `label_column`, target column or class labels
* `intent`, set as TRAIN or TEST depending on dataset purpose
* include `file_options` or `schema` depending on the data type
Other data types work similarly — follow the same configuration pattern with the `category`, `data_format` and options for your task from the table above.
### 2. Build Docker Image
With your script configured, the next step is to package it into a Docker image so it can run inside the Kubernetes cluster.
### Docker Hub Setup (first-time users)
The cluster pulls your ingestor image from a public Docker registry, so you need an account before you can push. If you already have one, skip to [Write the Dockerfile](#write-the-dockerfile).
1. **Create a Docker Hub account** at [hub.docker.com/signup](https://hub.docker.com/signup) and verify your email.
2. **Log in from your terminal** so the `docker push` command can authenticate:
```bash theme={null}
docker login
```
3. **Push the data ingestor image** to your account using the build/push commands in the next section. The image name takes the form `/:` — the username segment must match the account you just created.
4. **Make the image public** so the cluster can pull it without credentials:
* Go to [hub.docker.com/repositories](https://hub.docker.com/repositories), open the repository you just pushed.
* Click **Settings → Visibility settings → Make public**.
Keeping the image private is also fine, but then you must create a Kubernetes `imagePullSecret` named `regcred` in the client namespace (the `ingestor-job.yaml` already references it).
### Place data files on the client host
Datasets are **not** baked into the Docker image. They live on the client host in the per-workspace data directory and are mounted into the ingestor pod through the shared PVC (`client-pvc` → `/data/shared`).
Copy your dataset into the client's data directory, where `` is the workspace name you chose during client install (which is also the Helm release name and the Kubernetes namespace — the chart uses the same value for all three). The directory `~/.tracebloc//data/` is created automatically by the installer; just drop your files into it:
```bash theme={null}
# Host path on the machine where the tracebloc client is installed.
# HOST_DATA_DIR defaults to ~/.tracebloc; override only if you set it during install.
cp -R LOCAL_PATH/images ~/.tracebloc//data/
cp LOCAL_PATH/labels.csv ~/.tracebloc//data/
```
Inside the ingestor pod this directory is mounted at `/data/shared`, so the same files appear as `/data/shared/images/...` and `/data/shared/labels.csv`. Set `SRC_PATH` and `LABEL_FILE` in `ingestor-job.yaml` to point at those in-pod paths (see [Configure Kubernetes](#3-configure-kubernetes) below).
For tabular data the same rule applies — drop the single `labels.csv` (with features and labels) into `~/.tracebloc//data/`.
### Write the Dockerfile
The Dockerfile only needs to install the ingestor package and copy in your script — the dataset is mounted at runtime, so do **not** `COPY` data into the image:
```dockerfile theme={null}
FROM python:3.11-slim
WORKDIR /app
RUN pip install --no-cache-dir tracebloc-ingestor
# Copy the ingestion script into /app
COPY ingestor.py /app/ingestor.py
# Set the entrypoint
ENTRYPOINT ["python", "/app/ingestor.py"]
```
If the cluster enforces the `restricted` Pod Security Standard (see [Run as non-root](#run-as-non-root) below), also add a non-root user to the Dockerfile, **before** the `# Set the entrypoint` line:
```dockerfile theme={null}
RUN groupadd -g 1000 app && \
useradd -u 1000 -g 1000 -m -s /bin/bash app && \
chown -R 1000:1000 /app
USER 1000
# Set the entrypoint
```
### Build Docker Image
You need a docker user and password to proceed with the next step. Cloud platforms run a mix of x86 and ARM nodes (e.g. AWS Graviton, Azure Ampere, GCP Tau T2A). Building a multi-arch image with `--platform linux/amd64,linux/arm64` guarantees the image runs on either, particularly if you build on Apple Silicon (M1/M2) or other ARM-based systems. Build and push the image with a single command:
```bash theme={null}
docker buildx build --platform linux/amd64,linux/arm64 -t /: --push .
```
### 3. Configure Kubernetes
With the image generated and pushed to the registry, edit `ingestor-job.yaml` with your settings:
```yaml theme={null}
apiVersion: batch/v1
kind: Job
metadata:
name: # Set a job name e.g. ingestor-job-train
namespace: # Use the client namespace
spec:
template:
spec:
containers:
- name: api
image: /:latest # Your Docker image name and tag, e.g. "latest"
imagePullPolicy: Always # Use IfNotPresent only for local tests
# Required if the namespace enforces the `restricted` Pod Security Standard.
# See "Run as non-root" below.
securityContext:
allowPrivilegeEscalation: false
runAsNonRoot: true
capabilities:
drop:
- "ALL"
seccompProfile:
type: RuntimeDefault
volumeMounts:
- name: shared-volume
mountPath: "/data/shared" # Client shared PVC. Backed by ~/.tracebloc//data on the client host — read your dataset from here
env:
# Client credentials
- name: CLIENT_ENV
value: "prod"
- name: CLIENT_ID # Client credentials from tracebloc dashboard
value:
- name: CLIENT_PASSWORD # Client credentials from tracebloc dashboard
value:
# Storage configuration
- name: CLIENT_PVC # value has to match the shared data PVC name in the client values.yaml
value: "client-pvc"
# MySQL configuration
- name: MYSQL_HOST # value has to match the mysql deployment name in the client values.yaml
value: "mysql-client"
# Dataset information — paths inside the ingestor pod.
# /data/shared is the mount of the client-pvc, which is backed by
# ~/.tracebloc//data on the client host.
- name: SRC_PATH
value: "/data/shared" # Root of the mounted dataset directory
- name: LABEL_FILE
value: "/data/shared/labels.csv" # Path to the labels CSV inside the pod
- name: TABLE_NAME
value: # Different for train and test, no spaces
- name: TITLE
value: # Optional
- name: BATCH_SIZE
value: "4000" # Optional, defaults to 4000
- name: LOG_LEVEL
value: "DEBUG" # Set DEBUG, "WARNING", "INFO" or "ERROR"
imagePullSecrets:
- name: regcred
volumes:
- name: shared-volume
persistentVolumeClaim:
claimName: client-pvc # value has to match the shared data PVC name in the client values.yaml
restartPolicy: Never
```
**Specify:**
* `JOBNAME`, to distinguish between train and test data jobs.
* `NAMESPACE`, use the same as your client.
* `image`, your Docker image (imagePullPolicy: Always for DockerHub, IfNotPresent for local)
* `CLIENT_ID`, `CLIENT_PASSWORD` from the [tracebloc client view](https://ai.tracebloc.io/clients)
* `TABLE_NAME`, unique per dataset, train and test use different names, no spaces. Different names for train and test data is mandatory
* `LABEL_FILE`, path inside the ingestor pod (under `/data/shared`) to the CSV with file paths and labels — must match the location of the file you placed in `~/.tracebloc//data/`
* `SRC_PATH`, root inside the pod where the dataset directory is mounted (`/data/shared`)
* `BATCH_SIZE` is the number of entries sent to the server per request. Optional — defaults to 4000. Keep it consistent across data types. It depends on available CPU memory, not for example image size. Too large can exhaust memory. It was tested up to 10,000, but 5,000 is a safe default for most systems.
* `LOG_LEVEL`, "WARNING" for all warnings and errors, "INFO" for all logs, "ERROR" for errors only
### 4. Deploy
Run the ingestor as a Kubernetes Job:
```bash theme={null}
kubectl apply -f ingestor-job.yaml -n
kubectl wait -n --for=condition=complete job/
kubectl logs -n job/
# Delete the job only after verifying logs
kubectl delete -n job/
```
This will start a pod, run the ingestion process once, and once complete you can delete the job.
**IMPORTANT:** You must run this process twice — once for training data and once for test data. Use different `JOBNAME` and `TABLE_NAME` values for each run (e.g. `ingestor-job-train` / `ingestor-job-test`), and set `intent` to `TRAIN` or `TEST` accordingly in your ingestion script.
### Run as non-root
If the namespace enforces the `restricted` [Pod Security Standard](https://kubernetes.io/docs/concepts/security/pod-security-standards/), `kubectl apply` will be admitted but the pod will be rejected with a warning like:
```text theme={null}
Warning: would violate PodSecurity "restricted:latest":
allowPrivilegeEscalation != false (container "api" must set securityContext.allowPrivilegeEscalation=false),
unrestricted capabilities (container "api" must set securityContext.capabilities.drop=["ALL"]),
runAsNonRoot != true (pod or container "api" must set securityContext.runAsNonRoot=true),
seccompProfile (pod or container "api" must set securityContext.seccompProfile.type to "RuntimeDefault" or "Localhost")
job.batch/ingestor-job-train-data created
```
Two changes are needed:
**1. Add a `securityContext` block to the container in `ingestor-job.yaml`** (already shown in the YAML above):
```yaml theme={null}
securityContext:
allowPrivilegeEscalation: false
runAsNonRoot: true
capabilities:
drop:
- "ALL"
seccompProfile:
type: RuntimeDefault
```
**2. Run the container as a non-root user.** Add the following to the Dockerfile **before** the `# Set the entrypoint` line so the image ships with a UID that satisfies `runAsNonRoot: true`:
```dockerfile theme={null}
RUN groupadd -g 1000 app && \
useradd -u 1000 -g 1000 -m -s /bin/bash app && \
chown -R 1000:1000 /app
USER 1000
```
Rebuild and push the image, then re-apply the job.
The data ingestor always runs a validation step before ingestion and moving files.
#### Verify Deployment
Verify if jobs and pods are deployed successfully and running:
```bash theme={null}
kubectl get jobs,pods -n
kubectl logs -n
```
Look for "All records processed successfully" in the logs.
## Dataset Management Interface
View your datasets at [ai.tracebloc.io/data](https://ai.tracebloc.io/data) after successful deployment.
**Interface displays:**
* Dataset name, ID, and record count
* Data type (Tabular, Image, Text) and purpose (Training/Testing)
* Namespace and GPU requirements
## Best Practices
* Deploy jobs for training and testing simultaneously using different job names
* Use consistent, descriptive table names (e.g., `insurance-claims-train`, `insurance-claims-test`)
* Validate data schemas before deployment to prevent ingestion failures
* Clean data before ingestion - Participants cannot view, clean, or fix raw data, so model performance depends entirely on the quality of data you provide
## Troubleshooting
**Recommended for debugging:** Use [k9s](https://k9scli.io/), a terminal-based Kubernetes dashboard, to monitor jobs, pods, and logs in real time. Run `k9s -n ` to get a live view of resources, switch between them instantly, and inspect logs or events with a few keystrokes. Compared to kubectl, it is faster and more convenient.
**Stale Kubernetes Job preventing new Job execution:**
```bash theme={null}
kubectl delete job ingestor-job -n
kubectl logs
```
**Storage Issues:**
```bash theme={null}
kubectl get pvc -n
```
***
## Next Steps
* Define and publish your use case: [Define Use Case](/create-use-case/define)
***
## Need Help?
* Email us at [support@tracebloc.io](mailto:support@tracebloc.io)
# Prerequisites
Source: https://docs.tracebloc.io/create-use-case/prerequisites
Supported data types, tasks, and requirements for creating a use case on tracebloc.
tracebloc is continually expanding supported data types and tasks to enable your use cases. In case your use case is not yet supported, please reach out to us at [support@tracebloc.io](mailto:support@tracebloc.io).
Before you can create a use case on the [tracebloc website](https://ai.tracebloc.io/my-use-cases), make sure the following requirements are met:
* You are registered as a user on the tracebloc platform
* You have a client deployed using kubernetes either [locally or in the cloud](/environment-setup/setup-guide)
* Your dataset is cleaned and preprocessed
* You are familiar with the supported data types and tasks
Once these requirements are met, proceed with:
* [Preparing and ingesting the dataset](/create-use-case/prepare-dataset)
* [Defining the use case](/create-use-case/define)
* [Setting evaluation metrics](/create-use-case/define#supported-metrics-per-data-type-and-task)
* [Evaluating models](/create-use-case/evaluate-models)
## Supported Data Types and Tasks
The exact folder layout, CSV columns, `ingest.yaml` and validation rules for each task are on the [dataset template pages](/create-use-case/templates). Where this overview and a template page differ, the template page is authoritative — it is derived from the ingestor's own checks.
### Image Data
**Requirements for all image data tasks:** Uniform image sizes and uniform file types. For example all images as 256x256 rgb .jpg files. Convert files if necessary and in case your images do not fit the supported size, crop or resize accordingly.
**Filenames in the label CSV:** For all image and text tasks, the `filename` column in the label CSV **must not include the file extension** (e.g. use `cat01`, not `cat01.jpg`). The extension is configured once on the ingestor side via `file_options.extension` in the template and applied to every row at ingestion time.
All images are validated before ingestion by the data ingestor. The ingestion process only starts when every file meets the requirements. Fix or remove any invalid images, then retry.
| Task | Input file type | Color mode | Supported image size | Label file type | Requirements | Links |
| --------------------- | --------------- | ------------------------------------------------------------ | ----------------------- | --------------- | ----------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------- |
| Classification | PNG, JPG, JPEG | rgb (3 channels) or grayscale (1 channel), 8-bit per channel | Square (height = width) | CSV | Uniform image size and file type per dataset | [Detailed structure](#image-classification) [Template](/create-use-case/templates/image-classification) |
| Keypoint Detection | PNG, JPG, JPEG | rgb (3 channels) or grayscale (1 channel), 8-bit per channel | Square (height = width) | CSV | Uniform image sizes Same number of keypoints per image and class | [Detailed structure](#image-keypoint-detection) [Template](/create-use-case/templates/keypoint-detection) |
| Object Detection | PNG, JPG, JPEG | rgb (3 channels) or grayscale (1 channel), 8-bit per channel | Square (height = width) | Pascal VOC | Uniform image sizes, one xml per image | [Detailed structure](#image-object-detection) [Template](/create-use-case/templates/object-detection) |
| Semantic Segmentation | PNG, JPG, JPEG | rgb (3 channels) or grayscale (1 channel), 8-bit per channel | Square (height = width) | PNG, JPG, JPEG | Uniform image and mask sizes | [Detailed structure](#image-semantic-segmentation) [Template](/create-use-case/templates/semantic-segmentation) |
### Image Classification
```structure theme={null}
train/
labels.csv
images/
cat01.jpg
dog02.jpg
...
test/
...
```
```labels.csv theme={null}
filename,label
cat01,cat
dog02,dog
...
```
The `filename` column must not include the file extension. Set the expected extension once via `file_options.extension` in the ingestor template.
### Image Keypoint Detection
The number of keypoints per class and per image must be fixed. For example, in a person/car keypoint detection project, both classes must define the same keypoints, and every image must contain the full set for its class. You cannot mix classes with different keypoint counts (e.g., 16 for person and 32 for car) or annotate some images with fewer keypoints for the same class.
```structure theme={null}
train/
annotations.csv
images/
image01.png
image02.png
...
test/
...
```
```annotations.csv theme={null}
filename,label,x,y,visibility
image01,person,100,150,2
image01,car,120,140,1
image02,person,95,155,0
image02,car,115,145,2
```
* **X and Y** determine the X-/Y-coordinates of a keypoint
* **Visibility** indicates whether a keypoint is visible in the image or not: 0 = not visible (point outside the image or point is in the image but occluded), 1 = visible
* **Filename** should not include the file extension.
### Image Object Detection
The filename like "street01.png" specifies the link between images and annotations. XML-file annotations are in Pascal VOC format. The labels.csv contains a global list of all images and objects.
```structure theme={null}
train/
labels.csv
images/
street01.png
street02.png
...
annotations/
street01.xml
street02.xml
...
test/
...
```
```xml theme={null}
street01.jpg
```
Each row represents one detected object, not one image. An image with multiple objects will have multiple rows.
```labels.csv theme={null}
filename,image_label
street01,car
street01,car
street01,person
street02,car
...
```
The `filename` column links each row to its image and to the matching XML annotation file. The `image_label` column holds the class name for each object instance — one row per object. Filenames should not include the file extension.
### Image Semantic Segmentation
Each mask is an rgb image whose pixel values map to classes defined in labels.csv. The labels.csv contains a global list per image and class. All masks must exactly match their corresponding image sizes and file names.
For binary segmentation (two classes), provide a single-channel grayscale mask where background pixels are black (0) and foreground pixels are white (255).
For three or more classes, supply an RGB mask where each class is represented by a unique color (or pixel value). The filename should not include the file extension.
```structure theme={null}
train/
labels.csv
images/
scene01.png
scene02.png
...
masks/
scene01.png
scene02.png
...
test/
...
```
```labels.csv theme={null}
filename,mask_filename,label,colour
image1,mask1,road,#FFFFFF
image1,mask1,background,#000000
image2,mask2,background,#000000
image2,mask2,road,#FFFFFF
...
```
### Tabular Data
**Requirements for all tabular data tasks:** Each dataset must be provided as a single CSV file with a header row. Every column must contain uniform data types, for example numeric values for features and a categorical or alphanumeric column for labels. Use UTF-8 encoding with comma separators and validate that your schema matches the expected types. Invalid rows are skipped by the ingestor.
| Task | Data file type | Requirements | Links |
| ------------------------ | ------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------- |
| Classification | CSV (features and label in one single file) | Uniform data formats per column. Feature columns: Numeric Label columns: Alphanumeric | [Detailed structure](#tabular-classification) [Template](/create-use-case/templates/tabular-classification) |
| Regression | CSV (features and label in one single file) | Uniform data formats per column. Feature columns: Numeric Label column: Numeric (continuous target) | [Detailed structure](#tabular-regression) [Template](/create-use-case/templates/tabular-regression) |
| Time Series Forecasting | CSV (timestamp, features and target in one single file) | A timestamp column in a parsable format (e.g. `YYYY-MM-DD` or ISO 8601). Feature columns: Numeric Target column: Numeric | [Detailed structure](#time-series-forecasting) [Template](/create-use-case/templates/time-series-forecasting) |
| Time to Event Prediction | CSV (features, time and event in one single file) | A `time` column (duration until event or censoring, integer or numeric). An event column (binary 0/1 indicating whether the event occurred). Feature columns: Numeric | [Detailed structure](#time-to-event-prediction) [Template](/create-use-case/templates/time-to-event-prediction) |
### Tabular Classification
Include a header row with clear column names, using a dedicated column for the labels. An `id` column is recommended but not required.
```csv theme={null}
id,feature1,feature2,feature3,label
1,1.5,2.3,0.8,class_a
2,2.1,1.9,1.2,class_b
3,0.9,3.1,0.5,class_a
...
```
### Tabular Regression
Same structure as Tabular Classification, but the label column holds a continuous numeric target (not a class).
```csv theme={null}
id,square_feet,bedrooms,age,price
1,1668.08,3,15,285.50
2,1701.78,4,12,320.75
3,1697.01,2,8,245.30
...
```
### Time Series Forecasting
Provide a single CSV with a timestamp column, one or more numeric feature columns, and the numeric target column you want to forecast. Rows must be ordered by time and use a consistent timestamp format.
```csv theme={null}
timestamp,feature_1,feature_2,target
2023-10-01,7,1,125.50
2023-10-02,1,0,132.30
2023-10-03,2,0,128.75
...
```
### Time to Event Prediction
Provide a single CSV with feature columns, a `time` column (duration to event or censoring), and a binary event column (1 = event occurred, 0 = censored).
```csv theme={null}
age,feature_1,feature_2,time,event
75,0,1.9,4,1
55,0,1.1,6,1
65,0,1.3,7,0
...
```
### Text Data
| Task | Input files | Label file type | Requirements | Links |
| -------------- | ----------- | --------------- | -------------------------- | ----------------------------------------------------------------------------------------------------------- |
| Classification | TXT | CSV | Text file may not be empty | [Detailed structure](#text-classification) [Template](/create-use-case/templates/text-classification) |
### Text Classification
The `filename` column must not include the file extension. The extension is set once via `file_options.extension` in the ingestor template (e.g. `FileExtension.TXT`).
```structure theme={null}
train/
labels.csv
texts/
review01.txt
review02.txt
...
test/
...
```
```labels.csv theme={null}
filename,label
review01,positive
review02,negative
...
```
```text file example theme={null}
# review01.txt
This product is amazing! I love it.
```
***
## Next Steps
* Prepare and ingest your dataset: [Prepare Data](/create-use-case/prepare-dataset)
***
## Need Help?
* Email us at [support@tracebloc.io](mailto:support@tracebloc.io)
# Dataset templates
Source: https://docs.tracebloc.io/create-use-case/templates
The dataset layout, ingest.yaml and validation rules for every task tracebloc supports — one page per task, ready to copy.
Every task tracebloc supports has a dataset template: the exact folder layout the data ingestor expects, a labels or data CSV in the right shape, a ready-to-edit `ingest.yaml`, and the list of checks the ingestor runs before a single row is stored. Pick your task below, lay your data out the same way, and follow [Prepare Data](/create-use-case/prepare-dataset) to run the ingest.
## Templates by task
| Data | Task | Template |
| ----------- | --------------------------------- | ---------------------------------------------------------------------------- |
| Image | Image classification | [Layout and config](/create-use-case/templates/image-classification) |
| Image | Object detection | [Layout and config](/create-use-case/templates/object-detection) |
| Image | Keypoint detection | [Layout and config](/create-use-case/templates/keypoint-detection) |
| Image | Semantic segmentation | [Layout and config](/create-use-case/templates/semantic-segmentation) |
| Text | Text classification | [Layout and config](/create-use-case/templates/text-classification) |
| Text | Token classification | [Layout and config](/create-use-case/templates/token-classification) |
| Text | Sentence-pair classification | [Layout and config](/create-use-case/templates/sentence-pair-classification) |
| Text | Masked language modeling | [Layout and config](/create-use-case/templates/masked-language-modeling) |
| Text | Causal language modeling | [Layout and config](/create-use-case/templates/causal-language-modeling) |
| Text | Sequence-to-sequence | [Layout and config](/create-use-case/templates/seq2seq) |
| Text | Embeddings | [Layout and config](/create-use-case/templates/embeddings) |
| Tabular | Tabular classification | [Layout and config](/create-use-case/templates/tabular-classification) |
| Tabular | Tabular regression | [Layout and config](/create-use-case/templates/tabular-regression) |
| Time series | Time-series forecasting | [Layout and config](/create-use-case/templates/time-series-forecasting) |
| Time series | Time-series classification | [Layout and config](/create-use-case/templates/time-series-classification) |
| Time series | Survival analysis (time-to-event) | [Layout and config](/create-use-case/templates/time-to-event-prediction) |
## How the templates fit together
Every template follows the same three steps. The task pages only spell out what differs.
1. **Stage the files** on the shared data volume of your secure environment. Inside the ingestor they appear under `/data/shared/`; how to get them there is covered in [Prepare Data](/create-use-case/prepare-dataset#2-stage-your-data-on-the-clusters-shared-pvc).
2. **Write `ingest.yaml`** from the task page. The top of the file is identical for every task; the `category` field names the task and decides which folders, columns and checks apply.
3. **Run the ingest once per split** — once with `intent: train` and once with `intent: test`, each with its own `table` name:
```bash theme={null}
helm install tracebloc/ingestor \
--namespace \
--set-file ingestConfig=./ingest.yaml
```
The ingestor validates the whole dataset first, then copies files, stores rows and registers only metadata with the platform. Raw data never leaves your infrastructure. If any check fails, nothing is stored.
## Where files must live
For tasks with one file per sample (images and text), the ingestor reads every file from a **fixed subfolder name** next to the others: `images/`, `annotations/`, `masks/`, `texts/` or `sequences/`. It derives the dataset root from the folder you name in `ingest.yaml` (the parent of `images:`, `texts:`, and so on), so keep the subfolders side by side and spelled exactly like that:
```text theme={null}
/data/shared//
├── labels.csv # the manifest (any name; point `csv:` at it)
├── images/ # or texts/ or sequences/ — the per-sample files
└── annotations/ # only object detection (masks/ only semantic segmentation)
```
Tabular and time-series tasks have no per-sample files: the CSV named in `csv:` is the whole dataset.
## The `ingest.yaml` contract
These fields are shared by every task. Task-specific fields (`images`, `texts`, `schema`, `target_size`, ...) are explained on the task pages.
| Field | Required | Meaning |
| ----------------------------------------------------- | ------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `apiVersion` | yes | Always `tracebloc.io/v1`. |
| `kind` | yes | Always `IngestConfig`. |
| `category` | yes | The task. One of the 16 identifiers in the table above (for example `image_classification`). |
| `table` | yes | Name of the dataset table in your secure environment. Letters, digits and underscores only; must start with a letter or underscore. Use a different name for the train and test splits, and a new name for every new version — the ingestor refuses to write into a table folder that already holds data. |
| `intent` | yes | `train` or `test`. |
| `csv` | one of `csv` / `json` | Path (inside the ingestor) to the labels CSV or, for tabular tasks, the data CSV. Object detection has neither — see its page. |
| `json` | one of `csv` / `json` | Path to a JSON manifest (a top-level array of records, or one object) instead of a CSV. |
| `label` | depends on task | Either the column name (`label: label`) or an object `{column, policy}`. Regression-class tasks require the object form with an explicit `policy` (`bucket` or `passthrough` — see [Label policy](#label-policy-for-regression-class-tasks)); self-supervised text tasks must not set it at all. See each task page. |
| `schema` | tabular, time-series, semantic segmentation | Map of column name to SQL type. See [SQL types](#sql-types-for-schema). |
| `data_id` | no | How each stored row gets its id. `strategy: content_hash` (default) hashes the row content with a salt that never leaves your secure environment, so a retried run re-uses its rows instead of duplicating them; identical source rows collapse into one. `strategy: uuid` gives every row a fresh id. `strategy: column` with `column: ` copies a column of yours — only safe when that column carries no personal data. |
| `spec.csv_options` | no | `chunk_size` (default 1000), `delimiter` (`,`), `quotechar` (`"`), `escapechar` (`\`). Files are always read as UTF-8. |
| `spec.file_options` | no | `extension` (one of `.jpg`, `.jpeg`, `.png`, `.txt`, `.text`, `.xml`), `target_size` (`[width, height]`), `min_size` (`[width, height]`, default `[32, 32]`). Per-task defaults are listed on each page. |
| `columns` | no | Per-column facts the platform cannot infer, used when datasets are combined: `unit` (for example `years`, `USD`) and `ordinal` (category values in order, low to high). |
| `color_mode`, `bit_depth` | no | Image tasks: `RGB` or `grayscale`; `8` or `16`. |
| `language`, `normalization` | no | Text tasks: the dataset language (for example `en`) and the text normalization you applied (for example `lowercase`). |
| `time_unit`, `event_indicator` | no | Survival analysis: `days`, `weeks`, `months` or `years`, and the integer codes for an observed event and a censored case. |
| `positive_definition` | no | Embeddings: what counts as a positive pair. |
| `spec.validators`, `spec.sidecars`, `spec.processors` | no | Accepted by the schema but **not executed** by the current ingestor — it logs a warning and continues with the task defaults. Leave them out. |
## Rules that apply to every CSV
* The file must be valid UTF-8 without NUL bytes. Excel users: save as **CSV UTF-8**.
* Surrounding whitespace in header names is stripped; duplicate header names are rejected.
* The label column you configure is matched case- and whitespace-insensitively (`Label` satisfies `label: label`).
* Tasks with one file per sample need a column named exactly `filename` (lowercase). Its value may include the file extension or not: `cat1.jpeg` and `cat1` both resolve to `images/cat1.jpeg` when the configured extension is `.jpeg`. A value that already ends in `.jpeg`, `.jpg`, `.png`, `.xml`, `.txt` or `.text` (any case) is used as is.
* The manifest must contain at least one data row, and at least one referenced file must exist.
* String labels have surrounding whitespace stripped before storage. Only a label that is missing from the record is stored as NULL; an empty or whitespace-only label is stored as the empty string `""`. The NA tokens below apply to `schema` columns only — in a label column that is not declared in `schema`, an empty cell is `""` and `NA` or `null` is a genuine class value.
* The `id` column in the sample CSVs is not stored unless you declare it in `schema` — and you cannot declare it under that name. These column names are reserved by the ingestor and must not appear in `schema`: `id`, `created_at`, `updated_at`, `status`, `data_intent`, `data_id`, `filename`, `extension`, `annotation`, `ingestor_id`. The label column is not declared in `schema` either — `label:` names it.
* Column names may be at most 64 characters. Any other character is allowed.
## Checks every ingest runs
Task-specific checks are listed on each page. These run for every task:
| Check | What it rejects |
| ------------------ | ----------------------------------------------------------------------------------------------------------------------------------------------------- |
| Ingestable records | A CSV with a header but no rows; a per-sample-file task whose manifest lacks an exact `filename` column, or whose referenced files are all missing. |
| Label diversity | A classification dataset with fewer than 2 distinct label values (after whitespace stripping). Not run for regression-class or self-supervised tasks. |
| Table name | A `table` outside `^[a-zA-Z_][a-zA-Z0-9_]*$`. Common SQL keywords produce a warning. |
| Duplicate | A destination folder for this `table` that already contains data. Duplicate `filename` values within the manifest produce a warning. |
## Label policy for regression-class tasks
Tabular regression, time-series forecasting and survival analysis have a numeric target, and the platform must never receive raw target values. Their `label` therefore needs the object form with an explicit `policy`:
```yaml theme={null}
label:
column: price
policy: bucket
```
`bucket` replaces each value with one of 64 stable hash buckets **in the metadata sent to the platform only**. The rows stored in your secure environment keep the raw value, which is what training reads. A missing target is reported as bucket `-1`. `passthrough` sends raw values and is only appropriate when you have cleared that with your compliance owner.
## SQL types for `schema`
`VARCHAR(n)`, `CHAR(n)`, `TEXT`, `INT`, `INTEGER`, `TINYINT`, `SMALLINT`, `MEDIUMINT`, `BIGINT`, `FLOAT`, `DOUBLE`, `DECIMAL(p,s)`, `NUMERIC(p,s)`, `BOOLEAN`, `BOOL`, `DATE`, `DATETIME`, `TIMESTAMP`, `TIME`, `BLOB`, `LONGBLOB`. Type names are case-insensitive. A near-miss (`INTERGER`) is rejected with a "did you mean" hint. Every declared column is checked against its type across the whole file: non-numeric values in numeric columns, values over a `VARCHAR` length, `inf`, integers outside the 64-bit range and unparseable dates all fail the ingest. In `schema` columns, empty cells and exactly these tokens are stored as NULL: `NA`, `N/A`, `n/a`, `NULL`, `null`, `None`, `none`, `NaN`, `nan`, ``, `#N/A`. The match is case-sensitive — `Null`, `NONE`, `NAN` or `#n/a` are ordinary values and fail the type check in a numeric column.
## Next steps
* Run the ingest: [Prepare Data](/create-use-case/prepare-dataset)
* Define and publish your use case: [Define Use Case](/create-use-case/define)
## Need help?
Email [support@tracebloc.io](mailto:support@tracebloc.io).
# Causal language modeling
Source: https://docs.tracebloc.io/create-use-case/templates/causal-language-modeling
Dataset template for causal language modeling: raw text or prompt/completion pairs, one per .txt, a manifest CSV without labels, ingest.yaml and the checks the data ingestor runs.
Predict the next word. This task is **self-supervised**: there is no label. Each sample is one `.txt` file of raw text in one of two shapes:
* **Pretraining** — the whole file is plain text.
* **Instruction tuning (SFT)** — one line of the form `promptcompletion`. Everything before the first tab is the prompt, everything after is the completion.
A dataset may mix both shapes. The ingestor does not enforce either shape; it only checks that files are valid text.
## Folder layout
```text theme={null}
/data/shared/dolly-clm/
├── labels.csv
└── texts/
├── clm_0000001.txt # plain text
├── clm_0000002.txt # plain text
├── clm_0000004.txt # promptcompletion
└── ...
```
* The folder is named `texts` (raw text) and sits next to the manifest CSV.
* UTF-8 encoded; one extension across the dataset (`.txt` by default).
## Manifest CSV
```csv theme={null}
filename
clm_0000001
clm_0000002
clm_0000003
```
| Column | Required | Meaning |
| ----------- | ---------------------- | ------------------------------------------------------------------- |
| `filename` | yes, exactly this name | The text file, with or without extension. |
| `extension` | no | Present in the shipped sample (`'.txt'`); not read by the ingestor. |
There is no label column. Setting `label:` in `ingest.yaml` is rejected.
## ingest.yaml
```yaml theme={null}
apiVersion: tracebloc.io/v1
kind: IngestConfig
category: causal_language_modeling
table: dolly_clm_train
intent: train
csv: /data/shared/dolly-clm/labels.csv
texts: /data/shared/dolly-clm/texts/
```
| Field | Required | Meaning |
| ----------------------------- | ------------------- | ------------------------------------------- |
| `csv` | yes | Path to the manifest CSV. |
| `texts` | yes | The `texts/` folder. |
| `label` | **must not be set** | Self-supervised task. |
| `schema` | no | Extra typed columns only. Never `filename`. |
| `spec.file_options.extension` | no | `.txt` (default) or `.text`. |
| `language`, `normalization` | no | Dataset language and applied normalization. |
## What the ingestor checks
| Check | Rejects |
| ------------ | ------------------------------------------------------------------------------------- |
| File type | Files under `texts/` with a different extension than configured, or mixed extensions. |
| Text content | NUL bytes or invalid UTF-8; empty files produce a warning. |
| Data types | With a `schema`: values that do not match their declared type. |
Plus the [checks every ingest runs](/create-use-case/templates#checks-every-ingest-runs), except label diversity (no label). No tokenizer is needed at ingest; the ingestor records a data-derived text profile (script mix and length distribution, never the text) for the platform's tokenizer-fit warning.
## Sample dataset
The template ships five files: three plain-text passages and two `promptcompletion` pairs, with a five-row manifest. The `ingest.yaml` above ingests it with no overrides.
## Next steps
* Stage the data and run the ingest: [Prepare Data](/create-use-case/prepare-dataset)
* Shared rules for every template: [Dataset templates](/create-use-case/templates)
# Embeddings
Source: https://docs.tracebloc.io/create-use-case/templates/embeddings
Dataset template for contrastive embedding training: anchor/positive pairs or anchor/positive/negative triplets, one per .txt, ingest.yaml and the checks the data ingestor runs.
Learn vector representations from text pairs. This task is **self-supervised** (contrastive): there is no label column — the pairing is the supervision. Each sample is one `.txt` file with one tab-separated record:
* a **pair** `anchorpositive` — two texts that should embed close together, or
* a **triplet** `anchorpositivenegative` — with a hard negative that should embed far from the anchor.
Pairs and triplets may be mixed in one dataset.
## Folder layout
```text theme={null}
/data/shared/stsb-embeddings/
├── labels.csv
└── texts/
├── emb_0000001.txt # anchorpositive
├── emb_0000004.txt # anchorpositivenegative
└── ...
```
Example pair:
```text theme={null}
How do I reset my password? What are the steps to recover my account login?
```
* The folder is named `texts` and sits next to the manifest CSV.
* Exactly one tab between fields, all fields non-empty, one record per file on a single line. Unlike the other self-supervised text tasks, this structure **is enforced**.
* UTF-8 encoded; one extension across the dataset (`.txt` by default).
## Manifest CSV
```csv theme={null}
filename
emb_0000001
emb_0000002
emb_0000003
```
| Column | Required | Meaning |
| ----------- | ---------------------- | ------------------------------------------------------------------- |
| `filename` | yes, exactly this name | The text file, with or without extension. |
| `extension` | no | Present in the shipped sample (`'.txt'`); not read by the ingestor. |
There is no label column. Setting `label:` in `ingest.yaml` is rejected.
## ingest.yaml
```yaml theme={null}
apiVersion: tracebloc.io/v1
kind: IngestConfig
category: embeddings
table: stsb_embeddings_train
intent: train
csv: /data/shared/stsb-embeddings/labels.csv
texts: /data/shared/stsb-embeddings/texts/
```
| Field | Required | Meaning |
| ----------------------------- | ------------------- | --------------------------------------------------------------------------------------------------------------------------------------- |
| `csv` | yes | Path to the manifest CSV. |
| `texts` | yes | The `texts/` folder. |
| `label` | **must not be set** | No label column. |
| `positive_definition` | no | Free text describing what makes a pair positive (for example `paraphrase`), recorded for consistency checks when datasets are combined. |
| `schema` | no | Extra typed columns only. Never `filename`. |
| `spec.file_options.extension` | no | `.txt` (default) or `.text`. |
| `language`, `normalization` | no | Dataset language and applied normalization. |
## What the ingestor checks
| Check | Rejects |
| ----------------- | ------------------------------------------------------------------------------------------------------------------- |
| File type | Files under `texts/` with a different extension than configured, or mixed extensions. |
| Text content | NUL bytes or invalid UTF-8; empty files produce a warning. |
| Contrastive pairs | A referenced file that is missing, spans several lines, has a field count other than 2 or 3, or has an empty field. |
| Data types | With a `schema`: values that do not match their declared type. |
Plus the [checks every ingest runs](/create-use-case/templates#checks-every-ingest-runs), except label diversity (no label).
## Sample dataset
The template ships three pairs and two triplets with a five-row manifest. The `ingest.yaml` above ingests it with no overrides.
## Next steps
* Stage the data and run the ingest: [Prepare Data](/create-use-case/prepare-dataset)
* Shared rules for every template: [Dataset templates](/create-use-case/templates)
# Image classification
Source: https://docs.tracebloc.io/create-use-case/templates/image-classification
Dataset template for image classification: folder layout, labels CSV, ingest.yaml and the checks the data ingestor runs.
Sort images into classes. Each sample is one image file plus one row in a labels CSV that names the file and its class.
## Folder layout
```text theme={null}
/data/shared/cats-dogs/
├── labels.csv
└── images/
├── cat1.jpeg
├── cat2.jpeg
├── dog1.jpeg
└── ...
```
* The image folder must be named `images` and sit next to the labels CSV.
* Every image in the dataset must have the **same extension** and the **same width and height**. Mixed extensions or mixed resolutions fail the ingest.
* Supported extensions: `.jpg`, `.jpeg`, `.png`. The default is `.jpeg`; override it with `spec.file_options.extension`.
* The ingestor copies images as they are. It does not resize them — bring them to the target size before you ingest.
## Labels CSV
```csv theme={null}
filename,label
cat1.jpeg,cat
cat2.jpeg,cat
dog1.jpeg,dog
dog2.jpeg,dog
```
| Column | Required | Meaning |
| ---------- | ---------------------- | ----------------------------------------------------------------------------------------------------------------- |
| `filename` | yes, exactly this name | The image file, with or without extension (`cat1` resolves to `images/cat1.jpeg` under the default extension). |
| `label` | yes | The class of the image. Any column name works — set it with `label:`. At least two distinct classes are required. |
Extra columns are ignored unless you declare them in `schema`.
## ingest.yaml
```yaml theme={null}
apiVersion: tracebloc.io/v1
kind: IngestConfig
category: image_classification
table: cats_dogs_train
intent: train
csv: /data/shared/cats-dogs/labels.csv
images: /data/shared/cats-dogs/images/
label: label
```
| Field | Required | Meaning |
| ----------------------------- | -------- | ------------------------------------------------------------------------------------------------- |
| `csv` | yes | Path to the labels CSV inside the ingestor. |
| `images` | yes | The `images/` folder. Its parent becomes the dataset root. |
| `label` | yes | Name of the class column in the CSV. |
| `target_size` | no | `[width, height]` every image must have. Default `[256, 256]`. |
| `spec.file_options.extension` | no | `.jpg`, `.jpeg` or `.png`. Default `.jpeg`. |
| `spec.file_options.min_size` | no | Absolute minimum `[width, height]`; images with a smaller side are rejected. Default `[32, 32]`. |
| `color_mode`, `bit_depth` | no | Declare `RGB` or `grayscale` and `8` or `16` so combined datasets can be checked for consistency. |
To use images of another size, set `target_size` (or `spec.file_options.target_size`) to their exact dimensions:
```yaml theme={null}
target_size: [512, 512]
spec:
file_options:
extension: .png
```
## What the ingestor checks
| Check | Rejects |
| ---------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| File type | Any file under `images/` whose extension is not the configured one, or a mix of extensions. |
| Image resolution | Any image whose `(width, height)` differs from `target_size`, more than one resolution in the folder, images with a side below `min_size`, and unreadable or empty image files. |
| Label column | A CSV whose header has no column matching `label:`. |
| Label diversity | Fewer than two distinct label values. |
Plus the [checks every ingest runs](/create-use-case/templates#checks-every-ingest-runs).
## Sample dataset
The template ships six 256×256 RGB JPEG images (three cats, three dogs) and a labels CSV with the two classes `cat` and `dog`. The `ingest.yaml` above ingests it with no overrides.
## Next steps
* Stage the data and run the ingest: [Prepare Data](/create-use-case/prepare-dataset)
* Shared rules for every template: [Dataset templates](/create-use-case/templates)
# Keypoint detection
Source: https://docs.tracebloc.io/create-use-case/templates/keypoint-detection
Dataset template for keypoint detection: images plus per-image keypoint JSON in the labels CSV, ingest.yaml and the checks the data ingestor runs.
Locate landmark points, for example body joints. Each sample is one image plus one CSV row that carries the keypoint coordinates, their visibility flags and a class label.
## Folder layout
```text theme={null}
/data/shared/pose/
├── labels.csv
└── images/
├── person_001.jpg
├── person_002.jpg
└── ...
```
* The image folder must be named `images` and sit next to the labels CSV.
* All images share one extension (`.jpg`, `.jpeg` or `.png`; default `.jpg`) and one resolution, which you declare in `target_size`. The ingestor copies them unchanged.
## Labels CSV
```csv theme={null}
filename,Annotation,Visibility,image_label
person_001,"{""nose"": [0.50, 0.20], ""left_eye"": [0.46, 0.16], ""right_eye"": [0.54, 0.16], ""left_shoulder"": [0.37, 0.39], ""right_shoulder"": [0.63, 0.39], ""left_elbow"": [0.31, 0.59], ""right_elbow"": [0.68, 0.59], ""left_wrist"": [0.27, 0.76], ""right_wrist"": [0.72, 0.76]}","{""nose"": 1, ""left_eye"": 1, ""right_eye"": 1, ""left_shoulder"": 1, ""right_shoulder"": 1, ""left_elbow"": 1, ""right_elbow"": 1, ""left_wrist"": 1, ""right_wrist"": 1}",person
```
| Column | Required | Meaning |
| ------------- | ---------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `filename` | yes, exactly this name | The image file, with or without extension. |
| `Annotation` | yes, exactly this name | A JSON object mapping each keypoint name to `[x, y]` (a `{"x": .., "y": ..}` object is also accepted). Quote the JSON and double the inner quotes, as in the sample. |
| `Visibility` | yes, exactly this name | A JSON object with the **same keys** as `Annotation`, each `1` (visible) or `0` (occluded or out of frame). |
| `image_label` | yes | The class of the image. Any column name works — set it with `label:`. At least two distinct classes are required. |
Rules for `Annotation`:
* Every row must name exactly `number_of_keypoints` keypoints, and every row must use the same keypoint names as the first row.
* Coordinates must be numeric and non-negative, with `x < width` and `y < height` of `target_size`.
* At least two keypoints must differ in both x and y, so the keypoints span a real bounding box.
## ingest.yaml
```yaml theme={null}
apiVersion: tracebloc.io/v1
kind: IngestConfig
category: keypoint_detection
table: pose_train
intent: train
csv: /data/shared/pose/labels.csv
images: /data/shared/pose/images/
label: image_label
target_size: [448, 448] # width, height — must match your images
number_of_keypoints: 9 # 17 for COCO pose; 9 for the shipped sample
```
| Field | Required | Meaning |
| ----------------------------- | -------- | ---------------------------------------------------------------------------------------------- |
| `csv` | yes | Path to the labels CSV. |
| `images` | yes | The `images/` folder. |
| `label` | yes | Name of the class column. |
| `target_size` | **yes** | `[width, height]` of every image. There is no default for this task — your pose model decides. |
| `number_of_keypoints` | **yes** | Keypoints per sample. Every row's `Annotation` must have exactly this many entries. |
| `spec.file_options.extension` | no | `.jpg`, `.jpeg` or `.png`. Default `.jpg`. |
| `spec.file_options.min_size` | no | Minimum `[width, height]`. Default `[32, 32]`. |
| `color_mode`, `bit_depth` | no | `RGB` or `grayscale`; `8` or `16`. |
## What the ingestor checks
| Check | Rejects |
| ------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| File type | Files under `images/` with a different extension than configured, or mixed extensions. |
| Image resolution | Images whose size differs from `target_size`, mixed resolutions, images below `min_size`, unreadable files. |
| Keypoint annotation | A missing `Annotation` column; invalid JSON; a row with a keypoint count other than `number_of_keypoints`; keypoint names that differ between rows; non-numeric, negative or out-of-image coordinates; a degenerate bounding box. |
| Keypoint visibility | A missing `Visibility` column; invalid JSON; values other than `0`/`1`; keys that do not match the row's `Annotation` keys. |
| Label diversity | Fewer than two distinct label values. |
Plus the [checks every ingest runs](/create-use-case/templates#checks-every-ingest-runs).
## Sample dataset
The template ships three 448×448 RGB JPEG images with nine upper-body keypoints each (`nose`, `left_eye`, `right_eye`, `left_shoulder`, `right_shoulder`, `left_elbow`, `right_elbow`, `left_wrist`, `right_wrist`) and three classes. The `ingest.yaml` above ingests it as is.
## Next steps
* Stage the data and run the ingest: [Prepare Data](/create-use-case/prepare-dataset)
* Shared rules for every template: [Dataset templates](/create-use-case/templates)
# Masked language modeling
Source: https://docs.tracebloc.io/create-use-case/templates/masked-language-modeling
Dataset template for masked language modeling: one token sequence per .txt under sequences/, a manifest CSV without labels, ingest.yaml and the checks the data ingestor runs.
Predict masked-out words. This task is **self-supervised**: there is no label — the training side masks tokens on the fly. Each sample is one `.txt` file holding a space-separated token sequence, listed in a manifest CSV.
## Folder layout
```text theme={null}
/data/shared/primekg-mlm/
├── labels.csv
└── sequences/
├── seq_0000001.txt
├── seq_0000002.txt
└── ...
```
* The folder is named `sequences` (not `texts`) — the ingestor reserves `sequences/` for pre-tokenized data — and sits next to the manifest CSV.
* One sequence per file, tokens separated by spaces, UTF-8 encoded, one extension across the dataset (`.txt` by default). Example from the shipped sample: `Lepirudin indication Huntington phenotype_present Chorea associated_with Dystonia`.
* The shipped sample also contains a `tokenizer.json` next to the CSV. The current ingestor neither reads nor copies it; it is part of the sample only.
## Manifest CSV
```csv theme={null}
filename
seq_0000001
seq_0000002
seq_0000003
```
| Column | Required | Meaning |
| ----------- | ---------------------- | ------------------------------------------------------------------- |
| `filename` | yes, exactly this name | The sequence file, with or without extension. |
| `extension` | no | Present in the shipped sample (`'.txt'`); not read by the ingestor. |
There is no label column. Setting `label:` in `ingest.yaml` is rejected.
## ingest.yaml
```yaml theme={null}
apiVersion: tracebloc.io/v1
kind: IngestConfig
category: masked_language_modeling
table: primekg_mlm_train
intent: train
csv: /data/shared/primekg-mlm/labels.csv
sequences: /data/shared/primekg-mlm/sequences/
```
| Field | Required | Meaning |
| ----------------------------- | ------------------- | ------------------------------------------------------------- |
| `csv` | yes | Path to the manifest CSV. |
| `sequences` | yes | The `sequences/` folder. Its parent becomes the dataset root. |
| `label` | **must not be set** | Self-supervised task. |
| `schema` | no | Extra typed columns only. Never `filename`. |
| `spec.file_options.extension` | no | `.txt` (default) or `.text`. |
| `language`, `normalization` | no | Dataset language and applied normalization. |
## What the ingestor checks
| Check | Rejects |
| ------------ | ----------------------------------------------------------------------------------------- |
| File type | Files under `sequences/` with a different extension than configured, or mixed extensions. |
| Text content | NUL bytes or invalid UTF-8; empty files produce a warning. |
| Data types | With a `schema`: values that do not match their declared type. |
Plus the [checks every ingest runs](/create-use-case/templates#checks-every-ingest-runs), except label diversity (no label). For text tasks the ingestor also records a data-derived text profile — Unicode-script mix and document-length distribution, never the text itself — so the platform can warn when a model's tokenizer is a poor fit for the dataset.
## Sample dataset
The template ships five knowledge-graph random-walk sequences and a five-row manifest. The `ingest.yaml` above ingests it with no overrides.
## Next steps
* Stage the data and run the ingest: [Prepare Data](/create-use-case/prepare-dataset)
* Shared rules for every template: [Dataset templates](/create-use-case/templates)
# Object detection
Source: https://docs.tracebloc.io/create-use-case/templates/object-detection
Dataset template for object detection: images plus Pascal VOC XML annotations, ingest.yaml and the checks the data ingestor runs.
Draw boxes around objects. Each sample is one image plus one Pascal VOC XML file that lists the objects in it. There is **no labels CSV**: the ingestor reads the image list and the classes straight from the XML files.
## Folder layout
```text theme={null}
/data/shared/visdrone/
├── images/
│ ├── 0000001_02999_d_0000005.jpg
│ └── ...
└── annotations/
├── 0000001_02999_d_0000005.xml
└── ...
```
* Both folders must have exactly these names and sit side by side.
* Images and annotations pair by **file stem**: `images/frame01.jpg` belongs to `annotations/frame01.xml`. Every image needs its XML and every XML needs its image.
* All images share one extension (`.jpg`, `.jpeg` or `.png`; default `.jpg`) and one resolution. The ingestor copies them unchanged and does not resize.
## Annotation format (Pascal VOC)
```xml theme={null}
imagesframe01.jpgUnknownPASCAL VOC1920108030
```
Every element shown is required:
| Element | Rule |
| ----------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `folder`, `filename` | Non-empty text. |
| `source/database`, `source/annotation` | Non-empty text. |
| `size/width`, `size/height`, `size/depth` | Positive integers. `width` and `height` must equal the actual image dimensions. |
| `segmented` | `0` or `1`. |
| `object/name` | The class. Non-empty. |
| `object/pose` | Non-empty text (`Unspecified` is fine). |
| `object/truncated` | `0` or `1`. |
| `object/difficult` | A non-negative integer. Values other than `0`/`1` are accepted with a warning. |
| `object/bndbox` | Integer `xmin`, `ymin`, `xmax`, `ymax` with `xmin < xmax`, `ymin < ymax`, all non-negative, and `xmax`/`ymax` within the declared image size. A box with area below 10 pixels produces a warning. |
An XML file with no `