Define Use Case

This guide walks you through creating and publishing an AI use case on the tracebloc platform. The Create Use Case panel has two steps. Make sure you have a secure environment running and your training and test data is ingested. Go to the use cases section, click Create a Use Case and follow along.

Step 1: Define your use case

Use case title: Up to 82 characters.
Who can join this use case?: Private (invite-only) or Public (visible to all users).
What task should contributors work on?: The task, for example Object Detection. It filters the training datasets offered in step 2. See the full list of supported data types and tasks. If your use case is not yet supported, reach out to us at [email protected].

Click Next.

Step 2: Data & Evaluation

Select your training dataset: The training datasets you ingested in the Prepare Data step, for the task you picked.
Select your test dataset: The test and score lists appear after you pick the training dataset. The test list shows only test sets compatible with it. When the task has classes, the test set’s classes must be a subset of the training set’s classes. A test set missing from the list is not compatible.
Select your score: The metric that ranks submissions on the leaderboard. The list holds the metrics in the tables below for your task. If your metric is not supported, reach out to us at [email protected].
Upload EDA File (optional): A .ipynb or .json file, up to 25 MB, to help peers understand the data. Explore template use cases for inspiration.

Click Create Use Case. The use case is published at once; there is no draft. The panel sets these values for you:

Description: The title. Edit it on the use case’s Overview tab with Edit as admin.
Compute budget: 1 PF per peer. Change it on the Admin tab under Compute usage with Adjust limit.
End date: None, so the use case runs until you end it. Set one on the Admin tab under Duration.

To invite peers, open the use case and click the Invite team members icon next to the team avatars. For how peers join, see the join a use case guide.

Supported metrics per data type and task

The tables below list the metrics you can pick as the score for each task today. Each metric uses either higher is better or lower is better sort order on the leaderboard. Some tasks report more metrics during training than you can pick as the score; the notes under those tables name them.

Image Classification

Metric	Description	Sort Order
Accuracy	Proportion of predictions that exactly match the ground truth. Can be misleading on imbalanced datasets.	Higher is better
Precision	Measures the proportion of predicted positives that are actually positive. Important when false positives are costly.	Higher is better
Recall	Measures the proportion of actual positives correctly identified. Important when missing a positive instance is costly.	Higher is better
F1 Score	Balances precision and recall into a single metric. Especially useful for imbalanced datasets.	Higher is better
Loss	Quantifies the error between predicted outputs and actual values. A core metric used during training and optimization.	Lower is better
Log Loss	Measures how well a model predicts probability estimates for each class. Penalizes overconfident incorrect predictions.	Lower is better
AUC-ROC	Measures ability to distinguish between classes across all thresholds, independent of any single decision threshold.	Higher is better
AUC-PR	Measures precision-recall balance across different thresholds. Especially useful for highly imbalanced datasets.	Higher is better
Top-3 Accuracy	Measures how often the true class label appears among the model’s top three predictions.	Higher is better
Top-5 Accuracy	Measures how often the true class label appears among the model’s top five predictions.	Higher is better
Cohen’s Kappa	Measures agreement between predicted and ground truth labels while accounting for chance agreement.	Higher is better
Matthews Correlation Coefficient (MCC)	Classification quality using all parts of the confusion matrix. Balanced even with imbalanced classes. Ranges from -1 to 1.	Higher is better
Quadratic Weighted Kappa (QWK)	Measures agreement between predicted and ground truth labels, penalizing larger disagreements more heavily.	Higher is better
Brier Score	Mean squared difference between predicted probabilities and actual outcomes.	Lower is better

Text Classification

Metric	Description	Sort Order
Accuracy	Proportion of predictions that exactly match the ground truth. Can be misleading on imbalanced datasets.	Higher is better
Precision	Measures the proportion of predicted positives that are actually positive. Important when false positives are costly.	Higher is better
Recall	Measures the proportion of actual positives correctly identified. Important when missing a positive instance is costly.	Higher is better
F1 Score	Balances precision and recall into a single metric. Especially useful for imbalanced datasets.	Higher is better
F1 Weighted	Weights each class’s F1 Score by its support (number of true instances). Suitable for imbalanced multi-class classification.	Higher is better
Micro F1	Aggregates true positives, false positives, and false negatives across all classes, treating every prediction equally.	Higher is better
Loss	Quantifies the error between predicted outputs and actual values. A core metric used during training and optimization.	Lower is better
Log Loss	Measures how well a model predicts probability estimates for each class. Penalizes overconfident incorrect predictions.	Lower is better
AUC-ROC	Measures ability to distinguish between classes across all thresholds, independent of any single decision threshold.	Higher is better
Hamming Loss	Measures the fraction of labels incorrectly predicted. Commonly used in multi-label classification tasks.	Lower is better
Jaccard Score	Measures similarity between predicted and ground truth labels by comparing their intersection to their union.	Higher is better
Cohen’s Kappa	Measures agreement between predicted and ground truth labels while accounting for chance agreement.	Higher is better
Matthews Correlation Coefficient (MCC)	Classification quality using all parts of the confusion matrix. Balanced even with imbalanced classes. Ranges from -1 to 1.	Higher is better
Quadratic Weighted Kappa (QWK)	Measures agreement between predicted and ground truth labels, penalizing larger disagreements more heavily.	Higher is better
Brier Score	Mean squared difference between predicted probabilities and actual outcomes.	Lower is better

Sentence Pair Classification

Metric	Description	Sort Order
Accuracy	Proportion of sentence pairs whose predicted label exactly matches the ground truth.	Higher is better

Training also reports precision, recall, F1 score, MCC and other classification metrics, but Accuracy is the only score you can pick today.

Token Classification

Token-level scores are computed over labelled tokens only: padding, special tokens and sub-word continuations are excluded.

Metric	Description	Sort Order
Accuracy	Token accuracy: the fraction of labelled tokens whose predicted tag matches the ground truth tag (for example a BIO tag in named entity recognition).	Higher is better

Training also reports token accuracy (same value as Accuracy) and precision, recall and F1 score, macro-averaged across tag classes. Accuracy is the only score you can pick today.

Masked Language Modeling

Metric	Description	Sort Order
Masked Token Accuracy	Fraction of masked positions where the model predicts the original token. Only the masked positions (about 15% of tokens) are scored. 1.0 means every masked token is reconstructed.	Higher is better
Perplexity	Exponential of the average cross-entropy loss on the validation data: how “surprised” the model is by the correct tokens. 1.0 is a perfect prediction; a value near the vocabulary size means random guessing. Capped at e^100.	Lower is better

Causal Language Modeling

Causal language modeling is next-token prediction, so its scores are token-level. Each validation sequence is scored at every position that has a training target: the model’s most likely next token is compared with the actual next token. Padding is excluded, and so is the prompt part of a prompt<TAB>completion row, so only completion tokens count.

Metric	Description	Sort Order
Accuracy	Next-token accuracy: the fraction of scored positions where the model’s most likely next token equals the actual next token. It is not an exact match of whole generated completions: a completion with one wrong token still scores its other tokens as correct.	Higher is better
Perplexity	Exponential of the average cross-entropy loss on the validation data: how “surprised” the model is by the actual next tokens. 1.0 is a perfect prediction; a value near the vocabulary size means random guessing. Capped at e^100.	Lower is better

Seq2Seq

Seq2Seq scores are token-level over the target (decoder) sequence, with padding excluded.

Metric	Description	Sort Order
Accuracy	Token accuracy: the fraction of target positions where the model’s most likely token equals the reference token.	Higher is better

Training also reports token accuracy (same value as Accuracy) and perplexity, but Accuracy is the only score you can pick today.

Embeddings

Embeddings are trained with a contrastive objective on anchor/positive pairs, so the scores measure the embedding space rather than class labels.

The only score offered for embeddings today is Perplexity, and an embeddings run does not report it. The leaderboard of an embeddings use case therefore cannot rank submissions yet. Contact [email protected] before you create one.

An embeddings run reports these metrics during training, but you cannot pick them as the score today:

Metric	Description	Sort Order
Average Cosine Similarity	Mean cosine similarity between each anchor and its positive across the validation set. Roughly -1 to 1; it climbs toward 1 as matching pairs move closer together.	Higher is better
Retrieval Accuracy	Fraction of anchors whose nearest candidate (among the other pairs in the batch and any hard negatives) is its own positive. Ranges from 0 to 1.	Higher is better
Average Contrastive Loss	Mean in-batch contrastive (InfoNCE) loss over the validation set.	Lower is better

Tabular Classification

Metric	Description	Sort Order
Accuracy	Proportion of predictions that exactly match the ground truth. Can be misleading on imbalanced datasets.	Higher is better
Precision	Measures the proportion of predicted positives that are actually positive. Important when false positives are costly.	Higher is better
Recall	Measures the proportion of actual positives correctly identified. Important when missing a positive instance is costly.	Higher is better
F1 Score	Balances precision and recall into a single metric. Especially useful for imbalanced datasets.	Higher is better
Loss	Quantifies the error between predicted outputs and actual values. A core metric used during training and optimization.	Lower is better
AUC-ROC	Measures ability to distinguish between classes across all thresholds, independent of any single decision threshold.	Higher is better
AUC-PR	Measures precision-recall balance across different thresholds. Especially useful for highly imbalanced datasets.	Higher is better
Balanced Accuracy	Averages recall across all classes, ensuring each class contributes equally regardless of frequency.	Higher is better
Specificity (True Negative Rate)	Measures the proportion of actual negatives correctly identified. Important when false positives must be minimized.	Higher is better
NPV (Negative Predictive Value)	Measures the proportion of predicted negatives that are actually negative. Important when confirming absence matters.	Higher is better
F-beta Score (beta = 0.5)	Balances precision and recall with more emphasis on precision. Suitable when false positives are more costly.	Higher is better
F-beta Score (beta = 2)	Balances precision and recall with more emphasis on recall. Suitable when missing positive instances is more costly.	Higher is better
Hamming Loss	Measures the fraction of labels incorrectly predicted. Commonly used in multi-label classification tasks.	Lower is better
Jaccard Score	Measures similarity between predicted and ground truth labels by comparing their intersection to their union.	Higher is better
Cohen’s Kappa	Measures agreement between predicted and ground truth labels while accounting for chance agreement.	Higher is better
Matthews Correlation Coefficient (MCC)	Classification quality using all parts of the confusion matrix. Balanced even with imbalanced classes. Ranges from -1 to 1.	Higher is better
Quadratic Weighted Kappa (QWK)	Measures agreement between predicted and ground truth labels, penalizing larger disagreements more heavily.	Higher is better
Brier Score	Mean squared difference between predicted probabilities and actual outcomes.	Lower is better
Gini Coefficient	Measures discriminatory power between positive and negative classes. Closely related to AUC-ROC.	Higher is better
Normalized Gini	Scales the Gini Coefficient relative to a perfect model, enabling fair comparison across datasets. Ranges from -1 to 1.	Higher is better

Object Detection

Metric	Description	Sort Order
Accuracy	Share of images in which every ground-truth box is found by a prediction of the same class with IoU above 0.2. Extra boxes are not penalized and no confidence threshold applies, so read it as image-level recall. Measured on validation only, not during training.	Higher is better
Loss	Quantifies the error between predicted outputs and actual values. A core metric used during training and optimization.	Lower is better
Mean Average Precision (mAP)	COCO mAP: average precision averaged over IoU thresholds 0.50 to 0.95.	Higher is better
Mean Average Precision @ IoU 0.50	Evaluates object detection requiring at least 50% overlap between predicted and ground truth bounding boxes.	Higher is better
Mean Average Precision @ IoU 0.75	Stricter variant requiring at least 75% overlap between predicted and ground truth bounding boxes.	Higher is better
Intersection over Union (IoU)	Overlap between predicted and ground truth boxes. Pairs with IoU below 0.2 are discarded, and missed objects do not lower it.	Higher is better
GIoU (Generalized IoU)	Extends standard IoU by penalizing non-overlapping predictions using the smallest enclosing box. Ranges from -1 to 1.	Higher is better
Mean Average Recall @ 1 Detection	Measures how well the model retrieves ground truth objects when only the single top-scoring detection is allowed.	Higher is better
Mean Average Recall @ 10 Detections	Measures retrieval of ground truth objects when up to 10 detections per image are allowed.	Higher is better

For ranking detectors, pick Mean Average Precision @ IoU 0.50: unlike Accuracy and IoU, it penalizes both missed objects and false boxes.

Semantic Segmentation

Metric	Description	Sort Order
Accuracy	Proportion of predictions that exactly match the ground truth.	Higher is better
Precision	Measures the proportion of predicted positives that are actually positive.	Higher is better
Recall	Measures the proportion of actual positives correctly identified.	Higher is better
F1 Score	Balances precision and recall into a single metric.	Higher is better
Loss	Quantifies the error between predicted outputs and actual values.	Lower is better
Intersection over Union (IoU)	Measures overlap between predicted and ground truth regions.	Higher is better
Mean Intersection over Union (mIoU)	Averages IoU across all classes, evaluating how well a model predicts each class region.	Higher is better
Frequency Weighted IoU	Weights each class’s IoU by its relative frequency in the ground truth, giving more importance to dominant classes.	Higher is better
Dice Coefficient	Measures similarity between predicted and ground truth segmentation regions. Especially sensitive to small structures.	Higher is better
Pixel Accuracy	Proportion of correctly classified pixels across the entire image. Can be dominated by frequent classes.	Higher is better
Mean Pixel Accuracy	Averages pixel accuracy per class, giving equal importance to all classes regardless of frequency.	Higher is better
Boundary IoU	Measures how well predicted segmentation boundaries align with ground truth boundaries. Focuses on edge accuracy.	Higher is better
Boundary F1 Score	Combines boundary precision and recall to evaluate how accurately predicted boundaries match ground truth edges.	Higher is better
Hausdorff Distance	Maximum distance between predicted and ground truth boundaries. Captures the worst-case boundary mismatch.	Lower is better
Average Surface Distance (ASD)	Average distance between predicted and ground truth boundary points. Stable measure of overall boundary alignment.	Lower is better

Keypoint Detection

Metric	Description	Sort Order
Precision	Measures the proportion of predicted positives that are actually positive.	Higher is better
Recall	Measures the proportion of actual positives correctly identified.	Higher is better
F1 Score	Balances precision and recall into a single metric.	Higher is better
Loss	Quantifies the error between predicted outputs and actual values.	Lower is better
Mean Absolute Error (MAE)	Average magnitude of errors between predicted and actual values, expressed in the same units as the target.	Lower is better
[email protected]	Keypoints within a normalized distance threshold of 0.05 from ground truth. Strictest variant.	Higher is better
[email protected]	Keypoints within a normalized distance threshold of 0.10 from ground truth.	Higher is better
[email protected]	Keypoints within a normalized distance threshold of 0.20 from ground truth.	Higher is better
[email protected]	Keypoints within a normalized distance threshold of 0.30 from ground truth.	Higher is better
[email protected]	Keypoints within a normalized distance threshold of 0.50 from ground truth. Most lenient variant.	Higher is better
Object Keypoint Similarity (OKS)	Measures similarity between predicted and ground truth keypoints, accounting for scale and localization uncertainty.	Higher is better
Mean Per Joint Position Error (MPJPE)	Average Euclidean distance between predicted and ground truth joint positions. Standard for pose estimation.	Lower is better
Visibility Accuracy	Measures how correctly the model predicts the visibility status of keypoints, independent of spatial localization.	Higher is better

Tabular Regression

Metric	Description	Sort Order
Accuracy	For regression this is the R² value, not an exact-match rate. Pick R² instead: an upcoming release stops reporting R² as Accuracy.	Higher is better
Loss	Quantifies the error between predicted outputs and actual values.	Lower is better
Mean Squared Error (MSE)	Average squared difference between predicted and actual values. Penalizes larger errors more heavily.	Lower is better
Root Mean Squared Error (RMSE)	Square root of MSE, expressing prediction error in the same units as the target variable.	Lower is better
R² (Coefficient of Determination)	Measures how well a regression model explains variance in the target. A value of 1.0 indicates perfect prediction.	Higher is better
Root Mean Squared Logarithmic Error (RMSLE)	Measures error on a logarithmic scale. Useful when target values span several orders of magnitude.	Lower is better
Median Absolute Error (Median AE)	Uses the median of absolute errors instead of the mean. Highly robust to outliers.	Lower is better
Explained Variance	Measures how well the model captures the variance of the target, independent of systematic bias. Ranges up to 1.0.	Higher is better
Mean Bias Error (MBE)	Measures the average bias in predictions. Positive = overestimation, negative = underestimation. The best value is 0, but the leaderboard ranks the highest MBE first, so do not pick it as the score.	Higher ranks first
Max Absolute Error	The single largest absolute difference between predicted and actual values.	Lower is better

Time Series Forecasting

By default, when the data has feature columns besides timestamp and the target, the engine scales the target to the 0–1 range (MinMaxScaler) before training, so error metrics such as MAE and RMSE are in scaled units, not in your data’s units.

Metric	Description	Sort Order
Accuracy	Not an exact-match rate on the continuous target. Pick an error metric such as MAE or RMSE as the score.	Higher is better
Loss	Quantifies the error between predicted outputs and actual values.	Lower is better
Mean Absolute Error (MAE)	Average magnitude of errors between predicted and actual values.	Lower is better
Mean Squared Error (MSE)	Average squared difference between predicted and actual values. Penalizes larger errors more heavily.	Lower is better
Root Mean Squared Error (RMSE)	Square root of MSE.	Lower is better
R² (Coefficient of Determination)	Measures how well a regression model explains variance in the target. A value of 1.0 indicates perfect prediction.	Higher is better
Mean Absolute Percentage Error (MAPE)	Average percentage difference between predicted and actual values. Easy to interpret across different scales.	Lower is better
Symmetric Mean Absolute Percentage Error (SMAPE)	Symmetric variant of MAPE that reduces issues when actual values are close to zero.	Lower is better
Median Absolute Percentage Error (MdAPE)	Uses the median of absolute percentage errors. More robust to outliers.	Lower is better
Theil’s U (U2 Statistic)	Measures forecasting accuracy relative to a naive benchmark. Values below 1.0 mean the model outperforms the baseline.	Lower is better
Max Absolute Error	Captures the single largest absolute difference between predicted and actual values.	Lower is better

Training also reports direction accuracy (how often the model predicts the direction of change correctly), but you cannot pick it as the score today.

Time Series Classification

Time series classification offers the same metrics as Tabular Classification, computed per sequence instead of per row.

Time-to-Event Prediction

Metric	Description	Sort Order
Concordance Index (C-Index)	Measures how well predicted risk scores agree with the observed ordering of event times. Standard for survival analysis.	Higher is better

Next Steps

Once your use case is published, invite peers to train models on it. In the use case view, monitor

total resource consumption
daily submits and user activity
overall leaderboard and submissions

Once models have been submitted, you can compare them in the leaderboard section of a use case.

Need Help?

Email us at [email protected]

Overview

Create a Use Case

Join a Use Case

Environment Setup

Tools & Help

Releases

Step 1: Define your use case

Step 2: Data & Evaluation

Supported metrics per data type and task

Image Classification

Text Classification

Sentence Pair Classification

Token Classification

Masked Language Modeling

Causal Language Modeling

Seq2Seq

Embeddings

Tabular Classification

Object Detection

Semantic Segmentation

Keypoint Detection

Tabular Regression

Time Series Forecasting

Time Series Classification

Time-to-Event Prediction

Next Steps

Need Help?

​Step 1: Define your use case

​Step 2: Data & Evaluation

​Supported metrics per data type and task

​Image Classification

​Text Classification

​Sentence Pair Classification

​Token Classification

​Masked Language Modeling

​Causal Language Modeling

​Seq2Seq

​Embeddings

​Tabular Classification

​Object Detection

​Semantic Segmentation

​Keypoint Detection

​Tabular Regression

​Time Series Forecasting

​Time Series Classification

​Time-to-Event Prediction

​Next Steps

​Need Help?

Step 1: Define your use case

Step 2: Data & Evaluation

Supported metrics per data type and task

Image Classification

Text Classification

Sentence Pair Classification

Token Classification

Masked Language Modeling

Causal Language Modeling

Seq2Seq

Embeddings

Tabular Classification

Object Detection

Semantic Segmentation

Keypoint Detection

Tabular Regression

Time Series Forecasting

Time Series Classification

Time-to-Event Prediction

Next Steps

Need Help?