Skip to main content
This guide walks you through creating and publishing an AI use case on the tracebloc platform. The Create Use Case panel has two steps. Make sure you have a secure environment running and your training and test data is ingested. Go to the use cases section, click Create a Use Case and follow along.

Step 1: Define your use case

  • Use case title: Up to 82 characters.
  • What task should contributors work on?: The task, for example Object Detection. It filters the training datasets offered in step 2. See the full list of supported data types and tasks. If your use case is not yet supported, reach out to us at [email protected].
Click Next.

Step 2: Data & Evaluation

A note at the top of this step says what the panel sets for you: each peer gets a 50 PF compute budget, and you set an end date after publishing.
  • Select your training dataset: The training datasets you ingested in the Prepare Data step, for the task you picked.
  • Select your test dataset: The test and score lists appear after you pick the training dataset. The test list shows only test datasets compatible with it. When the task has classes, the test data’s classes must be a subset of the training data’s classes. Under it, a list names each of your other test datasets for the task and why it isn’t offered, for example “Not offered: class bicycle isn’t in the training data”.
  • Select your score: The metric that ranks submissions on the leaderboard. The list holds the metrics in the tables below for your task. If your metric is not supported, reach out to us at [email protected].
  • What does this use case give?: The use case’s output.
    • Report: the results as a document, with exact numbers ready to cite in a paper.
    • Weights download: each peer can download the weights files of their own experiments.
  • Who can join this use case?: Only you or Peers you invite. Peers join through your invitation; accepting it is accepting the use case’s contract.
  • Purpose: Research (non-commercial) or Commercial.
  • Upload EDA File (optional): A .ipynb or .json file, up to 25 MB, to help peers understand the data. Explore template use cases for inspiration.
The output, who can join and the purpose together are the use case’s type. All three are required, and the panel offers only what your data allows. A choice your data rules out stays visible but disabled, with the note “Only what your data allows is offered”. What your data allows follows the mark each dataset got when you ingested it: Patient data, Open data or Synthetic data. Every dataset of the use case must allow the type: Click Create Use Case. The use case is published at once; there is no draft. The panel sets these values for you:
  • Description: The title. Edit it on the use case’s Overview tab with Edit as admin.
  • Compute budget: 50 PF per peer. Change it on the Admin tab under Compute usage with Adjust limit.
  • End date: None, so the use case runs until you end it. Set one on the Admin tab under Duration.
To invite peers, open the use case and click the Invite team members icon next to the team avatars. For how peers join, see the join a use case guide.

Supported metrics per data type and task

The tables below list the metrics you can pick as the score for each task today. Each metric uses either higher is better or lower is better sort order on the leaderboard. Some tasks report more metrics during training than you can pick as the score; the notes under those tables name them.

Image Classification

Text Classification

Sentence Pair Classification

Training also reports precision, recall, F1 score, MCC and other classification metrics, but Accuracy is the only score you can pick today.

Token Classification

Token-level scores are computed over labelled tokens only: padding, special tokens and sub-word continuations are excluded. Training also reports token accuracy (same value as Accuracy) and precision, recall and F1 score, macro-averaged across tag classes. Accuracy is the only score you can pick today.

Masked Language Modeling

Causal Language Modeling

Causal language modeling is next-token prediction, so its scores are token-level. Each validation sequence is scored at every position that has a training target: the model’s most likely next token is compared with the actual next token. Padding is excluded, and so is the prompt part of a prompt<TAB>completion row, so only completion tokens count.

Seq2Seq

Seq2Seq scores are token-level over the target (decoder) sequence, with padding excluded. Training also reports token accuracy (same value as Accuracy) and perplexity, but Accuracy is the only score you can pick today.

Embeddings

Embeddings are trained with a contrastive objective on anchor/positive pairs, so the scores measure the embedding space rather than class labels. Perplexity is not a score for embeddings: an embeddings run does not report it.

Tabular Classification

Object Detection

For ranking detectors, pick Mean Average Precision @ IoU 0.50: unlike Accuracy and IoU, it penalizes both missed objects and false boxes.

Semantic Segmentation

Keypoint Detection

Tabular Regression

Time Series Forecasting

By default, when the data has feature columns besides timestamp and the target, the engine scales the target to the 0–1 range (MinMaxScaler) before training, so error metrics such as MAE and RMSE are in scaled units, not in your data’s units. Training also reports direction accuracy (how often the model predicts the direction of change correctly), but you cannot pick it as the score today.

Time Series Classification

Time series classification offers the same metrics as Tabular Classification, computed per sequence instead of per row.

Time-to-Event Prediction


Compute budget

The compute budget caps how much compute, in PetaFLOPs (PF), each team spends on the use case. You set it per peer; a team’s budget is that value times the number of peers in the team. One value applies to every team of the use case. To change it, open the use case’s Admin tab. Compute usage shows the compute all teams have used out of the use case’s total. Click Adjust limit, enter the allocation per peer in TF (1 PF is 1,000 TF), and click Save. The value must be at least 100 TF and fit what your account has left. You can’t set it below the most any team has already used per peer. The use case’s owners, and members of your account with write permission on use cases, can change it. When a team uses up its budget, the platform pauses the team’s running experiments, and the team can’t start new ones. Raising the budget doesn’t resume them: the team resumes each paused experiment on its Training tab. For what peers see, see When the compute budget runs out.

Next Steps

Once your use case is published, invite peers to train models on it. In the use case view, monitor
  • total resource consumption
  • daily submits and user activity
  • overall leaderboard and submissions
Once models have been submitted, you can compare them in the leaderboard section of a use case.

Need Help?