Step 1: Define your use case
- Use case title: Up to 82 characters.
- What task should contributors work on?: The task, for example Object Detection. It filters the training datasets offered in step 2. See the full list of supported data types and tasks. If your use case is not yet supported, reach out to us at [email protected].
Step 2: Data & Evaluation
A note at the top of this step says what the panel sets for you: each peer gets a 50 PF compute budget, and you set an end date after publishing.- Select your training dataset: The training datasets you ingested in the Prepare Data step, for the task you picked.
- Select your test dataset: The test and score lists appear after you pick the training dataset. The test list shows only test datasets compatible with it. When the task has classes, the test data’s classes must be a subset of the training data’s classes. Under it, a list names each of your other test datasets for the task and why it isn’t offered, for example “Not offered: class
bicycleisn’t in the training data”. - Select your score: The metric that ranks submissions on the leaderboard. The list holds the metrics in the tables below for your task. If your metric is not supported, reach out to us at [email protected].
- What does this use case give?: The use case’s output.
- Report: the results as a document, with exact numbers ready to cite in a paper.
- Weights download: each peer can download the weights files of their own experiments.
- Who can join this use case?: Only you or Peers you invite. Peers join through your invitation; accepting it is accepting the use case’s contract.
- Purpose: Research (non-commercial) or Commercial.
- Upload EDA File (optional): A
.ipynbor.jsonfile, up to 25 MB, to help peers understand the data. Explore template use cases for inspiration.
Click Create Use Case. The use case is published at once; there is no draft. The panel sets these values for you:
- Description: The title. Edit it on the use case’s Overview tab with Edit as admin.
- Compute budget: 50 PF per peer. Change it on the Admin tab under Compute usage with Adjust limit.
- End date: None, so the use case runs until you end it. Set one on the Admin tab under Duration.
Supported metrics per data type and task
The tables below list the metrics you can pick as the score for each task today. Each metric uses either higher is better or lower is better sort order on the leaderboard. Some tasks report more metrics during training than you can pick as the score; the notes under those tables name them.Image Classification
Text Classification
Sentence Pair Classification
Training also reports precision, recall, F1 score, MCC and other classification metrics, but Accuracy is the only score you can pick today.
Token Classification
Token-level scores are computed over labelled tokens only: padding, special tokens and sub-word continuations are excluded.
Training also reports token accuracy (same value as Accuracy) and precision, recall and F1 score, macro-averaged across tag classes. Accuracy is the only score you can pick today.
Masked Language Modeling
Causal Language Modeling
Causal language modeling is next-token prediction, so its scores are token-level. Each validation sequence is scored at every position that has a training target: the model’s most likely next token is compared with the actual next token. Padding is excluded, and so is the prompt part of aprompt<TAB>completion row, so only completion tokens count.
Seq2Seq
Seq2Seq scores are token-level over the target (decoder) sequence, with padding excluded.
Training also reports token accuracy (same value as Accuracy) and perplexity, but Accuracy is the only score you can pick today.
Embeddings
Embeddings are trained with a contrastive objective on anchor/positive pairs, so the scores measure the embedding space rather than class labels.
Perplexity is not a score for embeddings: an embeddings run does not report it.
Tabular Classification
Object Detection
For ranking detectors, pick Mean Average Precision @ IoU 0.50: unlike Accuracy and IoU, it penalizes both missed objects and false boxes.
Semantic Segmentation
Keypoint Detection
Tabular Regression
Time Series Forecasting
By default, when the data has feature columns besidestimestamp and the target, the engine scales the target to the 0–1 range (MinMaxScaler) before training, so error metrics such as MAE and RMSE are in scaled units, not in your data’s units.
Training also reports direction accuracy (how often the model predicts the direction of change correctly), but you cannot pick it as the score today.
Time Series Classification
Time series classification offers the same metrics as Tabular Classification, computed per sequence instead of per row.Time-to-Event Prediction
Compute budget
The compute budget caps how much compute, in PetaFLOPs (PF), each team spends on the use case. You set it per peer; a team’s budget is that value times the number of peers in the team. One value applies to every team of the use case. To change it, open the use case’s Admin tab. Compute usage shows the compute all teams have used out of the use case’s total. Click Adjust limit, enter the allocation per peer in TF (1 PF is 1,000 TF), and click Save. The value must be at least 100 TF and fit what your account has left. You can’t set it below the most any team has already used per peer. The use case’s owners, and members of your account with write permission on use cases, can change it. When a team uses up its budget, the platform pauses the team’s running experiments, and the team can’t start new ones. Raising the budget doesn’t resume them: the team resumes each paused experiment on its Training tab. For what peers see, see When the compute budget runs out.Next Steps
Once your use case is published, invite peers to train models on it. In the use case view, monitor- total resource consumption
- daily submits and user activity
- overall leaderboard and submissions
Need Help?
- Email us at [email protected]