Skip to main content
tracebloc ships as one platform, and this page is where its releases are read. Each entry carries a release number, a date and the release notes of everything that changed for you, grouped by where you notice it: Platform, Secure environment, Python SDK, CLI and Docs. A collapsed block at the end of every entry lists the Helm chart, CLI, Python SDK and data ingestor versions the release corresponds to, so you can tell which of them to update. How the number moves. The middle number (0.19.0 to 0.20.0) moves when a release changes something you can notice. The first number moves when a compatibility floor or an install-time contract changes: read those entries first, they name what you have to do. Releases that change nothing for you are not listed here. Where to act. When an entry asks you to update something:
  • the secure environment updates itself by default; to update by hand or pin a version, follow Upgrade, or roll back if you have to
  • update the tracebloc CLI with the one-liner under Install
  • update the Python SDK as described under Installation
  • setting up a new secure environment starts at the Quick Start
Follow releases. Every entry on this page is also published as an RSS item at https://docs.tracebloc.io/changelog/rss.xml; the RSS button at the top of this page links the same feed. Filter the entries by surface with the tags in the side panel.
​
Secure environmentPython SDK
2026-11-02
Secure environment
  • The outbound gate now runs under its own service account, which may only check which of the client’s own services is sending.
  • The install log now keeps the real error message from a failed step instead of hiding it behind progress-spinner characters.
  • Trainings on a Mac run on its CPU for now: training on the Mac’s Apple GPU is paused until a later release turns it back on.
Python SDK
  • When a model’s own code does not run with the libraries training uses, the upload now stops with a message naming the error and what to do, instead of an error trace.
  • Hugging Face models saved without a model type in their config, such as ProtBERT, now convert on upload.
​
CLI
2026-11-02
CLI
  • A time-to-event ingest no longer suggests leaving out its time column, which the command would have refused.
  • Guided data ingest now stops right away when you pass —leave-out for image or other non-table data, instead of showing it as left out and failing after you confirm.
​
PlatformCLI
2026-11-02
Platform
  • Inviting someone twice, or someone already on a use case, now says so instead of silently succeeding, and an unanswered invite no longer counts against the use case’s compute budget.
  • Uploading a PyTorch model that imports catboost, lifelines, lightgbm, sksurv or xgboost is now refused up front, with a message to upload it as a sklearn-family model instead.
  • A use case is now seen by who can join it: only you means you alone, and peers you invite means you, its admins and the peers you invite, never the rest of your account.
  • Pasting HTML into the use case description editor no longer keeps scripts, iframes or unsafe links in what you save.
  • A notebook cell’s fullscreen output now opens as a dialog that keeps keyboard focus inside it and returns focus when it closes.
  • Use case owner avatars use the platform’s identity colours, so some owners show a different colour.
  • A deleted dataset’s details panel no longer shows record counts, format, size or use cases for data that is gone.
CLI
  • Switching the task in the guided data flow no longer carries over a time column and then refuses the run when no task was given on the command line.
  • tracebloc outbound shows what left your secure environment, day by day, and what the outbound gate did with each message.
​
CLI
2026-11-01
CLI
  • When your data has a column that looks like email addresses, phone numbers, names or IDs, tracebloc warns you and offers to leave it out of the dataset; scripts can pass —leave-out.
  • A CSV separated by semicolons or tabs now gets an error that names the delimiter and says to re-save it with commas.
​
Secure environment
2026-11-01
Secure environment
  • A fresh install no longer fails when MySQL’s first start is slow; if a first start was cut short, the installer now says so and how to reset it.
​
PlatformSecure environmentCLI
2026-11-01
Platform
  • Starting a run or linking a dataset with a library model now refuses a request that also names another model under the model_name spelling, instead of silently accepting it.
Secure environment
  • DistilGPT2 now trains with its default settings on machines with 8 GB of memory instead of stopping with an out-of-memory error.
CLI
  • Starting a data copy no longer slips past the ingestor version check when the image registry is briefly slow or unreachable.
​
Secure environment
2026-11-01
Secure environment
  • Large models download their weights on your client without needing memory for the whole file.
  • A short backend outage no longer makes a training client give up, and a long wait for the previous global model no longer starves a later weights upload.
  • A training run waiting for the previous cycle’s weights or the shared statistics no longer waits past its time limit when the connection to tracebloc drops and returns.
  • A federated round no longer drops after a brief earlier connection outage when the edge is waiting for the previous global or the statistics barrier.
  • When setup stops, the installer now says what happened, what to do next, the command to run and where its log is, in the same short words on every screen.
  • The installer now asks for a different name when a secure environment name is only digits, and offers “machine-” plus the number when your computer’s name is all digits, instead of failing at install.
  • On Linux GPU servers without a desktop, the installer again installs NVIDIA’s server driver, and it no longer mistakes an older driver already on the machine for the one it just installed.
  • Your secure environment no longer switches its services to a newer build between releases; a service you run without a digest pin now updates when it restarts.
  • Every release now carries a signature for each chart, so you can check that a chart is signed by tracebloc before you install it.
  • When the installer’s connection check times out, it now says which addresses didn’t answer instead of claiming you’re offline or blocked.
  • Training results now leave your secure environment through the outbound gate, which keeps an outbound record of each one on your infrastructure. In this release the gate refuses nothing.
​
Secure environment
2026-11-01
Secure environment
  • A run on a Mac now continues within a cycle after the laptop wakes from sleep, instead of stalling until the client is restarted.
  • Respin, CPU-fallback, GPU-placement, Service Bus listener, failure-report and migration log lines are written to the jobs-manager log file again.
​
Platform
2026-10-31
Platform
  • Saving an example use case with empty type fields no longer fails with a “cannot be changed” error.
​
Platform
2026-10-31
Platform
  • The Create Use Case type step no longer says your data narrowed the choices before you have picked a dataset.
​
PlatformDocs
2026-10-31
Platform
  • A use case now has a type: what it gives, who can join and its purpose. On patient data only research use cases are offered.
  • Saving a use case that has no type yet no longer fails when the form sends its empty type fields back.
  • When you create a use case you now choose what it gives, who can join and its purpose. On patient data only research use cases are offered.
Docs
  • The security page now lists the egress gateway’s per-environment API hosts and no longer lists a host the chart does not permit.
​
PlatformCLI
2026-10-31
Platform
  • A refused save of a use case’s end date, description, title or team name now shows the reason the server gave.
  • Tab and Shift+Tab now stay inside the Create Use Case panel instead of moving to the page behind it.
  • Scrolling the page with a Create Use Case dropdown open no longer raises an error.
  • Keyboard users can load more environments in the environment switcher when the first page of the list is empty or after undoing a hide.
  • Saving an Admin setting during a server outage now shows a plain error message instead of raw page markup.
  • The dashboard no longer loads Google Tag Manager or Google Analytics.
  • Refused invites and participant changes on a use case now show the reason the server gives instead of failing silently.
  • The leaderboard’s Model column now shows the name of the model behind each team’s score, and ”—” for a team with no score yet.
CLI
  • When copying data into your secure environment fails or has no room, the message now says what happened and, on its own line, what to do next.
  • A CSV saved as Excel “CSV UTF-8” now ingests as a table, instead of being refused with a request to re-save it.
  • tracebloc data delete now refuses to delete a dataset that a use case is built from, and tells you which use case to archive first.
  • A missing label column in your dataset file now exits with code 3, like every other problem in the dataset’s own files.
  • A secure environment that isn’t linked to a cluster yet now tells you to run tracebloc client create on its machine, instead of saying it isn’t answering.
  • Picking another secure environment under “Change something” now checks your dataset name against that environment too.
  • Pressing Ctrl-C while data delete is looking up your datasets now exits quietly with the standard interrupt code instead of reporting that the target can’t be confirmed.
  • Re-ingesting a dataset under a name that is already taken now stops with a clear message instead of creating a duplicate, and --overwrite now really replaces the old dataset.
  • tracebloc data list and tracebloc data delete show your datasets again instead of listing them as System or finding none.
  • A scripted ingest now stops before the copy on a value its column can’t hold (a true/false or date that isn’t one, or text longer than its column), naming the line, and warns when the column to predict spells a class several ways.
  • data ingest now rejects a malformed --image-digest up front, accepts --intent in any letter case, and reports a whitespace-only CSV as empty.
  • A data CSV that is not valid text is now refused up front as an encoding problem, instead of ending in a spec validation error that names no flag.
  • A scripted ingest now stops before the copy on a forecasting file that isn’t in time order or has a date to come, a time column called date instead of timestamp, a time-to-event time of 0 or below, a text whose tags don’t match its words, or a file listed twice with different labels, naming the line and the fix.
  • Reusing a retry key with a different dataset name is now refused up front, before any files are copied, with a plain message.
  • When data ingest refuses a name that matches an existing dataset’s title, it now tells you tracebloc data delete can remove it.
  • tracebloc resources set now says a running training takes the new size from its next cycle, instead of promising it keeps its old size.
  • tracebloc data delete now works when you target a namespace with --namespace or --context, and --dry-run shows what would be deleted instead of refusing.
  • A scripted tracebloc data ingest --output-json run that stops on a task or label-column problem now reports the dataset name you passed, or none, instead of one guessed from the files.
  • A —label-column typed with a stray space around it is now found in the table instead of reported as missing.
  • Pressing Ctrl-C while tracebloc data delete looks up your secure environment now exits cleanly with the interrupt status instead of a sign-in error.
  • Preflight now catches a labels file that lists the same text file twice with different labels on token-classification and sentence-pair datasets.
  • When you ingest a dataset, tracebloc asks whether it holds patient data, open data or synthetic data (or takes --holds); a dataset nobody declared counts as patient data.
  • A reused —idempotency-key now gets a plain message saying the key was already used and what to do, instead of an internal server error.
  • When tracebloc delete can’t finish, it tells you where to send its log for help.
  • If your kubeconfig points at a stopped cluster that isn’t your secure environment, the CLI now says at once that it can’t reach it, instead of waiting 90 seconds first.
​
Python SDKDocs
2026-10-31
Python SDK
  • Converting a model whose layers module star-imports a library no longer fails late with a NameError; it is refused at the star line with a clear message.
Docs
  • A new page shows where to get help and how to delete tracebloc from your machine.
​
2026-10-30
​
PlatformPython SDK
2026-10-30
Platform
  • Runs started from the Run tab now hold out at least 10% of the data for validation by default, and tabular classification defaults to cross-entropy loss instead of mean squared error.
  • When you create a use case, the lists of training and test datasets to choose from load faster on accounts with many datasets.
  • Re-sending a model-weights part while the upload is being finished no longer risks joining the earlier copy of that part.
  • tracebloc team accounts can again submit inference, pause, stop and resume their runs, and comment on them.
  • Downloading a model’s trained weights from the platform starts at once, even for multi-GB models.
  • The leaderboard shows scores as the server sends them and energy with a unit, the leader no longer shows a 0 % change, and Eval headings use the metric’s name.
  • The account menu now says “Datasets created” instead of “Metadatasets created”.
  • Create Use Case now tells you when your account has no access to datasets and what to do about it, instead of asking you to refresh the page.
  • Notifications, such as team-join requests, now show their message as plain text instead of raw markup.
  • Archiving a use case while the Run drawer is open no longer brings the drawer back when it is unarchived.
  • Eco training can no longer be switched on for a run on an archived use case, where it would have scheduled training.
  • The leaderboard no longer shows the use case’s title in the Model column of every team.
  • Create Use Case now offers every training and test dataset in its lists, not only the 15 newest.
  • When a use case admin setting cannot be saved, the message now says why instead of “Unknown error occurred”.
  • The cover image upload now rejects files that are not a PNG or JPEG, or are over 1MB, and says why.
  • The End date calendar no longer lets you pick today or a past day.
  • Experiment notes now stop at 2000 characters and show a live character counter.
  • Cancelling an edit of a use case description now asks before discarding your changes.
  • While a run’s inference submit is in progress, its Submit button shows a spinner beside its label and stays reachable from the keyboard.
Python SDK
  • A large weights upload whose start the platform answered badly no longer leaves an unfinished upload behind.
  • Uploading your own model works when its class is named MyModel or when your package re-exports it with from .module import *; a star import the converter cannot follow is refused with the file and line.
  • Converting your own model code now stops with a clear message when it relies on a from x import * whose exported names cannot be read, instead of producing a package that fails on the server.
​
Docs
2026-10-30
Docs
  • The Windows, macOS and Linux prerequisites pages match the installer again: they no longer point to the retired CLI fallback for moving a k3d machine.
​
PlatformPython SDK
2026-10-30
Platform
  • Deleting a dataset that a live use case still uses is now refused, and the response names the use cases.
Python SDK
  • A model you upload shows its licence, and you confirm you may use its weights in a data owner’s secure environment; weights whose licence forbids redistribution (CC BY-NC-ND) are stopped before upload.
  • Some valid model repos that were refused before now convert: packages reached through nested imports, folders with a damaged best checkpoint next to a good one, and Hugging Face models whose tokenizer code is elsewhere.
  • ESM C protein models (300M, 600M, 6B) can be uploaded and fine-tuned for classification, token classification, masked language modelling and embeddings.
  • Linking a converted tabular model to a dataset with a different number of classes now shows the parameter mismatch report instead of stopping with a conversion error.
  • A failure while a model’s head is being fitted to its dataset at link time no longer leaves the package’s files out of step with each other.
  • Upload a model as a Hugging Face folder, a checkpoint, your own code or a live notebook object: the SDK converts it, checks it loads and behaves the same, and asks you to confirm its licence before uploading.
  • Models under the standard GPL-2.0, LGPL-2.1 and Open Font licences are no longer wrongly refused at upload.
  • Uploading a model with a dataset key reads the dataset’s task faster and also picks up its feature count.
  • The SDK’s README and its seed-weights hint point you to training from your own notebook instead of Google Colab.
  • In your notebook, fetch a model from your library or a ready-made model by the ids your library lists, and link it to a use case without uploading it again.
  • Model licences that ban redistribution now stop before upload even when another line of the same text mentions an exception.
  • When a converted model’s classes or features don’t match your dataset, linking rebuilds it for the dataset and links in one step, and a classifier whose classes are the dataset’s keeps its trained weights.
  • A converted model trains with its own image normalisation and its own sequence length, a converted model over 300M parameters trains with LoRA unless you turn it off, and you can set the batch size in the training plan.
  • Upload a torchvision or timm checkpoint, pathology encoders such as UNI included, and the SDK builds the model file for your use case with the model’s own image normalisation.
  • Upload a Hugging Face model folder, such as ESM-2, ProtBERT, Phikon or DINOv2, and the SDK builds it for your use case’s task, keeping its pretrained weights and starting a new head.
​
Secure environment
2026-10-30
Secure environment
  • Training uses a model’s own image normalisation values and text sequence length when the model declares them, instead of fixed defaults.
  • The installer’s internet check no longer reports reachable hosts as unreachable after a brief network hiccup.
  • Each data ingestor Helm chart version is published once; later releases no longer re-package it, so a version always downloads the same chart.
  • On Linux machines with an NVIDIA GPU, the installer now installs a driver your GPU actually supports, tells you plainly (with the command to run) when it could not, and no longer leaves the NVIDIA package source unreadable, so re-running it fixes a previously failed GPU install.
  • The installer now waits for the cluster to answer after a restart before setting up your example use case, so it no longer comes up empty.
  • The installer’s “What’s next” step now links to the current use-case creation page.
  • The installer now says when it could not read the NVIDIA driver list for your GPU in time and is choosing the driver from the package archive instead.
​
PlatformDocs
2026-10-30
Platform
  • Archived use cases now show a notice that no new training can start, and no longer offer Start training or the Run tab.
  • Opening a use case that no longer exists now shows a “That use case wasn’t found.” notice on the first-time Use Cases screen too.
  • Create Use Case no longer submits without your EDA file when you click it while the file is still being checked.
  • An archived use case no longer lets you resume a paused experiment from the Training tab.
  • The home page’s quick start sends you to the training notebook to run in your own Jupyter, instead of Google Colab.
Docs
  • The docs no longer send you to Google Colab: you train from the Run tab or from your own notebook.
  • The quickstart guide runs in your own Jupyter notebook; the Google Colab link is gone.
  • The model zoo’s README links the training notebook on GitHub, to run in your own Jupyter, instead of Google Colab.
​
PlatformCLI
2026-10-29
Platform
  • A bad EDA file now shows a clear message at the upload field when you attach it, and an attached EDA file can be removed.
  • My Data now counts every dataset in its header and lists active datasets before deleted ones.
  • A use case link with an unknown tab now opens the Overview tab, an unknown secure environment link says it fell back to yours, and a missing use case says it wasn’t found.
  • The Create Use Case form now tells you a title is required when you leave it empty or type only spaces.
  • Dropdowns in Create Use Case list only the real options; the “Select” line that looked like a choice is gone.
  • Choosing a wrong file for the EDA field no longer drops keyboard focus to the top of the page.
CLI
  • The CLI checks there’s room in your secure environment before copying, removes a copy that was stopped or failed, and says what Ctrl-C did at each step of an ingest.
  • A column name the database can’t store (too long, containing %( or %s, a name tracebloc keeps for itself, or two names that differ only in letter case), or a table too wide to store, now stops the ingest before the copy and names the column to rename.
  • data list now shows each dataset by its name instead of an internal table id.
  • Cancelling an ingest after a dataset went in now says plainly what stays and how to see it.
  • tracebloc data ingest --secure-environment <name> ingests into the secure environment you name, and refuses one this machine can’t reach.
  • tracebloc data list now tells you to upgrade the CLI when it is too old for tracebloc, instead of showing a generic “names couldn’t be read” notice.
  • On a Mac, the support bundle from tracebloc doctor --diagnose no longer contains training output: it lists each training log by name, size and time, and says where the logs are on the Mac.
  • Time-to-event tables whose duration column holds a different whole number in every row are now recognised, so the duration is filled in for you.
  • Error messages now start with a capital letter and end with a full stop.
  • A scripted ingest now stops before the copy when the data is too small to train on, and warns when a class has few examples.
  • The CLI now warns when a tabular classification label column holds decimal values, which would each become their own class.
  • --schema now rejects a column named twice instead of silently keeping the last type and losing rows.
  • tracebloc doctor --context <name> now says the context isn’t in your kubeconfig, naming it, instead of claiming there is no secure environment.
  • Pressing Ctrl-C while tracebloc data list runs now stops it quietly with exit code 130, without printing a partial list.
  • resources set --memory now tells you that memory is set in GiB and that Mi and other units are not accepted, instead of a generic “number of GiB” message.
  • data ingest --output-json now reports the dataset name you asked for in name, and table always means the created table, so it is empty when the run failed before one existed.
  • When you replace a dataset in the guided ingest and tracebloc could not be told, the CLI now warns you that My Data still lists the old data as active.
  • tracebloc data delete now accepts the dataset name that data list shows, and refuses a name shared by several datasets instead of guessing.
  • tracebloc data ingest --example now exits with code 11 while the example data is still being ingested on your secure environment, instead of 0.
​
PlatformPython SDKDocs
2026-10-29
Platform
  • The use case page no longer shifts its tab row when the Admin tab and “Edit as admin” appear, so clicks land on the tab you aimed at.
  • The use case header and the invite panel now say “Peers”, and the invite panel shows how many peers have joined and how many are still invited.
  • Filters on the Use Cases page now let you filter by task, including for use cases you created yourself.
Python SDK
  • You can upload a model you built yourself, a Hugging Face model that ships its own code, or a model object straight from your notebook, and tracebloc writes the model file for you.
  • Uploading a model folder with a corrupt config or weights file now reports it as unrecognised instead of crashing.
Docs
  • The changelog now says “experiment” instead of “run” in the 1.136.0 notes.
​
PlatformPython SDK
2026-10-29
Platform
  • Model files can now import typing, functools, dataclasses, einops and other plain Python modules.
  • Use case titles longer than 82 characters are now refused, and the compute-limit error reads “at least 100 TFLOPs”.
  • Use case titles containing letters such as ø, đ, ł or ß now keep them (transliterated) in their URL slug instead of losing them.
  • Tracebloc staff accounts can no longer change use cases through the platform API; every write is refused the same way inference submit already is.
  • Model uploads that reach a download or file load through another uploaded file are now refused, the same as direct calls.
  • Experiment cycles are listed in order, an unknown or hidden use case no longer reveals what kind of record the id was, and the daily inference limit message now says in plain words how many runs you have submitted today.
  • The training and test dataset pickers no longer list datasets from decommissioned secure environments.
  • A failed request no longer always says to check your internet connection; the message now only says that when your browser is actually offline.
  • The use case Overview no longer shows an empty “On this page” heading, and the Resources figure no longer ends in a stray ”| -” when no GPU is reported.
  • A use case title containing markup now shows as plain text in its default description instead of rendering as page elements.
  • A deleted secure environment no longer shows as “Never connected” in the environment switcher.
  • Pressing Escape with a dropdown open in Create Use Case now closes only the dropdown, closing the panel with entered data asks first, and open dropdowns close when the panel scrolls.
  • Inviting participants now says why an email was not added, and example addresses no longer look like removable entries.
  • Switching tabs on the Use Cases page now updates the address, so a refresh stays on the tab you chose.
  • Adjust limit now keeps the editor open and shows the reason when a save is refused, clears old errors when you edit or reopen it, and disables Save for values that are not a valid number.
  • Sending an invite now shows whether it was sent, and the same address typed in different capitals is added once.
  • The leaderboard no longer ranks or crowns a team that has no score yet, and the Model column no longer shows the team’s generated name.
  • A failed training run no longer shows a training time left.
  • Participants in a use case no longer see an “Invite others to join” button they cannot use.
  • “Shared with me” no longer lists public use cases that nobody shared with your account.
  • On the Metadata page, the type, stage and creator filters are now one Filters menu, and search updates as soon as you pause typing.
  • Code in the Run tab’s training plan and in evaluation code is now coloured the same way as every other code box, with numbers and method names easier to tell apart.
  • Invitation errors now always show a readable message, even when the server answers in an unexpected shape.
  • The Run tab’s Trainable check no longer fails on the first attempt when the notebook has only just started.
  • The use case header now shows the same CPU per run as the secure environment page.
  • The Secure environment page no longer scrolls sideways in narrower windows; wide tables scroll inside their own frame.
Python SDK
  • Text models now train on their own maximum sequence length instead of a fixed 512 tokens.
  • A text model whose tokenizer marks sequences with <cls> and <eos>, as ESM protein models do, now passes the upload check.
  • Uploading a fitted multiclass LogisticRegression with weights now passes the training check.
  • Small datasets now keep at least 10% of rows for validation by default, and tabular classification trains with cross-entropy loss by default.
  • Uploading a model folder that holds several weights files, none with a standard name, now stops with a message naming the files instead of silently dropping the weights.
​
Docs
2026-10-29
Docs
  • The dataset “Check by hand” pairing command now handles CSV files with Windows line endings, and the Setup Guide says how to install the NVIDIA driver yourself on Linux when the installer can’t.
  • The docs now explain deleting tracebloc from a machine with no secure environment linked, and that the Storage tile is measured on Windows.
​
PlatformSecure environment
2026-10-29
Platform
  • Inference notifications show scores to the same precision as My Submissions, and no longer quote error text from a secure environment in a partner account.
  • Uploaded weights files now appear in the model library under the file name you uploaded.
  • An experiment that fails while its models are being combined now says whether the cause was its settings, its model or the platform.
  • Inviting someone to your team now opens in its own window that is ready for typing and closes with Escape.
  • Headings and text on the theme settings page stay readable when the colour scheme changes.
  • The Use Cases search and industry filters stay in the page address, so reloading, going back or sharing the link keeps them.
  • Initials on project owners’ avatars are readable on every avatar colour.
  • The Run tab now says there was no answer from the server when the notebook kernel cannot be reached, instead of claiming the Python kernel did not start.
  • The experiment card no longer says it “started successfully” while its training image is still downloading; it now says the first training on a machine can take several minutes.
  • The experiment card now counts “Cycle X of N” and labels the time beside it as time left, which no longer shows once an experiment has finished.
  • Models without a parameter count, such as CatBoost and Decision Tree, no longer show “0 params” in the model picker.
  • Creating a Use Case now confirms with a “Your Use Case is Live!” message, and the new Use Case appears first in My Use Cases.
  • The task menu in the create use case panel is no longer cut off at the bottom of the panel.
Secure environment
  • When an experiment fails, the failure report sent to tracebloc no longer includes its last output lines.
  • The support bundle from the installer’s --diagnose no longer contains training log tails; it says where to read them on the machine.
​
PlatformSecure environment
2026-10-29
Platform
  • The Run tab now accepts models of up to about 350 million parameters (was 200 million).
  • A model file that writes its framework in any case, such as “PyTorch”, now gets every upload check, and a model’s own normalisation values reach training.
  • tracebloc no longer stores sample rows of your datasets; it keeps only their title, schema and counts.
  • Choosing test data for a use case is quicker when an account has a lot of test data.
  • A model that passes the upload checks is now checked against the same numpy, pandas and interpret versions it trains with.
  • New use cases, including the example use case, start with weights download switched off; the use case’s owner can switch it on in the use case’s admin settings.
Secure environment
  • Model files can now import typing, functools, dataclasses, einops and other plain Python modules.
  • The Windows one-line installer no longer downloads its 180 MB signature tool twice, and it says when it cannot remove its temporary folder instead of leaving it behind silently.
​
Platform
2026-10-29
Platform
  • Initials on green avatars are now dark, so they are easier to read.
​
PlatformCLI
2026-10-29
Platform
  • The experiments list on a use case loads faster when many runs are queued, stopped or paused.
  • Uploading a text model now needs its tokenizer, and PyTorch weights must be uploaded as SafeTensors; each refusal says what to upload instead.
  • The experiments tab of a busy use case loads several times faster.
  • The use case list loads faster.
  • The Admin tab’s compute line now shows how many people the use case’s compute total covers, so its numbers add up. The use case header shows “Team compute”, and a machine whose storage is not measured says so instead of asking you to update.
  • Moving through a dataset card with the keyboard now shows the same focus outline as the rest of the app.
  • Commands and code snippets now show in the app’s monospace typeface.
  • The Use Cases search box shows a search icon and its own clear button, and results update once you pause typing.
  • Keyboard and screen-reader users can now show and hide a notebook cell’s code and output in a use case’s data exploration tab.
  • Training pages catch up faster and lighter after a connection blip.
CLI
  • Pressing Ctrl-C while waiting for the secure environment to start no longer shows a “did not become Ready” timeout message.
  • The warnings shown when the CLI can’t verify which environment a data command targets now say “secure environment” instead of “cluster”.
  • tracebloc delete now lists the commands to finish by hand when it cannot read the container runtime or node from your install record.
​
Secure environmentCLI
2026-10-29
Secure environment
  • Deleting a dataset that has been trained on now works instead of failing on file permissions.
  • Training in your secure environment can now reach only the tracebloc API of its own environment, and no other address.
  • The first data ingest right after installing on a GPU machine no longer fails while the GPU image is still downloading.
CLI
  • Signing in offers to open the link when you press Enter, and to get a new code when one expires. tracebloc auth status exits 1 when your sign-in has expired. The home screen says why your secure environment is offline, and what to do about it.
  • tracebloc commands always reach your secure environment itself, whichever kubectl context you have selected.
  • tracebloc data ingest now stops before the copy and names any column whose name contains %( or %s, instead of letting your secure environment refuse the dataset afterwards.
  • Pressing Ctrl-C while the CLI waits for a restarting secure environment now exits quietly instead of printing a connection error.
  • tracebloc data ingest --dry-run now reports a secure environment it cannot verify as a finding instead of refusing, and the ingest prompts say which size is used by default, that replacing a dataset deletes the old one first, and what the bucket option sends.
  • A slow connection no longer cuts a large copy into your secure environment off at 30 minutes; a connection too slow to finish is stopped early and says how long it would take.
  • While copying data into your secure environment the CLI shows how much is left and how long it will take, and keeps your computer from going to sleep until the ingest ends.
  • When tracebloc data ingest stops following an ingest on its own (it hasn’t started in 5 minutes, or is still running after an hour), it now exits with code 11 instead of 0. --detach still exits 0.
  • A waiting spinner on a narrow terminal now animates on one line instead of filling the screen with copies of itself.
  • Pressing Ctrl-C during data ingest --dry-run now exits quietly instead of reporting a missing secure environment.
  • The runner’s message about skipping a VM sweep now speaks of the training in plain words.
  • tracebloc delete now removes the CLI and its local data when you’re signed out or have no secure environment on the machine.
  • On Windows, tracebloc upgrade now points to the signature-verified installer, and says when you’re already on the latest version.
  • Error messages now show Windows dataset paths as typed, without doubled backslashes.
  • The first tracebloc data ingest right after an install now waits while your secure environment finishes downloading images, instead of failing after a minute.
  • tracebloc data ingest --output-json now always reports exit_code in its result, matching the command’s exit code, so a script can tell a run you detached from one the CLI stopped following on its own.
​
Platform
2026-10-29
Platform
  • A dataset whose edge map holds a malformed entry no longer breaks the dataset list and detail pages.
  • A tracebloc staff account that tries to submit, pause, stop, resume, comment on or edit an experiment now sees why it is not allowed, instead of “not found”.
  • Use cases with many teams open their team list faster.
  • An experiment that was waiting for room on its secure environment no longer shows as waiting or stalled once it starts training.
  • The use case list loads faster.
​
PlatformSecure environmentDocs
2026-10-28
Platform
  • Times such as how long a run took or how long is left now read the same everywhere, for example “2 hr 3 min”.
  • Changing the model in the Run panel now updates the run’s name and the plan values you did not change, so a run is no longer named after an earlier model.
Secure environment
  • Object detection experiments now show a confusion matrix, with a background row and column for missed and spurious boxes.
  • The object detection confusion matrix counts only boxes the model scores 0.25 or more, so low-confidence boxes no longer crowd the background row.
  • An install whose first sign-in to tracebloc is briefly refused by a busy backend no longer says your Client ID or password was wrong.
  • Data ingestion on clients with service database accounts runs ingestor v0.8.96.
  • The Windows installer no longer leaves a 180 MB temporary folder behind or names a missing folder on the system PATH, and it reuses a signature tool it already verified instead of downloading it again (once the tracebloc.io one-line installer passes it down).
Docs
  • The docs now give the current score options for causal language modeling and embeddings, Windows ways to check data and to run kubectl, what each Environment status tile measures on Windows and Linux, the budget refusal as the Run tab shows it, and how to delete tracebloc by hand when tracebloc delete can’t run.
​
Secure environmentPython SDK
2026-10-28
Secure environment
  • A fresh install no longer restarts the requests proxy once while the database and jobs manager are still starting.
  • On Windows, the secure environment’s Storage tile now shows how much space the environment uses and how much it can still grow into.
  • On Windows and Linux, jobs waiting for an NVIDIA GPU now start in the order they were submitted, and each waiting job knows its place in the queue.
Python SDK
  • The link printed after start() now opens your use cases (find the run in the use case’s Training tab) instead of a “Page Not Found” page.
​
PlatformSecure environmentCLI
2026-10-28
Platform
  • Keyboard users now see a focus ring on the Create Use Case panel, the environment switcher, the Run tab’s code cell and the editor’s bookmark and card fields.
  • The email field in the Invite to team popover now shows where keyboard focus is, and screen readers announce its error message.
  • The Confusion Matrix tab says “not ready yet” only until a cycle is delivered; after that it says when a cycle has no confusion matrix instead of waiting forever.
  • The disabled weights Download button on an experiment now says why, and tells the use case’s owner where to turn weights download on.
  • Upvoting an experiment, posting a comment or reply, or changing an experiment that you are not allowed to change now leaves things as they were and says why, instead of looking like it worked.
  • Object-detection experiments now say their confusion matrix isn’t available, instead of waiting for one that never comes.
  • Secure environment now shows times like the rest of the app, for example “2 hr ago”, and the date for anything older than a week.
  • Keyboard and screen-reader users can now fold and unfold notebook sections in a use case’s data exploration tab.
  • The comment and reply Send buttons no longer drop keyboard focus while your comment is being saved.
Secure environment
  • The Mac installer shows one live line for each step with the time it takes, asks only for your secure environment’s name, and ends with a short summary of what is ready and what to do next.
CLI
  • The installer no longer hangs when the installed tracebloc CLI stops responding while its version is checked.
​
PlatformSecure environment
2026-10-28
Platform
  • A leaderboard that fails to load because of a server error no longer tells you to check your credentials.
  • Model Code and Download no longer appear on experiments you are not allowed to open.
  • If your experiments fail to load, the Training tab now says so and offers Retry instead of showing the getting-started guide.
  • Keyboard users who open the invite field from an example address now land in the field, and screen readers skip purely decorative images.
  • When a page fails to load, the side navigation stays in place, and you can press Try again or move to another page.
  • Comments, the Secure environment tables and the usage chart now show dates like the rest of the app, for example “4 Aug 2026, 11:15” or “3 days ago”.
  • The Run drawer no longer claims it set the output classes in your model; it tells you what to change.
  • Download and Model Code no longer disappear from an experiment after you try to download weights that aren’t ready yet.
  • On a phone, the Create Use Case panel now covers the top navigation bar instead of sitting under it.
  • When submitting a cycle for inference fails, you now see why instead of nothing happening.
  • When pausing, stopping, resuming or switching Green AI on an experiment is refused, you now see why instead of nothing happening.
Secure environment
  • The installer now calls the Mac GPU training component “the tracebloc runner” everywhere it prints it.
  • The chart’s configuration help now says “secure environment” instead of “edge”.
​
Secure environmentPython SDKCLI
2026-10-28
Secure environment
  • Forecasting runs now stop before training, naming how many rows are needed, when the data is too short for a model with a fixed window.
  • A training cycle that is stopped while its final scores are being computed is now delivered with its trained model when the training restarts, instead of being trained again or lost.
  • Stopping a multi-GPU training during cycle-end scoring now stops every GPU worker instead of letting some deliver a cancelled cycle.
  • Data ingestion on secure environments with service database accounts runs ingestor v0.8.95.
  • The installer now reports rejected credentials instead of printing “connected” when the secure environment keeps restarting because it cannot log in.
  • A model is no longer scored with data preparation learnt from the test set; such runs now stop with a clear reason.
Python SDK
  • A tabular model trained with one-hot encoding now links at its encoded input width on newly ingested datasets, including datasets combined across several secure environments.
CLI
  • A failed command now ends on one ✖ line that says what happened and what to do next, and piped or --plain output no longer shows the waiting lines, only each step’s ✔ line.
  • Support bundles from doctor no longer merge two different machines under one pseudonym when a machine is itself named like node-1.
  • A GPU training that has to wait for the VM to restart no longer first spends time zeroing memory that cannot make room.
  • The installer now refuses to move a machine off a secure environment running on k3d even when no record or kubeconfig mentions it.
​
Secure environment
2026-10-28
Secure environment
  • A training on a Mac GPU whose first step doesn’t match the CPU now moves to the CPU right away and frees the GPU for the next training, and models trained over several cycles keep training on the GPU.
  • Mac GPU trainings now report the engine’s CPU load instead of 0.0.
  • A Mac GPU training that needs an operation Apple’s GPU does not support now moves to the CPU instead of failing.
  • Machines that still run tracebloc on k3d are no longer supported. The installer stops on them, changes nothing and asks nothing. Moving one to k3s is a reinstall: delete its secure environment with tracebloc delete, then delete the old install record and the k3d-tracebloc entries from your kubeconfig if they are still there, and run the installer again.
  • Stopping the Mac installer before it finishes now stops the VM it started, so it no longer holds your Mac’s memory.
  • A new secure environment on Linux, Windows or a Mac no longer gives its Kubernetes node this machine’s hostname, so the hostname no longer reaches tracebloc in what the secure environment reports. The name is random (tracebloc- and 8 characters). An existing secure environment keeps its name, and the machine name you choose at install is unchanged.
  • On a machine that still runs tracebloc on k3d, the installer now stops while k3d’s cluster is still in Docker, even after its install record and kubeconfig entries were deleted by hand, so it never sets up a second Kubernetes beside it.
  • On a Linux machine with SELinux on, your secure environment’s services and trainings now run confined by SELinux.
  • Each dataset you ingest is now kept in its own table on your secure environment, so a new ingest never adds rows to an earlier dataset, and deleting a dataset removes all of its data.
​
2026-10-28
No customer-facing changes. Every component is at the same version as in 1.123.0.
​
Platform
2026-10-27
Platform
  • A secure environment can only fetch or report on experiments it trains or evaluates.
  • Another team’s model code and trained weights can no longer be read or downloaded from an open dataset or open use case; they stay with that team and its account.
  • Starting an experiment without naming a dataset now returns a clear “Mandatory Fields Missing” error instead of a server error.
  • Someone who has left a dataset or use case, or a staff account, can no longer pause, resume or submit experiments on it.
​
CLI
2026-10-27
CLI
  • The support bundle from tracebloc doctor now includes the Mac runner’s log and its most recent job logs.
  • On a machine where an administrator already removed the secure environment’s local setup, tracebloc delete no longer refuses over an experiment tracebloc still lists as running; it names the experiment in a warning and goes on.
​
Platform
2026-10-27
Platform
  • When the leaderboard fails to load, it now shows an error with a Retry button instead of “No submissions yet”.
  • Renaming a use case in the secure environment now shows the new name when you reopen it from an activity notice.
  • Links with tracking or search parameters, such as /?utm_source=… or /explore?search=…, now keep those parameters when they redirect, and the home page reaches the use cases page in one step.
  • Text and sentence-pair classification templates now read up to 512 tokens of each text instead of cutting it to 5.
​
CLI
2026-10-27
CLI
  • tracebloc data ingest now catches a blank label, a blank image filename, or a keypoint image missing from images/ before uploading, instead of failing in your secure environment after the copy.
  • tracebloc delete no longer refuses because of an experiment whose training has already stopped. When it does refuse, it names the experiments training on your secure environment.
  • The support bundle from tracebloc doctor --diagnose no longer contains your machine’s hostname.
  • Doctor run from a Mac against a remote cluster no longer reports that Mac’s GPU as unknown when the cluster’s machines can’t be listed; it reports the installer’s GPU record instead.
  • On a Mac, the GPU runner’s log now gives the time of every line, and says why each training waited, what memory it started with, when it moved to the CPU and when the VM restarted; Kubernetes library noise no longer appears in it.
  • When tracebloc data ingest refuses labels.csv rows with a missing image or a blank filename, or labels that are blank or a single class, the message now says the whole ingest would fail instead of naming internal infrastructure.
  • When tracebloc delete or tracebloc client remove warns about an experiment still listed as running, it now names the command you ran and the peer who can stop the experiment.
​
Platform
2026-10-27
Platform
  • The example use case set up after you install your secure environment now gives you 50 PetaFLOPs of compute instead of 10.
  • An experiment waiting for room on its secure environment now shows it on its card as soon as the wait starts or ends, without reloading the page.
  • When live updates are unavailable, an experiment’s card still shows the wait for room starting and ending without reloading the page.
​
Platform
2026-10-27
Platform
  • Replying to a reply on an experiment comment is now refused with a clear error, so comment threads stay one level deep.
  • The message when you may not use a dataset or experiment now says “don’t” instead of “dont”.
​
Platform
2026-10-27
Platform
  • A new use case now gives each peer 50 PetaFLOPs of compute instead of 1, so a first training no longer stops after a few minutes.
  • Signing in now shows a message instead of spinning forever when the service is busy, unavailable, or refuses the sign-in.
  • Logging out now signs you out even when the logout request fails, instead of leaving you stuck on pages that error.
  • After signing in from a link to a protected page, you now land on that page instead of the default one.
  • The cycle number box in a run’s Test row is now announced as “Cycle” by screen readers instead of “minus”.
​
PlatformSecure environmentPython SDK
2026-10-27
Platform
  • Comment authors without a profile photo now show their initials instead of a broken image.
Secure environment
  • The Mac installer no longer says GPU training is unsupported on a Mac that trains on its GPU, and the dashboard no longer shows that Mac as CPU-only.
  • When another Apple container system is already running on your Mac, the installer now says where it is installed and gives a command that stops it, even when its files are gone.
  • An install run over ssh, or with no terminal, no longer waits for Enter when the sign-in code expires (a fresh code follows by itself), and prints the dashboard link instead of opening a browser on that machine.
  • On a machine an administrator prepared with prepare-host, the NVIDIA GPU is now set up too, so the owner’s install trains on the GPU instead of running in CPU mode.
Python SDK
  • When a link is refused, the message now ends with where the dashboard shows the use case’s dataset key.
  • A refused link now calls the key a dataset key, the word the dashboard uses, instead of a dataset ID.
​
Docs
2026-10-27
Docs
  • The docs now cover the Run tab, LoRA learning rates, what the Environment status tiles measure, and what happens when a use case’s compute budget runs out.
​
PlatformSecure environmentCLI
2026-10-27
Platform
  • Model training and scoring no longer fail when a secure environment, a test set or the person who launched a training has since been deleted.
  • The sort and filter button on the Metadatasets page is highlighted while a sort or filter is applied.
  • The scrollbar is a lighter grey and the loading bar at the top of the page is brand green.
  • In a use case’s Training tab, each run’s status button, tags and upvote button have the new look, and the whole status button copies the run’s key.
  • On a use case’s Leaderboard, “Train a model” now opens the use case’s Run panel instead of an external Colab notebook, and so does “How to train a model” on an empty Training tab.
  • A queued experiment now tells you when it is waiting for room on its secure environment, and since when.
  • The use case page now shows the dataset key you pass to link_model_dataset when you train from the SDK, with a copy button, under Start training.
  • A training that is waiting for room on its secure environment now says so on its card, with the time it started waiting, instead of reading as in progress.
  • A run that is averaging results or scoring no longer shows “Unknown status” in its header.
  • Signing in on another tab while a device activation page was loading no longer leaves it stuck on “Checking your session”.
  • A run’s Test row, where you submit a cycle for inference and read its test scores, has the new look.
Secure environment
  • The token-classification and masked-language-model templates now train on a Mac’s GPU.
  • Deleting a dataset with the CLI now shows it as Deleted in My Data within minutes, instead of up to an hour later.
CLI
  • tracebloc data delete now tells tracebloc right away, so My Data shows the dataset as Deleted within minutes; if it can’t, it says when it will.
​
PlatformSecure environmentPython SDK
2026-10-27
Platform
  • Experiment comments are visible only to people who can see the experiment itself.
Secure environment
  • Your secure environment now refuses a downloaded weight file that tries to read, write or open a network connection while it loads.
  • Data ingestion on secure environments with service database accounts runs ingestor v0.8.94.
  • Windows installs no longer stop with “can’t read the tracebloc CLI’s minimum version”.
  • An ingest whose dataset registration stalls on the platform side now fails within about 15 minutes instead of hanging for hours.
  • Ingesting a causal or masked language modeling, seq2seq or embeddings dataset no longer ends with a misleading “empty label” warning.
  • An ingest that loses its database connection now fails promptly with “Database unavailable”, so you can run it again, instead of appearing stuck for hours.
Python SDK
  • The Python SDK’s console is quieter: a successful upload shows in green, a scikit-learn training plan explains epochs only when you ask for more than 1, and the dataset parameters appear once.
​
Platform
2026-10-27
Platform
  • When an experiment that trains in several secure environments is refused at launch, the message now says “secure environment” instead of “edge”.
​
CLI
2026-10-27
CLI
  • On a Mac, tracebloc doctor now says when training uses the Apple GPU and where those trainings’ logs are, and it names your secure environment the way you named it.
  • tracebloc upgrade keeps your secure environment on the environment it was installed for, whichever environment the CLI is signed in to.
  • A tracebloc data ingest that stops before the run starts now removes its temporary copy of your data instead of leaving it in your secure environment.
  • Data ingestion on a Mac runner is no longer cut short when the runner restarts its VM to re-measure it.
  • On a Mac, tracebloc doctor no longer reports “runs on CPU” for the GPU when it simply couldn’t list your secure environment’s machines; it says the GPU state is unknown.
  • On a Mac, the reasons a training waits for the Apple GPU, shown by tracebloc doctor and tracebloc resources, now speak of trainings.
  • tracebloc doctor now says “your secure environment’s machines” when it explains why it can’t tell whether a GPU is used.
  • Ingesting data right after your secure environment restarts now waits for it to be ready instead of failing.
​
PlatformSecure environment
2026-10-27
Platform
  • Federated training on tabular, time-series and time-to-event data now uses one shared preprocessing state across every secure environment taking part, so each one encodes its data the same way.
Secure environment
  • An install now stops, and tells you how to update, when the tracebloc CLI on your machine is too old to work and could not be updated automatically.
​
PlatformSecure environment
2026-10-27
Platform
  • A request that is too large, or otherwise malformed before it reaches the API, now gets a JSON error instead of an HTML page.
  • When more metadatasets fail to load, a message with a Retry button appears under the list, and removing a dataset keeps keyboard focus on the list.
  • The dashboard now calls the people you train with “peers” instead of “collaborators”.
Secure environment
  • Training on a secure environment installed with a chart version before 1.9.198 now needs that secure environment upgraded.
  • Your secure environment’s stage now appears under one setting, TRACEBLOC_ENV, in its Helm values.
​
Secure environment
2026-10-27
Secure environment
  • On GPUs without bf16 (such as T4 or V100) with mixed precision on, training no longer loses its first steps of every cycle while the loss scale settles, so small datasets learn from the first step.
  • When the platform’s message queue is unavailable, training no longer stalls for minutes at the end of each epoch while metrics are retried.
  • Object-detection training on a Mac’s Apple GPU no longer runs out of memory while scoring the finished cycle, and trained epochs are kept if scoring itself runs out of memory.
  • A training cycle that was interrupted after all its epochs had finished now resumes straight to scoring and upload instead of training every epoch again.
  • After a ‘tracebloc delete’ that could not finish, the installer now asks once whether to set the old install record aside and go on, instead of refusing forever.
  • On a Mac, the installer’s summary names the VM that runs tracebloc where it said “the macOS node”.
  • The installer no longer offers to set aside an old install record while a k3d cluster may still be running, and it names the command to run first.
  • When an earlier tracebloc install is still running on k3d, the installer no longer offers to set its record aside, even if the machine uses a different cluster name or kubeconfig file.
  • The installer no longer offers to set an old k3d install record aside while your kubeconfig can’t be read, so a live k3d cluster is not left beside a new native install.
​
PlatformPython SDK
2026-10-27
Platform
  • A Mac that trains on its Apple GPU now counts that GPU in the use case header.
  • The peer stack in a use case’s details on Secure environment now counts every peer working on it, showing “+N” for those past the first four.
  • The use case header now shows “1 GPU” with its memory for a Mac that trains on its Apple GPU, instead of “0 GPU”.
  • Double-clicking ✕ on the Run tab’s checks row now stops the checks and leaves them stopped, instead of starting them again.
  • After you delete a secure environment, the Secure Environment page keeps showing it with its CPU, GPU and memory history, its runs and its use cases, instead of going back to the install page.
Python SDK
  • When the platform refuses a model upload, the message names each refused import or call once, with its line number, and lists the allowed imports once.
  • A refused model upload shows the platform’s reason once, without a Python traceback, and a model’s repeated warnings during the upload checks show once.
​
Platform
2026-10-26
Platform
  • Inviting a team to merge no longer fails on an AI use case that has no end date.
​
CLI
2026-10-26
CLI
  • tracebloc data ingest stops at once with the reason when the folder you name on the command line is not laid out for the task, instead of asking the questions again; a --dry-run review now says it only runs the checks.
  • tracebloc data ingest now catches a broken object-detection annotation and manifest rows whose text file is missing before copying anything, and a failed ingest no longer prints an internal stack trace.
  • tracebloc delete now finds helm on a Mac when it runs in a shell whose PATH lacks ~/.local/bin, such as a non-login ssh, and removes the install record.
  • tracebloc resources set now shows the real memory maximum to a tenth of a GiB, and its dry run for the Apple GPU shows the memory limit the Mac will apply.
  • tracebloc data ingest no longer fails when it starts while your secure environment is restarting the runner; it waits up to 90 seconds for the service to come back.
  • Annotation files saved as UTF-16, as XML 1.1, or with entities defined in a DOCTYPE no longer fail the local check before ingest.
  • On a Mac, the lines that say why a training waits for the GPU now name the VM.
​
PlatformSecure environmentDocs
2026-10-26
Platform
  • The sort and filter menu on the Metadatasets page has the new look, and Escape or a click outside closes it.
  • After a run starts, you can start another run with the same model straight from the Run panel: switch between training and inference, change the settings and start again, with no re-upload and no new model checks.
  • Double-clicking “Run again” no longer starts a second Inference only run.
Secure environment
  • Experiments waiting for a Mac’s GPU now start in the order they were submitted, and each waiting experiment knows its place in the queue.
Docs
  • Prepare Data shows how to check a dataset with the tracebloc CLI before you set up a secure environment.
​
PlatformPython SDK
2026-10-26
Platform
  • Who can see, open and applaud an experiment now follows one rule across a use case’s experiment list, its details and its live updates.
  • Restarting an experiment no longer hangs when the connection to its secure environment stalls.
  • Pausing or stopping an experiment at the moment it finishes, or as its first training results arrive, no longer leaves it in the wrong state.
  • Experiment updates resume within seconds after a brief message-queue outage instead of after several minutes.
  • On a Mac, the GPU tile now shows how busy the Apple GPU is and draws its line over time, as it does for NVIDIA GPUs.
Python SDK
  • Training on part of a dataset without class labels, such as causal language modeling, now uses the number of items you ask for.
​
PlatformSecure environment
2026-10-26
Platform
  • When a training on a Mac’s GPU runs out of memory, the experiment now ends as failed with the reason instead of staying “Running” with nothing behind it.
  • Averaging recovers on its own within seconds after a short message-queue outage, and restarts instead of sitting idle when the queue stays unreachable.
  • On a use case’s Training tab, the sort buttons and the Archived button have the new look and work from the keyboard.
Secure environment
  • A secure environment that loses its connection to tracebloc now reconnects within seconds instead of stalling for minutes.
  • On a Mac whose GPU is turned off or can’t train, the platform no longer shows a GPU for that machine.
​
PlatformSecure environmentPython SDKDocs
2026-10-26
Platform
  • A Mac whose Apple GPU trains your models now shows that GPU and the memory each training can use on the platform instead of “no GPU”.
  • The feature icons on the Secure environment install page and the first-run My Use Cases screen use updated tints.
  • The tracebloc logo in the sidebar and the mobile header now uses the design system’s official mark and wordmark.
  • On the Metadatasets page, a dataset card’s Status row now stays inside the card.
  • Accepting or merging from the notifications inbox no longer closes the inbox, so the sending indicator stays visible.
  • Dataset cards show their created date as “15 Mar 2024, 09:30”, like the rest of the app.
  • After answering a team invitation fails, pressing the answer again now sends it.
  • Use case owners can raise each peer’s compute allocation from the Admin tab again: the field shows TF, has Save and Cancel buttons, and the new limit shows straight away in the page header.
  • The avatars of the people on a use case now show the correct “+N” count when more than five people work on it.
Secure environment
  • A secure environment whose tracebloc credentials are refused (wrong password, inactive or pending-deletion account) now stops at the first answer with a clear error instead of retrying the sign-in; when tracebloc is briefly unavailable (429/5xx), the pods monitor now retries too.
Python SDK
  • Asking for a training size the platform cannot apply now stops the run from starting and explains why, instead of quietly training on the whole dataset.
Docs
  • The docs now explain GPU training on a Mac with Apple silicon: which Macs, its limits, and where its logs are.
​
PlatformSecure environmentCLI
2026-10-26
Platform
  • Restarting a failed experiment no longer slows the platform down when some of your secure environments cannot be reached.
  • A use case whose test set is deleted from its secure environment is archived and stops taking submissions, and comes back when the dataset returns.
  • Experiments whose machines have gone offline are cleaned up reliably again, even when one of them fails to close.
  • A namespace you change for your secure environment is no longer reset by a heartbeat that arrives at the same moment.
  • A dataset that has disappeared from your secure environment no longer reappears as available when its metadata is refreshed at the same moment.
Secure environment
  • A successful data ingest no longer shows a misleading “host lookup failed” line in its output, and a slow reply from tracebloc during an ingest now shows a yellow “Near-miss” warning.
CLI
  • Ingesting data no longer shows a “host lookup failed” message while the ingestion starts.
​
PlatformSecure environment
2026-10-26
Platform
  • Registering a secure environment no longer fails when its confirmation email cannot be queued, and a registration that fails no longer sends that email.
Secure environment
  • The installer now honours DO_NOT_TRACK and the other telemetry opt-outs even when it runs in a shell with a custom word separator.
​
PlatformSecure environmentCLI
2026-10-26
Platform
  • The user card now shows up-to-date counts each time it opens instead of a snapshot up to 12 seconds old.
Secure environment
  • helm test storage check now reads the cluster correctly on k3s installs on one machine and on Mac and Windows installs instead of reporting every claim unreadable.
  • The installer now tells you when TRACEBLOC_K3D_VERSION is set that it no longer uses it, instead of ignoring it silently.
CLI
  • On a Mac, a training that waits for the GPU (heat, battery, Low Power Mode, another training) now says why: each reason is an event on its pod, and its log starts with how long it waited and why.
​
Secure environment
2026-10-26
Secure environment
  • The installer’s password prompt, the hint for releasing a secure environment you can’t sign in to, and prepare-host’s group warning name k3s, not Docker or k3d. When TRACEBLOC_SERVERS or TRACEBLOC_AGENTS is set, the installer now warns that it ignores it: native k3s runs one node.
​
CLI
2026-10-26
CLI
  • tracebloc delete and tracebloc doctor no longer handle the older setup (a local k3d cluster on Docker). On such a machine both commands stop before changing anything and name the earlier CLI release that removes that setup, and how to install it. With no install record, delete now uninstalls the release only on a cluster it can prove is this machine’s, and leaves the local cluster in place.
​
Secure environment
2026-10-26
Secure environment
  • On Windows, a machine that still runs tracebloc on Docker Desktop with k3d is now told to delete its secure environment with tracebloc delete, or with tracebloc CLI 0.11.35 where that refuses, then to run the installer again.
​
Secure environment
2026-10-26
Secure environment
  • On a machine that runs tracebloc on k3d, the installer no longer moves it: it stops before it changes anything and says how to move it, which is to delete that secure environment with tracebloc delete (tracebloc CLI 0.11.35 if tracebloc delete refuses), then run the installer again.
  • On Linux, preparing a machine for another user now stops before it changes anything when that user’s tracebloc still runs on k3d, and says how to move it to native k3s.
  • On an arm64 Linux machine that keeps MySQL 5.7, the installer now names the QEMU package to install for amd64 emulation instead of a Docker command.
  • TRACEBLOC_SUBSTRATE=k3d inside WSL now says to install from Windows with install.ps1, a record whose runtime cannot be read names k3s as the runtime to set, --help no longer lists TRACEBLOC_K3D_VERSION, and the host check on Linux no longer shows the Kernel, Subordinate IDs and uidmap helpers rows, which native k3s does not need.
​
Secure environment
2026-10-25
Secure environment
  • Upgrading an existing install no longer sets up the example use case again.
​
CLI
2026-10-25
CLI
  • A freshly installed 16 GB Mac now gets its GPU for training shortly after the install, instead of training on the CPU only.
​
Secure environmentCLIDocs
2026-10-25
Secure environment
  • The training log shows the PyTorch version line once per start instead of up to three times.
  • A training now retries a database connection that times out while connecting, instead of failing.
  • The hint to open a new terminal after an install names only kubectl and helm, the tools the installer puts on your PATH.
CLI
  • Setting the Apple GPU memory per training below 5 GiB is now refused with the minimum named, instead of being accepted and leaving every training on the CPU.
Docs
  • The setup, configuration, operations and troubleshooting pages describe k3s on Linux, Mac and Windows, and no longer tell you to install k3d, Docker Desktop or colima.
  • The quick start and setup guide list the memory and disk each OS needs before the installer will run.
​
Secure environment
2026-10-25
Secure environment
  • On Linux outside WSL, the installer no longer sets up k3d, and asking for k3d there is refused. On a machine that runs tracebloc on k3d, the next installer run or tracebloc upgrade offers to move it to native k3s. The move deletes that secure environment and its data, and it happens only after you type the secure environment’s name to confirm; a run with no terminal stops and changes nothing. The machine comes back as a new secure environment.
  • On a Mac, tracebloc now runs in Apple’s own container runtime and no longer installs or uses Docker Desktop, Colima, Homebrew or k3d. A Mac that already runs tracebloc on Docker Desktop and k3d keeps its secure environment until you agree to a reinstall, but the installer and tracebloc upgrade no longer update it. The reinstall deletes that environment and its data, and the Mac comes back as a new secure environment.
  • Re-running the installer while your secure environment’s hourly auto-upgrade is in progress no longer fails with an interrupted-update error.
  • On Windows, installing with only a Client ID now asks for its password and connects that secure environment, instead of stopping.
  • In the tracebloc/client repository on GitHub, the prerequisites and security guides now say what a Mac needs to train on its GPU, why a Mac may train on its CPU instead, and how a training on the Mac’s GPU is kept apart from the rest of the Mac.
  • On Windows, when the installer stops on a machine that still has tracebloc on Docker Desktop with k3d, it now names the file and kubeconfig entries to delete after the move, instead of saying the next run goes through as usual.
  • If the installer cannot check whether the hourly auto-upgrade is running, it leaves an interrupted update as it is and says so, instead of rolling it back.
  • --diagnose no longer tells an Intel Mac, or one below macOS 26, that it is ready to install; it says why the install will refuse it.
  • On a Mac that still runs tracebloc on Docker Desktop and k3d, the installer and the prerequisites page now say that neither the installer nor tracebloc upgrade updates its secure environment until it moves.
  • The Windows installer now spots a secure environment left on Docker Desktop with k3d whatever format your kubeconfig is written in, and stops with the way out instead of installing beside it.
  • The installer no longer rolls back your secure environment while it cannot confirm that the hourly auto-upgrade has finished.
  • On Linux, the installer’s machine check and its —diagnose report no longer look for Docker or k3d, which an install on k3s does not use.
  • In the tracebloc/client repository on GitHub, the install, security and seal-check guides now describe k3s on every OS, and no longer tell you to set up k3d or fix Docker.
​
CLI
2026-10-24
CLI
  • On a Mac where the Apple GPU would have too little memory to hold a model, a training now runs on the CPU instead of failing.
​
CLI
2026-10-24
CLI
  • Every message now tells you to run tracebloc …, also when you start the CLI with its short alias tb.
​
CLI
2026-10-24
CLI
  • On a Mac, tracebloc doctor and tracebloc resources set no longer point you at Docker Desktop: resources set explains that the limit is the virtual machine the installer sized for this Mac.
​
CLI
2026-10-24
CLI
  • Macs with Apple silicon can train on their own GPU. The installer sets up the tracebloc runner, which runs each GPU training in a sandboxed process on the Mac, and tracebloc resources set gains --gpu-jobs and --gpu-memory and turns the Apple GPU off and on with --gpus.
​
Platform
2026-10-24
Platform
  • Training on a dataset with many columns now shows its results instead of restarting until it stops.
​
Secure environmentPython SDK
2026-10-24
Secure environment
  • Training on a dataset with differential privacy enabled no longer crashes on its first step.
  • A tabular run using one-hot encoding with a model sized for the dataset’s raw columns now fails at the start with a message naming the width one-hot encoding produces and suggesting label encoding, instead of a shape error during training.
  • Survival analysis models now treat censored rows correctly. Before, Cox and survival-forest fits counted every row as an event, so their coefficients and risk scores were wrong.
  • Stopping an experiment while it is starting now ends it cleanly instead of reporting an input-width or model error.
  • On a Mac, an install or re-run that waits for the VM tracebloc runs in to restart no longer fails partway through and leaves the secure environment mid-upgrade.
  • Re-running the installer on a Mac while the secure environment is restarting now clears an interrupted earlier update instead of failing on it.
  • A secure environment no longer restarts the service that runs its experiments for a chart upgrade or an image refresh while an experiment is running, so a run across several secure environments no longer loses that secure environment for a cycle. A busy secure environment can take up to a day to get a new chart or image.
  • On Windows, the installer no longer moves a machine off Docker Desktop and k3d: it stops, changes nothing, and names the installer release that does the move. Inside a WSL distro other than tracebloc’s own, the installer stops and says to install from Windows with install.ps1. On a 16 GB Mac, the VM tracebloc runs in now takes 10 GB of memory instead of 11 GB.
Python SDK
  • When linking a tabular model fails on num_feature_points, the message now says the value must be the dataset’s raw column count, even with one-hot encoding.
​
CLI
2026-10-24
CLI
  • The CLI now uses the same words as the docs: secure environment, Online / Offline, delete, ingest and training.
​
Platform
2026-10-24
Platform
  • A model declaring the one-hot width of a merged tabular dataset now links instead of being refused.
​
PlatformPython SDK
2026-10-24
Platform
  • A finished experiment’s charts no longer change when a secure environment that was dropped from it comes back online and reports training progress late.
  • The Run tab’s model zoo no longer warns that the Logistic Regression Stability template cannot train across more than one secure environment. It now also lists the Random Forest Stability template, and the Gemma 2 template is gone.
  • When a secure environment misses a cycle’s deadline and the cycle is averaged without it, the experiment’s owner now gets a notification, and that secure environment’s late weights for the cycle are no longer accepted.
  • A refused experiment launch now says why it was refused and names the refusal’s code.
  • Sustainability scores no longer flicker for a moment after each training cycle.
  • Sustainability scores are now normalized against the dataset’s scored experiments only, so experiments that have not finished a cycle no longer skew them.
  • A stock lifelines survival template links to any time-to-event dataset without edits.
Python SDK
  • A tabular model trained with one-hot encoding can now declare its encoded input width as num_feature_points and still link to the dataset, when the dataset’s schema declares its categories.
  • A refused training launch now says why it was refused and names the refusal’s code, instead of always reporting missing fields.
  • When linking a tabular model fails on num_feature_points, the message now also names the dataset’s width under one-hot encoding when the dataset accepts it.
​
Secure environmentPython SDK
2026-10-24
Secure environment
  • Where your secure environment’s Helm chart sets no image digest map (for example, images pulled through your own registry mirror), an experiment now keeps one engine version from start to finish, even if a new release lands mid-run.
Python SDK
  • A survival model that fails to link no longer shows a class-count row that does not apply to it.
​
Secure environment
2026-10-24
Secure environment
  • A secure environment that comes back online no longer starts training for a run that has already finished. A secure environment dropped from a training round stops that round’s work instead of running it beside the next round.
​
Docs
2026-10-24
Docs
  • The model zoo index now lists the Logistic Regression Stability and Random Forest Stability templates as trainable across more than one secure environment.
​
Secure environment
2026-10-23
Secure environment
  • If the connection to tracebloc drops while your secure environment reports a failed experiment, it keeps the report and sends it again, so the experiment no longer stays Running with nothing behind it.
  • A GPU run that falls back to the CPU and then fails now says why it ran on the CPU, not only the CPU error.
​
Secure environmentPython SDK
2026-10-23
Secure environment
  • A request to the platform that can never succeed now fails within seconds instead of retrying for minutes.
  • A run whose dataset connection drops now fails and retries instead of hanging at “Training”.
  • Time-to-event experiments using lifelines Weibull, log-normal or log-logistic AFT models can now be trained across multiple secure environments; for lifelines experiments across several secure environments (Cox and AFT) the downloaded weights are a coefficients file (.safetensors) rather than a pickled lifelines model.
  • On a Mac, the installer now waits while tracebloc restarts the secure environment’s node or the tracebloc CLI changes the secure environment. If that takes longer than 5 minutes, the installer stops without changing the secure environment, says why, and asks you to re-run it once it has ended.
  • Installing the secure environment with the tracebloc runner and per-experiment database credentials on a Kubernetes cluster older than 1.30 now stops with a clear message naming the version needed, instead of failing on an unknown resource.
  • On Windows, the installer no longer starts the example use case before your secure environment has connected.
Python SDK
  • A lifelines survival model trained across several secure environments can now be loaded back from its downloaded coefficients file with load_lifelines_model, ready to predict.
​
Secure environment
2026-10-23
Secure environment
  • When the platform’s message queue is briefly unavailable, the secure environment no longer stops starting and monitoring experiments while it retries failure reports.
​
PlatformPython SDKCLIDocs
2026-10-23
Platform
  • A use case’s admins can now add and remove that use case’s other admins, without needing an account admin.
Python SDK
  • Starting an experiment across several secure environments with a scikit-learn model that cannot be averaged across secure environments, such as a voting ensemble or an SVR, now fails at launch with the reason, instead of failing after the first training cycle.
CLI
  • tracebloc resources set, tracebloc delete and tracebloc seal now wait, up to five minutes, while another tracebloc command is changing the secure environment on the same machine, and say so, instead of changing it at the same time.
Docs
  • The setup docs now say that a Mac needs Apple silicon and macOS 26 or later, and that a new Mac runs without Docker.
  • The setup docs now say that Windows runs tracebloc in its own WSL distro, without Docker Desktop, and where a Mac keeps your datasets.
​
PlatformCLI
2026-10-23
Platform
  • Experiments no longer lose a round’s results when the platform has a brief internal outage; the results are retried instead of dropped.
  • The Start, Pause, Stop and Eco training buttons keep keyboard focus while a run’s status change is saving.
  • Templates and My Use Cases are now one Use Cases page, with your use cases, the ones shared with you and the templates in three tabs.
CLI
  • On a Mac, a GPU job that waits while the runner restarts its idle node no longer stalls the job that asked for the restart.
  • tracebloc delete on a Mac set up over SSH now stops Apple’s container system, or refuses with the command to stop it, instead of reporting success.
  • On Windows, tracebloc delete no longer leaves an empty tracebloc folder or a stale PATH entry behind.
​
Platform
2026-10-23
Platform
  • An experiment’s scores have the new design-system look.
​
PlatformSecure environmentPython SDKCLI
2026-10-23
Platform
  • A team invitation that partly fails no longer fails as a whole: the other invitations still go out.
  • Experiments keep starting and reporting progress when one of your secure environments is unreachable.
Secure environment
  • After the install connects, your account’s example use case is set up for you once; set TRACEBLOC_SKIP_EXAMPLE=1 to skip it.
  • A jobs-manager that stops making progress can now restart itself on its own, once the new liveness check is turned on.
  • A secure environment configured with a custom ingestion port or the new ingestion-disable setting now reports ready correctly.
  • On a new install, the example use case’s data now loads, and a test dataset’s preview no longer shows its labels.
  • Upgrading tracebloc on Linux no longer ends with the first-install “ingest your data” steps; it says the upgrade is complete and your data is unchanged.
  • Ingesting a test dataset no longer sends its rows’ labels to the platform: its sample rows carry only their row id.
Python SDK
  • Starting an experiment across several secure environments now stops straight away, with the reason, when the scikit-learn model cannot be combined across secure environments (for example a single decision tree or a kernel SVM), instead of failing after the first training round.
CLI
  • A new secure environment’s example use case now comes with its example data, ingested in your own secure environment.
  • A new secure environment’s example use case starts training its model by itself right after install, so its first result is there within about a minute.
​
PlatformCLI
2026-10-23
Platform
  • An experiment’s controls (resume, pause, stop and eco training) have the new design-system look.
  • When saving a dataset card’s heading or description fails, the editor now shows an error message instead of silently staying open.
CLI
  • When data ingest --example targets a cluster your active secure environment isn’t on, the error no longer sends you to a command that was removed.
  • On Apple Silicon, a GPU experiment no longer falls back to the CPU when the runner’s idle node restart has not finished yet.
  • When data ingest --example refuses a cluster your active secure environment isn’t on, it no longer suggests a client create --context that can’t fix it, and a machine with no linked secure environment gets advice that fits it.
​
PlatformCLI
2026-10-23
Platform
  • If the Metadatasets page can’t load your datasets, it says so and lets you try again, instead of showing an empty page.
  • Editing a metadataset card’s heading and description now opens in a side panel, with Cancel and Update buttons.
  • Metadataset cards have a refreshed look; their edit, copy and details buttons work from the keyboard, and the extra secure environments are listed from a “+N” chip.
  • On a use case’s Training tab with no experiments yet, the three getting-started guides are now links you can reach and open from the keyboard.
CLI
  • When your secure environment’s data ingestor is too old for a dataset or for the example, the message now says the ingestor is pinned, that it changes when the secure environment is upgraded, and how to set up the example afterwards, instead of promising the hourly auto-upgrade.
​
PlatformPython SDK
2026-10-23
Platform
  • Experiment cards, notifications and the Secure Environment page now update as soon as something changes, without a reload.
  • Dates across the platform now read the same way: “21 Oct 2026”, “21 Oct 2026, 14:05”, or “2 min ago” for anything in the last week.
  • Removing a deleted dataset from the Metadatasets page now asks you to confirm first.
  • Train and run inference on your models straight from a use case, in the new Run panel.
  • The Metadatasets page shows its title while your datasets load.
  • Keyboard users now see a focus outline on buttons and controls that used to hide it (Explore cards, the admin edit bar, experiment charts, the notebook viewer and the use case editor).
  • When you choose Inference only, zoo models that cannot run it are greyed out with the reason, before you start.
  • Inference runs and submissions now count against your team’s compute budget, and the Run tab can show what an inference will cost and how many of today’s submissions are left before you start.
  • Errors from starting an inference-only run now include a stable error code, so the SDK and Run tab can react to each reason without reading the message text.
  • The Run tab’s dataset check now shows the class order your model’s outputs must follow.
  • Inference runs started from the Run tab now use the input size your model declares instead of a fixed 224.
  • Inference runs started from the Run tab now use the model’s image size, batch size and the use case’s task, the same as runs started from the SDK.
  • Before you run inference only, the Run tab shows what the run will cost and how many of your team’s daily submissions are left.
  • The “you need a dataset” and Run drawers now dim the page behind them and keep keyboard focus inside until you close them.
Python SDK
  • Inference-only runs can now be started with the model name the Run tab checked, without linking the model to the dataset first.
​
PlatformDocsOther
2026-10-23
Platform
  • A submission that could not be queued now shows an error and can be submitted again, instead of being blocked as already submitted.
  • A dataset no longer shows as used by a use case after that use case is deleted.
  • A team can no longer go over its daily inference limit by submitting several inferences at the same moment.
  • The dataset details on a use case’s Training tab now open in a side panel, and the copy confirmation for the dataset ID shows on top of it.
  • The Metadatasets page’s subtitle no longer has a typo.
  • You can filter the Metadatasets page by the person who created each dataset.
Docs
  • The EKS guide now explains how to revoke a secure environment that runs on EKS from your account.
Other
  • Saving a metadataset card’s heading updates the list even if you close the editor before the save finishes.
​
Secure environment
2026-10-23
Secure environment
  • An inference run where no sample could be scored (for example, every image file is missing) now fails with a reason instead of finishing as Completed with zero metrics.
  • The jobs-manager and resource monitor pods now carry the node name as TRACEBLOC_NODE_NAME only; NODE_NAME is no longer set.
  • The auto-upgrade job now carries the chart name and repository as TRACEBLOC_CHART_NAME and TRACEBLOC_HELM_REPO_URL only; CHART_NAME and REPO_URL are no longer set.
​
PlatformCLIOther
2026-10-23
Platform
  • Labels of a use case’s test data no longer appear in the sample rows its peers receive.
CLI
  • tracebloc doctor now checks the platform you are signed in to when it cannot read your secure environment, instead of always checking the production platform.
  • On Windows, tracebloc doctor now tells you how to start, inspect or restart k3s inside its WSL distro instead of giving Linux-only commands.
Other
  • On the Metadatasets page, a failed load shows as an error box, and the list’s end and no-match messages use the new type style.
​
PlatformOther
2026-10-23
Platform
  • On a use case’s Overview, the “On this page” links can be reached with the keyboard, and its buttons have the new design-system look.
  • On a use case’s Leaderboard, the table, its column explanations and the team avatars have the new design-system look, and Size and Energy sort by value.
  • The end-date picker on a use case has a refreshed look, from both the stats bar and the Admin tab’s Duration setting.
  • The model code panel of an experiment has the new design-system copy button and loader.
  • The team list and invite box on a dataset row have a refreshed look.
  • The profile card that opens from a run’s or a comment’s avatar has a refreshed look.
  • The empty Datasets page’s setup steps and “Create a Secure Metadataset” button now match the rest of the platform’s look.
  • The page headers on My Use Cases, Explore and Metadatasets use the refreshed button and tooltip style.
  • The search box on the Metadatasets list uses the refreshed input style.
  • Keyboard users can skip past the navigation rail with a “Skip to content” link.
  • Clicking delete on a disabled dataset card more than once no longer sends duplicate delete requests.
  • The edit and invite buttons on a use case’s banner now use the same icons as the rest of the platform.
  • With a single environment, the header no longer shows a switcher arrow that opens nothing, and notices no longer offer a “Switch environment” button with nowhere to go.
  • The edit button on a run’s notes can now be reached with the keyboard and is read out by screen readers.
  • The notifications list in the side rail opens above the bell, closes with Escape, and returns focus to the bell.
  • A dataset deleted in its secure environment no longer shows a made-up Metadata ID on the Metadatasets page, and its hidden secure environments can be listed from the “+N” chip.
  • Hovering a dataset owner’s avatar shows their name in the standard tooltip.
Other
  • Deleting a metadataset updates the list in place and keeps the confirmation on screen, instead of reloading the page.
  • The answer buttons on a notification (Accept, Deny, Merge, Reject) have a new look, and can be reached with the keyboard.
​
Python SDK
2026-10-22
Python SDK
  • Every pretrained zoo seed whose task head depends on your class count now ships as backbone only, so you can fine-tune it on a dataset with any number of classes.
​
Secure environment
2026-10-22
Secure environment
  • A training round that starts before the previous round’s merged model is ready now waits for it, for up to 10 minutes, instead of failing that secure environment’s round.
  • On smaller machines, such as Windows laptops, each experiment now gets all the memory the machine has room for, so GPU runs no longer run out of memory during validation.
  • A GPU device plugin image that cannot be pulled no longer blocks secure environment auto-upgrades on multi-node GPU clusters; the secure environment upgrades and those nodes fall back to CPU until the plugin runs.
  • A failed data ingest now says why its records failed.
  • The installer asks what to call your machine and who provides it; change either later with tracebloc machine set.
​
CLI
2026-10-22
CLI
  • Name your machine and its provider from the terminal with tracebloc machine set; tracebloc machine shows them.
​
Platform
2026-10-22
Platform
  • An experiment whose averaging step crashes the averaging service now fails with a clear reason instead of hanging.
​
PlatformCLI
2026-10-22
Platform
  • Archiving a dataset you do not have access to now reports that it was not found, and nothing is archived.
  • The Gemma 2 model-zoo template is retired; its pretrained seed is no longer served.
  • Signing in and registering keep working during a brief interruption of our caching service instead of failing with a server error.
  • On a use case’s Eval tab, the scoring code is shown in the new design-system code box, with a labelled Copy code button.
  • An experiment’s confusion matrix has the new design-system type and lines.
  • An experiment’s model details and notes editor have the new design-system look.
CLI
  • tracebloc client remove can now revoke a secure environment you set up on a managed cluster such as EKS from your machine, and then clears it as your active secure environment.
  • On Windows, tracebloc client status --seal now runs its checks without needing helm installed on Windows, and tracebloc doctor reports whether k3s is running inside the tracebloc WSL distro.
​
PlatformSecure environmentPython SDKCLIOther
2026-10-22
Platform
  • Someone who leaves an account can no longer see or revoke the secure environments they created there; members who created a secure environment can now revoke it themselves.
  • When one machine fails a training round of a multi-machine run, the run carries on and you now get a notification naming the machine and why.
  • On an experiment across several environments, each environment’s run history shows whether its own part failed or ran out of memory.
Secure environment
  • Training in secure environments whose storage is an EFS access point no longer fails when it saves the model weights.
  • Starting or stopping a run during a brief cluster or database hiccup now retries instead of failing the run, and a stop that cannot be applied is never reported as a failed run.
  • The one-line installers now check the tracebloc installer’s signature before running it, and stop if it does not verify.
Python SDK
  • The model zoo’s stability-selection template is back: with many candidate features and few rows, it reports which features truly predict the label and how many false finds to expect.
  • The Gemma 2 text-classification template is no longer in the model zoo.
  • The model zoo’s random-forest stability-selection template is back, beside the logistic one, and its trained model downloads as a plain scikit-learn pipeline with the stability report on it.
CLI
  • tracebloc resources set max now gives each experiment all the memory the machine has room for, instead of rounding it down to a whole GB.
  • On Windows, tracebloc resources set now changes an experiment’s resources without needing helm on Windows.
Other
  • In a shared use case, you now see only your own team’s experiment details, metrics and comments, and can clap only experiments you can see; the leaderboard still shows every team.
  • Archiving or commenting on an experiment you cannot see now reports it as not found.
​
Secure environment
2026-10-22
Secure environment
  • PyTorch experiments started after this release upload a compressed update from the second cycle on, about half the data or less, and FedAvg and FedProx experiments may end with slightly different results.
  • Training uses less memory when it sends its update at the end of every cycle after the first, so large language models are less likely to run out of memory there.
​
Platform
2026-10-22
Platform
  • The message after deleting a dataset now appears at the top right of the screen, like the app’s other notifications.
​
PlatformSecure environment
2026-10-22
Platform
  • Explore and My Use Cases are back to their familiar layout; an unfinished redesign is no longer shown.
  • A failed or paused experiment now shows its reason in a redesigned message card.
  • The Secure Environment page is redesigned: your environment’s status and resources, your data, your use cases and the people working with you on one page, with notices for what changed since your last visit.
  • The GPU notice now shows how much GPU memory is used and says new runs wait in the queue. On My Data, a dataset whose use cases you cannot see shows – instead of 0 use cases.
  • The Secure Environment page keeps loading when one environment reports a reading it cannot use, and a deleted environment shows as offline straight away.
  • You can now create a use case from your secure environment’s page without leaving it, and each use case shows the compute and submissions used so far.
Secure environment
  • A resumed experiment keeps the memory-saving settings it switched to after running out of GPU memory, so it does not run out of memory the same way again. An experiment paused while the secure environment updates restarts its current cycle once.
  • On macOS, a secure environment whose stopped node fails to restart with its local dashboard port now finishes installing instead of failing.
  • Each release now includes signature files for the Linux/macOS and Windows installers.
  • A failed data ingest can be run again on the same files straight away, and large JSON datasets no longer run out of memory before they ingest.
  • Ingesting into, and cleaning up after a failed ingest on, large shared dataset tables no longer scans the whole table or blocks training reads of it.
  • Segmentation masks whose file names contain extra dots, such as scan.001.png, are now paired with the right mask file instead of a similarly named one.
  • Validating a large JSON dataset no longer runs out of memory, and a JSON array item that isn’t an object is now rejected at validation instead of failing later during ingest.
  • Datasets with more columns, or wider rows, than the database can hold are now refused up front with a clear message, instead of failing with a database error.
  • Re-running a failed image or file ingest no longer needs the storage folder cleaned by hand first.
​
Platform
2026-10-21
Platform
  • The use-case Admin settings for title, cover image, duration and daily submissions have a refreshed look, and the daily-submissions picker now works with the keyboard.
  • After archiving one use case and opening another, the Admin tab’s archive switch now shows the second use case’s own state.
  • After switching use cases, the Admin tab’s compute allocation now shows the current use case’s values, not the previous one’s.
​
Platform
2026-10-21
Platform
  • Tooltips on an experiment’s status, author and charts now use the new design system’s style and are read out to screen readers.
​
Platform
2026-10-21
Platform
  • Loading indicators across experiments, datasets, comments and profiles now use the new design system’s animated dots, and say what they are loading to screen readers.
  • The loader on notification actions and the column tips on the use case leaderboard now use the new design system’s style, and a column’s tip also opens when you tab to its header.
  • The compute and archive settings in a use case’s Admin tab have a refreshed look.
  • The Secure Environment page’s environment switcher and drawer links now show the design system’s standard focus outline when you move through them with the keyboard.
  • Signing in or signing up with GitHub or Google now only completes when you started it yourself in the same browser tab.
  • The More options tooltip in the peers list always shows with its proper styling.
  • An experiment’s training plan sheet has the new design-system type.
  • The hover bubble on an experiment’s charts has the new design-system look.
  • The leaderboard preview on a collaboration’s tag has a refreshed look.
  • The dataset info dialog in your secure environment now shows the dataset’s exploratory data analysis instead of loading forever.
  • After you switch to another use case, its Admin tab now offers that use case’s compute allocation and closes any archive confirmation you left open, so Confirm no longer writes the previous use case’s value.
  • Switching use cases in Admin while a save is in flight no longer shows or applies that save on the new use case.
​
Python SDK
2026-10-21
Python SDK
  • When a model file has no MyModel, the error now names the main_class / main_method it looked for and lists the classes and functions the file defines; the LoRA error no longer suggests from_pretrained, which upload refuses.
​
Python SDK
2026-10-21
Python SDK
  • A model file with a missing or unusable header value (for example output_classes = len(CLASSES), image_size = (224, 224) or framework = "torch") now fails at upload with a message that names the line and what to write. Every PyTorch model file must now declare batch_size.
  • Uploading an sklearn model without a valid model_type now fails before training starts and lists the values you can use.
​
PlatformSecure environmentCLI
2026-10-21
Platform
  • Signing in with GitHub or Google now requires a verified email address on that account.
  • A secure environment that reports a GPU figure the platform cannot read keeps its live readings on the Secure Environment page, and the card shows as an unknown GPU.
  • The Secure Environment page no longer shows 0 runs when the secure environment could not read them.
Secure environment
  • Windows laptops with an NVIDIA GPU now show live GPU usage and memory on the dashboard instead of “usage not reported”.
  • fix: on a Mac, the node’s packet filter lets only this Mac reach the secure environment’s local API port.
CLI
  • An ingest no longer fails when the platform is briefly unreachable while the run is being submitted; the CLI retries.
​
CLIDocs
2026-10-21
CLI
  • The CLI is built with the latest Go security fix for certificate verification.
Docs
  • The training guide now explains that pausing keeps a PyTorch model’s finished epochs, and that models built with other frameworks restart the current cycle when resumed.
​
PlatformSecure environment
2026-10-21
Platform
  • Newsletter sign-up now answers an invalid email address with a clear error instead of failing, and treats addresses that differ only in capitalisation as the same subscriber.
  • The peers column in My Use Cases (“Collaborators” on screen) no longer counts people who were invited but have not joined.
  • Signing in with GitHub now explains a missing email instead of failing with a server error, and GitHub or Google sign-in refuses an inactive or deletion-locked account just as password sign-in does.
  • Downloading a model whose experiment was deleted after the download link was created now answers “not found” instead of a server error.
  • Submitting the same password reset twice at once now says the reset code is not valid instead of failing with a server error.
  • Combining datasets whose owner was removed or whose label metadata is incomplete now shows a clear error instead of an unexpected server failure.
  • The account panel at the bottom of the sidebar now opens above your name and matches the new design system.
  • In a use case’s Admin tab, the training, weight-file and secure environment controls use the new design system’s switches and buttons, and screen readers now announce what each switch controls.
  • The peers list in a use case’s Admin tab has a refreshed look.
  • On the onboarding page, “Sign in” now takes you straight to the sign-in page.
  • You can now sort the use case leaderboard from the keyboard with Enter or Space, and screen readers announce each column’s sort order.
  • My Secure Environment no longer scrolls into a blank space below the page, and no longer shows the sign-in page for a moment when you reload while signed in.
Secure environment
  • Resuming a paused PyTorch experiment now continues from the last finished epoch on production installs too; models built with other frameworks still restart the current cycle, and operators can turn this off per install.
​
Platform
2026-10-21
Platform
  • Sample training now runs for every model on each new dataset, including models whose names contain capital letters.
  • Combining datasets whose stored statistics are malformed now shows a clear error naming the dataset instead of failing with a server error.
  • Opening an invitation link whose email was never sent now shows the invitation as expired instead of an error.
  • Combining datasets that list the same dataset twice now says so, instead of reporting the dataset as not found.
  • Changing the compute allowance per user on a use case that has an empty team no longer fails with a server error.
  • An experiment on a dataset with no sample preview no longer fails when its cycle is sent for averaging.
​
Platform
2026-10-21
Platform
  • The reset and update password pages have a refreshed look, clearer error messages that stay until you try again, and they work with password managers; after changing your password you are asked to sign in again.
  • New team members who join by invitation add their name and get an email link to set their password; people who sign in with Google or GitHub aren’t sent one.
  • The dataset rows in a use case’s Admin tab use the new design system’s type, so titles, descriptions and dataset names read slightly tighter.
  • The request-a-use-case dialog now shows the platform’s reason when a request is rejected.
  • Notifications shown while a company settings dialog is open can now be read and dismissed.
​
Platform
2026-10-21
Platform
  • The Explore page has a refreshed look, with an Industry menu to filter templates.
  • Your password no longer needs a number and a lowercase letter when you sign up; it needs at least 8 characters and can’t be a common or all-numeric password, and signing in no longer refuses a password without a number.
​
CLI
2026-10-21
CLI
  • Pressing Ctrl-C while tracebloc delete checks a Windows machine for a WSL install now stops quietly instead of telling you to run the installer again.
​
PlatformCLI
2026-10-20
Platform
  • The Storage tile on a secure environment now shows how much space the environment itself uses and how much room it has left to grow, instead of the whole machine’s disk.
CLI
  • Ingesting data while a model trains no longer waits for the experiment to finish when the machine was sized with resources set max.
​
Secure environment
2026-10-20
Secure environment
  • Object-detection training now stops with a clear error when none of the training images can be read, instead of finishing the cycle with untrained weights.
  • A PyTorch tabular model trained elsewhere, with its data preparation written into the model file, can be scored on a use case’s test set.
  • A finished training cycle whose upload fails is no longer trained again when the job restarts; the restart delivers the result it already has.
  • During a platform outage, the end of a training cycle keeps retrying its results and weights for up to 15 minutes instead of failing the experiment after a few minutes.
  • Multi-GPU training no longer skips early stopping, learning-rate reduction and best-checkpoint saving for an epoch when only one of them watches a metric that cannot be combined across GPUs.
  • Failed runs are no longer blamed on your data or model because an unrelated word in the error message looked like a known cause.
  • Training on dense-output tasks such as segmentation no longer runs out of memory partway through an epoch.
  • Wide tabular datasets now load and preprocess using about a third of the memory, so they no longer run out of memory before training starts.
  • Training steps for tabular, time-series, survival and NLP models on GPU wait less on the host.
  • Training that falls back from multi-GPU sharding to a single process now keeps updating the model, with its optimizer state intact, instead of silently training nothing.
  • Keypoint R-CNN training runs up to about 2.6x faster per step, and its training accuracy now shows as ”—” (not measured) instead of a number. Validation accuracy and PCK are unchanged.
  • Training on segmentation and token-level NLP tasks no longer runs out of memory at the end of a cycle.
  • GPU training and inference runs now stay within the CPU and memory size shown for each run, so a GPU run can no longer take memory the platform needs.
  • Installing on a Mac with your own values file that keeps data in host folders no longer fails.
  • A secure environment installed on a Linux machine with an NVIDIA GPU now reports the machine’s GPU driver version.
  • The Windows installer now shows the sign-in and registration messages correctly in Windows PowerShell, with no garbled characters.
​
PlatformPython SDKCLI
2026-10-20
Platform
  • Resetting your password no longer undoes profile changes you saved at the same time.
  • The use case tab bar and the use cases filter tabs can be navigated with the arrow keys again, and Tab no longer steps through every tab.
  • Sign-up now shows several password problems as separate sentences instead of running them together with a comma.
Python SDK
  • The missing-tokenizer error no longer suggests setting a Hugging Face tokenizer ID, which model upload rejects.
CLI
  • On Windows, tracebloc delete now removes everything it installed, including the tracebloc program itself, and ends with a clean result. On a machine where tracebloc still runs but its install record is missing, it stops before changing anything and says what to do.
  • On a Mac, tracebloc delete no longer leaves an empty tracebloc folder in Application Support.
​
PlatformPython SDK
2026-10-20
Platform
  • An account verified just as its sign-up window runs out is no longer deleted by the inactive-account cleanup.
  • A training GPU limit of 0 on a GPU machine now stays 0 instead of being reset to the machine’s GPU count.
  • Sign-up now shows password requirements as plain messages, and a password with leading or trailing spaces works for sign-in.
  • On a use case page, the tab bar, the Join / Sign up button and the team’s avatars have the new design-system look.
  • The Secure Environment page shows times in your own time zone.
Python SDK
  • Uploading a model from Windows now works with model files that contain non-ASCII characters, are saved with a UTF-8 byte-order mark, or are given as a C:\... path.
​
Platform
2026-10-20
Platform
  • Trying to join a use case while signed out now asks you to sign in instead of failing with a server error.
  • The use-case table’s peers column (“Collaborators” on screen) no longer counts you on your own machine, so it matches Who Is Working With Me.
  • A use case’s total compute budget no longer counts the same peer twice when their email address was saved with different capital letters.
  • Pausing a use case’s experiments from a form submission now pauses them instead of leaving them running.
  • A team’s final score no longer disappears from a competition’s final leaderboard after support reruns its final inference.
  • The dataset picker in Create Use Case and the dataset list load faster when your account has many datasets.
  • The Copy Dataset Key hint on a use case’s stats bar is no longer cut off by the page card.
  • The Secure Environment page now counts peers the same way in every section, leaving out yourself and people who are only invited, and its trainings total shows your peers’ trainings.
  • Edit and copy icons on the use case stats bar render at their intended size again.
  • The Secure environment page keeps its status and readings current while it is open.
  • My Use Cases now shows a No data status for a use case whose data is in no secure environment, and lets you filter by it.
  • A use case on a machine that does not report its GPU memory now shows “1 GPU” instead of “1 GPU (0 GB)” under Resources.
  • Submissions and leaderboard lists that fail to load more rows now show a Retry button.
​
PlatformSecure environment
2026-10-20
Platform
  • A GPU that is not NVIDIA is no longer labelled “NVIDIA” on the Secure Environment page.
  • On a use case’s Submissions tab, every column can be sorted (Date and CO2e now reorder the list) and a run opens from the keyboard.
  • Reopening an experiment’s comments, or a submission’s details, now shows what changed since you last looked.
  • A newly installed GPU machine no longer shows “No GPU found” while it finishes connecting, and its GPU appears without a page refresh.
  • A missing date now shows as — instead of 01.01.1970 in My Submissions and in experiment model details.
  • Training cards keep showing a run’s latest epoch after you switch tabs and come back.
  • On a use case page, the daily submission limit opens as a menu under its pencil and can be picked with the keyboard.
Secure environment
  • When the platform closes a training round before your secure environment’s weights arrive, the experiment is no longer marked Failed, and uploads the platform rejects outright now fail at once instead of after minutes of retries.
  • When a time-series sequence carries more than one label, the error now points to the affected rows instead of quoting your sequence identifier.
  • The Secure Environment page now shows a Mac’s real disk size and free space instead of roughly double it.
  • On a Windows GPU install, the dashboard shows the GPU within one monitoring cycle (about 30 s) of the installer setting it up, not up to a minute later.
  • If a data ingest is killed partway through, you can ingest into the same table again within minutes, instead of waiting up to 12 hours.
  • The secure environment’s MySQL 8.4 image picks up the latest Oracle Linux security updates (OpenSSL, curl, GnuTLS, libxml2).
  • When an ingest pod is retried while the earlier run is still finishing, the earlier run no longer removes the retry’s table lock, so two ingests can no longer write one table at the same time.
  • Retrying an ingest after a crashed run no longer fails for hours with “Another ingest is already running”, and a long-running ingest can no longer be interrupted by a newer one on the same table.
  • A brief storage hiccup during an ingest no longer lets a second ingest of the same table take over its lock, or start without one, and remove the first ingest’s rows.
  • A rejected dataset registration no longer prints values from your data into the ingest logs or CLI output; the error still shows the HTTP status and which field was rejected.
​
PlatformPython SDKCLIDocs
2026-10-20
Platform
  • Models trained elsewhere can now be scored without training on time-series and time-to-event use cases, when they bring their own data preparation.
  • An experiment on several secure environments keeps going when one secure environment fails; that round is averaged from the secure environments that finished, and the failed secure environment rejoins the next round.
  • Inviting several people to a use case now delivers every invitation even if one address cannot be reached.
  • Large model downloads from the platform no longer fail partway or briefly take the platform offline.
  • Archiving a dataset from the platform can no longer archive an unrelated experiment instead.
  • Pausing, stopping or resuming an experiment from an invalid state no longer leaves its status blank in the experiment list.
  • Scikit-learn multi-output forests whose outputs have different numbers of classes now aggregate instead of failing the training cycle.
  • Models that read their own file location, or that ship as an archive with several model files, now load the same way on every aggregation run.
Python SDK
  • Linking a second fitted model in the same notebook no longer picks up a custom class from the first model’s folder.
  • Uploading a scikit-learn, lifelines or scikit-survival model and then linking a second model in the same session no longer picks up the first model’s code.
  • Models that bring their own data preparation (DATA_PREPARATION = "model") can now be uploaded for time-series classification, time-series forecasting and time-to-event; before, upload_model refused them at its training check.
  • The Trains check no longer fails raw-mode time-series and time-to-event models that upload successfully.
CLI
  • tracebloc data ingest on an image or text task without --label-column now tells you which flag to pass and lists the columns of your labels file, instead of printing a schema validation error.
Docs
  • The benchmark demo video starts faster and uses far less data.
​
PlatformOther
2026-10-20
Platform
  • Comment threads on an experiment now load the page you pick, not only the first one.
  • When a machine’s NVIDIA driver is too old for GPU training, the Secure Environment notice names the driver it has and the one it needs.
Other
  • The Update Profile page has a refreshed look, matching the rest of the new design.
​
Platform
2026-10-20
Platform
  • Switching to another use case while its Admin settings are still saving no longer shows the previous use case’s data or messages.
​
Platform
2026-10-20
Platform
  • Switching use cases while a Max submissions change is saving no longer leaves the setting stuck loading.
​
Platform
2026-10-20
Platform
  • The Duration setting now shows the selected use case’s end date after switching between use cases.
​
PlatformSecure environment
2026-10-20
Platform
  • When a Max submissions change on a use case’s Admin tab is rejected, the reason now shows next to the field and the previous limit is restored; a successful change is confirmed.
  • Editing a use case’s name, cover image or team name, and uploading its EDA file, now opens a refreshed side panel.
Secure environment
  • Compute usage is no longer under-counted when a usage report briefly fails to reach the platform.
  • Binary classification models that end in a sigmoid now train correctly when class balancing is applied, and configured loss options are kept.
  • Early stopping and learning-rate reduction now follow accuracy monitors instead of stopping or cutting the learning rate on runs that are still improving.
  • An experiment whose loss or metrics could not be computed now shows them as missing instead of as a perfect or zero score
  • A time-series classifier or forecaster trained elsewhere, together with its own data preparation, can be scored on a use case’s test set.
  • Training rounds whose model weights diverge to NaN or Inf now stop with a clear error instead of sending corrupted weights to averaging.
  • When a dataset fails to load, the reported error now names the database error type and code instead of the raw database message, which could include values from your data.
  • On multi-GPU secure environments, training no longer switches into a sharding mode that still ran on a single GPU with fixed optimizer settings.
  • An experiment whose update cannot be sent intact is now reported as failed instead of sending broken weights to the shared model or restarting into the same failure.
  • An experiment whose secure environment’s credential has been revoked now stops at once and reports that reason, instead of retrying the model upload for minutes.
  • Segmentation Hausdorff distance and average surface distance now leave out the background class and count a class the model misses or wrongly predicts in an image as the largest possible distance, so these scores read higher (worse) than in earlier runs, and a model that predicts no foreground at all shows no score.
  • When reading a dataset fails, the error and its log no longer include values from your data.
​
Secure environment
2026-10-19
Secure environment
  • Training runs on PyTorch 2.14.1 on every secure environment; GPU secure environments need an NVIDIA driver R580 or newer for GPU training, and train on the CPU with the reason shown until they have one.
  • an inference with no saved data preparation is refused instead of scoring with preparation fitted on the test set.
  • Training no longer fails and loses the finished cycle when the platform is briefly unreachable while the trained model is being uploaded.
  • On small machines, the installer now leaves room for data ingestion beside an experiment, and says so when an ingest will wait for training to finish.
  • On Windows, an install started with only a client ID now stops at once with a clear message, without leaving a half-installed WSL environment behind.
  • GPU training now needs NVIDIA driver 580 or newer. On an older driver the installer sets the machine up in CPU mode and says how to update.
  • On a Mac, other virtual machines on the same computer can no longer reach the tracebloc cluster’s kubelet, API or service ports.
  • On a Mac, training pods can resolve only the cluster’s own names, and no service port is published.
  • feat: on a Mac, training data, logs and resume points live in a folder on this Mac that only you can open, ~/.tracebloc/job-data, and tracebloc delete removes it.
  • fix: on a machine with restricted DNS on, every Mac included, an experiment leaves room for the resolver, so an experiment sized to the machine’s edge no longer sits Pending.
  • On Windows, the installer now shows the sign-in link and code, so you can sign in and finish installing.
  • On a Mac, the installer’s log now says why the tracebloc runner was not set up, instead of always blaming the CLI release.
  • Editing or reformatting your install record no longer makes the installer forget what it installed, so tracebloc delete still removes everything.
​
Platform
2026-10-19
Platform
  • The sign-up page has a refreshed look that matches the rest of the platform.
  • Check marks and close buttons use the refreshed icon style.
  • The sign-in and password pages have a refreshed look that matches the rest of the platform.
  • The former “Clients” page, which listed secure environments, now opens My Secure Environment.
  • After you resend the verification email from the sign-up confirmation page, the 30-second wait now counts down and the resend link comes back.
  • The onboarding page has a refreshed look that matches the rest of the platform.
  • Sign-up and onboarding now tell you when an account could not be created because of a connection or server problem, instead of silently doing nothing.
  • The start-training page has a refreshed look that matches the rest of the platform.
  • A run that moves to the CPU because the NVIDIA driver or GPU can’t be used now says why, and the dashboard no longer asks you to update a driver that isn’t the problem.
  • The dataset card and its edit dialog no longer show a “Requires GPU” setting that was never saved.
  • An archived use case now shows its Archive switch on after a reload.
  • Refreshing the secure environment status on a use case’s Admin tab now shows the current status instead of the old one.
  • Your secure environment’s page shows each machine by the name you gave it, and the memory of all your machines together.
​
Secure environment
2026-10-19
Secure environment
  • A GPU secure environment whose NVIDIA driver is too old for the GPU training image now trains GPU experiments on the CPU, and says why, instead of failing them. The GPU is used again on its own once the driver is updated.
  • The secure environment’s heartbeat now reports the NVIDIA driver version, and says why the GPU is not used when the driver is too old for the GPU training image.
  • An experiment whose NVIDIA GPU cannot run the GPU training image now trains on the CPU instead of failing.
​
Python SDK
2026-10-19
Python SDK
  • Linking a PyTorch model whose own data preparation reads a column the dataset lacks through .loc, .at or attribute access is now refused by naming that column, instead of being reported as a failure on sample values.
  • An SDK that is too old for the platform now tells you to upgrade when it checks a raw-mode model or loads a dataset’s features, instead of asking you to try again or carrying on with no features.
​
2026-10-19
​
Platform
2026-10-19
Platform
  • An inference run that fails in a secure environment now shows as failed, with its cause and the environment, and its owner is told, instead of staying “Running”.
  • A training or inference run on a dataset ingested before ingestor IDs existed now fails with a clear message telling you to re-ingest it with a current ingestor, instead of a confusing label-type error.
  • When an account is deleted, the models and weights that person uploaded, and the datasets they built, are now deleted too, unless something else in the account still uses them.
  • Use cases whose data was removed from a secure environment are now reliably archived instead of sometimes staying launchable.
  • Deleting a user who owns a data space is now refused with a clear reason, instead of locking their account with no way to finish the deletion.
  • When inference is refused because the weights hold only a backbone, the message now points you to Train instead of a Fine-tune option that does not exist.
​
Python SDKCLI
2026-10-19
Python SDK
  • Linking a scikit-learn model for a tabular use case that brings its own data preparation now tells you straight away if it cannot work with the dataset, for example when it reads a column the dataset does not have, predicts a class it does not have, or returns the wrong shape, and shows exactly what the model will receive.
CLI
  • tracebloc resources set max now leaves room to ingest data while a run trains, and tells you when the machine is too small to do both at once; tracebloc doctor reports the same.
  • tracebloc delete no longer lists a folder among what it will remove when the install never recorded where that folder is.
​
PlatformCLI
2026-10-19
Platform
  • With reduced motion turned on in your system settings, the colour background on the legal, blog, career and platform overview pages now stays still.
CLI
  • The “create the use case” link printed after a data ingest now opens the create-use-case wizard instead of the use case list.
​
Platform
2026-10-19
Platform
  • Accounts locked for deletion are now deleted automatically when the 30-day grace period ends, and the lock can no longer be cancelled after that.
  • When an account is deleted, its email address is also removed from invitations, notifications, contact records and the audit trail.
  • When a use case ends, teams that did not pick their finals now get their best submissions entered as finals automatically. This applies to use cases that end on or after 19 October 2026; earlier use cases keep their finals as they are.
​
CLIDocs
2026-10-19
CLI
  • Deleting a secure environment whose local k3d cluster was stopped now finishes cleanly instead of reporting incomplete cleanup and setting k3d up again on the next install.
Docs
  • The Helm setup guides now create your secure environment’s credentials with tracebloc client create and install it into the namespace the tracebloc CLI looks in.
​
CLIDocs
2026-10-19
CLI
  • tracebloc doctor now says an NVIDIA driver older than 580 leaves the GPU off (tracebloc’s GPU training needs CUDA 13), with how to update it (on Windows and WSL, NVIDIA’s download page), a reboot and the installer re-run.
Docs
  • The setup guide’s unattended install now creates the secure environment with tracebloc client create instead of the former “Clients” page.
​
Secure environment
2026-10-19
Secure environment
  • fix: an experiment now leaves room for the restricted-DNS resolver, so an experiment sized to the edge of a machine with restricted DNS on no longer sits Pending.
​
PlatformPython SDK
2026-10-19
Platform
  • The model zoo API now says, for each template, whether it can run inference only and, if not, why.
Python SDK
  • pip install "tracebloc[pytorch]" now installs torch 2.14 and torchvision 0.29, the versions training runs on.
​
CLI
2026-10-19
CLI
  • fix: machine-sized experiments leave room for the restricted-DNS resolver, and no longer reserve CPU for a data ingest, which requests none.
​
PlatformDocs
2026-10-19
Platform
  • The training countdown no longer shows impossible times such as “60m 59s”; it now counts down accurately.
Docs
  • The benchmark page loads faster and no longer starts downloading the demo video until you press play.
  • Pages on tracebloc.io load a little faster.
​
Platform
2026-10-18
Platform
  • A Mac in the secure environment switcher can show its Apple chip instead of a core count, once its secure environment reports the chip. Today’s secure environments don’t report it yet, so the switcher still shows the core count.
​
PlatformSecure environmentDocs
2026-10-18
Platform
  • Mask2Former and OneFormer experiments now average instead of failing.
Secure environment
  • Tabular boosting experiments whose secure environments hold different label sets keep training after the first averaging round instead of failing.
  • Data can now be ingested while a model trains, and a machine keeps its full CPU for training while it does.
  • On Windows, the installer reserves CPU and memory for k3s from a measured record of tracebloc’s WSL distro, instead of warning that none was measured.
Docs
  • The setup guides and FAQ now consistently say “experiment” where they used to say “training run”.
​
Secure environmentDocs
2026-10-18
Secure environment
  • Pausing an experiment now stops its pod cleanly, so it shows as completed instead of as an error.
Docs
  • The local machine guide lists what Windows needs for tracebloc in its own WSL distro.
  • The operations guide now shows how to stop and start your secure environment on a Mac that runs Apple’s container runtime.
​
Secure environment
2026-10-18
Secure environment
  • When the storage seal check cannot read a volume claim, its log now shows the actual error instead of reporting the claim as missing.
​
Secure environment
2026-10-18
Secure environment
  • On a Mac, when Apple’s notarisation check does not answer in time, the installer asks once more, and if there is still no answer it says it cannot tell whether Apple’s container package is notarised, instead of calling it untrusted.
​
Secure environment
2026-10-18
Secure environment
  • Object detection models that don’t declare a model type now train instead of failing with “Unsupported model type”, and keypoint models that declare none train as direct regressors whatever their name.
  • Keypoint detection precision, recall and F1 now show as not measured, rather than 0, when every keypoint in an evaluation set is occluded.
  • Training logs now name each mixed-precision (fp16) step that was skipped because its gradients overflowed, with the cycle, the epoch and the new loss scale.
  • Ingesting data now lists the count per class in the ingestion summary, so a class you did not expect shows up before you create a use case.
​
PlatformDocs
2026-10-18
Platform
  • XGBoost, LightGBM and CatBoost experiments no longer fail at averaging when one secure environment’s data is missing a class; the merged model predicts every class.
  • The install page now describes the new installer: no Docker Desktop on a Mac or Windows, what each machine needs, and the one macOS prompt to allow.
Docs
  • The local machine guide lists what a Mac needs for tracebloc on Apple’s container runtime.
​
Secure environmentDocs
2026-10-18
Secure environment
  • Changing an experiment’s resources or upgrading the secure environment restarts tracebloc’s jobs-manager about a minute faster.
Docs
  • The allowed-imports page now lists numpy for every task, matching what model uploads accept.
​
PlatformSecure environmentCLI
2026-10-18
Platform
  • Federated training of neural-network (MLP) classifiers no longer fails when secure environments hold different sets of labels.
  • Saving your profile without editing it keeps your name instead of asking you to type it again, and a new secure environment password is checked against the listed requirements before it is sent.
  • The loading screen and the “page not found” page have a refreshed look, and the loading screen is announced to screen readers.
Secure environment
  • Forecasting and regression models trained without a chosen loss function now learn: they train with mean squared error instead of a classification loss.
  • On an Intel Mac, or a Mac on macOS 15 or older, the installer stops before changing anything and says why. A tracebloc already installed there keeps the version it has.
  • On a Mac with Apple silicon and macOS 26 or later, a first install now sets up Kubernetes in Apple’s built-in container runtime, with no Docker, Homebrew or k3d. A Mac that already runs tracebloc stays on what it has.
CLI
  • The tracebloc CLI for macOS now needs an Apple-silicon Mac with macOS 26 or later. The installer refuses an Intel Mac or macOS 15 or older, and a tracebloc already installed there keeps the version it has, without updates.
​
Secure environment
2026-10-17
Secure environment
  • Keypoint detection experiments now report real precision, recall and F1 scores instead of always showing 0, including on competition leaderboards scored by them.
  • New secure environment installs pin the MySQL 8.4 image by digest instead of following a moving tag.
​
PlatformSecure environmentCLI
2026-10-17
Platform
  • After you activate your account, the confirmation screen now says it is taking you to Explore, where it actually sends you, instead of promising the login page.
Secure environment
  • A model whose output classes are in a different order than the use case’s is refused with a clear reason instead of being scored wrongly.
  • Federated training from scratch now starts every participating secure environment from the same initial weights, so the averaged model trains as expected.
  • Node CPU utilisation is no longer missing from resource monitoring when the cluster reports very small CPU usage.
  • The pods-monitor log now shows a failed training pod’s container output as one quoted, escaped line instead of many.
  • The Windows installer’s WSL distro moves to ubuntu-24.04.5-20261017-v1.36.5-k3s1 (was ubuntu-24.04.5-20261001-v1.36.5-k3s1), with the Ubuntu updates it declares.
  • Re-running the installer on a machine that already runs tracebloc keeps the MySQL engine that machine runs, on Linux, Mac and Windows.
CLI
  • tracebloc doctor explains a GPU your container runtime can’t see in plain words without naming a specific runtime, and on Windows it tells you how to use a GPU that runs in the TCC driver model or is a data-centre card.
​
PlatformSecure environmentDocs
2026-10-17
Platform
  • Linking a model whose label map lists the classes in a different order now succeeds with a warning, and starting an inference-only run with such a model is refused with a clear reason.
  • Inference runs started from the browser now feed images in the colour mode the model expects.
  • A round’s F1, precision, recall, AUC and every other reported score are now weighted by each secure environment’s training-data size, matching loss and accuracy, so secure environments with more data count for more.
  • Federated NearestCentroid training no longer fails when secure environments hold different subsets of the classes; the averaged model now predicts every class any secure environment saw.
  • Gradient-boosted survival models trained across several secure environments can now be loaded after they are merged, instead of being refused as unsafe.
  • Linear-model experiments (including LDA) now merge when secure environments hold different sets of classes, instead of failing at averaging.
  • Tooltips on dataset, experiment and leaderboard screens now have the new design-system look, a darker compact bubble with a caret.
Secure environment
  • A model can be scored on a use case’s test set without uploading weights; it is scored on its random initialisation, the same way every time.
  • A tabular model trained elsewhere, together with its own data preparation, can be scored on a use case’s test set.
  • Data ingestion started during a full-size experiment no longer waits until the experiment ends.
  • New installs run MySQL 8.4 on every machine; a secure environment still on MySQL 5.7 stays on 5.7 only when its settings pin it, and its upgrade stops before touching its data otherwise.
Docs
  • The privacy policy now lists every service and cookie the website uses, and returning visitors are asked for cookie consent once more.
​
PlatformPython SDK
2026-10-17
Platform
  • An inference-only run now always finishes as done, failed with a reason, timed out or stopped, instead of staying “running” with no end.
  • You now get a notification when an inference finishes with its score, fails, or times out, so you no longer need to keep the page open.
  • Results of runs that only score a model are labelled “Inference only” in My Submissions and on the leaderboard, with their metrics shown only to the precision the test set supports.
  • The owner’s runs list and the leaderboard say when a run only scored a model, and whether its weights were uploaded, picked from the model zoo or a random initialisation.
  • Validating a model now shows the order its outputs must follow and refuses a model whose labels are in a different order, instead of letting it score near chance.
  • A model trained elsewhere can now be scored on a tabular use case when it brings its own data preparation, and inference that would learn its preparation from the test set is refused with a clear reason before it starts.
  • An inference-only run with no weights now starts from the model’s random initialisation on the secure environment instead of looking for a weight file that does not exist.
  • Running inference on a model picked from the zoo no longer fails with “The model could not be staged for inference”.
  • Running inference on a zoo model whose weights hold only the backbone is now refused with the reason, instead of silently scoring an untrained model.
  • Starting inference on a scikit-learn-family model with no weights is now always refused with the reason, even when the request does not say which framework the model uses.
  • Starting inference without naming the use case’s dataset now returns a clear “datasetId is required” error instead of a server error.
  • A finished inference run no longer shows an error when a failure report arrives at the moment it completes.
  • A trained model is scored on every test secure environment with the data preparation it learnt in training, never with one learnt from the test set.
  • Each run in the All runs list can now report the CPU, memory, disk and GPUs its container actually got, including when the environment shrank it to fit.
  • Account admins can lock a member’s account for deletion and cancel it within 30 days.
  • Scoring final submissions after a use case ends no longer pauses your team’s training on your other use cases.
  • A use case’s final submission no longer reports “timed out” while it is still waiting for the use case to end.
  • A submission whose evaluation never completes now reliably triggers the timeout notice instead of waiting indefinitely.
Python SDK
  • Hugging Face models whose label names differ from the use case’s classes only in letter case or spacing now link for training instead of being refused.
​
PlatformSecure environment
2026-10-17
Platform
  • My Submissions now marks runs that only scored a model, without training it, as “Inference only”.
  • Notifications now appear in the top-right corner in the new design, and the in-page notices in team, company and secure environment dialogs use the new design too.
  • The account activation, device sign-in and team invitation pages have a refreshed look that matches the rest of the platform.
  • Pressing Escape now closes only the newest notification. Before, it closed every notification on screen at once.
Secure environment
  • When jobs-manager cannot list cluster nodes, its log now says an overdue GPU run is waiting for capacity (not stuck as running), and the edgeuser-narrowing log says the account is dropped only when DROP_EDGEUSER is on.
  • The NVIDIA GPU device plugin moves from v0.14.5 to v0.20.1, pinned by digest.
  • On a Mac or Windows machine, the installer now refuses (or sizes up) a Docker VM too small to fit a single experiment, instead of installing a secure environment that can never train.
  • On PowerShell 7.5 and later, the Windows installer no longer leaves an empty TUF_ROOT variable in your session after it checks the release signature.
  • The secure environment’s pods now carry only the TRACEBLOC_ names of its settings; scripts of your own that read a pod’s environment must use those names.
  • On Windows (WSL2), the memory check now reports how much memory the WSL2 VM can actually give an experiment instead of understating it by 2 GB.
  • On a Mac behind a network that inspects TLS, tracebloc now installs with your corporate CA (TRACEBLOC_CA_BUNDLE) instead of refusing it.
  • On a Mac behind a proxy, tracebloc’s built-in VM now uses your HTTP(S)_PROXY and NO_PROXY settings. They reach the VM in a file only you can read, never on a command line.
  • On a Mac, TRACEBLOC_STORAGE_MODE=hostpath and TRACEBLOC_HOST_DATASET_DIR now work with tracebloc’s built-in VM: it mounts those folders at the same path.
  • The native k3s install no longer needs iproute2: it reads the machine’s routes from the kernel, so a minimal Linux and tracebloc’s Windows distro install without it.
  • On an 8 GB Linux machine whose Docker memory limit is set too low, the installer now tells you to raise that limit instead of telling you to use a larger machine.
  • A data folder given as ~/… or relative to your home folder no longer stops a macOS install that uses hostPath storage.
  • The chart now refuses to install or upgrade when the tracebloc images would be pulled from Docker Hub (images.traceblocRegistry or global.imageRegistry set to docker.io). Docker Hub’s frozen control-plane images cannot sign in with this chart’s settings. The error names the fix: clear the value so the secure environment pulls from ghcr.io.
  • On OpenShift, enabling the chart’s GPU device plugin now stops the install with a message pointing at the GPU Operator, instead of leaving a DaemonSet that never schedules.
​
PlatformSecure environmentPython SDKCLI
2026-10-17
Platform
  • Federated training with naive Bayes or QDA models no longer fails when some secure environments have no examples of a class.
  • Random forest, extra trees and bagging classifiers now finish federated training when some secure environments have not seen every class.
  • In the environment switcher, the Online status now has a thin green outline.
  • The calendar for picking a use case’s end date has a refreshed look.
  • Changing a secure environment’s password no longer breaks the secure environments page when the platform refuses it; the error message is shown instead.
  • An end date you pick for a use case is now saved on the day you chose, whatever your time zone.
  • When Validate finds your model’s outputs in a different class order, the Run panel now lists the order your model must follow.
  • The All runs drawer on your secure environment now shows the GPUs, CPU cores, memory and disk each run actually got, and says when a run was shrunk to fit.
  • You can now change a secure environment’s password or delete a secure environment straight from its row in the secure environments table.
  • Company owners can now hand ownership to another admin from the Company page.
  • The old link to the federated-learning applications article now redirects permanently to its current address.
  • The homepage loads faster.
  • The healthcare page loads its main image much faster.
  • Blog posts and the benchmark and healthcare pages load faster.
  • Pages start rendering slightly sooner.
  • Scrolling the blog, careers and platform pages is smoother.
Secure environment
  • The secure environment now stores its credentials only under their TRACEBLOC_ names; update the tracebloc CLI to 0.10.63 or later, and point any script of your own that reads the secure environment’s Secret at the TRACEBLOC_ keys.
  • Re-running the installer now updates a tracebloc CLI older than 0.10.63, the oldest version the platform accepts.
Python SDK
  • You can score a model on a use case’s test set without training it, from the SDK with user.run_inference(...).
  • Starting an inference-only run from the SDK now works against the platform, and a refused run shows the platform’s own reason.
  • Inference-only runs now start for grayscale image models: pass data_type="grayscale" to run_inference().
  • Inference-only runs no longer send empty time-series settings that the platform could refuse.
  • The training plan now lists the class order your model’s outputs must follow, one index per class.
CLI
  • tracebloc resources set, tracebloc seal and tracebloc delete now use the helm the installer set up, even when it is not on your PATH.
​
PlatformSecure environmentCLI
2026-10-16
This release changes a compatibility floor or an install-time contract. Read the required actions before you update.Platform
  • Storage totals for an environment now count only the nodes that are currently reporting, so an offline or stale node no longer inflates them.
  • A use case archived because a data owner withdrew their dataset can no longer be un-archived by hand.
  • Scores on a test set that lacks some of the training classes are now computed against the right classes.
  • Data scientists can score a model on a use case’s test set without training it, from their own weights, a zoo model’s, or none.
  • An inference-only run’s score now lands on the team board and the leaderboard, and the run can be picked as a final submission.
  • Submitting a model for inference on a dataset that is not a use case now says why it cannot run, instead of appearing to start and never finishing.
  • A use case built through a data space no longer shows another organisation’s sample records on its combined dataset.
  • The Datasets page lists only your own organisation’s datasets again, so every row you see can be edited and deleted.
  • A use case that was archived by hand can no longer be un-archived while one of the datasets it uses is withdrawn from its data space.
  • Only people who can manage a dataset can now contribute it to a data space.
  • Archived use cases no longer accept new experiments, and experiments running on them are paused.
  • An experiment whose averaging step never finishes is now retried and then restarted automatically, instead of staying in “Averaging” indefinitely.
  • Paused experiments on an archived use case can no longer be resumed.
  • Runs that only score a model without training it are labelled “Inference only”, with where their weights came from, on the experiment card and the leaderboard.
  • Switching between Train and Inference only on the Run tab no longer prepares the training plan twice.
  • The side panels on the Secure Environment page slide in when they open, and their close button is easier to hit.
Secure environment
  • An inference run on weights that lack a trained task head now stops with a clear message instead of scoring a randomly initialised head.
  • GPU inference runs now get the CPU and memory size you set for inference.
  • Data ingestion no longer waits behind an experiment that uses the whole machine.
  • The Linux installer no longer hangs silently when a package mirror stalls while installing system packages; it stops or continues with a clear message instead.
  • On Linux, the installer now stops early when Docker’s memory budget is too small to run any experiment, instead of finishing an install that can never train.
  • The installer no longer leaves a ~/.sigstore folder in your home. cosign’s signature-check cache now lives in the installer’s temporary folder and is removed with it. A ~/.sigstore from an earlier install is left as it is, because it can’t be told apart from one your own cosign made.
  • Windows installs no longer stop at the certificate-trust step when WSL answers in UTF-16.
CLI
  • The CLI installer no longer leaves a ~/.sigstore folder in your home. cosign’s signature-check cache now lives in the installer’s temporary folder and is removed with it. A ~/.sigstore from an earlier install is left as it is, because it can’t be told apart from one your own cosign made.
  • tracebloc resources set now keeps room for a dataset ingestion beside a max-sized experiment, so an ingestion started while an experiment is going no longer waits forever.
Required actions
  • Update the tracebloc CLI to 0.10.63 or newer: the platform no longer accepts older versions.
​
PlatformCLI
2026-10-15
Platform
  • The Who Is Working With Me table on the Secure Environment page opens a peer from their name with the keyboard as well as the mouse, and loads more peers as you scroll.
  • The owner’s runs list says when a run only scored a model, and whether its weights were uploaded, picked from the model zoo or a random initialisation.
CLI
  • tracebloc delete now lists, under “Left alone”, the PATH lines the installer added to your shell profile, with the file they are in, so you can remove them yourself.
​
Secure environment
2026-10-15
Secure environment
  • The installer now stops on a Linux machine too small to run a single experiment, and says how much memory it needs.
​
Platform
2026-10-15
Platform
  • The My Data table on the Secure Environment page opens a dataset from its name with the keyboard as well as the mouse, and loads more datasets as you scroll.
  • Experiments on the use case page now fit narrow and laptop-sized screens instead of being cut off on the right.
  • On a narrow screen, every score in an experiment row now wraps onto its own line instead of three of them staying squeezed into a quarter of the row.
  • The My Use Cases table on the Secure Environment page opens a use case from its name with the keyboard as well as the mouse, and loads more use cases as you scroll.
  • The cookie banner asks again once your choice is 12 months old or our cookie policy changes.
  • Every sign-up and contact form now links to our privacy policy, and the cookie banner links it too and gives Reject the same weight as Accept.
  • You can now change or withdraw your cookie consent at any time from the “Cookie settings” link in the website footer.
  • The website no longer contacts Google Tag Manager until you accept cookies; rejecting or ignoring the cookie banner sends nothing to Google.
​
PlatformSecure environmentCLIDocs
2026-10-15
Platform
  • A secure environment can now reach only the experiments and test sets it takes part in.
  • The celebration when you create your first use case or add your first dataset is now a calmer message with an animated icon, without the full-page confetti.
Secure environment
  • Object detection experiments now report average precision and recall for each class, by class name (ap_class_<name>, ar_class_<name>), in training validation and inference results.
  • Causal language modeling experiments now report exact match (sequence accuracy): the share of answers the model gets exactly right.
  • Re-running the installer on native k3s no longer needs raw.githubusercontent.com, and it now checks up front the hosts a re-run does need: rpm.rancher.io on every run of an SELinux machine, and GitHub’s release hosts on a run that upgrades k3s. A re-run that used to stop partway, at the SELinux policy or the k3s download, now stops at the connectivity check and names the host.
  • The Windows (WSL) installer no longer reports the tracebloc CLI as freshly installed when it is actually falling back to an older copy already on your PATH; it now warns and names the copy it uses.
CLI
  • tracebloc doctor no longer says everything looks good when it couldn’t read your GPU’s details; it now says some checks couldn’t finish.
Docs
  • Blog posts no longer jump down the page while their images load.
  • Quick Start shows how to install a secure environment you created with + on the secure environments page, using the command that page gives you.
​
Platform
2026-10-15
Platform
  • After you add a secure environment with +, the secure environments page shows the install command for that secure environment, ready to copy.
​
Secure environment
2026-10-15
Secure environment
  • Training large text models without LoRA no longer runs out of memory at the end of a training round.
  • Re-running the Windows installer after its credential was refused now restarts the secure environment’s services onto the new credential when the secure environment was installed under a release name different from its namespace.
  • On RHEL-family Linux machines with a slow package mirror, the installer no longer hangs silently at “Installing Docker…” or “Installing kernel modules…” — it stops with a message naming the stalled download, or skips the optional kernel modules and carries on.
  • The installer now waits out a brief GitHub download outage instead of failing at “1. Downloading”.
  • After an upgrade that changes the database credential, the secure environment’s services restart onto the new credential even when the installer could not read the old one beforehand.
  • MySQL now replaces an old install-time root password with the one in mysql-credentials on its own at startup; the MySQL pod restarts once during the upgrade.
  • A Linux install that times out while installing packages no longer leaves the system package manager half-finished or locked, and if the package manager is still busy the installer tells you to wait for it before re-running.
  • The installer again retries a release download that drops partway through, or whose redirect to GitHub’s download servers never connects, instead of failing at “1. Downloading”.
​
PlatformCLI
2026-10-15
Platform
  • Starting an experiment now requires being on the use case’s team, so the run counts against your FLOPS budget; join the use case first if you see “Join the use case before starting an experiment on it.”
  • A training cycle whose averaging succeeded is no longer restarted, and each secure environment’s uploaded weights are cleaned up after its cycle whatever their file format.
  • A brief infrastructure hiccup while finishing a training round no longer forces that round to be retrained on every secure environment.
  • Machines with an NVIDIA GPU on an arm64 CPU now show the reason their GPU is not used for training, instead of a blank one.
  • A training cycle that finishes without reporting an epoch no longer restarts, and the epochs after it keep their running count.
  • An experiment whose averaging result arrives malformed is now restarted right away instead of staying in “Averaging”.
  • On the Secure Environment page, the “+N” after a dataset or use case is now a button you can tab to, and its tooltip lists the names it hides.
  • A deleted secure environment is listed on the secure environments page as Deleted instead of Offline.
  • The sidebar is simpler: Templates, My Use Cases, Secure Environment and Docs, with the page you are on marked in green and your name on the account item.
  • Renaming a secure environment or setting its location now sticks when you search, filter or sort the secure environments list, instead of snapping back until a reload.
  • A deleted secure environment no longer shows as online in a use case’s secure environment status or on the dataset card.
CLI
  • tracebloc doctor now shows your GPU’s model, driver and compute capability, and whether GPU training can use it.
  • tracebloc client remove no longer removes a secure environment that still has experiments in progress unless you pass —force.
​
CLI
2026-10-14
CLI
  • The one-line CLI installer always installs the newest stable release, even after an older release is republished.
  • tracebloc upgrade now tells you on a Mac running native k3s that it runs the full install and updates your secure environment’s services.
  • You can now remove a secure environment you no longer use from your account with tracebloc client remove <id>, without deleting anything on this machine.
  • tracebloc client remove takes either the client ID that tracebloc client create prints or the numeric ID shown on the secure environments page.
  • tracebloc client remove never removes this machine’s own secure environment, and asks for --force whenever it can’t confirm the secure environment you named runs somewhere else.
  • The installer now retries a failed download a few times before giving up, so a brief network blip no longer stops the install.
​
PlatformSecure environment
2026-10-14
Platform
  • Object detection experiments show average precision and recall for each class, and causal language model experiments show Exact Match. Your secure environment reports these figures from Helm chart 1.9.215 on.
Secure environment
  • On Windows, re-running the installer now reconnects a machine whose credential tracebloc stopped accepting.
  • Installing with the client ID from the secure environments page now asks for that secure environment’s password and connects that secure environment, instead of registering a new one.
  • Re-running the installer with a secure environment’s password now restarts the services so the machine connects with it.
​
PlatformSecure environment
2026-10-14
Platform
  • An experiment paused for its budget, a finished competition, the owner, or by you no longer restarts on its own when sustainable training sees the grid get greener.
  • Sustainable-training experiments now resume on their own when carbon intensity drops, even after an earlier resume attempt failed.
  • Starting a run whose estimated compute cost is larger than your team’s remaining compute budget now returns a warning naming both amounts and how to raise the budget; the run still starts.
  • Causal language modeling use cases can rank on Exact Match, and object detection runs accept per-class AP/AR scores, so a use case can score on one class. Your secure environment reports these figures from Helm chart 1.9.215 on.
  • Sustainable training pauses and resumes on current grid carbon readings again.
  • An experiment no longer sends the same training round to its secure environments twice when an averaging result is delivered again.
  • The experiment list API now records when a run’s training restarted after a crash or out-of-memory kill, and why (restart_count, last_restart_reason, restart_note).
  • Averaging jobs that run past the time limit now fail visibly instead of leaving the experiment stuck.
  • The side panel no longer shows a separate Help button; documentation stays one click away under Docs.
  • The home and benchmark pages no longer flash white before turning dark when they load.
  • Job titles on the careers page now match the job pages they link to.
  • Blog post dates now all appear in the same style.
  • Blog, careers and legal pages load a much smaller footer logo.
  • The site no longer runs Google analytics or advertising cookies before you make a choice in the cookie banner, and your Reject is respected.
  • On older browsers, the home and benchmark pages show their dark background again instead of white text on a white page.
Secure environment
  • On machines with 8 or more CPU cores, an experiment now gets 1 core and up to 1 GiB of memory less, so the hourly maintenance jobs always fit beside it.
​
Docs
2026-10-14
Docs
  • The training guide now explains that pausing loses the current epoch’s progress and that a pause not resumed within 7 days expires.
  • The Start Training and Hyperparameters pages now show how to train on part of a dataset that has no class labels.
​
Platform
2026-10-14
Platform
  • Pages opened after the home page show their scrollbars again, so wide code blocks and tables can be scrolled.
  • Blog posts show their table of contents as soon as the page loads, without waiting for scripts.
​
PlatformDocs
2026-10-14
Platform
  • The Run tab’s model check findings now show in a light table, with each failed or fixed value marked by an icon.
Docs
  • The training guide notebook shows how to train on part of a dataset, including one without class labels.
​
PlatformPython SDKCLI
2026-10-14
Platform
  • A long dataset name in the use case header now shortens with ”…” instead of pushing Submits onto a second line.
  • The two feature stability-selection tabular templates are no longer offered until they can be uploaded.
Python SDK
  • You can now train on a set number of records from a dataset without class labels (for example causal language modeling or regression) with training.training_size(300).
CLI
  • The ingest summary now lists how many records, boxes, tags or sequences each class has, so an unexpected class is visible straight away.
​
PlatformSecure environment
2026-10-14
Platform
  • The install page again says that a Linux machine on k3d gets Docker if it is missing and keeps it on removal.
Secure environment
  • Re-running the installer on a native k3s machine now checks it first, as on k3d: a credential tracebloc no longer accepts is renewed, and a stopped or half-set-up k3s gets the same messages and repair.
  • When the installer gives up waiting for the node to be Ready, it now says what the node last reported and how many times k3s restarted (on a Mac, whether the node is still running).
​
Platform
2026-10-14
Platform
  • An experiment whose paused run settles into a failure now switches its alert from Paused to Failed without a page reload.
​
PlatformCLI
2026-10-14
Platform
  • A resumed experiment no longer keeps showing its old pause notice.
CLI
  • tracebloc upgrade on Linux now says up front that on native k3s it runs the full install and updates your secure environment’s services.
​
PlatformSecure environmentCLIDocs
2026-10-13
Platform
  • A paused experiment now says why it was paused: your team’s compute budget is used up, the competition has ended, it is waiting out high grid carbon intensity, or the owner turned experiments off.
  • An experiment now shows when a training attempt restarted on its own and why, for example because it ran out of memory.
  • Keyboard focus is easier to see, success and warning text is easier to read, and Secure Environment tables now size their columns to their content.
  • Buttons that are loading keep their label and are announced as busy by screen readers.
  • Pressing Escape now closes the Create Use Case panel immediately after it opens, and keyboard focus moves into the panel.
  • A paused experiment now shows why it was paused, instead of a red “Failed” alert.
Secure environment
  • A paused experiment keeps its saved training progress for 7 days; after that, resuming restarts the current cycle.
  • A paused experiment keeps its resume point for the full 7 days, also after it was paused, resumed and paused again.
  • On development and staging installs, resuming a paused experiment continues from the last finished epoch; operators can turn this off per install.
  • The Mac installer no longer mistakes its own container system for someone else’s when your home folder path ends in a slash.
  • On dev, MySQL now replaces an old install-time root password with the one in mysql-credentials on its own at startup; the MySQL pod restarts once during the upgrade.
CLI
  • tracebloc upgrade --help now says that on a native k3s machine (the default on a new Linux machine) the upgrade always runs the full install, which also brings your secure environment’s services up to date.
Docs
  • The Linux install docs now describe native k3s, the default on a new Linux machine: what it needs (administrator rights once, or an administrator’s one-time prepare-host), what it installs, and the hosts it reaches.
  • The Operations page now says that on a native k3s machine (the default on a new Linux machine) tracebloc upgrade always runs the full install, which also brings your secure environment’s services up to date.
​
Platform
2026-10-13
Platform
  • A paused experiment that is not resumed within 7 days now shows that its pause expired; resuming it restarts the current cycle.
  • Deleting a user account no longer deletes the use cases they own or their comments on shared experiments.
​
Platform
2026-10-13
Platform
  • The pause button now warns that pausing loses the current epoch’s progress, and a paused experiment shows when its saved progress has expired.
  • A comment whose author’s account was deleted now shows “Deleted user” instead of breaking the experiment’s comments.
​
Secure environmentCLI
2026-10-13
Secure environment
  • Failed experiments now show the correct failure reason (for example model vs data) instead of “not classified”.
  • On a Mac with Apple silicon and macOS 26 or later, the installer can now set up your secure environment on k3s in Apple’s container runtime instead of Docker and k3d, by running it with TRACEBLOC_SUBSTRATE=k3s. It is opt-in, and k3d stays the default. It needs no administrator rights, Homebrew or Docker. macOS asks once to allow Local Network access for “container-runtime-linux”, which the cluster needs. A Mac installed this way keeps k3s on later runs without the variable.
  • A new Linux machine whose Docker refuses a read now says why, in Docker’s own words, instead of reporting that Docker did not answer.
  • Publishing an older release to the public download site no longer changes which installer the one-line install command downloads.
  • With self-rotation of the MySQL root password on, a root password you change by hand in mysql-credentials is now kept when MySQL restarts instead of being reverted.
  • Installing on a Mac behind a proxy set only through ALL_PROXY no longer stops with a false Local Network permission error.
CLI
  • tracebloc upgrade --plain now keeps all of its output free of colour codes, including lines printed while it repairs your secure environment.
​
Platform
2026-10-13
Platform
  • You can now see when each secure environment joined, in a new “Date joined” column on the secure environments page, and sort by it.
​
CLI
2026-10-13
CLI
  • If you signed out and back in, tracebloc doctor no longer sends you straight to the installer when it can’t see your secure environment. It first tells you to run tracebloc client create, which reconnects this machine to the secure environment already running on your cluster.
​
PlatformSecure environmentCLI
2026-10-13
Platform
  • The install page no longer tells Linux users to install or open Docker; a new Linux machine runs without it.
Secure environment
  • The one-line installer’s Linux prerequisites (native k3s, administrator rights or a one-time prepare-host, memory, disk and the hosts it needs) are now on one page, docs/PREREQUISITES.md.
CLI
  • On Linux, tracebloc doctor now says how to start Docker Engine instead of pointing at Docker Desktop, and the delete and prepare-host help describe native k3s, the default on a new Linux machine.
​
PlatformSecure environmentPython SDKDocs
2026-10-13
Platform
  • Model and data training failures now show their cause instead of “not classified”.
  • Secure environments now report their security posture to the platform, so hardening across a fleet can be verified.
  • A secure environment’s reported chart version, app versions and security posture no longer revert to an older value when two heartbeats arrive close together.
  • A secure environment’s security findings and component versions no longer disappear when a heartbeat could not check them.
  • Uploading a large PyTorch model (such as a GPT-2-sized language model) no longer makes the platform briefly unavailable during model checks.
  • A node’s reported cluster versions and security posture no longer lose a value when two of its heartbeats arrive close together.
  • A node’s reported cluster versions and security posture no longer show a blank value where an earlier report had a real one.
Secure environment
  • CatBoost experiments no longer fail at the start of the second training cycle.
  • A paused experiment now shows as completed instead of as an error.
  • An experiment whose pretrained seed was built for a different number of classes now fails with a message that names the seed, instead of a bare size-mismatch error.
  • A fresh Linux machine now installs your secure environment on native k3s by default, with no Docker, k3d or separate kubectl. Machines already installed keep what they run, on re-runs and upgrades alike. TRACEBLOC_SUBSTRATE=k3d keeps today’s k3d install. A Linux user without administrator rights who installed through a Docker they could already reach now needs administrator rights once, or TRACEBLOC_SUBSTRATE=k3d. A new Linux machine that sets TRACEBLOC_AGENTS, TRACEBLOC_SERVERS or K8S_VERSION to something native k3s does not run keeps k3d, and the installer says why.
  • The chart’s comments and value descriptions about MySQL root rotation and the edgeuser account now describe older secure environments’ credentials as the image defaults.
Python SDK
  • Training an SSDLite detector from its pretrained seed now works on a dataset with any number of classes.
  • Linking an image model to a dataset, and starting the run, now warns when the model trains at a different image size from the dataset’s images and says whether that is an upscale or a downscale.
Docs
  • The image-classification dataset template now shows the commands to ingest your training and test data.
​
Secure environment
2026-10-12
Secure environment
  • A paused experiment shows as stopped cleanly instead of as an error, once the secure environment enables it.
  • Re-running the installer now reconnects a secure environment whose credential tracebloc stopped accepting, instead of reporting it as healthy.
  • On machines the installer cannot install an NVIDIA driver for (yum, zypper), or when the driver install fails, the installer now says so and how to install it yourself, and continues CPU-only instead of asking for a reboot.
  • Reinstalling a secure environment over its existing data now keeps the local database passwords, so the reinstall opens the database instead of being refused.
  • Running prepare-host without naming a researcher now also installs the rootless-Docker helpers, so the next host check no longer lists them as a pending admin step.
  • The install command shown in the installer’s help and messages now points at the working one-line installer (tracebloc.io/i.sh, tracebloc.io/i.ps1) instead of a link that failed.
  • Pausing an experiment no longer leaves its pod marked as an error.
  • MySQL can now replace an old install-time root password with the one in the release Secret on its own, at startup, without anyone needing access to the cluster. It is off by default.
  • When tracebloc refuses this machine’s credential and your installed CLI is too old to renew it, the installer now stops and tells you to update the CLI instead of reporting a successful install while the machine stays offline.
​
PlatformCLI
2026-10-12
Platform
  • The use case page now fits tablets and phones without scrolling sideways.
  • The use case page’s icon buttons (edit pencils, invite, archive switch, more options) now have names for screen readers and work from the keyboard.
CLI
  • On macOS, a node that stops right after it is started is now restarted on the next check, about a minute later, instead of staying down for about four minutes.
​
Secure environmentCLI
2026-10-12
Secure environment
  • On Linux, native k3s installs now accept TRACEBLOC_STORAGE_MODE=hostpath and TRACEBLOC_HOST_DATASET_DIR, and keep the data in your data directory. Before, they had to fall back to k3d. A re-run or upgrade keeps the storage mode and directories the install was made with, and says so when you asked for others.
  • On macOS, inside WSL, or with a setting native k3s cannot use, TRACEBLOC_SUBSTRATE=k3s on a machine running tracebloc on k3d is now refused before anything changes, instead of removing the k3d secure environment first and then refusing native k3s.
  • Moving from k3d to native k3s no longer refuses a machine for disk space that the move itself frees.
CLI
  • tracebloc resources set max now checks the ceiling again after applying it. When applying it started more of tracebloc’s own services, it lowers the ceiling once so an experiment still fits, and tracebloc doctor no longer calls a fresh max over-committed.
​
Secure environmentCLI
2026-10-11
Secure environment
  • A node whose GPU appears after the secure environment starts no longer reports that GPU as idle.
CLI
  • tracebloc resources set max now picks a size that fits beside what is already running, so doctor no longer reports it as over-committed.
​
Secure environmentCLI
2026-10-11
Secure environment
  • GPU nodes whose GPU becomes available after the secure environment starts are now reported as GPU nodes without a restart.
  • An install from your own values file (dev mode) now sets up the GPU the way a normal install does. The GPU request, runtime class, device type and device plugin follow what the installer detected on this machine, over any GPU settings in the file.
  • The Windows installer no longer stalls or stops partway when it is run as a piped command.
CLI
  • On EKS with the recommended EFS storage, tracebloc data delete no longer refuses with “can’t remove the files” when the files can be removed.
​
Secure environment
2026-10-10
Secure environment
  • On a GPU machine the installer now downloads the shared GPU training base image in the background once setup finishes, so your first GPU training no longer starts with a 4 GB download.
  • On native k3s, prepare-host now records the tracebloc group in the user’s install record, and whether it created the group, so tracebloc delete takes the user out of it and removes the group only when prepare-host made it and no members are left.
  • A local dashboard node port outside 30000–32767 now turns the local dashboard link off with a warning instead of failing the install.
  • Re-running the installer after your local cluster was rebuilt no longer asks you to name a new secure environment; it tells you which one it is reconnecting.
  • A fresh local install now tells you which kubectl context was active before it switched you to the new cluster, so you can switch back.
  • Installing native k3s no longer stops with “cannot tell whether a k3d cluster is live” when Docker briefly fails to answer after k3d has already listed this machine’s clusters.
  • On Linux, a machine that runs tracebloc on k3d can now move to native k3s when you run the installer with TRACEBLOC_SUBSTRATE=k3s. With your consent, the installer first deletes the old secure environment, then sets up k3s. Without consent, nothing changes. A consented move registers the machine as a new secure environment, so its environment name gets a suffix (for example my-lab becomes my-lab-2).
  • On a machine with less memory than the smallest training size, the installer no longer tells you to train on a larger machine; it says training runs there with less memory per run, so a larger model can run out of memory.
  • Switching an install to native k3s no longer refuses over a k3d cluster that does not exist when Docker stops answering partway through listing its containers.
  • A consented reinstall now stops with a clear message instead of hanging when the old environment’s delete stops responding.
  • Re-running the installer on a native k3s machine after a k3s version bump now restarts k3s onto the new version, instead of leaving the old one running until a reboot.
  • When the installer cannot read Docker during a reinstall check, it now shows Docker’s own error and how to fix it, instead of saying Docker did not answer.
  • On macOS, an install that stops partway now reports a failure instead of finishing with success.
  • On macOS, an installer run that stops on an internal error now reports a failure instead of success.
  • Re-running the installer with TRACEBLOC_SUBSTRATE set on a machine that is already set up and healthy now reaches the runtime choice, instead of stopping at “Already set up and healthy”.
  • A machine an administrator prepares with prepare-host now records which user it was prepared for, so that user can run their own tracebloc delete without administrator rights. A machine prepared before this release needs prepare-host once more.
  • Object-detection training no longer fails when the validation set ends in a single image, and a one-image BatchNorm failure now says what caused it.
  • Training logs no longer mention LoRA for experiments that did not enable it.
  • Image-model training logs no longer show an albumentations “Error fetching version info” warning, and pods start up to 2 s faster.
​
CLI
2026-10-10
CLI
  • On a machine an administrator prepared for you, tracebloc delete now disconnects the machine and removes what is yours, then names the one command an administrator runs to remove k3s, instead of stopping at sudo -v.
​
PlatformCLI
2026-10-09
Platform
  • Experiments that turn on LoRA without setting rank and alpha now train with rank 8 and alpha 16, the same defaults the SDK documents.
  • In My Use Cases, the create wizard waits for you to fix a rejected field before you can submit again.
CLI
  • The CLI is built with the latest Go security fixes for HTTPS and TLS connections.
  • When data delete can’t remove a dataset’s folders, it no longer blames an old ingestor for folders your storage class created, and points you at the storage class instead.
  • Declining the name prompt while deleting for a reinstall now stops the reinstall instead of reporting success.
​
Platform
2026-10-09
Platform
  • The environment switcher opens smoothly in place and scrolls smoothly through your environments, and long tables show 15 rows before they scroll.
  • Your environments now read machine, operating system and last seen in the switcher, and header notices are spaced as designed.
  • An offline environment shows one clear notice instead of also warning about its GPU.
  • My Use Cases now shows “Created by You” on your own use cases, celebrates your first use case, opens a welcome page for your first invitation, and asks you to log in again when your session expires.
​
PlatformSecure environmentCLI
2026-10-09
Platform
  • The use-case details on the Secure Environment page show whether a use case is public or private and when it ends.
  • Machine memory and the per-training limit now show in GiB and the CPU an experiment actually gets.
  • The environment switcher menu now animates open and closed.
  • Secure environment memory now reads in GiB and CPU shows what one experiment gets, on the secure environments list, the dataset card and the Run tab, so a secure environment shows the same figures on every page.
Secure environment
  • Reinstalling a secure environment on the same cluster no longer fails with “cluster in use” when the previous secure environment has stopped heartbeating.
CLI
  • tracebloc doctor reads this machine’s credentials for your secure environment, warns when they expire within 120 days, and names the restart and installer re-run that renews them.
  • an administrator removes the native k3s prepared for a user with sudo tracebloc delete --prepared-user <user>, which reads only that user’s install record.
  • tracebloc delete removes the group prepare-host made for a user’s k3s once nobody is left in it, and always takes the user out of it; a group that was there before stays.
  • The image resolution question in tracebloc data ingest now accepts 416×416, 416X416 and 416 x 416, and a value it still cannot read tells you to use the letter x, e.g. 512x512.
  • Re-running the installer on a machine whose tracebloc install was removed now reconnects your existing secure environment instead of asking for credentials or creating a new one.
  • Switching a guided ingest from test to train (or back) no longer offers the previous dataset’s name and path as defaults.
  • After an ingest, What’s next now sends you to My Data on your Secure Environment page and says which other dataset a use case needs; support-only IDs and setup lines show with —verbose or when a run fails.
  • Reinstalling tracebloc after your local cluster was rebuilt now reconnects your existing secure environment, with its datasets and history, instead of creating a second one.
  • tracebloc client status now shows whether your secure environment is online right now, and when it was last seen if it isn’t, instead of an out-of-date “online”.
  • tracebloc upgrade no longer replaces a newer pre-release of the CLI with an older release.
  • Re-running the installer no longer replaces a newer pre-release of the tracebloc CLI with an older release.
  • The home screen now shows the same CPU and memory as tracebloc resources, and tracebloc auth status names your secure environment.
  • tracebloc doctor now reports your secure environment as offline when tracebloc hasn’t heard from it, says when it last did, and how to reconnect it.
  • Running tracebloc client create --yes at a terminal on a cluster whose install is gone now issues a new credential instead of finishing with nothing to connect.
  • When your CLI is too old to reconnect an existing secure environment, tracebloc client create now tells you to upgrade instead of telling you to re-run.
  • After an administrator removes the k3s they prepared for you, your own tracebloc delete now finishes and disconnects the machine instead of stopping with “this machine has no root copy”.
  • On macOS, the tracebloc CLI is signed by Tracebloc GmbH and notarised by Apple.
​
PlatformSecure environmentCLI
2026-10-08
Platform
  • A secure environment created but never installed now shows “Not installed yet” instead of “Offline” on the dashboard, with the command to install it.
  • Leaderboards ranked by an error metric now show each team’s best result instead of its worst.
  • A submission that is still running no longer shows a loss of 1.0 and accuracy of 0.0; it shows no values until its results arrive.
  • My Data now shows the use cases a test dataset is used by, instead of “0 use cases”.
  • The My Secure Environment page gets a dedicated endpoint that returns connection state, resource meters and GPU notices already worked out, so every page shows the same status for the same secure environment.
  • An account with one or two use cases now sees them as larger cards with an “Add More Use Cases” card beside them.
  • An offline secure environment’s memory, CPU, GPU and storage cards now show their capacity with a dash for usage, instead of “no usage yet”.
  • Deleted environments are now marked in the environment switcher, and a use case’s peers show as avatars in its details panel.
  • The secure environment page’s use case details now show who works on each use case, and its notices and lists read more cleanly.
  • Use case cards now show who owns or invited you to each use case, use smaller topic tags, and show the end date when you hover over them.
  • Finishing account activation, a password reset or a profile update no longer pulls you off a page you navigated to during the short redirect delay.
  • Copy ticks and confirmation messages no longer get cut short when you click twice, and leaving the password page right after saving no longer redirects you a second later.
Secure environment
  • An administrator can run prepare-host --print to see every command prepare-host would run as root, in order, before any of it runs.
  • A k3s install on a machine that already runs tracebloc on k3d is refused before anything changes, with what moving to native k3s would mean and how to keep today’s environment.
  • A small machine on native k3s (7 GB) is Ready right after a clean install; experiments ask for the memory the machine actually has free.
  • A successful install no longer tells you to open a new terminal when your current terminal already runs tracebloc, and its log no longer records a false error after the success screen.
  • On k3d, an experiment’s memory now leaves room for the cluster’s own system pods, so a machine that fits just barely is Ready right after install instead of turning Not ready once those pods start.
  • A data ingest no longer prints an internal telemetry configuration line.
  • Re-running the installer on a machine that already has your secure environment no longer asks you to name a new one; it reconnects the existing one and tells you so.
  • On an AMD GPU, or an arm64 machine with an NVIDIA GPU, the installer now sets up a CPU-only secure environment and tells you why, instead of configuring a GPU that can’t train.
CLI
  • A secure environment whose cluster has stopped now shows as offline in client status and client list within about 35 minutes of its last heartbeat, on every environment.
​
PlatformDocs
2026-10-08
Platform
  • Tabular regression experiments now chart training and validation R² for every epoch, negative values included, instead of showing ”—” with no curve.
  • The Secure Environment Storage card now shows how much of your available space your datasets use, like the other cards.
  • After switching accounts in the same tab, the Secure Environment page no longer briefly marks the previous account’s machine as “this machine”.
  • A dataset’s details now list the use cases it is used in, by name, and each one opens that use case; test datasets no longer show “0 use cases”.
  • Empty sections on Secure environment no longer show a “Watch how to” button before its video is available.
  • Dates in the Use Cases and peers (“Collaborators” on screen) tables no longer wrap onto two lines on smaller laptop screens.
  • You can now rename a use case from its details panel on Secure environment.
  • The use case details panel now shows who created the use case.
  • Fractions, subscripts and superscripts in notebook math render at the right size again.
  • Object detection experiments and submissions now show mAP instead of an accuracy figure that does not apply to detection, and segmentation labels its accuracy as pixel accuracy.
  • The Secure Environment page no longer shows memory and CPU usage for an environment that is offline.
  • Clicking outside a details panel on Secure environment now closes it.
  • Deleted datasets can no longer be edited from their details panel.
  • Sorted table columns on Secure environment now show they are sorted, and banners have corrected spacing.
  • The empty peers section now explains where to invite people from, and shows the privacy note.
  • Use case names in a dataset’s details now open the use case’s page.
  • The buttons at the bottom of Secure environment’s details panels now use the standard button style.
  • The environment switcher now shows each machine’s full details when opened.
Docs
  • The CLI guide now explains the image resolution question in guided data ingest and shows how to find an image’s width and height on macOS, Linux and Windows.
  • The setup guide now lists exactly which GPUs, drivers and platforms train on a GPU, and which don’t.
  • The CLI page explains every guided-ingest question, what you can type at each, and where the suggested answers come from.
  • The Prepare dataset page now points to My Data on your Secure Environment page to see ingested datasets.
​
Secure environment
2026-10-07
Secure environment
  • Secure environments on Helm chart 1.9.200 now train with the engine changes announced under 0.68.0–0.71.0. For example, training on a GPU the platform cannot use now fails at once instead of running on CPU, keypoint PCK and regression R² report real values, and metrics that can’t be measured show ”—”.
  • A native k3s install on a machine without the timeout command no longer hangs indefinitely when a package mirror stops responding; it gives up after 10 minutes.
  • The installer now tells you when your GPU isn’t NVIDIA or Docker can’t see it, instead of reporting no GPU.
  • The installer now reserves port 47910 on 127.0.0.1, reachable only from this machine, for a coming dashboard feature. Set TRACEBLOC_LOCAL_API=off to skip it or TRACEBLOC_LOCAL_API_HOST_PORT to move it; if the port is taken, the install finishes without it and says why.
  • Re-running the installer on an existing secure environment that runs on k3d (Mac, Windows) adds that port and restarts the environment once; training running at that moment restarts.
  • Updating a machine set up before the storage change no longer stops with a storage-topology error; it keeps the existing data folder and finishes.
  • Running the installer with —diagnose on a machine that is already set up now produces the support bundle instead of opening the home screen.
  • When the auto-upgrade job cannot read a stuck release’s history, its log now says the read failed and shows helm’s error, instead of claiming the history has no healthy revision.
​
PlatformPython SDKCLIDocs
2026-10-06
Platform
  • Regression experiments now record and show their R² for every epoch.
  • My Use Cases and Explore are back to the previous layout; the new use-case dashboard remains available at its own page.
  • The Secure Environment machine selector now fits its details, with long values shortened with ”…” and shown in full on hover.
  • A secure environment you created but never installed now shows “Not installed yet” with the command to install it, instead of looking like an offline machine.
  • Secure Environment table headers now sort in one click (largest and newest first where it makes sense) and a second click restores the original order.
  • The secure environments table sorts names A to Z and numbers largest first in one click, and a second click restores the original order.
  • Secure Environment no longer says “None of these people saw your data” when nobody is working on your data yet.
  • The environment switcher no longer labels a machine that was never set up as “Offline”.
  • Secure Environment’s Training Limit card now shows tidy figures, with the exact limit on hover.
  • In Create Use Case, the compute budget note now appears at the top of the Data & Evaluation step.
  • The Secure Environment dataset panel now shows who created each dataset.
Python SDK
  • Uploading a YOLO model without its loss.py now tells you to upload model.py and loss.py together as one zip, and names the model zoo’s ready-made YOLO zips.
  • The training plan no longer lists dataset classes for regression and forecasting experiments.
  • The quickstart notebook installs the same minimum tracebloc SDK version the docs name.
  • The Cascade R-CNN, Sparse R-CNN, YOLOv10-S and EfficientDet-D0 detection templates can now be uploaded.
  • The model zoo’s YOLO v1, v5 and v8 templates can now be uploaded straight from a clone: upload object_detection/pytorch/yolo_v8.zip (or yolo_v1.zip, yolo_v5.zip).
CLI
  • Creating a secure environment when no cluster is reachable now points you to the installer instead of making a secure environment that can never come online, and a printed secure environment credential now comes with the command that connects it.
  • When tracebloc data ingest asks its questions and the folder is laid out for a different task, the hint now says to pick that task in the menu instead of telling you to pass a --task flag.
  • tracebloc cluster info no longer reports a missing ingestor image setting on working installs, and shows a shorter summary; --verbose still shows the full detail.
  • Checking a large labels.csv before an ingest no longer uses memory in proportion to the file’s size.
Docs
  • The Define page and dataset templates now explain MBE ranking, forecasting scaling and the CLI’s --schema and --label-column behaviour correctly.
  • The CLI reference now lists every task that needs a label column, and the SDK page opens the maintained Colab notebook.
  • The segmentation and keypoint dataset docs now show working CLI commands and segmentation mask rules that match training.
  • The operations page now points to tracebloc doctor --diagnose for support bundles, and the move guide tells you to re-ingest datasets on the new machine.
  • The training docs now show a custom loss that works in a notebook, mark the TensorFlow-only augmentation forms, and say what the seed controls.
  • The security page’s firewall list now matches what a Mac install downloads, so an allowlist built from it no longer blocks a fresh macOS install.
  • The model docs now explain how to upload the model zoo’s YOLO v1, v5 and v8 templates (as their .zip) and your own YOLO model (model.py and loss.py in one flat .zip).
  • The training guide now shows how to upload the model zoo’s YOLO models: pass the .zip next to the YOLO folder, for example object_detection/pytorch/yolo_v8.zip.
​
Platform
2026-10-06
Platform
  • New embeddings use cases offer score formulas the leaderboard can compute, instead of one that scored every submission “Score not computed”.
  • A stopped secure environment now shows as offline within a few minutes instead of half an hour.
  • Fetching a zoo template’s pretrained seed by its full template id now returns that task’s own seed. For example, the sentence-pair BERT template no longer gets the text-classification seed.
  • Training cycle and inference results now show up about a minute sooner.
  • Changing a secure environment’s password now signs that secure environment out of every existing session, so it must log in again with the new password.
  • The secure environment page can now show that a secure environment’s GPU is not supported and that it runs on CPU instead.
  • A model upload that reaches system commands, environment variables or native code through an allowed library such as torch is now refused at upload, and the refusal names the line.
  • Model uploads with many Python files are no longer rejected with a “corrupted scan” error.
​
PlatformSecure environmentPython SDKCLIDocs
2026-10-06
Platform
  • Keypoint detection PCK scores now reflect real accuracy instead of showing a perfect 1.0 for models that have not learned.
  • Regression experiments no longer show R² as “accuracy”; R² is reported as r2 and accuracy is left empty.
  • Opening Explore or Templates now takes you to the Templates tab of My Use Cases.
  • The leaderboard’s Compute column now shows your team’s real compute use instead of “0.0”, and says which compute numbers are per run and which are team totals.
  • Tooltips, badges, tabs and other labels across the app now use the app’s normal font instead of a serif font.
  • Buttons, tables, avatars, calendars and form fields from the older component set no longer fall back to a serif font.
  • Inference results for language-model and regression experiments now show their own metrics instead of empty F1, precision and recall.
  • Notebook outputs show their table styling again.
  • Secure Environment status cards now show each total in the caption (“12 of 32 GiB used”), and machines without a GPU show four even cards instead of an empty GPU card.
  • Secure Environment now tells you when a machine has no GPU, and that training still runs on CPU, just slower.
  • Secure Environment now says when a machine’s GPU is too old for training and links to the GPU requirements.
  • In Secure Environment, a use case’s dataset on another machine now opens, with a button to switch to that machine.
  • Secure Environment now says when a machine’s NVIDIA driver is too old for training, and how to fix it.
  • In Secure Environment, a use case’s datasets now show which belong to another machine and which were deleted.
  • The secure environments table now shows a machine as offline as soon as it stops, matching the Secure Environment page.
Secure environment
  • The dashboard can now tell you exactly why a machine is running on CPU: an old GPU, an old driver, a GPU that isn’t NVIDIA, or Docker that can’t see the GPU.
  • On Linux with TRACEBLOC_SUBSTRATE=k3s, the installer now sets up your NVIDIA GPU for training through the NVIDIA Container Toolkit, instead of installing in CPU mode.
  • The installer pins k3s v1.36.5+k3s1 (was v1.36.3+k3s1).
  • A machine whose NVIDIA GPU is too old for GPU training (older than Turing), or whose NVIDIA driver is older than the CUDA 12 minimum, now installs in CPU mode and the installer says why, instead of setting the GPU up and failing later.
  • When a machine’s GPU is too old for GPU training, or its NVIDIA driver is too old, the secure environment now tells the platform why it runs on CPU, so the reason can be shown on the secure environment page.
  • Dev secure environments now give each ingestion run its own short-lived platform token; staging and production are unchanged.
  • The chart’s multi-GPU size floor can also be set as env.TRACEBLOC_MULTI_GPU_MIN_PARAMETERS; env.MULTI_GPU_MIN_PARAMETERS keeps working, and either one now reaches the secure environment under both names.
  • When an automatic upgrade times out or its rollback fails, the auto-upgrade job’s log now says which, what was not ready, and a failed rollback exits with code 3.
  • Automatic upgrades that were interrupted now recover to the last working version and wait for it to be healthy
Python SDK
  • Fetching the starting weights for a template such as sentence-pair BERT now downloads the weights built for that template, not another template’s weights that share the model name.
  • When a model needs an ML framework that isn’t installed, the model check now tells you which extra to install, for example pip install 'tracebloc[pytorch]'.
CLI
  • tracebloc delete removes a native k3s install from its install record (k3s, its files and its host firewall rule), asking sudo once before anything is removed, and tracebloc doctor reads k3s’s own service instead of Docker on such an install.
  • When one image in a dataset has a different size, tracebloc data ingest now uses the size most images share and names the odd image, instead of blaming every other image.
  • When a dataset is too big for one ingest, the CLI now suggests making it smaller (JPEG, lower resolution, fewer samples) or using the Helm chart, instead of splitting it into datasets that can’t be trained on together.
  • When tracebloc data ingest is run with the wrong --task, it now names the task the folder is laid out for (for example “did you mean —task object_detection?”) instead of only reporting a missing labels.csv.
  • When tracebloc data ingest can’t start because an experiment is using the secure environment’s CPU or memory, it now names the experiment that holds them and says to retry when it finishes.
  • The staged copy of an ingested dataset can now be removed by the ingest job once the ingest finishes, including after --detach, so it no longer uses disk space until the table is re-ingested or deleted.
  • tracebloc upgrade --plain now keeps the installer’s output plain instead of animating a spinner in your terminal.
  • tracebloc delete now removes the cluster your install set up, even when your kubeconfig’s current context points at a different one.
  • Ingesting a keypoint-detection dataset now keeps the per-keypoint Visibility column, so visibility accuracy is no longer scored as if every keypoint were visible.
  • After tracebloc delete, kubectl no longer switches to another cluster in your kubeconfig. It is left with no current context, and the output says so.
  • tracebloc upgrade output is clean in logs and plain terminals (no spinner frames), and it ends by saying which version you are on instead of pointing at a setup step that doesn’t apply.
Docs
  • The setup guide now lists the NVIDIA driver minimums and explains that GPUs older than Turing (for example GTX 10xx, GT 710, V100, P100) install CPU-only.
  • The object detection template now shows the CLI ingest command and explains the class-name and train/test rules that can make an ingest fail or hide a test set.
  • The FAQs now say plainly that model weight updates go to tracebloc for averaging, and the SDK page shows how to fetch pretrained seed weights for model zoo templates.
  • The text dataset template pages now show the exact CLI command for each task, say that files must end in .txt, and list the score formulas you can choose.
  • The training guide now explains object detection model settings and pretrained seeds, and gives the correct daily submission limit.
  • The security page now lists every kind of data that leaves your secure environment, how to opt out of logs and usage events, and the exact hosts the installer needs.
  • The Quick Start and setup guide now explain macOS admin rights, Docker Desktop’s first-run prompts, the installer’s memory offer, and unattended-install credentials.
  • The Operations guide now explains how to move a secure environment to a new machine without revoking it, and warns you to check your kubectl context after deleting the local cluster.
  • The Configuration page now matches the chart defaults: hourly auto-upgrade, egress locked down by default, optional registry credentials, and opt-in PriorityClass.
  • The Amazon EKS guide now matches the current chart defaults, names the right pods and services, and no longer points to a missing setup script.
  • The object detection XML example on the prerequisites page now passes ingestion, and the dataset page link and leaderboard column list match the app.
  • Troubleshooting now shows how to find your namespace, what to do when the secure environment is Offline after a reboot, and the correct pull-secret check.
  • The hyperparameter guide now shows training plan calls that the SDK accepts, with per-task loss defaults and the federated aggregation options.
  • The Define Use Case guide now matches the two-step Create Use Case panel and lists only the scores you can actually pick for each task.
  • The bare-metal, AKS and OpenShift guides now show how to configure CPU-only or GPU nodes, multi-node storage and metrics-server, so training pods schedule on the first try.
  • The tabular and time-series dataset template pages now show the CLI command for each task and the model settings that must match your data, and they say which survival data trains successfully.
  • The CLI reference now explains what to do when a dataset is over the size cap, shows an object-detection ingest, and lists the missing flags and exit codes.
  • The training guide now explains how object detection models are trained and scored, and where image size, batch size and the validation split come from.
  • The segmentation and keypoint templates now explain how class count and coordinate units must line up with training, and show the keypoint CLI command.
  • The security pages now say plainly that model weight updates go to tracebloc after each training cycle.
  • The EKS guide now names the Helm version the upgrade command needs.
​
PlatformSecure environment
2026-10-06
Platform
  • Time series classification and segmentation metrics now agree: F-beta and Jaccard use the same averaging as F1, and Dice matches F1 on binary masks.
  • A TensorFlow experiment that selects FedProx now fails right away with a clear reason, because FedProx’s proximal term is only applied when training PyTorch models, so the experiment would otherwise have run as plain FedAvg.
  • A stored model .zip now loads only its top-level .py source; an archive carrying compiled, native or .pth files, symlinks, directories or unsafe paths is refused on its first delivery with a reason naming the file.
Secure environment
  • Image-classification top-3/top-5 accuracy now shows ”—” (not measured) when the task has no more classes than k, instead of a meaningless 1.0.
​
PlatformSecure environmentDocs
2026-10-06
Platform
  • Time-to-event experiments no longer show a c-index of 0.0 when it can’t be computed; it is reported as not measured, and accuracy is no longer reported for time-to-event.
  • The Secure Environment page shows a machine as offline as soon as it stops checking in, instead of after a delay.
  • The secure environment page shows a stopped secure environment as offline within about 3 minutes instead of 30, and a GPU the machine advertises is shown even when its usage isn’t reported.
  • The Secure Environment page shows each machine’s location and specs instead of repeated or unhelpful names.
  • Counts of one on the Secure Environment page now read correctly, such as “1 use case”.
  • The secure environment page now tells you when your GPU is too old to use and that the secure environment is running on CPU, with a link to the list of supported GPUs.
  • Language-model experiments now show perplexity and token accuracy instead of empty precision, recall and F1.
  • Keypoint detection experiments now show their MAE score in the experiment list.
  • The Secure Environment page now shows the machine’s GPU or specs under its name when no host name is known.
  • Notebook outputs no longer apply style blocks embedded in the notebook’s HTML; tables and their contents display as before.
  • On the Secure Environment page, a use case’s datasets now open their own details when clicked.
  • On the Secure Environment page, a dataset now shows how many use cases it is used in.
  • Keyboard users now see a clear focus outline on blog and site controls, and screen readers announce when code or a link has been copied.
  • Links, job details and blog metadata are easier to read, with stronger text contrast across the site.
Secure environment
  • A successful ingest no longer prints an internal telemetry configuration warning, and the note about the temporary staged copy now says how it gets cleared after a --detach run.
  • Ingestion Jobs now also accept TRACEBLOC_MYSQL_PORT, TRACEBLOC_MYSQL_WAIT_SECONDS and TRACEBLOC_BACKEND_TOKEN_SCHEME; the old names keep working.
  • Ingestion jobs pointed at a MySQL server on a non-default port no longer time out waiting for the database before they start.
Docs
  • Code examples and template cards on the website are easier to read, with higher-contrast text in both light and dark themes.
​
PlatformSecure environmentPython SDKDocs
2026-10-06
Platform
  • An experiment whose aggregation hyperparameters include a key the chosen strategy doesn’t accept now fails at once, with a message naming the key and the keys the strategy does accept. Before, it silently ran with the defaults.
  • A scikit-learn or survival experiment that names an aggregation strategy other than FedAvg now fails right away with a clear reason, because those models are merged by their own rule and the chosen strategy would never run.
  • Rejecting cookies is now remembered, so the cookie banner no longer reappears on every page.
  • Sharing a job post on LinkedIn now shares the full page link, and links between blog posts open in the same tab.
  • The dropdowns in the healthcare page’s meeting form can now be opened, browsed and chosen with the keyboard, and screen readers announce them as lists of options.
  • Moving between pages from the site menu is faster, the current page stays highlighted, and the menus now work fully with a keyboard and screen readers.
  • Background animations now respect your reduced-motion setting, and the benchmark setup steps can be paused.
  • Share and LinkedIn buttons on blog posts and job pages now highlight when reached with the keyboard, not only on mouse hover.
Secure environment
  • Training on a GPU the platform cannot use now fails right away with a message naming the GPU, its compute capability, the driver and the supported generations, instead of silently training on CPU.
  • Object-detection experiments using torchvision detectors now show training accuracy as ”—” (not measured) instead of a misleading 0.0.
  • Pausing an experiment now stops its pod right away instead of after a 30-second delay.
  • A Windows (Docker Desktop / WSL2) machine with an NVIDIA GPU now reports its GPU to tracebloc instead of showing “no GPU”.
  • When a secure environment’s GPU is too old to train on, the platform can now show which card it is and why it was left off.
  • The secure environment also accepts TRACEBLOC_MULTI_GPU_MIN_PARAMETERS for the multi-GPU size floor; the existing name and the chart’s values key keep working.
Python SDK
  • The deprecation warning for the old tracebloc_package import no longer says it was removed in 1.0.0, so it now matches what you have installed.
  • Pretrained seeds for zoo templates no longer carry the class head once republished, so a model built for your dataset’s own class count passes model checks with the fetched seed.
Docs
  • Keyboard and screen-reader users can now skip straight to page content, hear the German legal text in German, and no longer hear collapsed blog FAQ answers.
  • The installation docs now use TRACEBLOC_-prefixed names for the installer options (TRACEBLOC_CLUSTER_NAME, TRACEBLOC_STORAGE_MODE, …); the older names keep working.
  • The custom ingestion Job example now authenticates with your secure environment’s platform token and database account, so a Job copied from the docs starts instead of being refused.
​
Secure environment
2026-10-03
Secure environment
  • Linux machines can now install your secure environment on native k3s instead of k3d by running the installer with TRACEBLOC_SUBSTRATE=k3s. It is opt-in and Linux only: k3d stays the default, and macOS and Windows keep k3d. A machine installed on k3s keeps k3s on later runs and upgrades without the variable. Native k3s runs as root, so the installer asks for administrator rights once.
  • Reinstalling on native k3s no longer wipes leftover data when the k3s service’s state can’t be read; it asks you to stop k3s first.
  • Installer options now have TRACEBLOC_-prefixed names, for example TRACEBLOC_STORAGE_MODE, TRACEBLOC_CLUSTER_NAME and TRACEBLOC_PF_MIN_MEM_GB; the old names, such as TB_STORAGE_MODE and CLUSTER_NAME, keep working.
  • The native k3s installer now quotes what k3d reported when its cluster check fails, instead of always asking you to start Docker.
  • On native k3s, the kubelet now holds back memory and CPU for k3s itself and for the host, so an experiment can no longer take the memory k3s needs to keep the node running.
  • The installer retries a dropped connection while it downloads its signature files, instead of stopping, and says whether a failure was the network or a release that isn’t signed.
​
PlatformSecure environmentCLIDocs
2026-10-03
Platform
  • Tabular, time-series and survival datasets now show a size in My Data: the size of the file they were ingested from.
Secure environment
  • Training logs now show the last batch of every training, validation and evaluation loop, so a short epoch no longer looks stuck after its first batch.
CLI
  • Links printed after a data ingest now open the web app of the environment you are signed in to.
Docs
  • The training guide now shows how to fetch a model zoo template’s pretrained weights before uploading it.
​
PlatformCLIDocs
2026-10-03
Platform
  • The weights download button on an experiment is now disabled, with an explanation, when the use case doesn’t allow weight downloads, instead of failing after you click it.
  • When a use case doesn’t allow weight downloads, the experiment’s download button now says so and turns off, instead of showing “Model is downloading” and then doing nothing.
CLI
  • After you sign out and back in, tracebloc doctor now warns that this machine isn’t linked to its secure environment and tells you to run tracebloc client create. tracebloc delete and tracebloc client status give the same fix instead of saying there is nothing to delete.
Docs
  • The Start Training guide now shows how to fetch a model zoo template’s pretrained weights, so pretrained templates no longer silently train from scratch.
​
PlatformSecure environmentPython SDKCLIDocs
2026-10-02
Platform
  • Carbon-intensity figures for your secure environments keep updating even when the grid data provider sends back an unreadable answer for one region.
Secure environment
  • The secure environment’s services and training pods also accept TRACEBLOC_-prefixed names for their tuning and proxy environment variables (for example TRACEBLOC_HTTP_PROXY_HOST); the existing names keep working, and the chart’s proxy settings in values.yaml are unchanged.
  • Ingestion Jobs now accept TRACEBLOC_-prefixed names for their settings (for example TRACEBLOC_TITLE, TRACEBLOC_BATCH_SIZE, TRACEBLOC_LOG_LEVEL); the old names keep working.
Python SDK
  • The memory warning shown before training starts no longer suggests LoRA or a shorter sequence length for image, tabular and time-series models.
  • Uploading a model zoo template that has pretrained weights, without those weights, now warns that the model will train from random weights and shows how to fetch the weights.
CLI
  • tracebloc --version now prints the CLI version — the same line as tracebloc version.
  • The fix-permissions command printed by a refused data delete now runs to the end when the blocked folders belong to more than one owner.
Docs
  • The dataset ingestion guide now shows the TRACEBLOC_-prefixed setting names; the older names still work.
​
PlatformSecure environmentPython SDKCLIDocs
2026-10-02
Platform
  • Fine-tuning GPT-2-family models with LoRA now adapts the attention and feed-forward layers, so causal language models actually learn instead of staying near their starting perplexity.
  • Experiments on large text models no longer run out of memory and restart while saving their weights at the end of each training round.
  • The secure environment page’s Storage card now shows how much data this secure environment holds and how much space is left for new data, instead of the whole machine’s disk.
  • The Run tab warns before you start training when the plan may need more memory than the secure environment gives each experiment, and says what to change.
  • You can create use cases for object detection, image, text and tabular classification, segmentation, regression and forecasting again.
  • When you create a use case, the training dataset list shows only datasets that fit the task you picked.
  • When you create a use case, you see its compute budget per peer and where to set an end date before you publish.
  • The “Explore all use cases” button on the home and benchmark pages is back to its compact size on phones instead of stretching across the screen.
  • The pricing plan buttons and the “Try a template” and “Read Docs” buttons on the platform overview are back to their normal size instead of stretching across their card.
Secure environment
  • When an experiment fails for good, the secure environment no longer also logs it as restarting.
Python SDK
  • When you start training, the SDK now warns if the experiment may need more memory than the secure environment gives each experiment, and says what to change.
CLI
  • tracebloc data delete no longer half-deletes a dataset whose files it can’t remove: it now checks first, leaves everything in place, and prints the one command that fixes the folders.
  • tracebloc data ingest now checks that your secure environment’s data ingestor can handle the dataset before copying anything, and tells you plainly when it needs an upgrade.
  • tracebloc upgrade --help now says plainly that it updates the CLI, and that a healthy secure environment’s services keep upgrading on their own every hour.
  • Warnings and errors from a quiet data ingest are no longer hidden when they arrive in the middle of a progress bar.
  • With —verbose, tracebloc data ingest no longer says you are only seeing warnings and errors while it shows the full log.
  • When the data ingestor is too old for a task, the error now says how it gets updated instead of pointing at a CLI upgrade that doesn’t change it.
Docs
  • The Hyperparameters and Start Training pages now explain that a copied or renamed text model file needs its matching _tokenizer.json copied and renamed too.
​
PlatformSecure environmentCLIDocs
2026-10-02
Platform
  • When you publish a use case you are now its first peer, so you can start training right away without joining it yourself. Your team takes one of the use case’s team slots, and its compute budget comes out of your account’s, as when you join. When no budget or slot is left, the use case still publishes and you can join it yourself later.
  • An experiment’s hardware details no longer go blank when its node briefly fails to report its CPU or memory.
  • Use cases no longer offer scoring formulas that a task’s experiments never report, and formulas built on PCK thresholds or F-beta 0.5 now produce a leaderboard score instead of “Score not computed”.
  • Small usage figures on the Secure Environment page now show one decimal, such as “0.2 Cores used”.
  • The “GPU memory is full” warning on the Secure Environment page now shows how much GPU memory is in use.
  • The Secure Environment page shows “Training Now” on a use case while an experiment is in progress.
  • The “Environment offline” banner title on the Secure Environment page now lines up with its button.
  • The Memory card on the Secure Environment page shows the same total in its title and its usage line.
  • On phones, the “View full setup guide” button on the benchmark page is back to its normal size instead of stretching across the screen.
Secure environment
  • Training logs now show a progress line about once a minute (“Batch n of M”), not only the first batch.
  • An experiment that fails on your machine now shows as failed instead of staying Running. It is reported once its retries are used up (up to 3 attempts), and the compute every attempt used counts toward your budget.
  • A machine whose hardware details can’t be read no longer shows up with 0 CPU cores and 0 GB of memory.
  • Stopping an experiment in your secure environment no longer marks the experiment as failed.
  • Stopping an experiment no longer shows it as failed, even when your machine is busy.
  • When an experiment is killed for running out of memory, the failure reason now says so even if only an earlier attempt was recorded as OOMKilled, and secure environment logs show every in-place restart.
  • After installing on a GPU machine, the installer now reports how many GPUs the node offers, and warns with a fix when it offers none instead of claiming the GPU works.
  • On Linux, the installer no longer warns about NVIDIA drivers 550 to 569; below 550 its warning now also suggests turning on persistence mode if you keep the driver.
  • Installing on a Mac no longer fails now and then while writing the cluster node’s kubelet settings, and never leaves a half-written settings file behind.
CLI
  • tracebloc data ingest now finds text files that aren’t plain UTF-8, or that mix .txt and .text, on your machine before copying anything.
  • tracebloc data ingest now removes the temporary copy of a run that failed for good, so failed ingests no longer leave data taking up space in your environment.
  • tracebloc delete now removes the PATH line the installer added to your shell profile, and tells you that the k3s/k3d images Docker keeps are left in place and how to remove them.
  • tracebloc data ingest shows one summary per run; the ingestor’s full log appears only with —verbose, while its warnings and errors still show.
  • tracebloc data list now shows the real number of classes for object detection and token classification datasets, matching the web app.
Docs
  • The setup and Operations pages now show your own namespace and say “secure environment” throughout. They also explain that tracebloc upgrade updates the CLI, while your secure environment’s services upgrade themselves every hour.
​
PlatformCLI
2026-10-02
Platform
  • The usage cards on the Secure Environment page now fill the page width on large screens.
CLI
  • tracebloc delete --for-reinstall deletes tracebloc from a machine for a reinstall: it removes nothing unless tracebloc revoked the machine’s credential, and it keeps the CLI, your sign-in and the install logs. --confirm-name confirms a delete without a terminal.
​
PlatformSecure environmentCLI
2026-10-02
Platform
  • My Use Cases has a new design: tabs for owned and invited use cases, filters, and a guided Create flow. The filter by creator and the oldest/newest sort are gone, and a use case’s cover image is now set afterwards, in its Admin settings.
  • The Memory meter on a secure environment now shows its fullest node, so a node close to running out of memory is no longer hidden behind idle ones.
Secure environment
  • The GPU test command printed at the end of installation now runs on current kubectl versions.
  • Re-running the installer on a GPU machine that already has the NVIDIA repository key no longer fails at the container toolkit step.
CLI
  • tracebloc delete removes exactly what the installer recorded on this machine, including a cluster or release under a non-default name, lists the tools it leaves in place, and stops before changing anything when the record can’t be read.
  • Guided prompts accept the option’s name (for example train or a task name) as well as its number.
  • tracebloc client status --context/--kubeconfig/--namespace without --seal now explains what client status and --seal do, and names tracebloc doctor --context … for checking a secure environment on a chosen cluster.
  • tracebloc resources now shows the most CPU and memory one experiment can get, instead of 1 CPU · 2 GiB.
  • tracebloc data ingest links the CLI guide and your task’s folder-layout page instead of the Helm data-preparation page.
  • Commands that take --context (data list/ingest/delete, resources, client status --seal, cluster info) now find your secure environment on that cluster when it runs in a namespace other than the context’s default, instead of reporting “no tracebloc client found”.
  • The welcome screen now shows your secure environment’s dashboard name next to the name you typed, so the same secure environment no longer appears under two names.
  • When your kubeconfig points at a cluster the CLI can’t reach, data list, data ingest and resources set now say which context and server they tried and that your active secure environment may run on a different cluster (with the --context fix), instead of a bare dial tcp … unreachable error.
  • tracebloc data ingest with every option given on the command line goes straight to the review instead of asking each question again.
  • tracebloc auth status now checks your session with tracebloc: an expired or revoked session shows session expired and tells you to run tracebloc login, instead of signed in.
  • tracebloc data ingest reports a dataset-folder problem, such as a missing labels.csv, right at the path question instead of after the review.
  • When a dataset is over the ingest size limit, tracebloc data ingest now says its size, the limit, and how to get under it.
  • The tracebloc delete confirmation now names the release and namespace it will actually remove, and lists a recorded release it leaves in place.
  • When tracebloc delete can’t finish cleaning up, it now keeps the install record, so whatever is still on the machine stays listed.
  • A guided ingest with every flag supplied now checks the data folder’s layout before the review, and asks for the path again when the folder does not fit the task. It no longer fails after you confirm.
  • tracebloc delete no longer promises to remove a release on a cluster it won’t touch: when your kubeconfig points at another cluster, the preview lists that release as left alone, with the command that removes it.
  • When tracebloc upgrade or tracebloc prepare-host fails, the command it tells you to run keeps the TRACEBLOC_SUBSTRATE you asked for, and prepare-host refuses a value tracebloc doesn’t support before changing anything.
​
Secure environmentCLI
2026-10-01
Secure environment
  • Ingestion now stops before loading anything when any file named in your manifest is missing, and tells you which ones, instead of registering a partial dataset and then failing.
  • Ingesting a masked or causal language modeling, seq2seq or embeddings dataset now warns you about manifest columns that will not be stored, such as a label column, instead of dropping them silently.
  • A JSON dataset with a record that names a file missing from the dataset folder, leaves its filename empty or has no filename at all, is now rejected before anything is ingested, and the message lists the missing files and counts the empty ones. JSON datasets with such records were accepted before.
  • Ingest now rejects a manifest with a blank filename cell up front instead of registering a partial dataset and failing.
CLI
  • A successful ingest no longer ends with a false warning about a leftover temporary copy, and a real cleanup failure now explains in plain words how to remove it.
  • Ingest error messages now name a file by its path inside your dataset folder (for example texts/b77_wrong_ext.text) instead of the internal staging path.
​
PlatformSecure environmentPython SDK
2026-10-01
Platform
  • A secure environment whose node briefly fails to read its CPU or memory details now keeps updating its live resource meters instead of freezing them.
  • An experiment that stops making progress now says so on the Training tab, with the time of its last progress and the reason when one is known.
  • Signing in from a “Create a Use Case” link now opens the create wizard instead of the use case list.
  • The Memory meter on a secure environment’s page now measures the memory in use against the memory available to training, summed over the secure environment’s machines, instead of the busiest machine against its total memory. On a single machine it now reads higher and can reach red. It switches over once every machine of the secure environment reports the new figure.
Secure environment
  • The installer’s Docker memory advice now names one size to set and says plainly what a smaller budget means for training.
  • Automatic secure environment upgrades that fail now roll back cleanly instead of leaving the installation stuck mid-upgrade.
Python SDK
  • The training plan now labels the input length as sequence_length for token classification, masked and causal language modeling, and seq2seq, in both the printed plan and the notebook view.
​
Platform
2026-10-01
Platform
  • When some of a dataset’s secure environments have been offline for more than 30 minutes, a new experiment trains on the online secure environments only and records which ones it left out. An experiment on part of a dataset then takes all of its requested rows from the online secure environments and is billed for all of them, about what it costs with every secure environment online. Secure environment liveness on a use case is shown only to its owner and peers.
  • The use case wizard no longer offers score formulas that a task’s experiments can never compute, so new use cases no longer show “Score not computed” for every submission.
  • An experiment that was just restarted no longer shows as stalled before it has had time to make progress.
  • The people working with your secure environment no longer include you, or a use case’s own owner shown as a pending invite, so the peer and invited counts are accurate.
​
PlatformSecure environmentPython SDKDocs
2026-10-01
Platform
  • The use case header now shows your dataset’s name instead of its internal key; hover over the name to see the key.
  • “Create a Use Case” on the Secure Environment page now opens the create wizard directly.
  • A secure environment you created but never installed now shows “Never connected”, with a hint to install it or delete it, instead of “Offline”.
  • The Storage card on the Secure Environment page shows how much space is used, out of the total.
  • The unavailable “Watch how to” buttons on the Secure Environment page now say “Coming soon”.
  • The Secure Environment page’s getting-started steps no longer skip a number on accounts with several machines.
Secure environment
  • A failed data ingestion now reports its error right away instead of being retried twice first.
  • A machine that briefly loses its connection to tracebloc starts reporting usage again on its own, without a restart.
  • On macOS without GNU coreutils, the installer’s Docker, k3d and helm checks now time out as they already do on Linux, instead of waiting forever on a Docker that does not answer.
  • New installs using host-path storage, and every Windows install, now create a single-node cluster, so the machine’s CPU and memory are no longer counted twice.
  • New installs can ingest object detection datasets (images plus Pascal VOC annotations) as the documentation describes.
  • When the installer cannot change Docker Desktop’s memory on a Mac, it now says why and how to allow it.
  • Starting the installer from a folder that no longer exists no longer prints repeated “getcwd” errors.
  • After installing, the summary tells you when kubectl was switched to the tracebloc cluster and how to switch back.
  • On a busy Mac, an install step that stops responding now gives up when its time limit is reached instead of running on past it.
Python SDK
  • A causal-LM, masked-LM, seq2seq, embeddings or text / token / sentence-pair classification plan now defaults to the AdamW optimizer at a constant 5e-5 learning rate (was SGD at 1e-3), and get_training_plan() no longer lists image-only fields for these tasks.
  • LoRA now defaults to rank 8 and alpha 16 (was 256 / 512) when enable_lora(True) is set without set_lora_parameters(...).
  • pip install "tracebloc[pytorch]" now installs the PyTorch 2.13 line that the training images run, so a model checked locally runs on the same PyTorch version as the experiment.
Docs
  • The CLI page now documents the current tracebloc data ingest / data list / data delete commands and all 16 tasks, including causal language modeling.
  • The Quick Start now matches the installer: sign in in your browser, no Client ID or password; check the install with tracebloc doctor; delete with tracebloc delete.
  • The docs now define the leaderboard metrics for causal language modeling and the other text tasks, list the model parameters for every task, and show a lighter venv setup for the training notebook.
  • Prepare Data now walks you through tracebloc data ingest first, states the 1 GiB / 500 MiB ingest limit, and the image size rules match the templates (any aspect ratio, one size per dataset).
  • The model file reference now lists every task each framework supports, and a broken link on How training works is fixed.
  • The docs now use one name, “secure environment”, for the tracebloc software you run on your own infrastructure.
​
PlatformSecure environmentPython SDK
2026-10-01
Platform
  • An experiment that has made no progress for an hour is now marked as stalled on the Training tab’s data, with the reason (for example, no secure environment holding the data has been online since a given time), and the mark clears by itself when progress resumes.
  • Installs on the quickstart cluster no longer all show the same generic host name in the environment switcher; each one shows its namespace instead.
  • The usage cards on the Secure Environment page now stay evenly sized on wide screens.
  • The Secure Environment page no longer shows “0 GiB” or “0 Cores” for a machine whose size isn’t known yet.
  • The usage cards on the Secure Environment page keep the same size on every screen, and machines without a GPU show an empty GPU card.
  • The environment switcher marks remote machines (servers and clusters) with a “Remote access” tag.
  • In a secure environment’s use case panel, “Visit” next to Leaderboard now opens the use case on its Leaderboard tab instead of the Overview.
Secure environment
  • Storage and other usage readings keep updating on the secure environment page while the cluster’s metrics service is unavailable, instead of the page falling back to “no usage yet”.
Python SDK
  • The training notebook and README now install the current tracebloc SDK (1.2.36 or later).
​
Platform
2026-10-01
Platform
  • The environment status dot on the Secure Environment page now has its light ring, and the status labels are spaced as designed.
​
PlatformSecure environmentPython SDK
2026-10-01
Platform
  • Starting an experiment is now refused when no secure environment holding the dataset has been online in the last 30 minutes, and the message says when it was last seen.
  • Inference is now billed for the CPU its pod actually used. A billing sample could measure CPU over only the time since the last batch, so inference CPU was under-billed before: inference on CPU and GPU machines now costs more, by the CPU share that was missed. Training and validation are billed as before.
  • CPU training images now share one copy of the torch stack, so a machine that runs more than one kind of CPU task downloads and stores it once.
  • The “Explore docs” button on the benchmark page is back to its original size on phones and tablets.
Secure environment
  • A node whose usage can’t be measured no longer shows as idle; its meters show no reading instead.
  • Your secure environment can now report your machine’s own name to the environment switcher instead of a generic cluster node name such as k3d-tracebloc-server-0.
Python SDK
  • User.fetch_seed_weights(template) downloads a model zoo template’s pretrained seed <stem>_weights.pkl, so upload_model(<stem>, weights=True) trains from it.
​
PlatformSecure environment
2026-10-01
Platform
  • The homepage no longer scrolls sideways on phones and tablets.
  • The newsletter form on the home and benchmark pages now shows the platform’s validation message when a subscription is rejected.
  • FAQ answers on the platform overview page no longer cut off their last line.
  • Blog posts with code samples no longer scroll sideways on phones.
Secure environment
  • The egress gateway now runs a tracebloc build of squid, ghcr.io/tracebloc/squid-tracebloc:7.6-r0-tb.2 (squid 7.6 on Alpine), pinned by digest and about 11 MB compressed. If you mirror images, copy it into your mirror before this upgrade reaches your secure environment. If you set egressProxy.image.tag or .repository alone, read MIGRATION.md 1.9.187 first.
  • GPU machines whose NVIDIA card is too old for the GPU images (compute capability below 7.5) now install CPU-only with an explanation; machines whose NVIDIA driver is older than the recommended minimum keep the GPU and are told how to upgrade.
  • Chart releases reach the Helm repository again. 1.9.185 was never published there; 1.9.189 is the first published chart that pins the prod control plane.
  • On GPU machines whose package sources do not offer the recommended NVIDIA driver, the installer now installs the newest driver that is available and warns, instead of stopping with an error.
​
Secure environment
2026-10-01
Secure environment
  • An experiment that fails while it is being stopped now reports the original error instead of looking like a clean stop.
  • Chart 1.9.185, listed below, was never published to the chart repository. Install 1.9.189 or later.
​
Secure environment
2026-10-01
Secure environment
  • The installer now prints a tracebloc-installer substrate=k3d line at the start of every install and keeps a record of what it created, including your secure environment’s Helm release, in ~/.tracebloc/install-record.json.
  • On macOS, the installer no longer risks waiting indefinitely while it checks a k3d node’s log after a Docker restart; that check now gives up after 90 seconds, as it already does on Windows.
  • Published chart releases pin the control plane of a prod secure environment to the exact image build the release was tested with, so upgrading or rolling back the chart moves the control plane with it. Chart 1.9.185, listed below, was never published to the chart repository; 1.9.189 is the first published chart that does this.
  • On Windows, when the system refuses to start the tracebloc CLI setup, the installer now tries another way to launch it before giving up.
​
PlatformPython SDKCLI
2026-09-30
Platform
  • Audit log entries now record the address a request really came from; a sender can no longer write a false one.
  • Training steps are faster on GPU: FLOPs usage is now measured on a background timer instead of at every batch, and a per-batch GPU memory cleanup that only served that measurement is gone. The GPU part of the bill is now read during training rather than right after a cache flush, so it can be slightly higher; in the experiment it was measured on, the GPU part was about 1% of the bill.
Python SDK
  • The ConvNeXt and Swin object-detection templates now start training from an ImageNet-pretrained backbone instead of random weights.
CLI
  • At a Windows console (Windows Terminal, VS Code or a classic console window), an arrow key at a guided prompt is now refused and asked again; before, down then Enter silently took the default.
  • Pressing Ctrl-C at a question in the CLI now cancels only that question, never work the command had already started.
​
Platform
2026-09-30
Platform
  • Training and inference compute are now billed for the time that actually passes, on CPU and GPU. Each batch used to be charged its own duration plus a fixed 1 second; that fixed second is now the time since the previous batch ended. Experiments whose batches follow each other with less than a second between them are billed less than before, and experiments that wait longer than a second between batches, for example on slow data loading, are billed more.
​
PlatformSecure environment
2026-09-30
Platform
  • Multi-GPU training no longer restarts its data-loading worker processes at every epoch, which removes several seconds per epoch on each GPU. Augmentation stays reproducible and changes every epoch as before.
  • a job whose queue lock is lost now stops at the next secure environment instead of finishing, publishing and failing to settle; the redelivery lands the cycle.
Secure environment
  • Ingestion now rejects a CSV that is missing a schema column before any table is created, including files with a multi-character separator or a blank first line.
  • Ingesting a dataset whose records overlap an earlier dataset in the same table no longer changes the earlier dataset’s labels or feature values.
​
Secure environment
2026-09-30
Secure environment
  • The installer now stops before creating the cluster if SERVERS is set above 1 or AGENTS above 1, because every extra k3d node counted this machine’s CPU and memory again and let Kubernetes schedule more work than the machine has.
​
Platform
2026-09-30
Platform
  • Model uploads to check-model are now refused with 413 when the request body is over the combined file limits, before it is written to disk. Chunked uploads get 411.
​
CLI
2026-09-30
CLI
  • Guided prompts are now numbered lists you answer by typing a number.
  • The CLI download is about a third of its former size: the installers now fetch a signed, compressed archive. 32-bit Linux and Windows 10 on ARM are no longer supported, and Windows 11 on ARM runs the x64 build.
  • A downloaded install.ps1 now runs under Windows PowerShell 5.1; it had not parsed there since at least v0.10.40. The irm | iex one-liner was not affected.
​
Platform
2026-09-30
Platform
  • Starting an experiment on a dataset that isn’t part of a competition no longer fails with an internal server error.
  • A model’s stored original copy is now security-scanned too, so a check can no longer be hidden from the scan by the entry-point rename.
  • Model names ending in _formatted, _tokenizer or _weights are now refused at upload, because those suffixes name a model’s stored copies.
  • The model-zoo tokenizer fetch now gives up on a slow DNS lookup or slow response headers at its 15 s deadline, and “already being fetched” 503s are logged as warnings instead of errors.
​
PlatformSecure environmentPython SDKDocs
2026-09-30
Platform
  • Averaged models now take integer buffers, such as a BatchNorm layer’s batch counter, from the first secure environment. Before, legacy uploads got an averaged value, which was then truncated. Rounds that mix legacy and upgraded weight uploads now produce exactly the same model as uniform rounds.
  • The averaging pod now requests 12Gi and is limited to 26Gi memory (was 8Gi/24Gi), and its OpenMP/BLAS thread pools size from its CPU request.
  • When averaging refuses a round because of the uploaded weights (mismatched shapes or parameters, or NaN/inf values), the failure message now uses one wording on every path and names every secure environment at fault, not just the first one found.
  • The newsletter forms on the blog, career, legal, home and benchmark pages now submit on Enter, show a single accessible result message, and use the design-system input and button.
  • Tapping Sign in or Explore the Hub in the mobile menu now closes the menu, so the next page scrolls normally.
Secure environment
  • A secure environment’s MySQL upgrade that cannot create its new storage volume now stops within minutes with the storage provider’s reason, instead of waiting out a 15-minute timeout with no explanation; the secure environment is put back exactly as it was either way.
  • After a Docker restart, re-running the installer now repairs a secure environment whose Kubernetes stopped because its nodes’ network addresses changed, instead of reporting it as running; new environments keep fixed addresses so this can’t happen to them.
  • The log Collector now runs a tracebloc build of the same OpenTelemetry Collector release, ghcr.io/tracebloc/otelcol-tracebloc:0.159.0-tb.2, pinned by digest and about 15 MB compressed. If you mirror images, copy it into your mirror before this upgrade reaches your secure environment. If you set telemetryCollector.image.tag or .repository alone, read MIGRATION.md 1.9.181 first.
Python SDK
  • When linking a model or starting an experiment fails with “Model File Load Failed”, the SDK now tells you to upload the model again, instead of saying the dataset ID is wrong or fields are missing.
  • When your own MyModel() fails to build during link or start(), the SDK no longer suggests re-uploading the model, which wouldn’t fix it.
Docs
  • The ResNet-50 Faster R-CNN, FCOS and RetinaNet object-detection templates are uploaded without a weight file and train from scratch.
  • The Keypoint R-CNN and Faster R-CNN SPPE keypoint-detection templates are uploaded without a weight file and train from scratch.
​
Secure environment
2026-09-30
Secure environment
  • The secure environment’s credential Secret now also stores each credential under a TRACEBLOC_-prefixed key with the same value; nothing changes for running installs, but if you create or rotate that Secret by hand, set both keys (the security guide shows the command).
​
Secure environment
2026-09-30
Secure environment
  • The hourly auto-upgrade now runs Helm 4. If you mirror images, copy alpine/helm:4.2.3 into your mirror before this upgrade reaches your secure environment.
​
PlatformCLI
2026-09-29
Platform
  • When the platform’s task queue stops responding, actions that start background work now return an error within seconds instead of hanging.
  • Model zip uploads with an oversized file list are now refused before the platform reads that list.
  • Creating an experiment now requires a model you uploaded yourself. A model name from another user, or from an upload made before this release, is refused.
  • The notebook image now ships tracebloc 1.2.28 with CPU torch 2.13.0 and can run as a JupyterHub 5 named server.
  • The platform cluster now has the EBS CSI driver and an encrypted gp3 StorageClass, recorded in the repo after they were applied by hand.
  • Creating or updating a final submission now returns that submission’s id instead of an empty value.
  • CPU training images are about 360 MB smaller to download on a fresh machine.
  • The YOLOv5 and YOLOv8 object-detection templates now train as uploaded; their first training step used to fail with a channel-count error.
CLI
  • tracebloc delete now fully deletes tracebloc from a machine — it decommissions the secure environment, finds the installed release even when your config no longer names it, removes the macOS login autostart, reclaims only the GPU node image, and exits non-zero when anything is left behind.
  • tracebloc delete no longer hangs when the cluster stops answering while it looks up the release to delete.
​
PlatformPython SDK
2026-09-29
Platform
  • Model uploads can no longer skip the security scan with # nosec comments, and uploads the scan could not fully read are now refused.
  • Your team’s final-submission list loads faster, and each entry now carries its final-submission id instead of an empty value.
  • A training cycle whose last secure environment reports after the cycle was already closed on time is no longer aggregated a second time.
  • Opening another account’s experiment you cannot access now shows “not found” instead of a server error.
  • CPU training images are smaller to download, by about 67 MB each and more on the NLP image.
Python SDK
  • When a model zoo template fetch refuses to overwrite a file of yours, its message no longer refers to a function you did not call.
​
Python SDK
2026-09-29
Python SDK
  • tracebloc.notebook.render(category, framework, context) renders the Run tab’s training-plan cell for any supported task and framework, from templates that now ship with the SDK.
  • notebook.render treats empty context fields as not set instead of rejecting them, and can read its context from the TRACEBLOC_RENDER_CONTEXT environment variable.
​
PlatformSecure environmentCLI
2026-09-29
Platform
  • The API now lets the web app read Retry-After on busy or rate-limited responses, and the check-model schema types datasetId as the dataset key string.
  • The Run tab’s model zoo list no longer offers templates that are too large for its checks to accept (such as YOLO v1 and Gemma 2), and the notebook can no longer fetch them.
  • Competition leaderboards and experiment sustainability scores for runs since January 2026 now include compute (FLOPs) that was reported late and previously left out.
  • Linking a model to a dataset now works only for a model you uploaded yourself. A model uploaded before this release has to be uploaded again before it can be linked.
  • A zipped model’s stored copy now holds only its Python source files. Folders such as __pycache__ and __MACOSX are dropped instead of being passed on to training.
  • Federated Cox proportional-hazards survival models now predict from the combined coefficients of every secure environment, instead of matching a single secure environment’s model.
  • Averaging now runs your model’s code once per cycle instead of up to six times, so a constructor with side effects or random initialization runs once.
  • A loss or score that could not be computed now shows — instead of 0.00, and a real 0 score is no longer hidden.
Secure environment
  • A metric that could not be computed for a training or inference cycle is no longer reported as a score of 0.0, which for error metrics read as a perfect result.
  • Classifiers without probability outputs no longer report misleading AUC, Gini and Brier scores, and margin-based classifiers now report AUC from their decision scores.
  • Data ingestion runs no longer fail at submission with an internal server error.
  • The local database starts using about 235 MiB less memory after its next restart.
CLI
  • When your local secure environment is stopped, tracebloc doctor now tells you to start every stopped node, including worker nodes, instead of only the control plane.
​
PlatformCLIDocs
2026-09-29
Platform
  • When an averaging job fails for a reason that retrying cannot fix, such as mismatched model weights, a model file that does not load, or missing experiment data, the experiment now fails straight away with a clear reason instead of retrying for minutes first.
  • Changing the model or weights after pressing Start no longer silently drops the run; use Start over instead.
  • Switching to an environment that has no use cases yet now says so, instead of showing the first-time “create your first use case” step.
  • If you haven’t set up a secure environment yourself but your team has, the secure environments page now says so instead of showing the first-time setup guide.
CLI
  • When your local secure environment stops answering, tracebloc doctor now tells you whether Docker, the cluster, or Kubernetes inside it is down, instead of always saying to start Docker Desktop.
Docs
  • The installation docs now use TRACEBLOC_HOST_DATA_DIR for the data directory and TRACEBLOC_RELEASE_VERSION for pinning the Windows CLI version; the older names keep working.
​
Secure environment
2026-09-29
Secure environment
  • tracebloc prepare-host now grants container access only to the user named on the command line or in TB_PREPARE_USER, and warns when TRACEBLOC_PREPARE_USER is set.
​
Platform
2026-09-29
Platform
  • FedAdam, FedYogi and FedAdagrad experiments keep half as much optimizer state per experiment, and experiments already running continue where they left off.
  • Opening a use case directly on its Run view no longer makes the page flash or reload its content.
  • The filter chips on the Use Cases page now match the design’s width, and creator initials use the design’s text colour.
  • In the Create Use Case wizard, Next and Create stay disabled until the required fields are filled, and a required field you skip is highlighted.
  • Button text in the Create Use Case wizard, the “add a dataset” drawer and template cards is now vertically centred.
  • Leaving the Run drawer while a model is still validating no longer leaves a notebook kernel running in the background.
  • The page behind the “You need a dataset” panel is now dimmed, like the Create Use Case wizard.
  • Leaving the Run tab while a notebook is still starting no longer leaves a Python kernel running in the background.
​
PlatformDocs
2026-09-29
Platform
  • Carbon (gCO2) totals no longer count the same training time twice when several secure environments report an epoch at once.
  • Loading a model zoo template’s tokenizer in a notebook now fails within seconds with a retryable error when the zoo is unreachable, instead of hanging for minutes or silently finishing without the tokenizer.
  • Model uploads now accept only Python source files (.py, or a .zip of them). Other file types in an upload are refused, not loaded.
  • A retried averaging job for a cycle that already finished now republishes that cycle’s result instead of averaging it again, so the model and its privacy noise do not change on a retry.
Docs
  • Blog posts no longer scroll sideways on phones.
​
Python SDK
2026-09-29
Python SDK
  • User.fetch_zoo_template(template_id, dataset) downloads the model zoo template you picked in the Run tab into the notebook’s working directory.
  • Installing the SDK no longer adds stray tests and scripts packages to your Python environment, which could shadow your own project’s tests package.
  • Starting an experiment from Python now gives back the experiment’s id, and a refused experiment raises an error you can catch instead of only printing it.
​
Platform
2026-09-29
Platform
  • Anonymous visitors and other accounts no longer see a use case’s secure environment identity, location, status or node names; they get the compute summary only.
  • The Run tab’s Fits check now runs on the platform using the uploaded model’s own declared values. The browser sends only the dataset and the model name.
​
PlatformCLI
2026-09-29
Platform
  • The model list on the Run tab refreshes after a platform update and is never reused for a different user signing in on the same browser.
  • GPU memory limits for secure environment compute now always show on the compute card instead of intermittently appearing empty.
  • Checking a model against, or launching an experiment on, a public dataset shared with several users no longer fails with a server error.
  • Competition leaderboards and experiment sustainability scores now include compute (FLOPs) reported after an inference or training cycle was processed, instead of freezing at the value seen at that moment.
  • Compute a team uses while it is merging into another team is no longer dropped from the merged team’s budget.
  • Training rounds no longer stall waiting for results when the platform is updated while a round is finishing.
  • Leaving the invitation page before its automatic redirect no longer pulls you back to the login or use-case page a moment later.
CLI
  • The CLI installers now also accept TRACEBLOC_RELEASE_VERSION, TRACEBLOC_INSTALL_PREFIX and (Linux/macOS) TRACEBLOC_COSIGN_VERSION; the existing unprefixed variables keep working.
​
PlatformSecure environmentPython SDK
2026-09-29
Platform
  • The filter chips on the Use Cases page now line up exactly with the design.
  • Opening an invite link you already accepted now takes you to log in and then to the use case, instead of stopping on a blank “Thank you” page.
Secure environment
  • The installer accepts TRACEBLOC_NAMESPACE to name your secure environment; TB_NAMESPACE keeps working until the end of 2026.
Python SDK
  • User(stage=...) now also accepts the short stage names dev, stg and prod; every stage name that worked before still points at the same place.
​
PlatformPython SDK
2026-09-28
Platform
  • TensorFlow models averaged by the platform are now saved in the numeric precision your secure environments trained in (for example float32), instead of being widened to float64.
  • Averaging rounds that can never succeed, such as secure environments that trained different model shapes or reported zero training samples, now fail right away with the reason, instead of retrying for many minutes first.
  • Averaging jobs sent with invalid input now fail right away instead of being retried several times first.
  • FedAdam, FedYogi and FedAdagrad training no longer restarts its optimizer momentum when the number of secure environments contributing to a round changes, on models with integer buffers such as BatchNorm layers.
  • Moving between use cases no longer leaves the Run drawer open, or closed, from the previous use case.
  • Confusion matrices for completed training cycles now display in the experiment view instead of staying on “not ready yet”, with normalized values rounded for readability.
Python SDK
  • fetch_model now fails with a clear error when a download is cut short, instead of saving a truncated weights file that breaks training later.
​
PlatformPython SDK
2026-09-28
Platform
  • The averaging service no longer restarts when a message’s lock is lost mid-job; it logs the failure and carries on with the next job.
  • A training cycle whose results list the same secure environment more than once now fails with a clear error naming that secure environment, instead of silently counting it twice in the global model and its metrics.
  • Scikit-learn and survival-model experiments across several secure environments no longer fail an averaging round when the formatted model file is missing or can no longer be imported. Averaging only needs the secure environments’ trained models.
  • An averaging job for a framework the platform doesn’t support now fails immediately with a clear error listing the supported frameworks, instead of appearing to succeed with a meaningless result.
  • A mistyped or invalid aggregation hyperparameter no longer runs silently. An invalid value (wrong type, a fraction where a whole number is needed, true/false, or out of range) now fails the experiment’s averaging step immediately with a message naming the setting. An unrecognized setting name is logged as a warning for now and will be refused in a later release.
  • A training cycle with only one secure environment is no longer published when that secure environment’s model weights are invalid (NaN/infinite), and Krum aggregation now refuses a cycle with one secure environment as it already refuses one with two.
  • Aggregated confusion matrices built from normalized (fractional) values now show their real values instead of being rounded down to zeros. A cycle whose secure environments report mismatched class labels now fails straight away with a message naming the secure environment, instead of being retried repeatedly first.
Python SDK
  • User.fetch_model(model_name) downloads the model you uploaded from the Run tab into the notebook’s working directory.
​
Python SDK
2026-09-28
Python SDK
  • tracebloc.checks.trains(model_name) runs the same local training step as upload_model without uploading anything, and returns (and in a notebook shows) whether the model trains, and how long it took.
​
Platform
2026-09-28
Platform
  • When linking a model to a dataset fails for more than one reason, the Run tab now lists every reason instead of only the first.
  • Averaging no longer fails for models with BatchNorm or other scalar tensors while some secure environments have been upgraded to the new weight format and others have not.
  • The Create buttons on the Use Cases page now show a spinner and stay disabled while we check your datasets, so a click is never left without feedback.
  • Button text is now vertically centred, including the Filters Reset and Apply buttons.
​
Secure environment
2026-09-28
Secure environment
  • The Windows installer now adds its tools folder to the system PATH without turning %SystemRoot%-style entries into fixed paths.
  • Logs from the automatic upgrade and image-refresh jobs are no longer lost when the telemetry collector restarts while one of those jobs is running.
  • Installs that set fullnameOverride are now recognised as healthy by the installer: re-running it no longer reinstalls a working secure environment, and the support bundle includes the secure environment’s logs.
  • On a machine too small to run training beside the platform, the installer now stops before creating anything, so there is no empty cluster left to delete.
  • The installer now recognises an existing secure environment installed with a custom name override when its id is stored only in the cluster Secret.
  • Support bundles now include the resource monitor’s logs.
  • Upgrading to chart 1.9.158 or later: MySQL now requests 1 GiB, and experiments on a secure environment sized to its old maximum start slightly smaller instead of being refused. See MIGRATION.md.
​
PlatformSecure environment
2026-09-28
Platform
  • Federated experiments using FedAdam, FedYogi or FedAdagrad on models with batch normalization no longer lose their optimizer momentum when secure environments move to the new weight format, so convergence continues where it left off.
  • Clicking Create a Use Case with no datasets now shows how to add your first dataset, instead of a wizard you can’t finish.
  • The page behind the Create Use Case wizard is now dimmed, so the wizard stands out.
  • When a use case can’t be created, the field that needs changing is now highlighted in red next to the error message.
Secure environment
  • After an upgrade that grows the platform’s own memory use, experiments on a secure environment that was sized to its old maximum now start with a slightly smaller envelope instead of all being refused.
​
Platform
2026-09-28
Platform
  • The environment switcher now shows when each machine was last seen, its host name, where it runs, and an icon for the kind of machine.
  • The Use Cases dashboard now keeps its designed width on wide monitors, instead of stretching cards across the whole screen.
​
Python SDK
2026-09-28
Python SDK
  • Training-plan submissions that include LoRA seed, update-set, delta-transmission or ZCA-epsilon settings are no longer rejected by the duplicate-plan check.
​
Docs
2026-09-28
Docs
  • The changelog and the training guides now say “secure environment” wherever they used to say “edge”.
​
Docs
2026-09-27
Docs
  • The changelog now says “experiment” wherever older entries said training run, training job or training plan.
  • The 0.12.0 changelog now tells you to set TRACEBLOC_AMP=false in your secure environment to keep full fp32 training.
​
Secure environmentCLI
2026-09-27
Secure environment
  • When a training image cannot be pulled, the failure now says why: the registry’s rate limit, access refused, a missing tag or repository, or the registry being unreachable. It no longer shows a generic pull error.
CLI
  • tracebloc login now accepts staging, development and production as the environment name (from --env, TRACEBLOC_ENV or CLIENT_ENV), the same spelling the installer uses, and says which source it read when it rejects a value.
​
Secure environment
2026-09-27
Secure environment
  • Experiments on production secure environments now run one exact, verified engine image for every cycle of an experiment, and nodes that already hold that image no longer download it again.
  • Values set under env.RESTRICTED_DNS_NAMESERVER or env.RESTRICTED_DNS_CLUSTER_DOMAIN no longer change where training and ingestion pods resolve DNS; configure the restricted resolver through restrictedDns.clusterIP and restrictedDns.clusterDomain.
​
CLI
2026-09-27
CLI
  • The largest experiment tracebloc resources set max offers is up to 320 MiB smaller, because tracebloc now keeps 2.06 GiB of memory for itself instead of 1.75 GiB. On an 8-core / 32 GiB machine the maximum goes from 30 GiB to 29 GiB.
​
PlatformDocs
2026-09-26
Platform
  • API requests no longer fail with a 502 error while the platform is being updated.
  • An experiment that cannot start its next training cycle, for example because the team’s compute budget is used up, now shows as paused with the reason, instead of staying “running” with nothing happening.
  • The homepage’s Regulatory & compliance testing card now opens with a compliance scenario instead of copy from another card.
Docs
  • The install commands on the benchmark page and in the LLM benchmarking blog post now work; before, they downloaded an error page instead of the installer.
​
Platform
2026-09-26
Platform
  • Several blog posts now show working links, accurate image descriptions, corrected typos, FAQ sections without repeated questions, and comparison images that had been broken.
​
PlatformSecure environmentPython SDKCLI
2026-09-26
Platform
  • LoRA fine-tuning can now upload only the trained adapter factor instead of the whole model, cutting per-cycle upload size for large models while producing the same combined model.
  • Object detection models that state their own model type are now trained with the detection family they name, instead of one guessed from the model’s name.
  • LoRA fine-tuning on the B-only wire can now download each cycle’s model as the trained adapter update instead of the whole model, when your secure environment still holds the previous cycle’s model, with the same combined model as a result.
  • Refreshed the Use Cases page — updated spacing, hover states, filters, search, empty states, and a redesigned first-time setup screen.
  • Opening or copying the “Pricing” link from the navigation on any page now takes you to the homepage’s pricing section, instead of a broken link.
  • The Healthcare industry page is easier to use with a screen reader: each chart now has its own description, the two use-case cards no longer repeat the same bullet, and the mobile view no longer has a duplicate page heading.
  • Fixed a dead button on the benchmark page, corrected an outdated FAQ date, and made the demo video’s play button work properly with keyboard and screen readers.
  • The FAQ on the platform overview page can now be opened and closed with the keyboard.
  • Some Privacy Policy section links changed, and footer placement across pages is now corrected for accessibility.
  • Share, apply, and browse buttons across job and template pages now work with keyboard and screen reader navigation, not just a mouse click.
Secure environment
  • With mixed precision switched on, training uses bf16 on Ampere and newer NVIDIA GPUs and on CPUs with AMX, and fp16 with a gradient scaler on older GPUs (T4, V100); other CPUs stay in fp32. Mixed precision is off by default, so a secure environment trains in full fp32 unless you set TRACEBLOC_AMP=1 in it.
  • LoRA training no longer fails at setup when the experiment leaves the LoRA seed blank; the experiment’s shared seed is used instead.
  • An experiment now prints a one-line warning when logging to Application Insights cannot be attached after sign-in, instead of going quiet with logging degraded.
  • XGBoost models that use an evaluation-monitor callback now complete federated training, averaging included, instead of failing to load.
  • Retrying a dataset ingestion after restricted DNS was switched on no longer silently reuses an earlier job that cannot reach the database; it now reports a conflict instead.
  • The installer no longer tells you to delete your cluster without first releasing its secure environment, so recreating or switching a cluster stops leaving stale records on your dashboard.
  • Training and data-ingestion pods can now be confined to an in-cluster DNS resolver that answers only cluster names and an explicit allowlist, so they cannot look up names outside your cluster. It is opt-in and off by default.
Python SDK
  • When linking a model fails because the dataset’s label information is missing or unreadable, you now see the reason instead of an empty error.
  • The dataset size shown when you link a model now counts dataset records instead of individual labels, so token-classification and object-detection datasets no longer report an inflated total.
  • Fixing a rejected custom loss with a valid loss_function() call now lets start() run instead of reporting invalid training parameters.
  • Sessions on a non-production platform, including the in-app notebook, now show the platform’s actual error message when a sub-dataset or experiment request fails.
CLI
  • tracebloc client create now warns you when your machine’s previous secure environment will be left behind on your dashboard, and tells you how to release it first next time.
​
PlatformSecure environment
2026-09-25
Platform
  • Probability metrics (AUC-ROC, AUC-PR, Gini and Brier) now appear for scikit-learn tabular classification experiments in secure environments that saw only some of the dataset’s classes, where they were previously left out of the results.
  • LoRA fine-tuning now shares one fixed adapter factor across every dataset and trains only the other one, so an experiment updates about half as many adapter parameters and its results combine cleanly across datasets.
  • LoRA fine-tuning of NLP models now updates only the adapter weights, instead of silently retraining the whole base model.
  • An experiment that asks for more GPUs than any single machine can provide is now refused right away, with a message naming the limit and how to stay within it, instead of waiting indefinitely with no explanation.
Secure environment
  • Training results and FLOPs relayed from your secure environment no longer stall or drop while the messaging credential rotates.
  • Setting an image block to null in the secure environment’s chart values no longer fails the install or upgrade; the chart’s default image is used instead.
​
Secure environment
2026-09-25
Secure environment
  • A dataset with a column name containing %( or %s (for example a%(b)s or %share) is now rejected before ingestion starts, with an error naming the column. Before, the table was created and every row then failed to insert, or the dataset ingested and training on it failed. A % followed by anything else (for example %rate or 50%) still works.
​
2026-09-25
No customer-facing changes. This release carried internal test changes only; every component is at the same version as in 0.8.0.
​
PlatformSecure environment
2026-09-25
Platform
  • When the service is briefly at capacity, a request now comes back as a clear “too many requests” with a Retry-After hint to wait on, instead of an unexplained gateway error.
  • Banners and toasts on the Secure Environment and Run screens have a refreshed, flatter look.
Secure environment
  • A brief database connection drop at the end of an ingest no longer deletes the rows that were just ingested; the ingestor retries and registers the dataset.
  • A CSV with a repeated column name is now rejected even when it uses a multi-character or regex separator. Before, it could ingest with its columns silently mismatched.
  • A classification dataset with a blank label cell now fails the preflight check and lists the affected rows. Before this, the dataset registered with an extra, empty class.
  • An ingest stopped by the platform (eviction, node drain, deadline) while it was registering its dataset no longer leaves its rows stuck; the next ingest into the same table cleans them up.
  • A semicolon-delimited (or other non-comma) labels file is now validated correctly for image classification, keypoint detection, all text tasks, embeddings and time-to-event prediction. Such files were previously rejected at preflight, or passed a text-content check without any file being checked.
  • A re-ingest into an existing table can no longer delete an earlier dataset’s rows when it fails, the reported record count now matches the rows actually stored, and duplicate keys that collapse into one row are shown as “Collapsed Duplicates” in the summary.
​
PlatformSecure environment
2026-09-24
Platform
  • Flash messages are flatter and more compact — the coloured bar down the left edge is gone and the spacing is tighter.
  • Long tooltips now wrap onto a second line instead of stretching across the screen.
  • Code blocks now use a monospace font, and the one-line command box has slightly taller line spacing.
Secure environment
  • File copies during an ingest now run 4 at a time by default, which is where throughput on network storage such as EFS stops improving. Set TRACEBLOC_COPY_WORKERS (1–16) to tune it; 1 copies one file at a time.
  • The dashboard’s machine switcher now shows each machine’s name and whether it is a workstation, a GPU server or a server.
  • On Windows, the installer now retries the tracebloc CLI install when Windows refuses to start it, and the final summary tells you if the CLI still needs installing.
​
Docs
2026-09-24
Docs
  • The docs now list the Python packages a model file may import for each task, and explain what an upload refusal means.
​
Platform
2026-09-24
Platform
  • When a model upload is refused for importing a package its task does not allow, the message now links to the page listing every task’s allowed packages.
​
PlatformSecure environmentCLI
2026-09-24
Platform
  • The environment switcher now shows each machine’s hardware (GPU or CPU) under its name.
Secure environment
  • Your secure environment now reports its host name and machine type, so the dashboard’s environment switcher can show which machine each one runs on.
  • Large tabular datasets ingest about twice as fast, because rows are written to the database with far less per-batch overhead.
  • A semicolon-delimited time-series (or tabular) CSV declared with csv_options.delimiter now passes the pre-ingest checks instead of failing with “Required column ‘timestamp’ not found” or “Schema columns not present in CSV”.
  • An object-detection ingest now fails, instead of completing with a missing annotation, when an annotation file is removed after validation.
CLI
  • The tracebloc resources set wizard’s “Use as much as possible” option now refuses a ceiling no experiment could be admitted at, exactly as tracebloc resources set max does, instead of saving it.
​
Platform
2026-09-23
Platform
  • If your account can’t be loaded because of a connection problem, protected pages now show an error with a “Try again” button instead of a blank screen.
​
Secure environmentDocs
2026-09-23
Secure environment
  • The release page of every chart version on the public mirror now links the tracebloc changelog entry that release belongs to.
Docs
  • The docs now have a changelog page at /changelog that lists every tracebloc release that changes something for you, with an RSS feed to follow it.
​
PlatformCLI
2026-09-23
Platform
  • The Secure Environment page has a refreshed header showing your environment’s namespace and status, and ended or archived items are now clearly greyed out.
CLI
  • The release page of every CLI version on the public mirror now links the tracebloc changelog entry that release belongs to.