Skip to main content
tracebloc ships as one platform, and this page is where its releases are read. Each entry carries a release number, a date and the release notes of everything that changed for you, grouped by where you notice it: Platform, Secure environment, Python SDK, CLI and Docs. A collapsed block at the end of every entry lists the Helm chart, CLI, Python SDK and data ingestor versions the release corresponds to, so you can tell which of them to update. How the number moves. The middle number (0.19.0 to 0.20.0) moves when a release changes something you can notice. The first number moves when a compatibility floor or an install-time contract changes: read those entries first, they name what you have to do. Releases that change nothing for you are not listed here. Where to act. When an entry asks you to update something:
  • the secure environment updates itself by default; to update by hand or pin a version, follow Upgrade, or roll back if you have to
  • update the tracebloc CLI with the one-liner under Install
  • update the Python SDK as described under Installation
  • setting up a new secure environment starts at the Quick Start
Follow releases. Every entry on this page is also published as an RSS item at https://docs.tracebloc.io/changelog/rss.xml; the RSS button at the top of this page links the same feed. Filter the entries by surface with the tags in the side panel.
​
Secure environment
2026-10-12
Secure environment
  • A paused training run shows as stopped cleanly instead of as an error, once the edge enables it.
  • Re-running the installer now reconnects a secure environment whose credential tracebloc stopped accepting, instead of reporting it as healthy.
  • On hosts the installer cannot install an NVIDIA driver for (yum, zypper), or when the driver install fails, the installer now says so and how to install it yourself, and continues CPU-only instead of asking for a reboot.
  • Reinstalling a secure environment over its existing data now keeps the local database passwords, so the reinstall opens the database instead of being refused.
  • Running prepare-host without naming a researcher now also installs the rootless-Docker helpers, so the next host check no longer lists them as a pending admin step.
  • The install command shown in the installer’s help and messages now points at the working one-line installer (tracebloc.io/i.sh, tracebloc.io/i.ps1) instead of a link that failed.
  • Pausing a training run no longer leaves its pod marked as an error.
  • MySQL can now replace an old install-time root password with the one in the release Secret on its own, at startup, without anyone needing access to the cluster. It is off by default.
  • When tracebloc refuses this machine’s credential and your installed CLI is too old to renew it, the installer now stops and tells you to update the CLI instead of reporting a successful install while the machine stays offline.
​
PlatformCLI
2026-10-12
Platform
  • The use case page now fits tablets and phones without scrolling sideways.
  • The use case page’s icon buttons (edit pencils, invite, archive switch, more options) now have names for screen readers and work from the keyboard.
CLI
  • On macOS, a node that stops right after it is started is now restarted on the next check, about a minute later, instead of staying down for about four minutes.
​
Secure environmentCLI
2026-10-12
Secure environment
  • On Linux, native k3s installs now accept TRACEBLOC_STORAGE_MODE=hostpath and TRACEBLOC_HOST_DATASET_DIR, and keep the data in your data directory. Before, they had to fall back to k3d. A re-run or upgrade keeps the storage mode and directories the install was made with, and says so when you asked for others.
  • On macOS, inside WSL, or with a setting native k3s cannot use, TRACEBLOC_SUBSTRATE=k3s on a machine running tracebloc on k3d is now refused before anything changes, instead of removing the k3d client first and then refusing native k3s.
  • Moving from k3d to native k3s no longer refuses a machine for disk space that the move itself frees.
CLI
  • tracebloc resources set max now checks the ceiling again after applying it. When applying it started more of tracebloc’s own services, it lowers the ceiling once so a training run still fits, and tracebloc doctor no longer calls a fresh max over-committed.
​
Secure environmentCLI
2026-10-11
Secure environment
  • A node whose GPU appears after the client starts no longer reports that GPU as idle.
CLI
  • tracebloc resources set max now picks a size that fits beside what is already running, so doctor no longer reports it as over-committed.
​
Secure environmentCLI
2026-10-11
Secure environment
  • GPU nodes whose GPU becomes available after the client starts are now reported as GPU nodes without a restart.
  • An install from your own values file (dev mode) now sets up the GPU the way a normal install does. The GPU request, runtime class, device type and device plugin follow what the installer detected on this machine, over any GPU settings in the file.
  • The Windows installer no longer stalls or stops partway when it is run as a piped command.
CLI
  • On EKS with the recommended EFS storage, tracebloc data delete no longer refuses with “can’t remove the files” when the files can be removed.
​
Secure environment
2026-10-10
Secure environment
  • On a GPU machine the installer now downloads the shared GPU training base image in the background once setup finishes, so your first GPU training no longer starts with a 4 GB download.
  • On native k3s, prepare-host now records the tracebloc group in the user’s install record, and whether it created the group, so tracebloc delete takes the user out of it and removes the group only when prepare-host made it and no members are left.
  • A local dashboard node port outside 30000–32767 now turns the local dashboard link off with a warning instead of failing the install.
  • Re-running the installer after your local cluster was rebuilt no longer asks you to name a new secure environment; it tells you which one it is reconnecting.
  • A fresh local install now tells you which kubectl context was active before it switched you to the new cluster, so you can switch back.
  • Installing native k3s no longer stops with “cannot tell whether a k3d cluster is live” when Docker briefly fails to answer after k3d has already listed this machine’s clusters.
  • On Linux, a machine that runs tracebloc on k3d can now move to native k3s when you run the installer with TRACEBLOC_SUBSTRATE=k3s. With your consent, the installer first offboards the old secure environment, then sets up k3s. Without consent, nothing changes. A consented move registers the machine as a new client, so its environment name gets a suffix (for example my-lab becomes my-lab-2).
  • On a machine with less memory than the smallest training size, the installer no longer tells you to train on a larger machine; it says training runs there with less memory per run, so a larger model can run out of memory.
  • Switching an install to native k3s no longer refuses over a k3d cluster that does not exist when Docker stops answering partway through listing its containers.
  • A consented reinstall now stops with a clear message instead of hanging when the old environment’s offboard stops responding.
  • Re-running the installer on a native k3s machine after a k3s version bump now restarts k3s onto the new version, instead of leaving the old one running until a reboot.
  • When the installer cannot read Docker during a reinstall check, it now shows Docker’s own error and how to fix it, instead of saying Docker did not answer.
  • On macOS, an install that stops partway now reports a failure instead of finishing with success.
  • On macOS, an installer run that stops on an internal error now reports a failure instead of success.
  • Re-running the installer with TRACEBLOC_SUBSTRATE set on a machine that is already set up and healthy now reaches the runtime choice, instead of stopping at “Already set up and healthy”.
  • A machine an administrator prepares with prepare-host now records which user it was prepared for, so that user can run their own tracebloc delete without administrator rights. A machine prepared before this release needs prepare-host once more.
  • Object-detection training no longer fails when the validation set ends in a single image, and a one-image BatchNorm failure now says what caused it.
  • Training logs no longer mention LoRA for experiments that did not enable it.
  • Image-model training logs no longer show an albumentations “Error fetching version info” warning, and pods start up to 2 s faster.
​
CLI
2026-10-10
CLI
  • On a machine an administrator prepared for you, tracebloc delete now disconnects the machine and removes what is yours, then names the one command an administrator runs to remove k3s, instead of stopping at sudo -v.
​
PlatformCLI
2026-10-09
Platform
  • Experiments that turn on LoRA without setting rank and alpha now train with rank 8 and alpha 16, the same defaults the SDK documents.
  • In My Use Cases, the create wizard waits for you to fix a rejected field before you can submit again.
CLI
  • The CLI is built with the latest Go security fixes for HTTPS and TLS connections.
  • When data delete can’t remove a dataset’s folders, it no longer blames an old ingestor for folders your storage class created, and points you at the storage class instead.
  • Declining the name prompt while offboarding for a reinstall now stops the reinstall instead of reporting success.
​
Platform
2026-10-09
Platform
  • The environment switcher opens smoothly in place and scrolls smoothly through your environments, and long tables show 15 rows before they scroll.
  • Your environments now read machine, operating system and last seen in the switcher, and header notices are spaced as designed.
  • An offline environment shows one clear notice instead of also warning about its GPU.
  • My Use Cases now shows “Created by You” on your own use cases, celebrates your first use case, opens a welcome page for your first invitation, and asks you to log in again when your session expires.
​
PlatformSecure environmentCLI
2026-10-09
Platform
  • The use-case details on the Secure Environment page show whether a use case is public or private and when it ends.
  • Machine memory and the per-training limit now show in GiB and the CPU a training run actually gets.
  • The environment switcher menu now animates open and closed.
  • Edge memory now reads in GiB and CPU shows what one training run gets, on the edge devices list, the dataset card and the Run tab, so a device shows the same figures on every page.
Secure environment
  • Reinstalling a client on the same cluster no longer fails with “cluster in use” when the previous client has stopped heartbeating.
CLI
  • tracebloc doctor reads this machine’s credentials for your secure environment, warns when they expire within 120 days, and names the restart and installer re-run that renews them.
  • an administrator removes the native k3s prepared for a user with sudo tracebloc delete --prepared-user <user>, which reads only that user’s install record.
  • tracebloc delete removes the group prepare-host made for a user’s k3s once nobody is left in it, and always takes the user out of it; a group that was there before stays.
  • The image resolution question in tracebloc data ingest now accepts 416×416, 416X416 and 416 x 416, and a value it still cannot read tells you to use the letter x, e.g. 512x512.
  • Re-running the installer on a machine whose tracebloc install was removed now reconnects your existing secure environment instead of asking for credentials or creating a new one.
  • Switching a guided ingest from test to train (or back) no longer offers the previous dataset’s name and path as defaults.
  • After an ingest, What’s next now sends you to My Data on your Secure Environment page and says which other dataset a use case needs; support-only IDs and setup lines show with —verbose or when a run fails.
  • Reinstalling tracebloc after your local cluster was rebuilt now reconnects your existing secure environment, with its datasets and history, instead of creating a second one.
  • tracebloc client status now shows whether your secure environment is connected right now, and when it was last seen if it isn’t, instead of an out-of-date “online”.
  • tracebloc upgrade no longer replaces a newer pre-release of the CLI with an older release.
  • Re-running the installer no longer replaces a newer pre-release of the tracebloc CLI with an older release.
  • The home screen now shows the same CPU and memory as tracebloc resources, and tracebloc auth status names your secure environment.
  • tracebloc doctor now reports your secure environment as not connected when tracebloc hasn’t heard from it, says when it last did, and how to reconnect it.
  • Running tracebloc client create --yes at a terminal on a cluster whose install is gone now issues a new credential instead of finishing with nothing to connect.
  • When your CLI is too old to reconnect an existing client, tracebloc client create now tells you to upgrade instead of telling you to re-run.
  • After an administrator removes the k3s they prepared for you, your own tracebloc delete now finishes and disconnects the machine instead of stopping with “this machine has no root copy”.
  • On macOS, the tracebloc CLI is signed by Tracebloc GmbH and notarised by Apple.
​
PlatformSecure environmentCLI
2026-10-08
Platform
  • A client created but never installed now shows “Not installed yet” instead of “Offline” on the dashboard, with the command to install it.
  • Leaderboards ranked by an error metric now show each team’s best result instead of its worst.
  • A submission that is still running no longer shows a loss of 1.0 and accuracy of 0.0; it shows no values until its results arrive.
  • My Data now shows the use cases a test dataset is used by, instead of “0 use cases”.
  • The My Secure Environment page gets a dedicated endpoint that returns connection state, resource meters and GPU notices already worked out, so every page shows the same status for the same client.
  • An account with one or two use cases now sees them as larger cards with an “Add More Use Cases” card beside them.
  • An offline secure environment’s memory, CPU, GPU and storage cards now show their capacity with a dash for usage, instead of “no usage yet”.
  • Deleted environments are now marked in the environment switcher, and a use case’s collaborators show as avatars in its details panel.
  • The secure environment page’s use case details now show who works on each use case, and its notices and lists read more cleanly.
  • Use case cards now show who owns or invited you to each use case, use smaller topic tags, and show the end date when you hover over them.
  • Finishing account activation, a password reset or a profile update no longer pulls you off a page you navigated to during the short redirect delay.
  • Copy ticks and confirmation messages no longer get cut short when you click twice, and leaving the password page right after saving no longer redirects you a second later.
Secure environment
  • An administrator can run prepare-host --print to see every command prepare-host would run as root, in order, before any of it runs.
  • A k3s install on a machine that already runs tracebloc on k3d is refused before anything changes, with what moving to native k3s would mean and how to keep today’s environment.
  • A small machine on native k3s (7 GB) is Ready right after a clean install; training runs ask for the memory the machine actually has free.
  • A successful install no longer tells you to open a new terminal when your current terminal already runs tracebloc, and its log no longer records a false error after the success screen.
  • On k3d, a training run’s memory now leaves room for the cluster’s own system pods, so a machine that fits just barely is Ready right after install instead of turning Not ready once those pods start.
  • A data ingest no longer prints an internal telemetry configuration line.
  • Re-running the installer on a machine that already has your secure environment no longer asks you to name a new one; it reconnects the existing one and tells you so.
  • On an AMD GPU, or an arm64 machine with an NVIDIA GPU, the installer now sets up a CPU-only client and tells you why, instead of configuring a GPU that can’t train.
CLI
  • A client whose cluster has stopped now shows as offline in client status and client list within about 35 minutes of its last heartbeat, on every environment.
​
PlatformDocs
2026-10-08
Platform
  • Tabular regression experiments now chart training and validation R² for every epoch, negative values included, instead of showing ”—” with no curve.
  • The Secure Environment Storage card now shows how much of your available space your datasets use, like the other cards.
  • After switching accounts in the same tab, the Secure Environment page no longer briefly marks the previous account’s machine as “this machine”.
  • A dataset’s details now list the use cases it is used in, by name, and each one opens that use case; test datasets no longer show “0 use cases”.
  • Empty sections on Secure environment no longer show a “Watch how to” button before its video is available.
  • Dates in the Use Cases and Collaborators tables no longer wrap onto two lines on smaller laptop screens.
  • You can now rename a use case from its details panel on Secure environment.
  • The use case details panel now shows who created the use case.
  • Fractions, subscripts and superscripts in notebook math render at the right size again.
  • Object detection experiments and submissions now show mAP instead of an accuracy figure that does not apply to detection, and segmentation labels its accuracy as pixel accuracy.
  • The Secure Environment page no longer shows memory and CPU usage for an environment that is disconnected.
  • Clicking outside a details panel on Secure environment now closes it.
  • Deleted datasets can no longer be edited from their details panel.
  • Sorted table columns on Secure environment now show they are sorted, and banners have corrected spacing.
  • The empty collaborators section now explains where to invite people from, and shows the privacy note.
  • Use case names in a dataset’s details now open the use case’s page.
  • The buttons at the bottom of Secure environment’s details panels now use the standard button style.
  • The environment switcher now shows each machine’s full details when opened.
Docs
  • The CLI guide now explains the image resolution question in guided data ingest and shows how to find an image’s width and height on macOS, Linux and Windows.
  • The setup guide now lists exactly which GPUs, drivers and platforms train on a GPU, and which don’t.
  • The CLI page explains every guided-ingest question, what you can type at each, and where the suggested answers come from.
  • The Prepare dataset page now points to My Data on your Secure Environment page to see ingested datasets.
​
Secure environment
2026-10-07
Secure environment
  • Secure environments on Helm chart 1.9.200 now train with the engine changes announced under 0.68.0–0.71.0. For example, training on a GPU the platform cannot use now fails at once instead of running on CPU, keypoint PCK and regression R² report real values, and metrics that can’t be measured show ”—”.
  • A native k3s install on a host without the timeout command no longer hangs indefinitely when a package mirror stops responding; it gives up after 10 minutes.
  • The installer now tells you when your GPU isn’t NVIDIA or Docker can’t see it, instead of reporting no GPU.
  • The installer now reserves port 47910 on 127.0.0.1, reachable only from this machine, for a coming dashboard feature. Set TRACEBLOC_LOCAL_API=off to skip it or TRACEBLOC_LOCAL_API_HOST_PORT to move it; if the port is taken, the install finishes without it and says why.
  • Re-running the installer on an existing secure environment that runs on k3d (Mac, Windows) adds that port and restarts the environment once; training running at that moment restarts.
  • Updating a machine set up before the storage change no longer stops with a storage-topology error; it keeps the existing data folder and finishes.
  • Running the installer with —diagnose on a machine that is already set up now produces the support bundle instead of opening the home screen.
  • When the auto-upgrade job cannot read a stuck release’s history, its log now says the read failed and shows helm’s error, instead of claiming the history has no healthy revision.
​
PlatformPython SDKCLIDocs
2026-10-06
Platform
  • Regression experiments now record and show their R² for every epoch.
  • My Use Cases and Explore are back to the previous layout; the new use-case dashboard remains available at its own page.
  • The Secure Environment machine selector now fits its details, with long values shortened with ”…” and shown in full on hover.
  • A client you created but never installed now shows “Not installed yet” with the command to install it, instead of looking like an offline machine.
  • Secure Environment table headers now sort in one click (largest and newest first where it makes sense) and a second click restores the original order.
  • The Clients table sorts names A to Z and numbers largest first in one click, and a second click restores the original order.
  • Secure Environment no longer says “None of these people saw your data” when nobody is working on your data yet.
  • The environment switcher no longer labels a machine that was never set up as “Offline”.
  • Secure Environment’s Training Limit card now shows tidy figures, with the exact limit on hover.
  • In Create Use Case, the compute budget note now appears at the top of the Data & Evaluation step.
  • The Secure Environment dataset panel now shows who created each dataset.
Python SDK
  • Uploading a YOLO model without its loss.py now tells you to upload model.py and loss.py together as one zip, and names the model zoo’s ready-made YOLO zips.
  • The training plan no longer lists dataset classes for regression and forecasting experiments.
  • The quickstart notebook installs the same minimum tracebloc SDK version the docs name.
  • The Cascade R-CNN, Sparse R-CNN, YOLOv10-S and EfficientDet-D0 detection templates can now be uploaded.
  • The model zoo’s YOLO v1, v5 and v8 templates can now be uploaded straight from a clone: upload object_detection/pytorch/yolo_v8.zip (or yolo_v1.zip, yolo_v5.zip).
CLI
  • Creating a client when no cluster is reachable now points you to the installer instead of making a client that can never come online, and a printed client credential now comes with the command that connects it.
  • When tracebloc data ingest asks its questions and the folder is laid out for a different task, the hint now says to pick that task in the menu instead of telling you to pass a --task flag.
  • tracebloc cluster info no longer reports a missing ingestor image setting on working installs, and shows a shorter summary; --verbose still shows the full detail.
  • Checking a large labels.csv before an ingest no longer uses memory in proportion to the file’s size.
Docs
  • The Define page and dataset templates now explain MBE ranking, forecasting scaling and the CLI’s --schema and --label-column behaviour correctly.
  • The CLI reference now lists every task that needs a label column, and the SDK page opens the maintained Colab notebook.
  • The segmentation and keypoint dataset docs now show working CLI commands and segmentation mask rules that match training.
  • The operations page now points to tracebloc doctor --diagnose for support bundles, and the move guide tells you to re-ingest datasets on the new machine.
  • The training docs now show a custom loss that works in a notebook, mark the TensorFlow-only augmentation forms, and say what the seed controls.
  • The security page’s firewall list now matches what a Mac install downloads, so an allowlist built from it no longer blocks a fresh macOS install.
  • The model docs now explain how to upload the model zoo’s YOLO v1, v5 and v8 templates (as their .zip) and your own YOLO model (model.py and loss.py in one flat .zip).
  • The training guide now shows how to upload the model zoo’s YOLO models: pass the .zip next to the YOLO folder, for example object_detection/pytorch/yolo_v8.zip.
​
Platform
2026-10-06
Platform
  • New embeddings use cases offer score formulas the leaderboard can compute, instead of one that scored every submission “Score not computed”.
  • A stopped client now shows as disconnected within a few minutes instead of half an hour.
  • Fetching a zoo template’s pretrained seed by its full template id now returns that task’s own seed. For example, the sentence-pair BERT template no longer gets the text-classification seed.
  • Training cycle and inference results now show up about a minute sooner.
  • Changing a client’s password now signs that client out of every existing session, so it must log in again with the new password.
  • The client page can now show that a client’s GPU is not supported and that it runs on CPU instead.
  • A model upload that reaches system commands, environment variables or native code through an allowed library such as torch is now refused at upload, and the refusal names the line.
  • Model uploads with many Python files are no longer rejected with a “corrupted scan” error.
​
PlatformSecure environmentPython SDKCLIDocs
2026-10-06
Platform
  • Keypoint detection PCK scores now reflect real accuracy instead of showing a perfect 1.0 for models that have not learned.
  • Regression experiments no longer show R² as “accuracy”; R² is reported as r2 and accuracy is left empty.
  • Opening Explore or Templates now takes you to the Templates tab of My Use Cases.
  • The leaderboard’s Compute column now shows your team’s real compute use instead of “0.0”, and says which compute numbers are per run and which are team totals.
  • Tooltips, badges, tabs and other labels across the app now use the app’s normal font instead of a serif font.
  • Buttons, tables, avatars, calendars and form fields from the older component set no longer fall back to a serif font.
  • Inference results for language-model and regression experiments now show their own metrics instead of empty F1, precision and recall.
  • Notebook outputs show their table styling again.
  • Secure Environment status cards now show each total in the caption (“12 of 32 GiB used”), and machines without a GPU show four even cards instead of an empty GPU card.
  • Secure Environment now tells you when a machine has no GPU, and that training still runs on CPU, just slower.
  • Secure Environment now says when a machine’s GPU is too old for training and links to the GPU requirements.
  • In Secure Environment, a use case’s dataset on another machine now opens, with a button to switch to that machine.
  • Secure Environment now says when a machine’s NVIDIA driver is too old for training, and how to fix it.
  • In Secure Environment, a use case’s datasets now show which belong to another machine and which were deleted.
  • The Clients page now shows a machine as offline as soon as it stops, matching Secure Environment.
Secure environment
  • The dashboard can now tell you exactly why a machine is running on CPU: an old GPU, an old driver, a GPU that isn’t NVIDIA, or Docker that can’t see the GPU.
  • On Linux with TRACEBLOC_SUBSTRATE=k3s, the installer now sets up your NVIDIA GPU for training through the NVIDIA Container Toolkit, instead of installing in CPU mode.
  • The installer pins k3s v1.36.5+k3s1 (was v1.36.3+k3s1).
  • A machine whose NVIDIA GPU is too old for GPU training (older than Turing), or whose NVIDIA driver is older than the CUDA 12 minimum, now installs in CPU mode and the installer says why, instead of setting the GPU up and failing later.
  • When a machine’s GPU is too old for GPU training, or its NVIDIA driver is too old, the client now tells the platform why it runs on CPU, so the reason can be shown on the client page.
  • Dev edges now give each ingestion run its own short-lived backend token; staging and production are unchanged.
  • The chart’s multi-GPU size floor can also be set as env.TRACEBLOC_MULTI_GPU_MIN_PARAMETERS; env.MULTI_GPU_MIN_PARAMETERS keeps working, and either one now reaches the secure environment under both names.
  • When an automatic upgrade times out or its rollback fails, the auto-upgrade job’s log now says which, what was not ready, and a failed rollback exits with code 3.
  • Automatic upgrades that were interrupted now recover to the last working version and wait for it to be healthy
Python SDK
  • Fetching the starting weights for a template such as sentence-pair BERT now downloads the weights built for that template, not another template’s weights that share the model name.
  • When a model needs an ML framework that isn’t installed, the model check now tells you which extra to install, for example pip install 'tracebloc[pytorch]'.
CLI
  • tracebloc delete removes a native k3s install from its install record (k3s, its files and its host firewall rule), asking sudo once before anything is removed, and tracebloc doctor reads k3s’s own service instead of Docker on such an install.
  • When one image in a dataset has a different size, tracebloc data ingest now uses the size most images share and names the odd image, instead of blaming every other image.
  • When a dataset is too big for one ingest, the CLI now suggests making it smaller (JPEG, lower resolution, fewer samples) or using the Helm chart, instead of splitting it into datasets that can’t be trained on together.
  • When tracebloc data ingest is run with the wrong --task, it now names the task the folder is laid out for (for example “did you mean —task object_detection?”) instead of only reporting a missing labels.csv.
  • When tracebloc data ingest can’t start because a training run is using the secure environment’s CPU or memory, it now names the run that holds them and says to retry when it finishes.
  • The staged copy of an ingested dataset can now be removed by the ingest job once the load finishes, including after --detach, so it no longer uses disk space until the table is re-ingested or deleted.
  • tracebloc upgrade --plain now keeps the installer’s output plain instead of animating a spinner in your terminal.
  • tracebloc delete now removes the cluster your install set up, even when your kubeconfig’s current context points at a different one.
  • Ingesting a keypoint-detection dataset now keeps the per-keypoint Visibility column, so visibility accuracy is no longer scored as if every keypoint were visible.
  • After tracebloc delete, kubectl no longer switches to another cluster in your kubeconfig. It is left with no current context, and the output says so.
  • tracebloc upgrade output is clean in logs and plain terminals (no spinner frames), and it ends by saying which version you are on instead of pointing at a setup step that doesn’t apply.
Docs
  • The setup guide now lists the NVIDIA driver minimums and explains that GPUs older than Turing (for example GTX 10xx, GT 710, V100, P100) install CPU-only.
  • The object detection template now shows the CLI ingest command and explains the class-name and train/test rules that can make an ingest fail or hide a test set.
  • The FAQs now say plainly that model weight updates go to tracebloc for averaging, and the SDK page shows how to fetch pretrained seed weights for model zoo templates.
  • The text dataset template pages now show the exact CLI command for each task, say that files must end in .txt, and list the score formulas you can choose.
  • The training guide now explains object detection model settings and pretrained seeds, and gives the correct daily submission limit.
  • The security page now lists every kind of data that leaves your secure environment, how to opt out of logs and usage events, and the exact hosts the installer needs.
  • The Quick Start and setup guide now explain macOS admin rights, Docker Desktop’s first-run prompts, the installer’s memory offer, and unattended-install credentials.
  • The Operations guide now explains how to move a secure environment to a new machine without revoking it, and warns you to check your kubectl context after deleting the local cluster.
  • The Configuration page now matches the chart defaults: hourly auto-upgrade, egress locked down by default, optional registry credentials, and opt-in PriorityClass.
  • The Amazon EKS guide now matches the current chart defaults, names the right pods and services, and no longer points to a missing setup script.
  • The object detection XML example on the prerequisites page now passes ingestion, and the dataset page link and leaderboard column list match the app.
  • Troubleshooting now shows how to find your namespace, what to do when the client is Offline after a reboot, and the correct pull-secret check.
  • The hyperparameter guide now shows training plan calls that the SDK accepts, with per-task loss defaults and the federated aggregation options.
  • The Define Use Case guide now matches the two-step Create Use Case panel and lists only the scores you can actually pick for each task.
  • The bare-metal, AKS and OpenShift guides now show how to configure CPU-only or GPU nodes, multi-node storage and metrics-server, so training pods schedule on the first try.
  • The tabular and time-series dataset template pages now show the CLI command for each task and the model settings that must match your data, and they say which survival data trains successfully.
  • The CLI reference now explains what to do when a dataset is over the size cap, shows an object-detection ingest, and lists the missing flags and exit codes.
  • The training guide now explains how object detection models are trained and scored, and where image size, batch size and the validation split come from.
  • The segmentation and keypoint templates now explain how class count and coordinate units must line up with training, and show the keypoint CLI command.
  • The security pages now say plainly that model weight updates go to tracebloc after each training cycle.
  • The EKS guide now names the Helm version the upgrade command needs.
​
PlatformSecure environment
2026-10-06
Platform
  • Time series classification and segmentation metrics now agree: F-beta and Jaccard use the same averaging as F1, and Dice matches F1 on binary masks.
  • A TensorFlow experiment that selects FedProx now fails right away with a clear reason, because FedProx’s proximal term is only applied when training PyTorch models, so the experiment would otherwise have run as plain FedAvg.
  • A stored model .zip now loads only its top-level .py source; an archive carrying compiled, native or .pth files, symlinks, directories or unsafe paths is refused on its first delivery with a reason naming the file.
Secure environment
  • Image-classification top-3/top-5 accuracy now shows ”—” (not measured) when the task has no more classes than k, instead of a meaningless 1.0.
​
PlatformSecure environmentDocs
2026-10-06
Platform
  • Time-to-event experiments no longer show a c-index of 0.0 when it can’t be computed; it is reported as not measured, and accuracy is no longer reported for time-to-event.
  • The Secure Environment page shows a machine as offline as soon as it stops checking in, instead of after a delay.
  • The client page shows a stopped client as disconnected within about 3 minutes instead of 30, and a GPU the machine advertises is shown even when its usage isn’t reported.
  • The Secure Environment page shows each machine’s location and specs instead of repeated or unhelpful names.
  • Counts of one on the Secure Environment page now read correctly, such as “1 use case”.
  • The client page now tells you when your GPU is too old to use and that the client is running on CPU, with a link to the list of supported GPUs.
  • Language-model experiments now show perplexity and token accuracy instead of empty precision, recall and F1.
  • Keypoint detection experiments now show their MAE score in the experiment list.
  • The Secure Environment page now shows the machine’s GPU or specs under its name when no host name is known.
  • Notebook outputs no longer apply style blocks embedded in the notebook’s HTML; tables and their contents display as before.
  • On the Secure Environment page, a use case’s datasets now open their own details when clicked.
  • On the Secure Environment page, a dataset now shows how many use cases it is used in.
  • Keyboard users now see a clear focus outline on blog and site controls, and screen readers announce when code or a link has been copied.
  • Links, job details and blog metadata are easier to read, with stronger text contrast across the site.
Secure environment
  • A successful ingest no longer prints an internal telemetry configuration warning, and the note about the temporary upload copy now says how it gets cleared after a --detach run.
  • Ingestion Jobs now also accept TRACEBLOC_MYSQL_PORT, TRACEBLOC_MYSQL_WAIT_SECONDS and TRACEBLOC_BACKEND_TOKEN_SCHEME; the old names keep working.
  • Ingestion jobs pointed at a MySQL server on a non-default port no longer time out waiting for the database before they start.
Docs
  • Code examples and template cards on the website are easier to read, with higher-contrast text in both light and dark themes.
​
PlatformSecure environmentPython SDKDocs
2026-10-06
Platform
  • An experiment whose aggregation hyperparameters include a key the chosen strategy doesn’t accept now fails at once, with a message naming the key and the keys the strategy does accept. Before, it silently ran with the defaults.
  • A scikit-learn or survival experiment that names an aggregation strategy other than FedAvg now fails right away with a clear reason, because those models are merged by their own rule and the chosen strategy would never run.
  • Rejecting cookies is now remembered, so the cookie banner no longer reappears on every page.
  • Sharing a job post on LinkedIn now shares the full page link, and links between blog posts open in the same tab.
  • The dropdowns in the healthcare page’s meeting form can now be opened, browsed and chosen with the keyboard, and screen readers announce them as lists of options.
  • Moving between pages from the site menu is faster, the current page stays highlighted, and the menus now work fully with a keyboard and screen readers.
  • Background animations now respect your reduced-motion setting, and the benchmark setup steps can be paused.
  • Share and LinkedIn buttons on blog posts and job pages now highlight when reached with the keyboard, not only on mouse hover.
Secure environment
  • Training on a GPU the platform cannot use now fails right away with a message naming the GPU, its compute capability, the driver and the supported generations, instead of silently training on CPU.
  • Object-detection experiments using torchvision detectors now show training accuracy as ”—” (not measured) instead of a misleading 0.0.
  • Pausing a training run now stops its pod right away instead of after a 30-second delay.
  • A Windows (Docker Desktop / WSL2) machine with an NVIDIA GPU now reports its GPU to tracebloc instead of showing “no GPU”.
  • When an edge’s GPU is too old to train on, the platform can now show which card it is and why it was left off.
  • The secure environment also accepts TRACEBLOC_MULTI_GPU_MIN_PARAMETERS for the multi-GPU size floor; the existing name and the chart’s values key keep working.
Python SDK
  • The deprecation warning for the old tracebloc_package import no longer says it was removed in 1.0.0, so it now matches what you have installed.
  • Pretrained seeds for zoo templates no longer carry the class head once republished, so a model built for your dataset’s own class count passes model checks with the fetched seed.
Docs
  • Keyboard and screen-reader users can now skip straight to page content, hear the German legal text in German, and no longer hear collapsed blog FAQ answers.
  • The installation docs now use TRACEBLOC_-prefixed names for the installer options (TRACEBLOC_CLUSTER_NAME, TRACEBLOC_STORAGE_MODE, …); the older names keep working.
  • The custom ingestion Job example now authenticates with your secure environment’s backend token and database account, so a Job copied from the docs starts instead of being refused.
​
Secure environment
2026-10-03
Secure environment
  • Linux hosts can now install the tracebloc client on native k3s instead of k3d by running the installer with TRACEBLOC_SUBSTRATE=k3s. It is opt-in and Linux only: k3d stays the default, and macOS and Windows keep k3d. A machine installed on k3s keeps k3s on later runs and upgrades without the variable. Native k3s runs as root, so the installer asks for administrator rights once.
  • Reinstalling on native k3s no longer wipes leftover data when the k3s service’s state can’t be read; it asks you to stop k3s first.
  • Installer options now have TRACEBLOC_-prefixed names, for example TRACEBLOC_STORAGE_MODE, TRACEBLOC_CLUSTER_NAME and TRACEBLOC_PF_MIN_MEM_GB; the old names, such as TB_STORAGE_MODE and CLUSTER_NAME, keep working.
  • The native k3s installer now quotes what k3d reported when its cluster check fails, instead of always asking you to start Docker.
  • On native k3s, the kubelet now holds back memory and CPU for k3s itself and for the host, so a training job can no longer take the memory k3s needs to keep the node running.
  • The installer retries a dropped connection while it downloads its signature files, instead of stopping, and says whether a failure was the network or a release that isn’t signed.
​
PlatformSecure environmentCLIDocs
2026-10-03
Platform
  • Tabular, time-series and survival datasets now show a size in My Data: the size of the file they were ingested from.
Secure environment
  • Training logs now show the last batch of every training, validation and evaluation loop, so a short epoch no longer looks stuck after its first batch.
CLI
  • Links printed after a data ingest now open the web app of the environment you are signed in to.
Docs
  • The training guide now shows how to fetch a model zoo template’s pretrained weights before uploading it.
​
PlatformCLIDocs
2026-10-03
Platform
  • The weights download button on an experiment is now disabled, with an explanation, when the use case doesn’t allow weight downloads, instead of failing after you click it.
  • When a use case doesn’t allow weight downloads, the experiment’s download button now says so and turns off, instead of showing “Model is downloading” and then doing nothing.
CLI
  • After you sign out and back in, tracebloc doctor now warns that this machine isn’t linked to its client and tells you to run tracebloc client create. tracebloc delete and tracebloc client status give the same fix instead of saying there is nothing to offboard.
Docs
  • The Start Training guide now shows how to fetch a model zoo template’s pretrained weights, so pretrained templates no longer silently train from scratch.
​
PlatformSecure environmentPython SDKCLIDocs
2026-10-02
Platform
  • Carbon-intensity figures for your edges keep updating even when the grid data provider sends back an unreadable answer for one region.
Secure environment
  • The secure environment’s services and training pods also accept TRACEBLOC_-prefixed names for their tuning and proxy environment variables (for example TRACEBLOC_HTTP_PROXY_HOST); the existing names keep working, and the chart’s proxy settings in values.yaml are unchanged.
  • Ingestion Jobs now accept TRACEBLOC_-prefixed names for their settings (for example TRACEBLOC_TITLE, TRACEBLOC_BATCH_SIZE, TRACEBLOC_LOG_LEVEL); the old names keep working.
Python SDK
  • The memory warning shown before training starts no longer suggests LoRA or a shorter sequence length for image, tabular and time-series models.
  • Uploading a model zoo template that has pretrained weights, without those weights, now warns that the model will train from random weights and shows how to fetch the weights.
CLI
  • tracebloc --version now prints the CLI version — the same line as tracebloc version.
  • The fix-permissions command printed by a refused data delete now runs to the end when the blocked folders belong to more than one owner.
Docs
  • The dataset ingestion guide now shows the TRACEBLOC_-prefixed setting names; the older names still work.
​
PlatformSecure environmentPython SDKCLIDocs
2026-10-02
Platform
  • Fine-tuning GPT-2-family models with LoRA now adapts the attention and feed-forward layers, so causal language models actually learn instead of staying near their starting perplexity.
  • Training runs on large text models no longer run out of memory and restart while saving their weights at the end of each training round.
  • The client page’s Storage card now shows how much data this client holds and how much space is left for new data, instead of the whole machine’s disk.
  • The Run tab warns before you start training when the plan may need more memory than the client gives each run, and says what to change.
  • You can create use cases for object detection, image, text and tabular classification, segmentation, regression and forecasting again.
  • When you create a use case, the training dataset list shows only datasets that fit the task you picked.
  • When you create a use case, you see its compute budget per participant and where to set an end date before you publish.
  • The “Explore all use cases” button on the home and benchmark pages is back to its compact size on phones instead of stretching across the screen.
  • The pricing plan buttons and the “Try a template” and “Read Docs” buttons on the platform overview are back to their normal size instead of stretching across their card.
Secure environment
  • When a training run fails for good, the client no longer also logs it as restarting.
Python SDK
  • When you start training, the SDK now warns if the run may need more memory than the client gives each run, and says what to change.
CLI
  • tracebloc data delete no longer half-deletes a dataset whose files it can’t remove: it now checks first, leaves everything in place, and prints the one command that fixes the folders.
  • tracebloc data ingest now checks that your secure environment’s data ingestor can handle the dataset before copying anything, and tells you plainly when it needs an upgrade.
  • tracebloc upgrade --help now says plainly that it updates the CLI, and that a healthy secure environment’s services keep upgrading on their own every hour.
  • Warnings and errors from a quiet data ingest are no longer hidden when they arrive in the middle of a progress bar.
  • With —verbose, tracebloc data ingest no longer says you are only seeing warnings and errors while it shows the full log.
  • When the data ingestor is too old for a task, the error now says how it gets updated instead of pointing at a CLI upgrade that doesn’t change it.
Docs
  • The Hyperparameters and Start Training pages now explain that a copied or renamed text model file needs its matching _tokenizer.json copied and renamed too.
​
PlatformSecure environmentCLIDocs
2026-10-02
Platform
  • When you publish a use case you are now its first participant, so you can start training right away without joining it yourself. Your team takes one of the use case’s team slots, and its compute budget comes out of your account’s, as when you join. When no budget or slot is left, the use case still publishes and you can join it yourself later.
  • An experiment’s hardware details no longer go blank when its node briefly fails to report its CPU or memory.
  • Use cases no longer offer scoring formulas that a task’s training runs never report, and formulas built on PCK thresholds or F-beta 0.5 now produce a leaderboard score instead of “Score not computed”.
  • Small usage figures on the Secure Environment page now show one decimal, such as “0.2 Cores used”.
  • The “GPU memory is full” warning on the Secure Environment page now shows how much GPU memory is in use.
  • The Secure Environment page shows “Training Now” on a use case while a training run is in progress.
  • The “Environment offline” banner title on the Secure Environment page now lines up with its button.
  • The Memory card on the Secure Environment page shows the same total in its title and its usage line.
  • On phones, the “View full setup guide” button on the benchmark page is back to its normal size instead of stretching across the screen.
Secure environment
  • Training logs now show a progress line about once a minute (“Batch n of M”), not only the first batch.
  • A training run that fails on your machine now shows as failed instead of staying Running. It is reported once its retries are used up (up to 3 attempts), and the compute every attempt used counts toward your budget.
  • A machine whose hardware details can’t be read no longer shows up with 0 CPU cores and 0 GB of memory.
  • Stopping a training run on your cluster no longer marks the experiment as failed.
  • Stopping a training run no longer shows it as failed, even when your machine is busy.
  • When a training run is killed for running out of memory, the failure reason now says so even if only an earlier attempt was recorded as OOMKilled, and edge logs show every in-place restart.
  • After installing on a GPU machine, the installer now reports how many GPUs the node offers, and warns with a fix when it offers none instead of claiming the GPU works.
  • On Linux, the installer no longer warns about NVIDIA drivers 550 to 569; below 550 its warning now also suggests turning on persistence mode if you keep the driver.
  • Installing on a Mac no longer fails now and then while writing the cluster node’s kubelet settings, and never leaves a half-written settings file behind.
CLI
  • tracebloc data ingest now finds text files that aren’t plain UTF-8, or that mix .txt and .text, on your machine before copying anything.
  • tracebloc data ingest now removes the temporary copy of a run that failed for good, so failed ingests no longer leave data taking up space in your environment.
  • tracebloc delete now removes the PATH line the installer added to your shell profile, and tells you that the k3s/k3d images Docker keeps are left in place and how to remove them.
  • tracebloc data ingest shows one summary per run; the ingestor’s full log appears only with —verbose, while its warnings and errors still show.
  • tracebloc data list now shows the real number of classes for object detection and token classification datasets, matching the web app.
Docs
  • The setup and Operations pages now show your own namespace and say “secure environment” throughout. They also explain that tracebloc upgrade updates the CLI, while your secure environment’s services upgrade themselves every hour.
​
PlatformCLI
2026-10-02
Platform
  • The usage cards on the Secure Environment page now fill the page width on large screens.
CLI
  • tracebloc delete --for-reinstall offboards a machine for a reinstall: it removes nothing unless tracebloc revoked the machine’s credential, and it keeps the CLI, your sign-in and the install logs. --confirm-name confirms an offboard without a terminal.
​
PlatformSecure environmentCLI
2026-10-02
Platform
  • My Use Cases has a new design: tabs for owned and invited use cases, filters, and a guided Create flow. The filter by creator and the oldest/newest sort are gone, and a use case’s cover image is now set afterwards, in its Admin settings.
  • The Memory meter on a secure environment now shows its fullest node, so a node close to running out of memory is no longer hidden behind idle ones.
Secure environment
  • The GPU test command printed at the end of installation now runs on current kubectl versions.
  • Re-running the installer on a GPU host that already has the NVIDIA repository key no longer fails at the container toolkit step.
CLI
  • tracebloc delete removes exactly what the installer recorded on this machine, including a cluster or release under a non-default name, lists the tools it leaves in place, and stops before changing anything when the record can’t be read.
  • Guided prompts accept the option’s name (for example train or a task name) as well as its number.
  • tracebloc client status --context/--kubeconfig/--namespace without --seal now explains what client status and --seal do, and names tracebloc doctor --context … for checking a client on a chosen cluster.
  • tracebloc resources now shows the most CPU and memory one training run can get, instead of 1 CPU · 2 GiB.
  • tracebloc data ingest links the CLI guide and your task’s folder-layout page instead of the Helm data-preparation page.
  • Commands that take --context (data list/ingest/delete, resources, client status --seal, cluster info) now find the tracebloc client on that cluster when it runs in a namespace other than the context’s default, instead of reporting “no tracebloc client found”.
  • The welcome screen now shows your client’s dashboard name next to the name you typed, so the same secure environment no longer appears under two names.
  • When your kubeconfig points at a cluster the CLI can’t reach, data list, data ingest and resources set now say which context and server they tried and that your active client may run on a different cluster (with the --context fix), instead of a bare dial tcp … unreachable error.
  • tracebloc data ingest with every option given on the command line goes straight to the review instead of asking each question again.
  • tracebloc auth status now checks your session with tracebloc: an expired or revoked session shows session expired and tells you to run tracebloc login, instead of signed in.
  • tracebloc data ingest reports a dataset-folder problem, such as a missing labels.csv, right at the path question instead of after the review.
  • When a dataset is over the ingest size limit, tracebloc data ingest now says its size, the limit, and how to get under it.
  • The tracebloc delete confirmation now names the release and namespace it will actually remove, and lists a recorded release it leaves in place.
  • When tracebloc delete can’t finish cleaning up, it now keeps the install record, so whatever is still on the machine stays listed.
  • A guided ingest with every flag supplied now checks the data folder’s layout before the review, and asks for the path again when the folder does not fit the task. It no longer fails after you confirm.
  • tracebloc delete no longer promises to remove a release on a cluster it won’t touch: when your kubeconfig points at another cluster, the preview lists that release as left alone, with the command that removes it.
  • When tracebloc upgrade or tracebloc prepare-host fails, the command it tells you to run keeps the TRACEBLOC_SUBSTRATE you asked for, and prepare-host refuses a value tracebloc doesn’t support before changing anything.
​
Secure environmentCLI
2026-10-01
Secure environment
  • Ingestion now stops before loading anything when any file named in your manifest is missing, and tells you which ones, instead of registering a partial dataset and then failing.
  • Ingesting a masked or causal language modeling, seq2seq or embeddings dataset now warns you about manifest columns that will not be stored, such as a label column, instead of dropping them silently.
  • A JSON dataset with a record that names a file missing from the dataset folder, leaves its filename empty or has no filename at all, is now rejected before anything is ingested, and the message lists the missing files and counts the empty ones. JSON datasets with such records were accepted before.
  • Ingest now rejects a manifest with a blank filename cell up front instead of registering a partial dataset and failing.
CLI
  • A successful ingest no longer ends with a false warning about a leftover temporary copy, and a real cleanup failure now explains in plain words how to remove it.
  • Ingest error messages now name a file by its path inside your dataset folder (for example texts/b77_wrong_ext.text) instead of the internal staging path.
​
PlatformSecure environmentPython SDK
2026-10-01
Platform
  • A client whose node briefly fails to read its CPU or memory details now keeps updating its live resource meters instead of freezing them.
  • An experiment that stops making progress now says so on the Training tab, with the time of its last progress and the reason when one is known.
  • Signing in from a “Create a Use Case” link now opens the create wizard instead of the use case list.
  • The Memory meter on a client’s page now measures the memory in use against the memory available to training, summed over the client’s machines, instead of the busiest machine against its total memory. On a single machine it now reads higher and can reach red. It switches over once every machine of the client reports the new figure.
Secure environment
  • The installer’s Docker memory advice now names one size to set and says plainly what a smaller budget means for training.
  • Automatic client upgrades that fail now roll back cleanly instead of leaving the installation stuck mid-upgrade.
Python SDK
  • The training plan now labels the input length as sequence_length for token classification, masked and causal language modeling, and seq2seq, in both the printed plan and the notebook view.
​
Platform
2026-10-01
Platform
  • When some of a dataset’s clients have been offline for more than 30 minutes, a new experiment trains on the online clients only and records which ones it left out. An experiment on part of a dataset then takes all of its requested rows from the online clients and is billed for all of them, about what it costs with every client online. Client liveness on a use case is shown only to its owner and participants.
  • The use case wizard no longer offers score formulas that a task’s experiments can never compute, so new use cases no longer show “Score not computed” for every submission.
  • An experiment that was just restarted no longer shows as stalled before it has had time to make progress.
  • The people working with your secure environment no longer include you, or a use case’s own owner shown as a pending invite, so the collaborator and invited counts are accurate.
​
PlatformSecure environmentPython SDKDocs
2026-10-01
Platform
  • The use case header now shows your dataset’s name instead of its internal key; hover over the name to see the key.
  • “Create a Use Case” on the Secure Environment page now opens the create wizard directly.
  • A client you created but never installed now shows “Never connected”, with a hint to install it or delete it, instead of “Offline”.
  • The Storage card on the Secure Environment page shows how much space is used, out of the total.
  • The unavailable “Watch how to” buttons on the Secure Environment page now say “Coming soon”.
  • The Secure Environment page’s getting-started steps no longer skip a number on accounts with several machines.
Secure environment
  • A failed data ingestion now reports its error right away instead of being retried twice first.
  • A machine that briefly loses its connection to tracebloc starts reporting usage again on its own, without a restart.
  • On macOS without GNU coreutils, the installer’s Docker, k3d and helm checks now time out as they already do on Linux, instead of waiting forever on a Docker that does not answer.
  • New installs using host-path storage, and every Windows install, now create a single-node cluster, so the machine’s CPU and memory are no longer counted twice.
  • New installs can ingest object detection datasets (images plus Pascal VOC annotations) as the documentation describes.
  • When the installer cannot change Docker Desktop’s memory on a Mac, it now says why and how to allow it.
  • Starting the installer from a folder that no longer exists no longer prints repeated “getcwd” errors.
  • After installing, the summary tells you when kubectl was switched to the tracebloc cluster and how to switch back.
  • On a busy Mac, an install step that stops responding now gives up when its time limit is reached instead of running on past it.
Python SDK
  • A causal-LM, masked-LM, seq2seq, embeddings or text / token / sentence-pair classification plan now defaults to the AdamW optimizer at a constant 5e-5 learning rate (was SGD at 1e-3), and get_training_plan() no longer lists image-only fields for these tasks.
  • LoRA now defaults to rank 8 and alpha 16 (was 256 / 512) when enable_lora(True) is set without set_lora_parameters(...).
  • pip install "tracebloc[pytorch]" now installs the PyTorch 2.13 line that the training images run, so a model checked locally runs on the same PyTorch version as the training job.
Docs
  • The CLI page now documents the current tracebloc data ingest / data list / data delete commands and all 16 tasks, including causal language modeling.
  • The Quick Start now matches the installer: sign in in your browser, no Client ID or password; check the install with tracebloc doctor; uninstall with tracebloc delete.
  • The docs now define the leaderboard metrics for causal language modeling and the other text tasks, list the model parameters for every task, and show a lighter venv setup for the training notebook.
  • Prepare Data now walks you through tracebloc data ingest first, states the 1 GiB / 500 MiB ingest limit, and the image size rules match the templates (any aspect ratio, one size per dataset).
  • The model file reference now lists every task each framework supports, and a broken link on How training works is fixed.
  • The docs now use one name, “secure environment”, for the tracebloc software you run on your own infrastructure.
​
PlatformSecure environmentPython SDK
2026-10-01
Platform
  • An experiment that has made no progress for an hour is now marked as stalled on the Training tab’s data, with the reason (for example, no client holding the data has been online since a given time), and the mark clears by itself when progress resumes.
  • Installs on the quickstart cluster no longer all show the same generic host name in the environment switcher; each one shows its namespace instead.
  • The usage cards on the Secure Environment page now stay evenly sized on wide screens.
  • The Secure Environment page no longer shows “0 GiB” or “0 Cores” for a machine whose size isn’t known yet.
  • The usage cards on the Secure Environment page keep the same size on every screen, and machines without a GPU show an empty GPU card.
  • The environment switcher marks remote machines (servers and clusters) with a “Remote access” tag.
  • In a secure environment’s use case panel, “Visit” next to Leaderboard now opens the use case on its Leaderboard tab instead of the Overview.
Secure environment
  • Storage and other usage readings keep updating on the client page while the cluster’s metrics service is unavailable, instead of the page falling back to “no usage yet”.
Python SDK
  • The training notebook and README now install the current tracebloc SDK (1.2.36 or later).
​
Platform
2026-10-01
Platform
  • The environment status dot on the Secure Environment page now has its light ring, and the status labels are spaced as designed.
​
PlatformSecure environmentPython SDK
2026-10-01
Platform
  • Starting an experiment is now refused when no client holding the dataset has been online in the last 30 minutes, and the message says when it was last seen.
  • Inference is now billed for the CPU its pod actually used. A billing sample could measure CPU over only the time since the last batch, so inference CPU was under-billed before: inference on CPU and GPU machines now costs more, by the CPU share that was missed. Training and validation are billed as before.
  • CPU training images now share one copy of the torch stack, so a machine that runs more than one kind of CPU task downloads and stores it once.
  • The “Explore docs” button on the benchmark page is back to its original size on phones and tablets.
Secure environment
  • A node whose usage can’t be measured no longer shows as idle; its meters show no reading instead.
  • Your client can now report your machine’s own name to the environment switcher instead of a generic cluster node name such as k3d-tracebloc-server-0.
Python SDK
  • User.fetch_seed_weights(template) downloads a model zoo template’s pretrained seed <stem>_weights.pkl, so upload_model(<stem>, weights=True) trains from it.
​
PlatformSecure environment
2026-10-01
Platform
  • The homepage no longer scrolls sideways on phones and tablets.
  • The newsletter form on the home and benchmark pages now shows the server’s validation message when a subscription is rejected.
  • FAQ answers on the platform overview page no longer cut off their last line.
  • Blog posts with code samples no longer scroll sideways on phones.
Secure environment
  • The egress gateway now runs a tracebloc build of squid, ghcr.io/tracebloc/squid-tracebloc:7.6-r0-tb.2 (squid 7.6 on Alpine), pinned by digest and about 11 MB compressed. If you mirror images, copy it into your mirror before this upgrade reaches your edge. If you set egressProxy.image.tag or .repository alone, read MIGRATION.md 1.9.187 first.
  • GPU hosts whose NVIDIA card is too old for the GPU images (compute capability below 7.5) now install CPU-only with an explanation; hosts whose NVIDIA driver is older than the recommended minimum keep the GPU and are told how to upgrade.
  • Chart releases reach the Helm repository again. 1.9.185 was never published there; 1.9.189 is the first published chart that pins the prod control plane.
  • On GPU machines whose package sources do not offer the recommended NVIDIA driver, the installer now installs the newest driver that is available and warns, instead of stopping with an error.
​
Secure environment
2026-10-01
Secure environment
  • A training run that fails while it is being stopped now reports the original error instead of looking like a clean stop.
  • Chart 1.9.185, listed below, was never published to the chart repository. Install 1.9.189 or later.
​
Secure environment
2026-10-01
Secure environment
  • The installer now prints a tracebloc-installer substrate=k3d line at the start of every install and keeps a record of what it created, including the client’s Helm release, in ~/.tracebloc/install-record.json.
  • On macOS, the installer no longer risks waiting indefinitely while it checks a k3d node’s log after a Docker restart; that check now gives up after 90 seconds, as it already does on Windows.
  • Published chart releases pin the control plane of a prod secure environment to the exact image build the release was tested with, so upgrading or rolling back the chart moves the control plane with it. Chart 1.9.185, listed below, was never published to the chart repository; 1.9.189 is the first published chart that does this.
  • On Windows, when the system refuses to start the tracebloc CLI setup, the installer now tries another way to launch it before giving up.
​
PlatformPython SDKCLI
2026-09-30
Platform
  • Audit log entries now record the address a request really came from; a sender can no longer write a false one.
  • Training steps are faster on GPU: FLOPs usage is now measured on a background timer instead of at every batch, and a per-batch GPU memory cleanup that only served that measurement is gone. The GPU part of the bill is now read during training rather than right after a cache flush, so it can be slightly higher; in the experiment it was measured on, the GPU part was about 1% of the bill.
Python SDK
  • The ConvNeXt and Swin object-detection templates now start training from an ImageNet-pretrained backbone instead of random weights.
CLI
  • At a Windows console (Windows Terminal, VS Code or a classic console window), an arrow key at a guided prompt is now refused and asked again; before, down then Enter silently took the default.
  • Pressing Ctrl-C at a question in the CLI now cancels only that question, never work the command had already started.
​
Platform
2026-09-30
Platform
  • Training and inference compute are now billed for the time that actually passes, on CPU and GPU. Each batch used to be charged its own duration plus a fixed 1 second; that fixed second is now the time since the previous batch ended. Experiments whose batches follow each other with less than a second between them are billed less than before, and experiments that wait longer than a second between batches, for example on slow data loading, are billed more.
​
PlatformSecure environment
2026-09-30
Platform
  • Multi-GPU training no longer restarts its data-loading worker processes at every epoch, which removes several seconds per epoch on each GPU. Augmentation stays reproducible and changes every epoch as before.
  • a job whose queue lock is lost now stops at the next edge instead of finishing, publishing and failing to settle; the redelivery lands the cycle.
Secure environment
  • Ingestion now rejects a CSV that is missing a schema column before any table is created, including files with a multi-character separator or a blank first line.
  • Ingesting a dataset whose records overlap an earlier dataset in the same table no longer changes the earlier dataset’s labels or feature values.
​
Secure environment
2026-09-30
Secure environment
  • The installer now stops before creating the cluster if SERVERS is set above 1 or AGENTS above 1, because every extra k3d node counted this machine’s CPU and memory again and let Kubernetes schedule more work than the machine has.
​
Platform
2026-09-30
Platform
  • Model uploads to check-model are now refused with 413 when the request body is over the combined file limits, before it is written to disk. Chunked uploads get 411.
​
CLI
2026-09-30
CLI
  • Guided prompts are now numbered lists you answer by typing a number.
  • The CLI download is about a third of its former size: the installers now fetch a signed, compressed archive. 32-bit Linux and Windows 10 on ARM are no longer supported, and Windows 11 on ARM runs the x64 build.
  • A downloaded install.ps1 now runs under Windows PowerShell 5.1; it had not parsed there since at least v0.10.40. The irm | iex one-liner was not affected.
​
Platform
2026-09-30
Platform
  • Starting an experiment on a dataset that isn’t part of a competition no longer fails with an internal server error.
  • A model’s stored original copy is now security-scanned too, so a check can no longer be hidden from the scan by the entry-point rename.
  • Model names ending in _formatted, _tokenizer or _weights are now refused at upload, because those suffixes name a model’s stored copies.
  • The model-zoo tokenizer fetch now gives up on a slow DNS lookup or slow response headers at its 15 s deadline, and “already being fetched” 503s are logged as warnings instead of errors.
​
PlatformSecure environmentPython SDKDocs
2026-09-30
Platform
  • Averaged models now take integer buffers, such as a BatchNorm layer’s batch counter, from the first participant. Before, legacy uploads got an averaged value, which was then truncated. Rounds that mix legacy and upgraded weight uploads now produce exactly the same model as uniform rounds.
  • The averaging pod now requests 12Gi and is limited to 26Gi memory (was 8Gi/24Gi), and its OpenMP/BLAS thread pools size from its CPU request.
  • When averaging refuses a round because of the uploaded weights (mismatched shapes or parameters, or NaN/inf values), the failure message now uses one wording on every path and names every participant at fault, not just the first one found.
  • The newsletter forms on the blog, career, legal, home and benchmark pages now submit on Enter, show a single accessible result message, and use the design-system input and button.
  • Tapping Sign in or Explore the Hub in the mobile menu now closes the menu, so the next page scrolls normally.
Secure environment
  • An edge’s MySQL upgrade that cannot create its new storage volume now stops within minutes with the storage provider’s reason, instead of waiting out a 15-minute timeout with no explanation; the edge is put back exactly as it was either way.
  • After a Docker restart, re-running the installer now repairs a secure environment whose Kubernetes stopped because its nodes’ network addresses changed, instead of reporting it as running; new environments keep fixed addresses so this can’t happen to them.
  • The log Collector now runs a tracebloc build of the same OpenTelemetry Collector release, ghcr.io/tracebloc/otelcol-tracebloc:0.159.0-tb.2, pinned by digest and about 15 MB compressed. If you mirror images, copy it into your mirror before this upgrade reaches your edge. If you set telemetryCollector.image.tag or .repository alone, read MIGRATION.md 1.9.181 first.
Python SDK
  • When linking a model or starting an experiment fails with “Model File Load Failed”, the SDK now tells you to upload the model again, instead of saying the dataset ID is wrong or fields are missing.
  • When your own MyModel() fails to build during link or start(), the SDK no longer suggests re-uploading the model, which wouldn’t fix it.
Docs
  • The ResNet-50 Faster R-CNN, FCOS and RetinaNet object-detection templates are uploaded without a weight file and train from scratch.
  • The Keypoint R-CNN and Faster R-CNN SPPE keypoint-detection templates are uploaded without a weight file and train from scratch.
​
Secure environment
2026-09-30
Secure environment
  • The client’s credential Secret now also stores each credential under a TRACEBLOC_-prefixed key with the same value; nothing changes for running installs, but if you create or rotate that Secret by hand, set both keys (the security guide shows the command).
​
Secure environment
2026-09-30
Secure environment
  • The hourly auto-upgrade now runs Helm 4. If you mirror images, copy alpine/helm:4.2.3 into your mirror before this upgrade reaches your edge.
​
PlatformCLI
2026-09-29
Platform
  • When the platform’s task queue stops responding, actions that start background work now return an error within seconds instead of hanging.
  • Model zip uploads with an oversized file list are now refused before the server reads that list.
  • Creating an experiment now requires a model you uploaded yourself. A model name from another user, or from an upload made before this release, is refused.
  • The notebook image now ships tracebloc 1.2.28 with CPU torch 2.13.0 and can run as a JupyterHub 5 named server.
  • The platform cluster now has the EBS CSI driver and an encrypted gp3 StorageClass, recorded in the repo after they were applied by hand.
  • Creating or updating a final submission now returns that submission’s id instead of an empty value.
  • CPU training images are about 360 MB smaller to download on a fresh machine.
  • The YOLOv5 and YOLOv8 object-detection templates now train as uploaded; their first training step used to fail with a channel-count error.
CLI
  • tracebloc delete now fully offboards a machine — it decommissions the client, finds the installed release even when your config no longer names it, removes the macOS login autostart, reclaims only the GPU node image, and exits non-zero when anything is left behind.
  • tracebloc delete no longer hangs when the cluster stops answering while it looks up the release to uninstall.
​
PlatformPython SDK
2026-09-29
Platform
  • Model uploads can no longer skip the security scan with # nosec comments, and uploads the scan could not fully read are now refused.
  • Your team’s final-submission list loads faster, and each entry now carries its final-submission id instead of an empty value.
  • A training cycle whose last client reports after the cycle was already closed on time is no longer aggregated a second time.
  • Opening another account’s experiment you cannot access now shows “not found” instead of a server error.
  • CPU training images are smaller to download, by about 67 MB each and more on the NLP image.
Python SDK
  • When a model zoo template fetch refuses to overwrite a file of yours, its message no longer refers to a function you did not call.
​
Python SDK
2026-09-29
Python SDK
  • tracebloc.notebook.render(category, framework, context) renders the Run tab’s training-plan cell for any supported task and framework, from templates that now ship with the SDK.
  • notebook.render treats empty context fields as not set instead of rejecting them, and can read its context from the TRACEBLOC_RENDER_CONTEXT environment variable.
​
PlatformSecure environmentCLI
2026-09-29
Platform
  • The API now lets the web app read Retry-After on busy or rate-limited responses, and the check-model schema types datasetId as the dataset key string.
  • The Run tab’s model zoo list no longer offers templates that are too large for its checks to accept (such as YOLO v1 and Gemma 2), and the notebook can no longer fetch them.
  • Competition leaderboards and experiment sustainability scores for runs since January 2026 now include compute (FLOPs) that was reported late and previously left out.
  • Linking a model to a dataset now works only for a model you uploaded yourself. A model uploaded before this release has to be uploaded again before it can be linked.
  • A zipped model’s stored copy now holds only its Python source files. Folders such as __pycache__ and __MACOSX are dropped instead of being passed on to training.
  • Federated Cox proportional-hazards survival models now predict from the combined coefficients of every participant, instead of matching a single participant’s model.
  • Averaging now runs your model’s code once per cycle instead of up to six times, so a constructor with side effects or random initialization runs once.
  • A loss or score that could not be computed now shows — instead of 0.00, and a real 0 score is no longer hidden.
Secure environment
  • A metric that could not be computed for a training or inference cycle is no longer reported as a score of 0.0, which for error metrics read as a perfect result.
  • Classifiers without probability outputs no longer report misleading AUC, Gini and Brier scores, and margin-based classifiers now report AUC from their decision scores.
  • Data ingestion runs no longer fail at submission with an internal server error.
  • The local database starts using about 235 MiB less memory after its next restart.
CLI
  • When your local secure environment is stopped, tracebloc doctor now tells you to start every stopped node, including worker nodes, instead of only the control plane.
​
PlatformCLIDocs
2026-09-29
Platform
  • When an averaging job fails for a reason that retrying cannot fix, such as mismatched model weights, a model file that does not load, or missing experiment data, the experiment now fails straight away with a clear reason instead of retrying for minutes first.
  • Changing the model or weights after pressing Start no longer silently drops the run; use Start over instead.
  • Switching to an environment that has no use cases yet now says so, instead of showing the first-time “create your first use case” step.
  • If you haven’t set up a client yourself but your team has, the Clients page now says so instead of showing the first-time setup guide.
CLI
  • When your local secure environment stops answering, tracebloc doctor now tells you whether Docker, the cluster, or Kubernetes inside it is down, instead of always saying to start Docker Desktop.
Docs
  • The installation docs now use TRACEBLOC_HOST_DATA_DIR for the data directory and TRACEBLOC_RELEASE_VERSION for pinning the Windows CLI version; the older names keep working.
​
Secure environment
2026-09-29
Secure environment
  • tracebloc prepare-host now grants container access only to the user named on the command line or in TB_PREPARE_USER, and warns when TRACEBLOC_PREPARE_USER is set.
​
Platform
2026-09-29
Platform
  • FedAdam, FedYogi and FedAdagrad experiments keep half as much optimizer state per experiment, and experiments already running continue where they left off.
  • Opening a use case directly on its Run view no longer makes the page flash or reload its content.
  • The filter chips on the Use Cases page now match the design’s width, and creator initials use the design’s text colour.
  • In the Create Use Case wizard, Next and Create stay disabled until the required fields are filled, and a required field you skip is highlighted.
  • Button text in the Create Use Case wizard, the “add a dataset” drawer and template cards is now vertically centred.
  • Leaving the Run drawer while a model is still validating no longer leaves a notebook kernel running in the background.
  • The page behind the “You need a dataset” panel is now dimmed, like the Create Use Case wizard.
  • Leaving the Run tab while a notebook is still starting no longer leaves a Python kernel running in the background.
​
PlatformDocs
2026-09-29
Platform
  • Carbon (gCO2) totals no longer count the same training time twice when several edges report an epoch at once.
  • Loading a model zoo template’s tokenizer in a notebook now fails within seconds with a retryable error when the zoo is unreachable, instead of hanging for minutes or silently finishing without the tokenizer.
  • Model uploads now accept only Python source files (.py, or a .zip of them). Other file types in an upload are refused, not loaded.
  • A retried averaging job for a cycle that already finished now republishes that cycle’s result instead of averaging it again, so the model and its privacy noise do not change on a retry.
Docs
  • Blog posts no longer scroll sideways on phones.
​
Python SDK
2026-09-29
Python SDK
  • User.fetch_zoo_template(template_id, dataset) downloads the model zoo template you picked in the Run tab into the notebook’s working directory.
  • Installing the SDK no longer adds stray tests and scripts packages to your Python environment, which could shadow your own project’s tests package.
  • Starting a training run from Python now gives back the run’s id, and a refused run raises an error you can catch instead of only printing it.
​
Platform
2026-09-29
Platform
  • Anonymous visitors and other accounts no longer see a use case’s edge device identity, location, status or node names; they get the compute summary only.
  • The Run tab’s Fits check now runs on the backend using the uploaded model’s own declared values. The browser sends only the dataset and the model name.
​
PlatformCLI
2026-09-29
Platform
  • The model list on the Run tab refreshes after a platform update and is never reused for a different user signing in on the same browser.
  • GPU memory limits for edge compute now always show on the compute card instead of intermittently appearing empty.
  • Checking a model against, or launching an experiment on, a public dataset shared with several users no longer fails with a server error.
  • Competition leaderboards and experiment sustainability scores now include compute (FLOPs) reported after an inference or training cycle was processed, instead of freezing at the value seen at that moment.
  • Compute a team uses while it is merging into another team is no longer dropped from the merged team’s budget.
  • Training rounds no longer stall waiting for results when the platform is updated while a round is finishing.
  • Leaving the invitation page before its automatic redirect no longer pulls you back to the login or use-case page a moment later.
CLI
  • The CLI installers now also accept TRACEBLOC_RELEASE_VERSION, TRACEBLOC_INSTALL_PREFIX and (Linux/macOS) TRACEBLOC_COSIGN_VERSION; the existing unprefixed variables keep working.
​
PlatformSecure environmentPython SDK
2026-09-29
Platform
  • The filter chips on the Use Cases page now line up exactly with the design.
  • Opening an invite link you already accepted now takes you to log in and then to the use case, instead of stopping on a blank “Thank you” page.
Secure environment
  • The installer accepts TRACEBLOC_NAMESPACE to name your secure environment; TB_NAMESPACE keeps working until the end of 2026.
Python SDK
  • User(stage=...) now also accepts the short stage names dev, stg and prod; every stage name that worked before still points at the same place.
​
PlatformPython SDK
2026-09-28
Platform
  • TensorFlow models averaged by the platform are now saved in the numeric precision your edges trained in (for example float32), instead of being widened to float64.
  • Averaging rounds that can never succeed, such as edges that trained different model shapes or reported zero training samples, now fail right away with the reason, instead of retrying for many minutes first.
  • Averaging jobs sent with invalid input now fail right away instead of being retried several times first.
  • FedAdam, FedYogi and FedAdagrad training no longer restarts its optimizer momentum when the number of clients contributing to a round changes, on models with integer buffers such as BatchNorm layers.
  • Moving between use cases no longer leaves the Run drawer open, or closed, from the previous use case.
  • Confusion matrices for completed training cycles now display in the experiment view instead of staying on “not ready yet”, with normalized values rounded for readability.
Python SDK
  • fetch_model now fails with a clear error when a download is cut short, instead of saving a truncated weights file that breaks training later.
​
PlatformPython SDK
2026-09-28
Platform
  • The averaging service no longer restarts when a message’s lock is lost mid-job; it logs the failure and carries on with the next job.
  • A training cycle whose results list the same edge more than once now fails with a clear error naming that edge, instead of silently counting it twice in the global model and its metrics.
  • Multi-participant scikit-learn and survival-model experiments no longer fail an averaging round when the formatted model file is missing or can no longer be imported. Averaging only needs the participants’ trained models.
  • An averaging job for a framework the platform doesn’t support now fails immediately with a clear error listing the supported frameworks, instead of appearing to succeed with a meaningless result.
  • A mistyped or invalid aggregation hyperparameter no longer runs silently. An invalid value (wrong type, a fraction where a whole number is needed, true/false, or out of range) now fails the experiment’s averaging step immediately with a message naming the setting. An unrecognized setting name is logged as a warning for now and will be refused in a later release.
  • A training cycle with only one participant is no longer published when that participant’s model weights are invalid (NaN/infinite), and Krum aggregation now refuses a one-participant cycle as it already refuses a two-participant one.
  • Aggregated confusion matrices built from normalized (fractional) values now show their real values instead of being rounded down to zeros. A cycle whose edges report mismatched class labels now fails straight away with a message naming the edge, instead of being retried repeatedly first.
Python SDK
  • User.fetch_model(model_name) downloads the model you uploaded from the Run tab into the notebook’s working directory.
​
Python SDK
2026-09-28
Python SDK
  • tracebloc.checks.trains(model_name) runs the same local training step as upload_model without uploading anything, and returns (and in a notebook shows) whether the model trains, and how long it took.
​
Platform
2026-09-28
Platform
  • When linking a model to a dataset fails for more than one reason, the Run tab now lists every reason instead of only the first.
  • Averaging no longer fails for models with BatchNorm or other scalar tensors while some edges have been upgraded to the new weight format and others have not.
  • The Create buttons on the Use Cases page now show a spinner and stay disabled while we check your datasets, so a click is never left without feedback.
  • Button text is now vertically centred, including the Filters Reset and Apply buttons.
​
Secure environment
2026-09-28
Secure environment
  • The Windows installer now adds its tools folder to the system PATH without turning %SystemRoot%-style entries into fixed paths.
  • Logs from the automatic upgrade and image-refresh jobs are no longer lost when the telemetry collector restarts while one of those jobs is running.
  • Installs that set fullnameOverride are now recognised as healthy by the installer: re-running it no longer reinstalls a working client, and the support bundle includes the client’s logs.
  • On a machine too small to run training beside the platform, the installer now stops before creating anything, so there is no empty cluster left to delete.
  • The installer now recognises an existing client installed with a custom name override when its id is stored only in the cluster Secret.
  • Support bundles now include the resource monitor’s logs.
  • Upgrading to chart 1.9.158 or later: MySQL now requests 1 GiB, and training runs on an edge sized to its old maximum start slightly smaller instead of being refused. See MIGRATION.md.
​
PlatformSecure environment
2026-09-28
Platform
  • Federated runs using FedAdam, FedYogi or FedAdagrad on models with batch normalization no longer lose their optimizer momentum when participants move to the new weight format, so convergence continues where it left off.
  • Clicking Create a Use Case with no datasets now shows how to add your first dataset, instead of a wizard you can’t finish.
  • The page behind the Create Use Case wizard is now dimmed, so the wizard stands out.
  • When a use case can’t be created, the field that needs changing is now highlighted in red next to the error message.
Secure environment
  • After an upgrade that grows the platform’s own memory use, training runs on an edge that was sized to its old maximum now start with a slightly smaller envelope instead of all being refused.
​
Platform
2026-09-28
Platform
  • The environment switcher now shows when each machine was last seen, its host name, where it runs, and an icon for the kind of machine.
  • The Use Cases dashboard now keeps its designed width on wide monitors, instead of stretching cards across the whole screen.
​
Python SDK
2026-09-28
Python SDK
  • Training-plan submissions that include LoRA seed, update-set, delta-transmission or ZCA-epsilon settings are no longer rejected by the duplicate-plan check.
​
Docs
2026-09-28
Docs
  • The changelog and the training guides now say “secure environment” wherever they used to say “edge”.
​
Docs
2026-09-27
Docs
  • The changelog now says “experiment” wherever older entries said training run, training job or training plan.
  • The 0.12.0 changelog now tells you to set TRACEBLOC_AMP=false in your secure environment to keep full fp32 training.
​
Secure environmentCLI
2026-09-27
Secure environment
  • When a training image cannot be pulled, the failure now says why: the registry’s rate limit, access refused, a missing tag or repository, or the registry being unreachable. It no longer shows a generic pull error.
CLI
  • tracebloc login now accepts staging, development and production as the environment name (from --env, TRACEBLOC_ENV or CLIENT_ENV), the same spelling the client installer uses, and says which source it read when it rejects a value.
​
Secure environment
2026-09-27
Secure environment
  • Training jobs on production workspaces now run one exact, verified engine image for every cycle of an experiment, and nodes that already hold that image no longer download it again.
  • Values set under env.RESTRICTED_DNS_NAMESERVER or env.RESTRICTED_DNS_CLUSTER_DOMAIN no longer change where training and ingestion pods resolve DNS; configure the restricted resolver through restrictedDns.clusterIP and restrictedDns.clusterDomain.
​
CLI
2026-09-27
CLI
  • The largest training run tracebloc resources set max offers is up to 320 MiB smaller, because tracebloc now keeps 2.06 GiB of memory for itself instead of 1.75 GiB. On an 8-core / 32 GiB machine the maximum goes from 30 GiB to 29 GiB.
​
PlatformDocs
2026-09-26
Platform
  • API requests no longer fail with a 502 error while the platform is being updated.
  • An experiment that cannot start its next training cycle, for example because the team’s compute budget is used up, now shows as paused with the reason, instead of staying “running” with nothing happening.
  • The homepage’s Regulatory & compliance testing card now opens with a compliance scenario instead of copy from another card.
Docs
  • The install commands on the benchmark page and in the LLM benchmarking blog post now work; before, they downloaded an error page instead of the installer.
​
Platform
2026-09-26
Platform
  • Several blog posts now show working links, accurate image descriptions, corrected typos, FAQ sections without repeated questions, and comparison images that had been broken.
​
PlatformSecure environmentPython SDKCLI
2026-09-26
Platform
  • LoRA fine-tuning can now upload only the trained adapter factor instead of the whole model, cutting per-cycle upload size for large models while producing the same combined model.
  • Object detection models that state their own model type are now trained with the detection family they name, instead of one guessed from the model’s name.
  • LoRA fine-tuning on the B-only wire can now download each cycle’s model as the trained adapter update instead of the whole model, when your secure environment still holds the previous cycle’s model, with the same combined model as a result.
  • Refreshed the Use Cases page — updated spacing, hover states, filters, search, empty states, and a redesigned first-time setup screen.
  • Opening or copying the “Pricing” link from the navigation on any page now takes you to the homepage’s pricing section, instead of a broken link.
  • The Healthcare industry page is easier to use with a screen reader: each chart now has its own description, the two use-case cards no longer repeat the same bullet, and the mobile view no longer has a duplicate page heading.
  • Fixed a dead button on the benchmark page, corrected an outdated FAQ date, and made the demo video’s play button work properly with keyboard and screen readers.
  • The FAQ on the platform overview page can now be opened and closed with the keyboard.
  • Some Privacy Policy section links changed, and footer placement across pages is now corrected for accessibility.
  • Share, apply, and browse buttons across job and template pages now work with keyboard and screen reader navigation, not just a mouse click.
Secure environment
  • With mixed precision switched on, training uses bf16 on Ampere and newer NVIDIA GPUs and on CPUs with AMX, and fp16 with a gradient scaler on older GPUs (T4, V100); other CPUs stay in fp32. Mixed precision is off by default, so a secure environment trains in full fp32 unless you set TRACEBLOC_AMP=1 in it.
  • LoRA training no longer fails at setup when the experiment leaves the LoRA seed blank; the experiment’s shared seed is used instead.
  • An experiment now prints a one-line warning when logging to Application Insights cannot be attached after sign-in, instead of going quiet with logging degraded.
  • XGBoost models that use an evaluation-monitor callback now complete federated training, averaging included, instead of failing to load.
  • Retrying a dataset ingestion after restricted DNS was switched on no longer silently reuses an earlier job that cannot reach the database; it now reports a conflict instead.
  • The installer no longer tells you to delete your cluster without first releasing its secure environment, so recreating or switching a cluster stops leaving stale records on your dashboard.
  • Training and data-ingestion pods can now be confined to an in-cluster DNS resolver that answers only cluster names and an explicit allowlist, so they cannot look up names outside your cluster. It is opt-in and off by default.
Python SDK
  • When linking a model fails because the dataset’s label information is missing or unreadable, you now see the reason instead of an empty error.
  • The dataset size shown when you link a model now counts dataset records instead of individual labels, so token-classification and object-detection datasets no longer report an inflated total.
  • Fixing a rejected custom loss with a valid loss_function() call now lets start() run instead of reporting invalid training parameters.
  • Sessions on a non-production deployment, including the in-app notebook, now show the server’s actual error message when a sub-dataset or experiment request fails.
CLI
  • tracebloc client create now warns you when your machine’s previous secure environment will be left behind on your dashboard, and tells you how to release it first next time.
​
PlatformSecure environment
2026-09-25
Platform
  • Probability metrics (AUC-ROC, AUC-PR, Gini and Brier) now appear for scikit-learn tabular classification experiments in secure environments that saw only some of the dataset’s classes, where they were previously left out of the results.
  • LoRA fine-tuning now shares one fixed adapter factor across every data source and trains only the other one, so an experiment updates about half as many adapter parameters and its results combine cleanly across sources.
  • LoRA fine-tuning of NLP models now updates only the adapter weights, instead of silently retraining the whole base model.
  • An experiment that asks for more GPUs than any single machine can provide is now refused right away, with a message naming the limit and how to stay within it, instead of waiting indefinitely with no explanation.
Secure environment
  • Training results and FLOPs relayed from your cluster no longer stall or drop while the messaging credential rotates.
  • Setting an image block to null in the client chart values no longer fails the install or upgrade; the chart’s default image is used instead.
​
Secure environment
2026-09-25
Secure environment
  • A dataset with a column name containing %( or %s (for example a%(b)s or %share) is now rejected before ingestion starts, with an error naming the column. Before, the table was created and every row then failed to insert, or the dataset ingested and training on it failed. A % followed by anything else (for example %rate or 50%) still works.
​
2026-09-25
No customer-facing changes. This release carried internal test changes only; every component is at the same version as in 0.8.0.
​
PlatformSecure environment
2026-09-25
Platform
  • When the service is briefly at capacity, a request now comes back as a clear “too many requests” with a Retry-After hint to wait on, instead of an unexplained gateway error.
  • Banners and toasts on the Secure Environment and Run screens have a refreshed, flatter look.
Secure environment
  • A brief database connection drop at the end of an ingest no longer deletes the rows that were just ingested; the ingestor retries and registers the dataset.
  • A CSV with a repeated column name is now rejected even when it uses a multi-character or regex separator. Before, it could ingest with its columns silently mismatched.
  • A classification dataset with a blank label cell now fails the preflight check and lists the affected rows. Before this, the dataset registered with an extra, empty class.
  • An ingest stopped by the platform (eviction, node drain, deadline) while it was registering its dataset no longer leaves its rows stuck; the next ingest into the same table cleans them up.
  • A semicolon-delimited (or other non-comma) labels file is now validated correctly for image classification, keypoint detection, all text categories, embeddings and time-to-event prediction. Such files were previously rejected at preflight, or passed a text-content check without any file being checked.
  • A re-ingest into an existing table can no longer delete an earlier dataset’s rows when it fails, the reported record count now matches the rows actually stored, and duplicate keys that collapse into one row are shown as “Collapsed Duplicates” in the summary.
​
PlatformSecure environment
2026-09-24
Platform
  • Flash messages are flatter and more compact — the coloured bar down the left edge is gone and the spacing is tighter.
  • Long tooltips now wrap onto a second line instead of stretching across the screen.
  • Code blocks now use a monospace font, and the one-line command box has slightly taller line spacing.
Secure environment
  • File copies during an ingest now run 4 at a time by default, which is where throughput on network storage such as EFS stops improving. Set TRACEBLOC_COPY_WORKERS (1–16) to tune it; 1 copies one file at a time.
  • The dashboard’s machine switcher now shows each machine’s name and whether it is a workstation, a GPU server or a server.
  • On Windows, the installer now retries the tracebloc CLI install when Windows refuses to start it, and the final summary tells you if the CLI still needs installing.
​
Docs
2026-09-24
Docs
  • The docs now list the Python packages a model file may import for each task, and explain what an upload refusal means.
​
Platform
2026-09-24
Platform
  • When a model upload is refused for importing a package its task does not allow, the message now links to the page listing every task’s allowed packages.
​
PlatformSecure environmentCLI
2026-09-24
Platform
  • The environment switcher now shows each machine’s hardware (GPU or CPU) under its name.
Secure environment
  • Your secure environment now reports its host name and machine type, so the dashboard’s environment switcher can show which machine each one runs on.
  • Large tabular datasets ingest about twice as fast, because rows are written to the database with far less per-batch overhead.
  • A semicolon-delimited time-series (or tabular) CSV declared with csv_options.delimiter now passes the pre-ingest checks instead of failing with “Required column ‘timestamp’ not found” or “Schema columns not present in CSV”.
  • An object-detection ingest now fails, instead of completing with a missing annotation, when an annotation file is removed after validation.
CLI
  • The tracebloc resources set wizard’s “Use as much as possible” option now refuses a ceiling no experiment could be admitted at, exactly as tracebloc resources set max does, instead of saving it.
​
Platform
2026-09-23
Platform
  • If your account can’t be loaded because of a connection problem, protected pages now show an error with a “Try again” button instead of a blank screen.
​
Secure environmentDocs
2026-09-23
Secure environment
  • The release page of every chart version on the public mirror now links the tracebloc changelog entry that release belongs to.
Docs
  • The docs now have a changelog page at /changelog that lists every tracebloc release that changes something for you, with an RSS feed to follow it.
​
PlatformCLI
2026-09-23
Platform
  • The Secure Environment page has a refreshed header showing your environment’s namespace and status, and ended or archived items are now clearly greyed out.
CLI
  • The release page of every CLI version on the public mirror now links the tracebloc changelog entry that release belongs to.