Skip to main content
Everything you do after your secure environment is running. The Helm and kubectl commands below use <namespace>: your secure environment’s namespace. The one-line installer names it after your client and gives the Helm release the same name, so <namespace> is both. tracebloc cluster info shows it. For a manual Helm install, it’s the namespace and release name you chose.
Day-to-day management — inspecting the cluster, ingesting and removing data — is easiest with the tracebloc CLI. The Helm/kubectl commands here cover the lifecycle bits the CLI doesn’t (upgrade, stop/start, uninstall) and deeper debugging.

Which context kubectl and Helm use

Every kubectl and helm command on this page that reaches your secure environment names tracebloc’s context: kubectl --context k3s-tracebloc … and helm --kube-context k3s-tracebloc …. A command that names its context reaches your secure environment whatever else your kubeconfig points at. The installer adds that context to your kubeconfig, the first file KUBECONFIG names, else ~/.kube/config. It is k3s-tracebloc, or k3s-<name> if you installed with TRACEBLOC_CLUSTER_NAME: write that name in place of k3s-tracebloc below. On a machine an older installer set up on k3d, it is k3d-tracebloc (k3d-<name>). To list the contexts in your kubeconfig, run kubectl config get-contexts. On Linux, kubectl is k3s’s own, and with no KUBECONFIG set it reads k3s’s own file, which has no k3s-tracebloc. If your shell had no KUBECONFIG when you installed, the install summary printed an export KUBECONFIG=… line that names the merged file: run it, and add it to your profile to keep it. For a Helm install into a cluster of your own, name that cluster’s context in its place.

kubectl and Helm on Windows

On Windows, kubectl and helm are inside tracebloc’s WSL distro, not on Windows: the installer puts neither on Windows. Run a kubectl command from PowerShell with wsl -d tracebloc -- in front of it. In the distro, k3s’s own kubeconfig, /etc/rancher/k3s/k3s.yaml, holds only your secure environment’s cluster, and its context is named default, not k3s-tracebloc. For a command you run in the distro, write default where this page says k3s-tracebloc:
Helm in the distro needs k3s’s kubeconfig named:
For a command with a pipe, such as the Secret reads under Move to another machine, open the distro’s shell with wsl -d tracebloc, run export KUBECONFIG=/etc/rancher/k3s/k3s.yaml, then type the command as it is on this page, with default in place of k3s-tracebloc. The installer also adds its context, k3s-tracebloc, to your Windows kubeconfig (%USERPROFILE%\.kube\config) and makes it current, so a kubectl you installed on Windows yourself reaches your secure environment too: name it with --context k3s-tracebloc.

Which version am I on?

The install summary also prints the version. For support, tracebloc doctor --diagnose writes a redacted support bundle, tracebloc-doctor-<timestamp>.txt, to the current directory; it includes the chart version.

Health & status

For deeper debugging:
Then check My Secure Environment: your secure environment should read Active now under its name.

Environment status

The Environment status tiles on My Secure Environment show what your secure environment reports. It sends a reading every minute. When several machines run it, each tile shows the busiest one. The line under CPU, GPU and Memory covers the last 3 days. Each point is the peak of its period, not the average, so a short burst shows as a spike. Runs counts an experiment only while its training is running on your secure environment. One that is queued, waiting for a free GPU, or between cycles is not counted, so the tile can read None running while the Training tab lists the experiment as training. On an older release, a training on a Mac’s GPU is not counted either: upgrade to count it. A tile reads No reading when your secure environment is offline or sent nothing for 10 minutes.

Logs

Stop & start (local machine)

Free up your machine without losing anything — data persists between stops. On Linux, native k3s runs as the k3s system service:
On a Mac, k3s runs in a VM in Apple’s container runtime (see Mac prerequisites). The login item io.tracebloc.node checks every minute and starts the node again, so unload it first. Then stop the node and the container system with the runtime they run on:
If launchctl print finds no com.apple.container.apiserver, the container system and the node are already stopped. The node is named after kubectl’s context, k3s-tracebloc, or k3s-<name> if you installed with TRACEBLOC_CLUSTER_NAME. To start again, load the login item. It starts the container system, then the node:
The stop lasts until you log in again: the login item loads at every login and brings the node back. On a machine an older installer set up on k3d, k3d cluster stop tracebloc and k3d cluster start tracebloc stop and start it (with your TRACEBLOC_CLUSTER_NAME in place of tracebloc, if you set one). Moving such a machine to k3s is a reinstall: see Linux prerequisites and Mac prerequisites.

Upgrade

The auto-upgrade CronJob keeps your secure environment current by default: every hour it checks for a new release and upgrades the services. To update the CLI now:
It re-runs the official installer (signature-verified) and brings the CLI to the latest release. On Linux and on a Mac it runs the full install, which also brings the services up to date. On Windows it prints the command to run in a new PowerShell window. Safe to re-run. On a machine an older installer set up on k3d, it does not update the secure environment: the installer stops and changes nothing. Moving that machine to k3s is a reinstall: delete its secure environment with tracebloc delete, then run the installer again (see Linux prerequisites and Mac prerequisites). To upgrade the services right away, use Helm:
--reset-then-reuse-values preserves the values the installer applied. It needs Helm 3.14 or later (Helm 4 works; the installer installs Helm 4). Append --version <x.y.z> to pin a specific release.

Roll back

Move to another machine

Your secure environment’s identity is its Client ID, not the machine. To relocate, reuse the old host’s credentials:
  1. On the old host, read the Client ID and its password from the secure environment’s Secret:
  2. On the new host, pass them to the installer. Without them, the installer signs you in through the browser and creates a new secure environment instead:
  3. Re-ingest your datasets on the new host. Ingested data doesn’t move with the credentials, and copying ~/.tracebloc across doesn’t carry it over.
  4. Once the new host shows Online, remove tracebloc from the old one with the steps under Delete it by hand. Don’t revoke it afterwards: the new host uses the same credential.
Don’t run tracebloc delete on the old host. It revokes the machine credential first, and that is now the credential the new host uses, so the new host goes offline too.

Uninstall

To remove tracebloc from this machine, run:
It shows what it will remove and keep, then asks you to type the name of your secure environment to confirm. --yes skips the confirmation (for automation). With a secure environment linked to this machine, it needs you signed in, since its first step revokes the machine credential on tracebloc. On a machine with no secure environment linked, see With no secure environment linked.
  • Removed from this machine: the machine credential is revoked, the Helm release is uninstalled, the local cluster is deleted, the tracebloc container images are reclaimed, ~/.tracebloc is cleared, and the tracebloc CLI removes itself.
  • Kept on tracebloc: your use cases, the catalog entries of your datasets, and the models trained here.
  • Left in place: NVIDIA drivers. On a machine an older installer set up on k3d, also Docker, kubectl, k3d and helm. Remove them yourself if nothing else uses them.
  • On Windows: the logon task tracebloc-keepalive is removed, then tracebloc’s WSL distro with everything inside it and its folder, %LOCALAPPDATA%\tracebloc\wsl. The CLI, and the entry its installer added to your Path, go once the command exits.
  • On Linux: k3s itself is removed too, through k3s’s own uninstall, so tracebloc delete asks for administrator rights once and lists every command it runs as root before it runs any. A k3s that an administrator prepared for you stays; removing it takes that administrator. --keep-data copies your volumes to ~/.tracebloc/kept-data/<time>/ first.
This can’t be undone. Datasets stored on this machine are erased. tracebloc delete --keep-data keeps ~/.tracebloc (local config and any datasets stored there on the host), but datasets kept inside the cluster go with it. On Linux, --keep-data copies them out first (see above). On a Mac set up with version 1.12 or later, the datasets, logs and resume points training uses are in ~/.tracebloc/job-data, and --keep-data keeps that folder. A Mac set up before version 1.12 has no such folder: its datasets are in the node’s volume, which goes with the node, so --keep-data keeps none of them. The delete preview lists what it keeps.
tracebloc delete refuses while a training runs or waits to run on your secure environment, and names each experiment it belongs to (use case, start time and cycle). Wait for them to finish, or pass --force to stop them. An experiment can stay listed as running after its training on this machine has ended, for example when the training ran out of memory. Such an experiment doesn’t block tracebloc delete: it warns, names the experiment, says nothing is training there, and goes on to the confirmation. Deleting stops nothing for such an experiment. It stays listed on tracebloc until its owner stops it. When neither your secure environment’s report nor the machine it runs on says what is training (for example, it is offline and unreachable), tracebloc delete refuses while tracebloc lists any experiment as running. It targets your secure environment as recorded at install time, not kubectl’s current context, and it refuses if what it reaches is not your secure environment. Pass --context only if the install record is missing.
After tracebloc delete, check kubectl config current-context before you run another kubectl or helm command: when the deleted cluster’s context was the current one, kubectl has no current context. On a machine an older installer set up on k3d, k3d cluster delete removes the cluster’s context and may make another cluster in your kubeconfig current, such as a remote or production cluster. Switch with kubectl config use-context <context>, or clear it with kubectl config unset current-context.
helm --kube-context k3s-tracebloc uninstall <namespace> -n <namespace> removes only the release (on a machine an older installer set up on k3d, k3d cluster delete tracebloc removes the cluster; check kubectl config current-context afterwards, as the warning above says). The machine credential stays live, so your secure environment stays registered on your dashboard, and the tracebloc CLI and ~/.tracebloc stay on disk. Prefer tracebloc delete.After a helm uninstall, PVCs are annotated helm.sh/resource-policy: keep, so your data survives. To remove it too: kubectl --context k3s-tracebloc delete pvc --all -n <namespace>.

With no secure environment linked

Signing out clears this machine’s link to its secure environment along with your sign-in, and a machine where you never created a secure environment has no link. tracebloc delete then reads the install record, needs no sign-in and asks tracebloc nothing:
  • The install record lists a secure environment: it still runs here, so tracebloc delete refuses and removes nothing. Only a delete with the link can check for running trainings and revoke the machine credential. Signed out, it names the way forward: tracebloc login, then tracebloc client create to relink it (it adopts the secure environment already here), then tracebloc delete again. Signed in, it stops with no active secure environment on this machine and names the same relink.
  • No secure environment here: it removes what is left on this machine: ~/.tracebloc (kept with --keep-data), the tracebloc CLI and the entry its installer added to your PATH, on a Mac the login item older installers added (io.tracebloc.runtime), and on Windows %LOCALAPPDATA%\tracebloc. It revokes nothing. To confirm, type tracebloc, since there is no secure environment name; --yes and --confirm-name tracebloc work as usual. Signed out, it ends with a reminder: if a secure environment of yours ran here, revoke it from the dashboard, or with tracebloc client remove as under Delete it by hand.
  • No install record, but tracebloc still runs here: on Windows, tracebloc’s WSL distro or its tracebloc-keepalive logon task is still there; on Linux, k3s is. tracebloc delete refuses and removes nothing, since it removes them only as a record lists them. Run the installer again (on Linux, tracebloc upgrade), which writes the record, then tracebloc delete again, or delete it by hand.

Delete it by hand

When tracebloc delete refuses and you can’t take the way forward it names, for example because the install record is gone or the machine can’t be relinked, these commands remove what the one-line installer put on this machine. They also finish what a tracebloc delete with no secure environment linked leaves: the k3s-tracebloc entries in your kubeconfig, and on a Mac any node VM, login items and ~/Library/Application Support/tracebloc still there. They erase your secure environment’s datasets, as tracebloc delete does. The install record, ~/.tracebloc/install-record.json (%USERPROFILE%\.tracebloc\install-record.json on Windows), lists everything the installer made here: read it first, and skip a line whose item it doesn’t list. The names below are the defaults: with TRACEBLOC_CLUSTER_NAME set, the context is k3s-<name>, and with TRACEBLOC_INSTALL_PREFIX set, the CLI is in that folder.
Run these in a normal PowerShell window, as the user who installed tracebloc. None of them needs administrator rights.
The k3s-tracebloc context, cluster and user stay in your Windows kubeconfig, %USERPROFILE%\.kube\config. Remove them with a kubectl you installed yourself (kubectl config delete-context k3s-tracebloc, then delete-cluster and delete-user with the same name), or leave them. On a machine an older installer set up, C:\Program Files\tracebloc\bin may also be on the system Path; removing it takes an administrator.
Deleting by hand doesn’t revoke the machine credential: your secure environment stays on tracebloc and reads Offline. Your use cases, the catalog entries of your datasets and the models trained here are kept, as with tracebloc delete. To revoke it as well, sign in on any machine with the CLI: tracebloc client list shows your secure environment’s id, and tracebloc client remove <id> revokes it and keeps its history. If it refuses because it can’t confirm that secure environment isn’t on the machine you run it from, and its machine is gone, pass --force. Skip this if you moved it to another machine: the new host uses the same credential.

Back up

Your data lives in the secure environment’s PVCs: the dataset PVC and the MySQL PVC, which holds the ingested tables (kubectl --context k3s-tracebloc get pvc -n <namespace> lists them). Back up those volumes on your normal schedule — tracebloc keeps nothing of yours off your infrastructure. On a default Linux install, those volumes sit in k3s’s storage directory, not in ~/.tracebloc (see Linux prerequisites). On a Mac set up with version 1.12 or later, the datasets, logs and resume points training uses are in ~/.tracebloc/job-data, and the rest sits in the node’s volume. On a Mac set up before version 1.12, all of it sits in the node’s volume (see Mac prerequisites). To back them up as a host directory, install with TRACEBLOC_STORAGE_MODE=hostpath: your data then lives under ~/.tracebloc (or TRACEBLOC_HOST_DATA_DIR).

Delete a dataset

tracebloc data delete <name> removes a dataset’s table and files from your secure environment (see List and delete datasets). Right after the delete, your secure environment reports its datasets to tracebloc, and My Data shows the dataset as Deleted within a few minutes. If the CLI says tracebloc wasn’t told yet, nothing needs doing: the next scheduled report, about hourly while your secure environment runs, marks it Deleted. The dataset’s record on tracebloc is kept, never removed.