Installer Options
Override defaults by setting environment variables before the install command. Useful for a custom cluster name, extra worker nodes, or a different data directory. Each variable used to go by a shorter name: the same name withoutTRACEBLOC_ (CLUSTER_NAME), or with TB_ in its place (TB_STORAGE_MODE). The older names still work. If you set both, the TRACEBLOC_ one wins.
Example — custom cluster name, with datasets kept on the host:
Cluster Management
The installer creates a k3d cluster that runs inside Docker. You can stop it to free resources, start it again later, or delete it entirely. Your datasets survive stop/start cycles. Deleting the cluster wipes them in the default node-local storage mode; withTRACEBLOC_STORAGE_MODE=hostpath they stay on the host under TRACEBLOC_HOST_DATA_DIR.
View logs
The jobs manager is the main tracebloc process. Check its logs when debugging connectivity or job execution issues:Useful commands
Common kubectl commands for inspecting cluster state:~/.tracebloc/install-*.log.
GPU Support
Training on a GPU needs an NVIDIA GPU of the Turing generation or newer, a driver that runs CUDA 12, and an x86_64 machine. Which GPUs, drivers and platforms are supported, and which aren’t, is in GPU support. This section is what the installer does about it.NVIDIA (Linux)
Automatic. The installer:- Detects NVIDIA GPUs via
nvidia-smiorlspci, and checks the driver version and the GPU’s compute capability - Installs the driver if it’s missing (Ubuntu, RHEL/CentOS and other
dnfdistributions, Arch). On any other distribution, install the driver yourself first - Installs the NVIDIA Container Toolkit and configures Docker
- Deploys the NVIDIA k8s device plugin into the cluster
- Passes
--gpus=allto k3d
AMD and Intel
Not supported for training: the training images are built for CUDA, and there are no ROCm images. From the installer release after v1.9.200, an AMD GPU installs CPU-only on Linux too: the installer installs no ROCm and no device plugin, and prints why. Re-running it on a machine an older installer set up switches that machine to CPU and leaves ROCm installed. With an older installer, runtracebloc resources set --gpus 0 to train on CPU. On Windows, an AMD GPU installs CPU-only. Intel GPUs, integrated ones included, install CPU-only on every platform.
macOS
CPU only, on both Intel and Apple silicon. Training runs inside Linux containers, which on macOS run in a virtual machine with no access to the Apple GPU — so macOS machines train on CPU by design, regardless of device settings. For GPU training, use a machine with an NVIDIA GPU (Linux, or Windows via WSL2), or deploy on AWS (EKS).Windows
The installer does not install GPU drivers on Windows. Pre-install the NVIDIA Windows driver (570.65 or newer recommended) before running the installer. The installer detects it vianvidia-smi and gives the cluster the GPU through Docker Desktop’s WSL 2 backend.
Manual Deployment
Skip the installer entirely. Use this if you already have a Kubernetes cluster, need custom resource limits, or want full control over the Helm deployment. A single chart —tracebloc/client — supports AKS, EKS, bare-metal, and OpenShift; choose your platform via values overrides. Every knob and its default is in the chart’s values.yaml.
Add the Helm repository
Get default values
Export the chart’s default configuration to customize it:Configure values.yaml
Authentication
Set your Client ID and password from the clients page:Resource Limits for Training Jobs
The chart leavesRESOURCE_REQUESTS and RESOURCE_LIMITS unset. Unset, each experiment gets cpu=1,memory=2Gi. The one-line installer sets both to fit your machine; on an installed secure environment, check and change them with tracebloc resources and tracebloc resources set. With Helm, override them per job size; for GPU support, requests and limits must be equal:
Storage
Storage class and PVC sizes:The database must stay on local disk. MySQL/InnoDB is unsafe on NFS/CIFS, so the database and logs always use the local
/tracebloc tree. To place large datasets on a network mount, set the installer’s TRACEBLOC_HOST_DATASET_DIR (the older HOST_DATASET_DIR still works) together with TRACEBLOC_STORAGE_MODE=hostpath — it relocates only the dataset volume (the chart’s hostPath.datasetPath → /tracebloc-data) and runs ingestion as the mount’s owner uid, so writes succeed under NFS root_squash.Docker Registry
The chart pulls public images fromghcr.io, so it needs no registry credentials by default. Add them only for a private mirror, and use a token, not a plaintext password:
create: true, the chart creates a secret named {{ .Release.Name }}-regcred. To use a pull secret you created yourself, set dockerRegistry.existingSecret: <secret-name> instead of the credentials. Omit the dockerRegistry block entirely for public images.
Proxy (optional)
Auto-upgrade (on by default)
Releases of chart1.3.0+ install a <release>-auto-upgrade CronJob that polls https://tracebloc.github.io/client hourly and runs helm upgrade --reset-then-reuse-values whenever a newer chart version is published — so secure environments auto-update instead of staying pinned to the version they were installed with.
cluster-admin ClusterRole because the chart templates cluster-scoped resources (PriorityClass, StorageClass, ClusterRoleBinding, optionally Namespace). Disable if you need a manual approval gate on upgrades.
A second CronJob, imageRefresh (on by default, schedule: "7,22,37,52 * * * *"), picks up new tracebloc service images published under the same chart version every 15 minutes. Set imageRefresh.enabled: false to turn it off. Pinning the jobs manager, pods monitor and resource monitor images by digest also skips it.
NetworkPolicy hardening for training pods
Training pods run untrusted ML code. The chart applies a NetworkPolicy that denies all ingress and restricts egress — arbitrary pod-to-pod traffic and the Kubernetes API are blocked, while the in-cluster MySQL that serves the training data and the proxy that reports results stay reachable. By default, training pods have no direct internet: their outbound HTTPS goes through the in-cluster egress gateway, which only allows the hosts on its allowlist (see Security & data handling). SetallowExternalHttps: true to opt a cluster out.
The policy is on by default. After you install, run
helm test <release> -n <namespace>: its enforcement probe fails when your CNI does not enforce the lockdown.
Resource Monitor and node-agents namespace
Thetracebloc-resource-monitor DaemonSet collects node-level CPU/memory metrics. It mounts hostPath volumes (/proc, /sys) which Pod Security Admission’s restricted profile bans — so the chart isolates it in a dedicated privileged namespace (default tracebloc-node-agents).
create: false, create the namespace yourself with the required PSA labels:
metrics-server. It is bundled on k3d/k3s/AKS, present on OpenShift, and must be installed manually on EKS (kubectl apply -f https://github.com/kubernetes-sigs/metrics-server/releases/latest/download/components.yaml).
Pod Security Admission labels
Training Jobs run untrusted user-supplied ML code. The chart can create the release namespace with Pod Security Admissionwarn/audit/enforce labels at the restricted profile for defense-in-depth:
create: false (default) and you want PSA labels on an existing namespace:
Image digest pinning
Pin images by content hash for reproducible deploys. Whendigest is set, tag is ignored and imagePullPolicy drops to IfNotPresent.
PriorityClass and PodDisruptionBudgets
By default the chart creates no PriorityClass. On a busy cluster, you can give the MySQL pod one so the scheduler keeps it running under pressure: setpriorityClass.create: true and a name. To use a PriorityClass your platform already manages, keep create: false and set its name.
PDBs for MySQL and the jobs manager default to auto (create: null): the chart adds them when hostPath.enabled is false (multi-node clusters such as AKS, EKS and OpenShift) and skips them on hostPath installs (single-node, bare-metal), where they would block kubectl drain. Set true or false to force either way:
Deploy
Install the chart into a new namespace:Upgrade
The auto-upgrade CronJob handles routine version bumps. To upgrade manually:When upgrading into chart 1.3.0 from 1.2.x, use
--reset-then-reuse-values (not plain --reuse-values) — the new autoUpgrade block did not exist in 1.2.x and a plain reuse fails template rendering.Uninstall
helm.sh/resource-policy: keep so your data and shared cluster resources survive uninstall. To remove them too:
Migrating from legacy charts
If you installed before chart 1.3.x usingtracebloc/aks, tracebloc/eks, or tracebloc/bm, see the migration guide in the client repo. Key changes:
- 4 charts → 1 chart (
tracebloc/client) with platform values overrides - Auth keys flattened:
jobsManager.env.CLIENT_ID+secrets.clientPassword→ top-levelclientId+clientPassword - PVC keys flattened:
clientData/clientLogsPvc/mysqlPvc(withname,storage,hostPath) →pvc.{data,logs,mysql}(size only) +hostPath.enabledfor bare-metal - ServiceAccount renamed from
defaultto{{ .Release.Name }}-jobs-manager - Pull-secret renamed from hard-coded
regcredto{{ .Release.Name }}-regcred - The
namespacevalue in the legacyvalues.yamlis gone — usehelm install -n <ns>instead