Skip to main content
The installer uses sensible defaults; this page covers what you can change. Installed with the one-liner? See Installer Options, Cluster Management, and GPU Support. Deploying into your own cluster with Helm (EKS, AKS, bare-metal)? Jump to Manual Deployment.

Installer Options

Override defaults by setting environment variables before the install command. Useful for a custom cluster name, extra worker nodes, or a different data directory. Each variable used to go by a shorter name: the same name without TRACEBLOC_ (CLUSTER_NAME), or with TB_ in its place (TB_STORAGE_MODE). The older names still work. If you set both, the TRACEBLOC_ one wins. Example — custom cluster name, with datasets kept on the host:

Cluster Management

The installer creates a k3d cluster that runs inside Docker. You can stop it to free resources, start it again later, or delete it entirely. Your datasets survive stop/start cycles. Deleting the cluster wipes them in the default node-local storage mode; with TRACEBLOC_STORAGE_MODE=hostpath they stay on the host under TRACEBLOC_HOST_DATA_DIR.
Deleting the cluster alone leaves the machine credential live, so your secure environment stays registered on your dashboard, and it leaves the tracebloc CLI and ~/.tracebloc on disk. To remove tracebloc from this machine, run tracebloc delete instead — see Uninstall.

View logs

The jobs manager is the main tracebloc process. Check its logs when debugging connectivity or job execution issues:

Useful commands

Common kubectl commands for inspecting cluster state:
Install logs are saved to ~/.tracebloc/install-*.log.

GPU Support

Training on a GPU needs an NVIDIA GPU of the Turing generation or newer, a driver that runs CUDA 12, and an x86_64 machine. Which GPUs, drivers and platforms are supported, and which aren’t, is in GPU support. This section is what the installer does about it.

NVIDIA (Linux)

Automatic. The installer:
  1. Detects NVIDIA GPUs via nvidia-smi or lspci, and checks the driver version and the GPU’s compute capability
  2. Installs the driver if it’s missing (Ubuntu, RHEL/CentOS and other dnf distributions, Arch). On any other distribution, install the driver yourself first
  3. Installs the NVIDIA Container Toolkit and configures Docker
  4. Deploys the NVIDIA k8s device plugin into the cluster
  5. Passes --gpus=all to k3d
A reboot may be required after driver installation. Re-run the installer afterward — it picks up where it left off. A GPU older than Turing, or a driver below the CUDA 12 minimum, installs CPU-only, and the installer prints why.

AMD and Intel

Not supported for training: the training images are built for CUDA, and there are no ROCm images. From the installer release after v1.9.200, an AMD GPU installs CPU-only on Linux too: the installer installs no ROCm and no device plugin, and prints why. Re-running it on a machine an older installer set up switches that machine to CPU and leaves ROCm installed. With an older installer, run tracebloc resources set --gpus 0 to train on CPU. On Windows, an AMD GPU installs CPU-only. Intel GPUs, integrated ones included, install CPU-only on every platform.

macOS

CPU only, on both Intel and Apple silicon. Training runs inside Linux containers, which on macOS run in a virtual machine with no access to the Apple GPU — so macOS machines train on CPU by design, regardless of device settings. For GPU training, use a machine with an NVIDIA GPU (Linux, or Windows via WSL2), or deploy on AWS (EKS).

Windows

The installer does not install GPU drivers on Windows. Pre-install the NVIDIA Windows driver (570.65 or newer recommended) before running the installer. The installer detects it via nvidia-smi and gives the cluster the GPU through Docker Desktop’s WSL 2 backend.

Manual Deployment

Skip the installer entirely. Use this if you already have a Kubernetes cluster, need custom resource limits, or want full control over the Helm deployment. A single chart — tracebloc/client — supports AKS, EKS, bare-metal, and OpenShift; choose your platform via values overrides. Every knob and its default is in the chart’s values.yaml.

Add the Helm repository

Get default values

Export the chart’s default configuration to customize it:

Configure values.yaml

Authentication

Set your Client ID and password from the clients page:

Resource Limits for Training Jobs

The chart leaves RESOURCE_REQUESTS and RESOURCE_LIMITS unset. Unset, each experiment gets cpu=1,memory=2Gi. The one-line installer sets both to fit your machine; on an installed secure environment, check and change them with tracebloc resources and tracebloc resources set. With Helm, override them per job size; for GPU support, requests and limits must be equal:

Storage

Storage class and PVC sizes:
The database must stay on local disk. MySQL/InnoDB is unsafe on NFS/CIFS, so the database and logs always use the local /tracebloc tree. To place large datasets on a network mount, set the installer’s TRACEBLOC_HOST_DATASET_DIR (the older HOST_DATASET_DIR still works) together with TRACEBLOC_STORAGE_MODE=hostpath — it relocates only the dataset volume (the chart’s hostPath.datasetPath → /tracebloc-data) and runs ingestion as the mount’s owner uid, so writes succeed under NFS root_squash.
Platform snippets (drop into your values file):

Docker Registry

The chart pulls public images from ghcr.io, so it needs no registry credentials by default. Add them only for a private mirror, and use a token, not a plaintext password:
With create: true, the chart creates a secret named {{ .Release.Name }}-regcred. To use a pull secret you created yourself, set dockerRegistry.existingSecret: <secret-name> instead of the credentials. Omit the dockerRegistry block entirely for public images.

Proxy (optional)

Auto-upgrade (on by default)

Releases of chart 1.3.0+ install a <release>-auto-upgrade CronJob that polls https://tracebloc.github.io/client hourly and runs helm upgrade --reset-then-reuse-values whenever a newer chart version is published — so secure environments auto-update instead of staying pinned to the version they were installed with.
The CronJob’s ServiceAccount is bound to the built-in cluster-admin ClusterRole because the chart templates cluster-scoped resources (PriorityClass, StorageClass, ClusterRoleBinding, optionally Namespace). Disable if you need a manual approval gate on upgrades. A second CronJob, imageRefresh (on by default, schedule: "7,22,37,52 * * * *"), picks up new tracebloc service images published under the same chart version every 15 minutes. Set imageRefresh.enabled: false to turn it off. Pinning the jobs manager, pods monitor and resource monitor images by digest also skips it.

NetworkPolicy hardening for training pods

Training pods run untrusted ML code. The chart applies a NetworkPolicy that denies all ingress and restricts egress — arbitrary pod-to-pod traffic and the Kubernetes API are blocked, while the in-cluster MySQL that serves the training data and the proxy that reports results stay reachable. By default, training pods have no direct internet: their outbound HTTPS goes through the in-cluster egress gateway, which only allows the hosts on its allowlist (see Security & data handling). Set allowExternalHttps: true to opt a cluster out.
Requires a CNI that enforces NetworkPolicy: The policy is on by default. After you install, run helm test <release> -n <namespace>: its enforcement probe fails when your CNI does not enforce the lockdown.
The chart’s training-pod egress lockdown only blocks traffic if your CNI enforces NetworkPolicy. Verify your CNI before relying on it.

Resource Monitor and node-agents namespace

The tracebloc-resource-monitor DaemonSet collects node-level CPU/memory metrics. It mounts hostPath volumes (/proc, /sys) which Pod Security Admission’s restricted profile bans — so the chart isolates it in a dedicated privileged namespace (default tracebloc-node-agents).
When create: false, create the namespace yourself with the required PSA labels:
The DaemonSet requires metrics-server. It is bundled on k3d/k3s/AKS, present on OpenShift, and must be installed manually on EKS (kubectl apply -f https://github.com/kubernetes-sigs/metrics-server/releases/latest/download/components.yaml).

Pod Security Admission labels

Training Jobs run untrusted user-supplied ML code. The chart can create the release namespace with Pod Security Admission warn/audit/enforce labels at the restricted profile for defense-in-depth:
When create: false (default) and you want PSA labels on an existing namespace:

Image digest pinning

Pin images by content hash for reproducible deploys. When digest is set, tag is ignored and imagePullPolicy drops to IfNotPresent.

PriorityClass and PodDisruptionBudgets

By default the chart creates no PriorityClass. On a busy cluster, you can give the MySQL pod one so the scheduler keeps it running under pressure: set priorityClass.create: true and a name. To use a PriorityClass your platform already manages, keep create: false and set its name. PDBs for MySQL and the jobs manager default to auto (create: null): the chart adds them when hostPath.enabled is false (multi-node clusters such as AKS, EKS and OpenShift) and skips them on hostPath installs (single-node, bare-metal), where they would block kubectl drain. Set true or false to force either way:

Deploy

Install the chart into a new namespace:

Upgrade

The auto-upgrade CronJob handles routine version bumps. To upgrade manually:
When upgrading into chart 1.3.0 from 1.2.x, use --reset-then-reuse-values (not plain --reuse-values) — the new autoUpgrade block did not exist in 1.2.x and a plain reuse fails template rendering.

Uninstall

PVCs, and the PriorityClass if you created one, are annotated helm.sh/resource-policy: keep so your data and shared cluster resources survive uninstall. To remove them too:

Migrating from legacy charts

If you installed before chart 1.3.x using tracebloc/aks, tracebloc/eks, or tracebloc/bm, see the migration guide in the client repo. Key changes:
  • 4 charts → 1 chart (tracebloc/client) with platform values overrides
  • Auth keys flattened: jobsManager.env.CLIENT_ID + secrets.clientPassword → top-level clientId + clientPassword
  • PVC keys flattened: clientData / clientLogsPvc / mysqlPvc (with name, storage, hostPath) → pvc.{data,logs,mysql} (size only) + hostPath.enabled for bare-metal
  • ServiceAccount renamed from default to {{ .Release.Name }}-jobs-manager
  • Pull-secret renamed from hard-coded regcred to {{ .Release.Name }}-regcred
  • The namespace value in the legacy values.yaml is gone — use helm install -n <ns> instead

Security

What leaves your network, how the secure environment is isolated, and what the installer changes on your machine are all on Security & data handling.