k9s --context k3s-tracebloc -n <namespace> to get a live view of pods, logs, and events.
Your namespace is the Environment shown in the summary at the end of the install (or run kubectl --context k3s-tracebloc get ns). Use it wherever this page says <namespace>.
Every kubectl command on this page names tracebloc’s context, k3s-tracebloc (k3s-<name> if you installed with TRACEBLOC_CLUSTER_NAME): see Which context kubectl and Helm use. On Windows, kubectl and helm run inside tracebloc’s WSL distro, where the context is default: see kubectl and Helm on Windows.
Stuck? Generate a support bundle. Re-run the installer with It writes a redacted
--diagnose:~/.tracebloc/tracebloc-diagnose-<timestamp>.tgz — logs, pod status, and versions with credentials removed — that you can send to support. The first line of output shows your installer version.Quick Checks
Error Messages
General
These errors typically indicate storage pressure on your Kubernetes nodes:Local
Issues specific to local deployments:Debugging Commands
When the quick checks don’t resolve the issue, use these commands to dig deeper.Pod status and logs
Logs of a GPU training on a Mac
A GPU training on a Mac runs on theapple-gpu node, the tracebloc runner, as a process on the Mac rather than in a pod of the VM. Nothing on the Mac takes incoming connections, so kubectl logs can’t reach it and fails with 502 Bad Gateway:
- The training’s log, written while it runs:
~/.tracebloc/job-data/<namespace>/logs/<experiment key>/<cycle>/<experiment id>/client_training.log - Everything the process printed, one file per attempt:
~/.tracebloc/job-data/<namespace>/logs/<experiment key>/<cycle>/tracebloc-runner/<pod>.<attempt>.log, where the first attempt is0
kubectl logs reads it as usual. kubectl --context k3s-tracebloc get pods -n <namespace> -o wide shows which node each attempt ran on.
Resource usage
See if your nodes or pods are running out of CPU or memory:Storage
Check that persistent volume claims are bound and have enough capacity:Image pull credentials
The default images onghcr.io are public and need no pull secret. If you set registry credentials (dockerRegistry.create: true) and pods fail with ErrImagePull, verify that the chart created the secret <release>-regcred (the installer names the release after the namespace):
dockerRegistry.existingSecret, check that name instead.
CPU and Memory Optimization
Hitting resource limits during training? Two levers:- Reduce data size — smaller batches, lower resolution, shorter sequences
- Smaller models — fewer layers, smaller hidden dimensions
Memory Consumption (RAM / GPU VRAM)
Understanding what drives memory usage helps you right-size your resource limits:- Batch size — memory scales roughly linearly with batch size
- Model size — more parameters = more memory for weights, activations, and gradients
- Precision — FP16/BF16 uses half the memory of FP32. Mixed-precision training helps significantly
- Optimizer — Adam requires ~2-3x the memory of SGD (stores running averages)
- Input dimensions — transformer attention grows quadratically with sequence length; CNN memory grows quadratically with image resolution
Compute Consumption (CPU / GPU)
If training is slow, these are the factors to look at:- Batch size — larger batches increase GPU utilization up to memory saturation
- Model complexity — transformer attention is O(seq_len² x hidden_dim); CNNs scale with kernel x feature map x filters
- Precision — FP16/BF16 can speed up training 2-3x on modern GPUs
- Data pipeline — slow CPU preprocessing (augmentation, tokenization) can bottleneck training
- Parallelization — data parallelism splits batches across GPUs; model parallelism splits the model itself