Skip to main content
Most issues fall into a few categories: pods not starting, the secure environment not connecting, or resource limits being hit. Start with the quick checks below — they cover the most common problems. For real-time cluster monitoring, try k9s — run k9s -n <namespace> to get a live view of pods, logs, and events. Your namespace is the Environment shown in the summary at the end of the install (or run kubectl get ns). Use it wherever this page says <namespace>.
Stuck? Generate a support bundle. Re-run the installer with --diagnose:
It writes a redacted ~/.tracebloc/tracebloc-diagnose-<timestamp>.tgz — logs, pod status, and versions with credentials removed — that you can send to support. The first line of output shows your installer version.

​
Quick Checks

​
Error Messages

​
General

These errors typically indicate storage pressure on your Kubernetes nodes:

​
Local

Issues specific to local deployments:

​
Debugging Commands

When the quick checks don’t resolve the issue, use these commands to dig deeper.

​
Pod status and logs

​
Logs of a GPU training on a Mac

A GPU training on a Mac runs on the apple-gpu node, the tracebloc runner, as a process on the Mac rather than in a pod of the VM. Nothing on the Mac takes incoming connections, so kubectl logs can’t reach it and fails with 502 Bad Gateway:
The log is on the Mac, in your job data folder:
  • The training’s log, written while it runs: ~/.tracebloc/job-data/<namespace>/logs/<experiment key>/<cycle>/<experiment id>/client_training.log
  • Everything the process printed, one file per attempt: ~/.tracebloc/job-data/<namespace>/logs/<experiment key>/<cycle>/tracebloc-runner/<pod>.<attempt>.log, where the first attempt is 0
When the training ends, its last 100 lines (at most 4 KB) are also in the pod’s status:
A training that moved to the CPU runs in a pod in the VM, and kubectl logs reads it as usual. kubectl get pods -n <namespace> -o wide shows which node each attempt ran on.

​
Resource usage

See if your nodes or pods are running out of CPU or memory:

​
Storage

Check that persistent volume claims are bound and have enough capacity:

​
Image pull credentials

The default images on ghcr.io are public and need no pull secret. If you set registry credentials (dockerRegistry.create: true) and pods fail with ErrImagePull, verify that the chart created the secret <release>-regcred (the installer names the release after the namespace):
If you use your own secret through dockerRegistry.existingSecret, check that name instead.

​
CPU and Memory Optimization

Hitting resource limits during training? Two levers:
  • Reduce data size — smaller batches, lower resolution, shorter sequences
  • Smaller models — fewer layers, smaller hidden dimensions

​
Memory Consumption (RAM / GPU VRAM)

Understanding what drives memory usage helps you right-size your resource limits:
  • Batch size — memory scales roughly linearly with batch size
  • Model size — more parameters = more memory for weights, activations, and gradients
  • Precision — FP16/BF16 uses half the memory of FP32. Mixed-precision training helps significantly
  • Optimizer — Adam requires ~2-3x the memory of SGD (stores running averages)
  • Input dimensions — transformer attention grows quadratically with sequence length; CNN memory grows quadratically with image resolution

​
Compute Consumption (CPU / GPU)

If training is slow, these are the factors to look at:
  • Batch size — larger batches increase GPU utilization up to memory saturation
  • Model complexity — transformer attention is O(seq_len² x hidden_dim); CNNs scale with kernel x feature map x filters
  • Precision — FP16/BF16 can speed up training 2-3x on modern GPUs
  • Data pipeline — slow CPU preprocessing (augmentation, tokenization) can bottleneck training
  • Parallelization — data parallelism splits batches across GPUs; model parallelism splits the model itself