Skip to main content
Most issues fall into a few categories: pods not starting, the secure environment not connecting, or resource limits being hit. Start with the quick checks below — they cover the most common problems. For real-time cluster monitoring, try k9s — run k9s --context k3s-tracebloc -n <namespace> to get a live view of pods, logs, and events. Your namespace is the Environment shown in the summary at the end of the install (or run kubectl --context k3s-tracebloc get ns). Use it wherever this page says <namespace>. Every kubectl command on this page names tracebloc’s context, k3s-tracebloc (k3s-<name> if you installed with TRACEBLOC_CLUSTER_NAME): see Which context kubectl and Helm use. On Windows, kubectl and helm run inside tracebloc’s WSL distro, where the context is default: see kubectl and Helm on Windows.
Stuck? Generate a support bundle. Re-run the installer with --diagnose:
It writes a redacted ~/.tracebloc/tracebloc-diagnose-<timestamp>.tgz — logs, pod status, and versions with credentials removed — that you can send to support. The first line of output shows your installer version.

Quick Checks

Error Messages

General

These errors typically indicate storage pressure on your Kubernetes nodes:

Local

Issues specific to local deployments:

Debugging Commands

When the quick checks don’t resolve the issue, use these commands to dig deeper.

Pod status and logs

Logs of a GPU training on a Mac

A GPU training on a Mac runs on the apple-gpu node, the tracebloc runner, as a process on the Mac rather than in a pod of the VM. Nothing on the Mac takes incoming connections, so kubectl logs can’t reach it and fails with 502 Bad Gateway:
The log is on the Mac, in your job data folder:
  • The training’s log, written while it runs: ~/.tracebloc/job-data/<namespace>/logs/<experiment key>/<cycle>/<experiment id>/client_training.log
  • Everything the process printed, one file per attempt: ~/.tracebloc/job-data/<namespace>/logs/<experiment key>/<cycle>/tracebloc-runner/<pod>.<attempt>.log, where the first attempt is 0
When the training ends, its last 100 lines (at most 4 KB) are also in the pod’s status:
A training that moved to the CPU runs in a pod in the VM, and kubectl logs reads it as usual. kubectl --context k3s-tracebloc get pods -n <namespace> -o wide shows which node each attempt ran on.

Resource usage

See if your nodes or pods are running out of CPU or memory:

Storage

Check that persistent volume claims are bound and have enough capacity:

Image pull credentials

The default images on ghcr.io are public and need no pull secret. If you set registry credentials (dockerRegistry.create: true) and pods fail with ErrImagePull, verify that the chart created the secret <release>-regcred (the installer names the release after the namespace):
If you use your own secret through dockerRegistry.existingSecret, check that name instead.

CPU and Memory Optimization

Hitting resource limits during training? Two levers:
  • Reduce data size — smaller batches, lower resolution, shorter sequences
  • Smaller models — fewer layers, smaller hidden dimensions

Memory Consumption (RAM / GPU VRAM)

Understanding what drives memory usage helps you right-size your resource limits:
  • Batch size — memory scales roughly linearly with batch size
  • Model size — more parameters = more memory for weights, activations, and gradients
  • Precision — FP16/BF16 uses half the memory of FP32. Mixed-precision training helps significantly
  • Optimizer — Adam requires ~2-3x the memory of SGD (stores running averages)
  • Input dimensions — transformer attention grows quadratically with sequence length; CNN memory grows quadratically with image resolution

Compute Consumption (CPU / GPU)

If training is slow, these are the factors to look at:
  • Batch size — larger batches increase GPU utilization up to memory saturation
  • Model complexity — transformer attention is O(seq_len² x hidden_dim); CNNs scale with kernel x feature map x filters
  • Precision — FP16/BF16 can speed up training 2-3x on modern GPUs
  • Data pipeline — slow CPU preprocessing (augmentation, tokenization) can bottleneck training
  • Parallelization — data parallelism splits batches across GPUs; model parallelism splits the model itself