Skip to main content
Most issues fall into a few categories: pods not starting, client not connecting, or resource limits being hit. Start with the quick checks below — they cover the most common problems. For real-time cluster monitoring, try k9s — run k9s -n <workspace> to get a live view of pods, logs, and events.
Stuck? Generate a support bundle. Re-run the installer with --diagnose:
It writes a redacted ~/.tracebloc/tracebloc-diagnose-<timestamp>.tgz — logs, pod status, and versions with credentials removed — that you can send to support. The first line of output shows your client version.

​
Quick Checks

​
Error Messages

​
General

These errors typically indicate storage pressure on your Kubernetes nodes:

​
Local

Issues specific to local (k3d) deployments:

​
Debugging Commands

When the quick checks don’t resolve the issue, use these commands to dig deeper.

​
Pod status and logs

​
Resource usage

See if your nodes or pods are running out of CPU or memory:

​
Storage

Check that persistent volume claims are bound and have enough capacity:

​
Image pull credentials

If pods fail with ErrImagePull, verify that the Docker registry secret exists:

​
CPU and Memory Optimization

Hitting resource limits during training? Two levers:
  • Reduce data size — smaller batches, lower resolution, shorter sequences
  • Smaller models — fewer layers, smaller hidden dimensions

​
Memory Consumption (RAM / GPU VRAM)

Understanding what drives memory usage helps you right-size your resource limits:
  • Batch size — memory scales roughly linearly with batch size
  • Model size — more parameters = more memory for weights, activations, and gradients
  • Precision — FP16/BF16 uses half the memory of FP32. Mixed-precision training helps significantly
  • Optimizer — Adam requires ~2-3x the memory of SGD (stores running averages)
  • Input dimensions — transformer attention grows quadratically with sequence length; CNN memory grows quadratically with image resolution

​
Compute Consumption (CPU / GPU)

If training is slow, these are the factors to look at:
  • Batch size — larger batches increase GPU utilization up to memory saturation
  • Model complexity — transformer attention is O(seq_len² x hidden_dim); CNNs scale with kernel x feature map x filters
  • Precision — FP16/BF16 can speed up training 2-3x on modern GPUs
  • Data pipeline — slow CPU preprocessing (augmentation, tokenization) can bottleneck training
  • Parallelization — data parallelism splits batches across GPUs; model parallelism splits the model itself