Azure AKS

​

Prerequisites

• An AKS cluster, and kubectl pointed at it (az aks get-credentials …).
• Helm 3.14 or later (Helm 4 works).
• The tracebloc CLI, signed in (tracebloc login). It creates your secure environment’s Client ID and password; see Configuration → Authentication.

​

Install

tracebloc client create --credential-file tb.env   # Client ID, password and namespace, see below
helm repo add tracebloc https://tracebloc.github.io/client
helm repo update
helm show values tracebloc/client > values.yaml   # edit per below
. ./tb.env
helm upgrade --install "$TRACEBLOC_NAMESPACE" tracebloc/client \
  -n "$TRACEBLOC_NAMESPACE" --create-namespace -f values.yaml

​

Verify

kubectl get pods -n "$TRACEBLOC_NAMESPACE"

​

Environment-specific config

storageClass:
  create: true
  provisioner: file.csi.azure.com
  parameters:
    skuName: Standard_LRS
  mountOptions: [dir_mode=0750, file_mode=0640, uid=999, gid=999, mfsymlinks, cache=strict, actimeo=30]
clusterScope: true

• NetworkPolicy: create the AKS cluster with --network-policy azure (Azure NPM) or Calico — otherwise the training-pod egress lockdown won’t enforce.
• metrics-server is bundled on AKS.
• GPU or CPU-only. In the env block of values.yaml, declare what your nodes have. Without GPU_LIMITS, every training pod requests one NVIDIA GPU, and on CPU-only nodes it stays Pending:
  env:
    TRACEBLOC_DDP: "1"           # from the chart's env block; keep it
    GPU_LIMITS: ""               # CPU-only nodes. For NVIDIA GPUs: "nvidia.com/gpu=1"
  
  The chart’s values.yaml writes env in braces (env: { … }), where a new line without a comma is a YAML error. Replace that whole block with the one above, and carry over any other keys you’ve set. With GPUs, use x86_64 nodes with an NVIDIA GPU of the Turing generation or newer (GPU support). The nodes also need the NVIDIA device plugin. Install it yourself, or let the chart run it:
  gpu:
    devicePlugin:
      enabled: true
      vendor: nvidia
  

​

Production notes

• Use GPU node pools for training workloads, with the NVIDIA device plugin running on them (see above); size requests/limits per job.
• Day-2 upgrades and rollbacks: see Operations.