Prepare Data - tracebloc

Overview

Make your data available to the Kubernetes cluster so it can be used for training and evaluation. Whether your client runs on Azure, AWS, Google Cloud, or a local Minikube setup, the process of ingesting datasets works the same way. The data ingestor is a lightweight service that bridges your raw data and the cluster’s persistent storage. Every supported task has a dataset template — the folder layout, the labels CSV and a ready-to-edit ingest.yaml — that you lay your own data out against. By containerizing the ingestion step, the ingestor validates data format and schema, enforces consistency, and transfers the dataset securely into cluster’s SQL storage where it becomes accessible to all training and evaluation jobs. This guide covers:

Laying your data out to match the dataset template for your task (tabular, images, text, time series)
Deploying the data ingestor for training and test data using Kubernetes
Managing datasets through the tracebloc interface

IMPORTANT Make sure that the data format and ML task is supported and that data standards are met by reviewing the docs. You must run the process twice, once to ingest training and once to ingest testing data.

Setup options

You can ingest data into your client in two ways:

Declarative YAML (recommended, simpler) — describe your dataset in ~8 lines of ingest.yaml, then helm install. No Dockerfile, no custom Python script. The official ingestor image runs it for you. Use this for any dataset that fits a supported category.
Custom Python script + Kubernetes Job (advanced) — install the tracebloc-ingestor Python package, write a short ingestion script against it, build and push a Docker image, then kubectl apply an ingestor-job.yaml. Use this when the declarative schema can’t express what your data needs — e.g. non-trivial preprocessing, a custom validator, or a BaseProcessor subclass.

Start with the declarative method below. Drop down to the custom-script flow only if you need it.

Declarative YAML (recommended)

Describe your dataset in ~8 lines of YAML, then helm install. The official ingestor image (published as ghcr.io/tracebloc/ingestor) runs it. No Dockerfile, no Python script.

Before you run any commands in this section: if you installed the client via the one-liner (curl -fsSL https://tracebloc.io/i.sh | bash), every later helm upgrade <workspace> tracebloc/client … must include --reset-then-reuse-values, otherwise the upgrade drops the values the installer applied and breaks the workspace:

helm upgrade <workspace> tracebloc/client -n <namespace> --reset-then-reuse-values

Append --version <version-number> to pin a specific chart version. This caveat only affects upgrades of the parent tracebloc/client chart, not the helm install tracebloc/ingestor runs below.

1. Add the chart repo (one-time)

helm repo add tracebloc https://tracebloc.github.io/client
helm repo update

The tracebloc/client parent chart bootstraps the cluster (jobs-manager, MySQL, RBAC). The tracebloc/ingestor subchart submits per-dataset ingestion runs against it.

2. Stage your data on the cluster’s shared PVC

The chart doesn’t transport data into the cluster — it points at data already accessible to the cluster’s shared PVC (client-pvc by default, mounted at /data/shared/ inside the ingestor Pod). Before installing, get your raw files there. For a single-node workspace (the default install), the PVC is backed by a host directory the installer created at ~/.tracebloc/<workspace>/data/. Drop your files into a per-dataset subdirectory:

# Host path on the machine where the tracebloc client is installed.
# Pick a <prefix> per dataset — it becomes the path you reference in ingest.yaml.
mkdir -p ~/.tracebloc/<workspace>/data/<prefix>
cp -R LOCAL_PATH/images   ~/.tracebloc/<workspace>/data/<prefix>/
cp    LOCAL_PATH/labels.csv ~/.tracebloc/<workspace>/data/<prefix>/

Inside the ingestor Pod those files appear at /data/shared/<prefix>/... — that’s what you’ll put in ingest.yaml below.

For multi-node or EKS deployments where the PVC isn’t backed by a local host path, use a throwaway kubectl cp Pod or a cloud-storage init container instead. See the client ingestor README for those recipes.

3. Write your `ingest.yaml`

The example below is for image_classification. Other tasks require different fields — e.g. tabular_classification has no images: and instead needs a typed schema: block. Don’t copy this one blindly; open the dataset template for your task — one page per task with the folder layout, the CSV columns, a ready-to-edit ingest.yaml and the checks the ingestor runs — and edit from there.

apiVersion: tracebloc.io/v1
kind: IngestConfig
category: image_classification
table: cats_dogs_train
intent: train
csv: /data/shared/cats-dogs/labels.csv
images: /data/shared/cats-dogs/images/
label: label

The top-level shape (apiVersion, kind, category, table, intent, label) is the same for every category; the category field picks the validator set, file-extension defaults, and column conventions. The data-source fields (csv:, images:, schema:, …) vary per category. The paths are paths inside the ingestor Pod, which is the PVC mount you populated in step 2.

4. Install once per dataset

The ingestor runs once: validates your data, copies files into the destination directory on the PVC, inserts rows into MySQL, sends metadata to the tracebloc backend, then exits. Run it twice per dataset — once with intent: train, once with intent: test — using distinct table: names. The example below shows both releases:

# Train release — points at the ingest.yaml from step 3 (table: cats_dogs_train, intent: train)
helm install cats-dogs-train tracebloc/ingestor \
  --namespace <workspace> \
  --set-file ingestConfig=./ingest-train.yaml

# Test release — same shape, with table: cats_dogs_test and intent: test
helm install cats-dogs-test tracebloc/ingestor \
  --namespace <workspace> \
  --set-file ingestConfig=./ingest-test.yaml

Each helm install is a separate release (the first argument is the release name), so the two runs don’t collide. The ingestor Pod picks up CLIENT_ID / CLIENT_PASSWORD automatically from the Kubernetes Secret the parent tracebloc/client chart created in <workspace> at install time — you don’t pass credentials on the helm install command.

Validation error like '<your_category>' is not one of [...] or Additional properties are not allowed (<field> was unexpected)? This comes from the cluster’s jobs-manager validating against its own bundled schema at submit time — the deployed schema is older than the ingestor image you’re installing. helm repo update won’t fix it (that only refreshes the local chart index, not the running server). The fix is on the cluster side: upgrade the parent chart so jobs-manager redeploys with the current schema.

helm upgrade <workspace> tracebloc/client \
  -n <workspace> --reset-then-reuse-values

Then re-run the helm install command above.

Full chart docs (data-staging recipe, schema, every category, update model, verification, override knobs) → client ingestor README.

Custom Python script (advanced)

Use this flow when the declarative schema can’t express what your data needs — typically when you have non-trivial preprocessing logic, a custom validator, or a BaseProcessor subclass. The sections below — Quick Setup and Detailed Setup — both describe this advanced path.

Quick Setup

Use this quick setup if you already have an ingestor configured and just want to switch datasets or toggle between training and testing. If you are setting up for the first time, go to the next section for the detailed walkthrough.

Steps

Edit your ingestion script (the one you wrote in Configure a script below)

Update csv options and data_path
Only for tabular data: Update schema
Set schema and CSVIngestor()parameters like category, intent, label_column, etc. to match data type, task and train/test purpose

ingestor = CSVIngestor(
    ...
    category=TaskCategory.TABULAR_CLASSIFICATION, # Adjust for your task
    csv_options=csv_options,      # Defined above
    label_column="ColumnName",    # Target column
    intent=Intent.TRAIN,          # TRAIN or TEST
)

Build and push docker image:

Make sure Docker is running on your system (e.g. by starting Docker Desktop), then execute the following command:

# Build for cloud (multi-arch) and push directly to registry
docker buildx build --platform linux/amd64,linux/arm64 -t <your-username>/<image-name>:<tag> --push .

Edit ingestor-job.yaml:

metadata.name: Unique job name (e.g. ingestor-job-train and ingestor-job-test)
image: The tag you built and pushed
LABEL_FILE: Path inside the pod to the labels CSV, under the PVC mount (e.g. /data/shared/labels.csv). For tabular data, this is the same file that contains both labels and features.
TABLE_NAME: Unique table name (no spaces, one per dataset). Title is optional
SRC_PATH: Root of the mounted dataset directory inside the pod (/data/shared, backed by ~/.tracebloc/<workspace>/data on the client host)

Deploy to Kubernetes

`kubectl apply -f ingestor-job.yaml -n <workspace>`

Detailed Setup

1. Configure a script

This section walks you through the step-by-step setup of a data ingestor. You will install the ingestor package, lay your data out against the dataset template for your task, and write a short ingestion script that matches it. Follow this guide if you are setting up an ingestor for the first time or need full control beyond the quick setup.

Install the ingestor package

The ingestion library is published on PyPI as tracebloc-ingestor (import name tracebloc_ingestor). It is the same code the official ingestor image runs, so a script written against it behaves exactly like the declarative path. It needs Python 3.11 or newer:

pip install tracebloc-ingestor

IMPORTANT: Datasets must be cleaned and preprocessed before ingestion. Participants cannot view, clean or fix raw data, so model performance will only be as good as the data you provide.

Pick the dataset template for your task

Open the dataset template for your task. It gives you the folder layout and the labels CSV the ingestor expects — lay your data out the same way, then set the matching category and data_format in your script. The rows below cover the most common tasks; every other task on the templates index works the same way, and its TaskCategory constant is the upper-cased category identifier from its page (for example keypoint_detection → TaskCategory.KEYPOINT_DETECTION).

Data Type	Dataset template	Data Category	Data Format
Tabular	Tabular classification	`TaskCategory.TABULAR_CLASSIFICATION`	`DataFormat.TABULAR`
Tabular	Tabular regression	`TaskCategory.TABULAR_REGRESSION`	`DataFormat.TABULAR`
Tabular	Time-series forecasting	`TaskCategory.TIME_SERIES_FORECASTING`	`DataFormat.TABULAR`
Tabular	Survival analysis (time-to-event)	`TaskCategory.TIME_TO_EVENT_PREDICTION`	`DataFormat.TABULAR`
Image	Image classification	`TaskCategory.IMAGE_CLASSIFICATION`	`DataFormat.IMAGE`
Image	Object detection	`TaskCategory.OBJECT_DETECTION`	`DataFormat.IMAGE`
Text	Text classification	`TaskCategory.TEXT_CLASSIFICATION`	`DataFormat.TEXT`

High Level Script Structure

Every ingestion script follows the same structure — save it as ingestor.py next to your Dockerfile:

import logging

from tracebloc_ingestor import Config, Database, APIClient, CSVIngestor, run_ingestion
from tracebloc_ingestor.utils.logging import setup_logging
from tracebloc_ingestor.utils.constants import TaskCategory, Intent, DataFormat

config = Config()
setup_logging(config)
logger = logging.getLogger(__name__)


def main():
    # Initialize components
    database = Database(config)
    # Initialize API client
    api_client = APIClient(config)

    # Define csv_options and schema (schema is only needed for tabular data)
    csv_options = {...}
    schema = {...}

    # Initialize ingestor
    ingestor = CSVIngestor(...)

    # Run and ingest data
    run_ingestion(ingestor, config.LABEL_FILE, batch_size=config.BATCH_SIZE, logger=logger)


if __name__ == "__main__":
    main()

Both Database, APIClient and other values are configured automatically from the environment variables defined in ingestor_job.yaml.

config.LABEL_FILE: Path to local csv label file
config.BATCH_SIZE: Batch size used during ingestion

Customize the script

The structure above is a starting point, but every dataset has its own format and labels. In this step you adapt the script to your data by tuning CSV ingestion options and setting the ingestor parameters (category, label column, intent, data path and schema). The following example shows how to ingest a tabular dataset, but the setup works the same way for image or text data.

Needed for Tabular Data: Define Schema

Define the dataset schema as a Python dictionary, mapping each column to its SQL type and constraints. Do not include IDs or the label column into the schema.

# Schema definition for tabular data
schema = {
  "feature_00": "FLOAT ",
  "feature_01": "FLOAT ",
  "feature_02": "FLOAT ",
  ...
}

Needed for Image Classification Data: Define Image Options

Define image size and file extension.

# Image specific options including CSV options
image_options = {
    # Image processing options
    "target_size": (512, 512),  # Define image size. Height = Width
    "extension": FileExtension.JPG,  # allowed extension for images: jpeg, jpg, png
}

Needed for Object Detection Data: Define Image Options

Define file extension.

# Object detection specific options including CSV options
object_detection_options = {
    # Image processing options
    "target_size": (448, 448),  # Resize images to this fixed dimension. Dimension is not changeable.
    "extension": FileExtension.JPG,  # allowed extension for images: jpeg, jpg, png
}

Needed for Text Data: Define File Extension

Define file extensions.

text_options = {"extension": FileExtension.TXT}  # Allowed text file extensions

Set CSV ingestion options

Customize parsing, memory handling, and data cleaning with the csv_options dictionary:

csv_options = {
    "chunk_size": 1000,          # Process rows in batches for efficiency
    "delimiter": ",",            # Column separator
    "quotechar": '"',            # Quoted field character
    "escapechar": "\\",          # Escape character for quotes
    "encoding": "utf-8",         # File encoding
    "on_bad_lines": "warn",      # Log malformed rows instead of failing
    "skip_blank_lines": True,    # Ignore empty rows
    "na_values": ["", "NA", "NULL", "None"]  # Treat these as missing values
}

Set Up the Ingestor

Define the Ingestor instance with the required configuration. See the tabular data example below:

ingestor = CSVIngestor(
    database=database,                  # From ingestor-job.yaml
    api_client=api_client,              # From ingestor-job.yaml
    table_name=config.TABLE_NAME,       # From ingestor-job.yaml
    schema=schema,                      # Defined above, only needed for tabular data
    data_format=DataFormat.TABULAR,     # Set the data format for the task
    category=TaskCategory.TABULAR_CLASSIFICATION, # Adjust for your task
    csv_options=csv_options,            # Defined above
    file_options={"number_of_columns": len(schema)}, # Don´t change
    label_column="ColumnName",          # Target column
    intent=Intent.TRAIN,                # TRAIN or TEST
)

Specify:

category, choose the ML task type (TABULAR_CLASSIFICATION, IMAGE_CLASSIFICATION, OBJECT_DETECTION)
label_column, target column or class labels
intent, set as TRAIN or TEST depending on dataset purpose
include file_options or schema depending on the data type

Other data types work similarly — follow the same configuration pattern with the category, data_format and options for your task from the table above.

2. Build Docker Image

With your script configured, the next step is to package it into a Docker image so it can run inside the Kubernetes cluster.

Docker Hub Setup (first-time users)

The cluster pulls your ingestor image from a public Docker registry, so you need an account before you can push. If you already have one, skip to Write the Dockerfile.

Create a Docker Hub account at hub.docker.com/signup and verify your email.
Log in from your terminal so the docker push command can authenticate:
docker login
Push the data ingestor image to your account using the build/push commands in the next section. The image name takes the form <your-docker-username>/<image-name>:<tag> — the username segment must match the account you just created.
Make the image public so the cluster can pull it without credentials:
- Go to hub.docker.com/repositories, open the repository you just pushed.
- Click Settings → Visibility settings → Make public.
Keeping the image private is also fine, but then you must create a Kubernetes imagePullSecret named regcred in the client namespace (the ingestor-job.yaml already references it).

Place data files on the client host

Datasets are not baked into the Docker image. They live on the client host in the per-workspace data directory and are mounted into the ingestor pod through the shared PVC (client-pvc → /data/shared). Copy your dataset into the client’s data directory, where <workspace> is the workspace name you chose during client install (which is also the Helm release name and the Kubernetes namespace — the chart uses the same value for all three). The directory ~/.tracebloc/<workspace>/data/ is created automatically by the installer; just drop your files into it:

# Host path on the machine where the tracebloc client is installed.
# HOST_DATA_DIR defaults to ~/.tracebloc; override only if you set it during install.
cp -R LOCAL_PATH/images   ~/.tracebloc/<workspace>/data/
cp    LOCAL_PATH/labels.csv ~/.tracebloc/<workspace>/data/

Inside the ingestor pod this directory is mounted at /data/shared, so the same files appear as /data/shared/images/... and /data/shared/labels.csv. Set SRC_PATH and LABEL_FILE in ingestor-job.yaml to point at those in-pod paths (see Configure Kubernetes below). For tabular data the same rule applies — drop the single labels.csv (with features and labels) into ~/.tracebloc/<workspace>/data/.

Write the Dockerfile

The Dockerfile only needs to install the ingestor package and copy in your script — the dataset is mounted at runtime, so do not COPY data into the image:

FROM python:3.11-slim
WORKDIR /app

RUN pip install --no-cache-dir tracebloc-ingestor

# Copy the ingestion script into /app
COPY ingestor.py /app/ingestor.py

# Set the entrypoint
ENTRYPOINT ["python", "/app/ingestor.py"]

If the cluster enforces the restricted Pod Security Standard (see Run as non-root below), also add a non-root user to the Dockerfile, before the # Set the entrypoint line:

RUN groupadd -g 1000 app && \
    useradd -u 1000 -g 1000 -m -s /bin/bash app && \
    chown -R 1000:1000 /app

USER 1000

# Set the entrypoint

Build Docker Image

You need a docker user and password to proceed with the next step. Cloud platforms run a mix of x86 and ARM nodes (e.g. AWS Graviton, Azure Ampere, GCP Tau T2A). Building a multi-arch image with --platform linux/amd64,linux/arm64 guarantees the image runs on either, particularly if you build on Apple Silicon (M1/M2) or other ARM-based systems. Build and push the image with a single command:

docker buildx build --platform linux/amd64,linux/arm64 -t <your-username>/<image-name>:<tag> --push .

3. Configure Kubernetes

With the image generated and pushed to the registry, edit ingestor-job.yaml with your settings:

apiVersion: batch/v1
kind: Job
metadata:
  name: <JOBNAME> # Set a job name e.g. ingestor-job-train
  namespace: <workspace> # Use the client namespace
spec:
  template:
    spec:
      containers:
      - name: api
        image: <YOUR_DOCKER_USER>/<YOUR_IMAGE_NAME>:latest # Your Docker image name and tag, e.g. "latest"
        imagePullPolicy: Always  # Use IfNotPresent only for local tests
        # Required if the namespace enforces the `restricted` Pod Security Standard.
        # See "Run as non-root" below.
        securityContext:
          allowPrivilegeEscalation: false
          runAsNonRoot: true
          capabilities:
            drop:
              - "ALL"
          seccompProfile:
            type: RuntimeDefault
        volumeMounts:
          - name: shared-volume
            mountPath: "/data/shared" # Client shared PVC. Backed by ~/.tracebloc/<workspace>/data on the client host — read your dataset from here
        env:
        # Client credentials
        - name: CLIENT_ENV
          value: "prod"
        - name: CLIENT_ID # Client credentials from tracebloc dashboard
          value: <YOUR_CLIENT_ID>
        - name: CLIENT_PASSWORD # Client credentials from tracebloc dashboard
          value: <YOUR_CLIENT_PASSWORD>

        # Storage configuration
        - name: CLIENT_PVC # value has to match the shared data PVC name in the client values.yaml
          value: "client-pvc"

        # MySQL configuration
        - name: MYSQL_HOST # value has to match the mysql deployment name in the client values.yaml
          value: "mysql-client"

        # Dataset information — paths inside the ingestor pod.
        # /data/shared is the mount of the client-pvc, which is backed by
        # ~/.tracebloc/<workspace>/data on the client host.
        - name: SRC_PATH
          value: "/data/shared" # Root of the mounted dataset directory
        - name: LABEL_FILE
          value: "/data/shared/labels.csv" # Path to the labels CSV inside the pod
        - name: TABLE_NAME
          value: <UNIQUE_TABLE_NAME> # Different for train and test, no spaces
        - name: TITLE
          value: <DATASET_TITLE> # Optional
        - name: BATCH_SIZE
          value: "4000" # Optional, defaults to 4000
        - name: LOG_LEVEL
          value: "DEBUG" # Set DEBUG, "WARNING", "INFO" or "ERROR"
      imagePullSecrets:
      - name: regcred
      volumes:
        - name: shared-volume
          persistentVolumeClaim:
            claimName: client-pvc # value has to match the shared data PVC name in the client values.yaml
      restartPolicy: Never

Specify:

JOBNAME, to distinguish between train and test data jobs.
NAMESPACE, use the same as your client.
image, your Docker image (imagePullPolicy: Always for DockerHub, IfNotPresent for local)
CLIENT_ID, CLIENT_PASSWORD from the tracebloc client view
TABLE_NAME, unique per dataset, train and test use different names, no spaces. Different names for train and test data is mandatory
LABEL_FILE, path inside the ingestor pod (under /data/shared) to the CSV with file paths and labels — must match the location of the file you placed in ~/.tracebloc/<workspace>/data/
SRC_PATH, root inside the pod where the dataset directory is mounted (/data/shared)
BATCH_SIZE is the number of entries sent to the server per request. Optional — defaults to 4000. Keep it consistent across data types. It depends on available CPU memory, not for example image size. Too large can exhaust memory. It was tested up to 10,000, but 5,000 is a safe default for most systems.
LOG_LEVEL, “WARNING” for all warnings and errors, “INFO” for all logs, “ERROR” for errors only

4. Deploy

Run the ingestor as a Kubernetes Job:

kubectl apply -f ingestor-job.yaml -n <workspace>
kubectl wait -n <workspace> --for=condition=complete job/<INGESTOR_JOB_NAME>
kubectl logs -n <workspace> job/<INGESTOR_JOB_NAME>

# Delete the job only after verifying logs
kubectl delete -n <workspace> job/<INGESTOR_JOB_NAME>

This will start a pod, run the ingestion process once, and once complete you can delete the job. IMPORTANT: You must run this process twice — once for training data and once for test data. Use different JOBNAME and TABLE_NAME values for each run (e.g. ingestor-job-train / ingestor-job-test), and set intent to TRAIN or TEST accordingly in your ingestion script.

Run as non-root

If the namespace enforces the restricted Pod Security Standard, kubectl apply will be admitted but the pod will be rejected with a warning like:

Warning: would violate PodSecurity "restricted:latest":
  allowPrivilegeEscalation != false (container "api" must set securityContext.allowPrivilegeEscalation=false),
  unrestricted capabilities (container "api" must set securityContext.capabilities.drop=["ALL"]),
  runAsNonRoot != true (pod or container "api" must set securityContext.runAsNonRoot=true),
  seccompProfile (pod or container "api" must set securityContext.seccompProfile.type to "RuntimeDefault" or "Localhost")
job.batch/ingestor-job-train-data created

Two changes are needed: 1. Add a securityContext block to the container in ingestor-job.yaml (already shown in the YAML above):

securityContext:
  allowPrivilegeEscalation: false
  runAsNonRoot: true
  capabilities:
    drop:
      - "ALL"
  seccompProfile:
    type: RuntimeDefault

2. Run the container as a non-root user. Add the following to the Dockerfile before the # Set the entrypoint line so the image ships with a UID that satisfies runAsNonRoot: true:

RUN groupadd -g 1000 app && \
    useradd -u 1000 -g 1000 -m -s /bin/bash app && \
    chown -R 1000:1000 /app

USER 1000

Rebuild and push the image, then re-apply the job. The data ingestor always runs a validation step before ingestion and moving files.

Verify Deployment

Verify if jobs and pods are deployed successfully and running:

kubectl get jobs,pods -n <workspace>
kubectl logs -n <workspace> <pod-name>

Look for “All records processed successfully” in the logs.

Dataset Management Interface

View your datasets at ai.tracebloc.io/data after successful deployment. Interface displays:

Dataset name, ID, and record count
Data type (Tabular, Image, Text) and purpose (Training/Testing)
Namespace and GPU requirements

Best Practices

Deploy jobs for training and testing simultaneously using different job names
Use consistent, descriptive table names (e.g., insurance-claims-train, insurance-claims-test)
Validate data schemas before deployment to prevent ingestion failures
Clean data before ingestion - Participants cannot view, clean, or fix raw data, so model performance depends entirely on the quality of data you provide

Troubleshooting

Recommended for debugging: Use k9s, a terminal-based Kubernetes dashboard, to monitor jobs, pods, and logs in real time. Run k9s -n <workspace> to get a live view of resources, switch between them instantly, and inspect logs or events with a few keystrokes. Compared to kubectl, it is faster and more convenient. Stale Kubernetes Job preventing new Job execution:

kubectl delete job ingestor-job -n <workspace>
kubectl logs <pod-name>

Storage Issues:

kubectl get pvc -n <workspace>

Next Steps

Define and publish your use case: Define Use Case

Need Help?

Email us at [email protected]

​Overview

​Setup options

​Declarative YAML (recommended)

​1. Add the chart repo (one-time)

​2. Stage your data on the cluster’s shared PVC

​3. Write your ingest.yaml

​4. Install once per dataset

​Custom Python script (advanced)

​Quick Setup

​Steps

​Detailed Setup

​1. Configure a script

​Install the ingestor package

​Pick the dataset template for your task

​High Level Script Structure

​Customize the script

​Needed for Tabular Data: Define Schema

​Needed for Image Classification Data: Define Image Options

​Needed for Object Detection Data: Define Image Options

​Needed for Text Data: Define File Extension

​Set CSV ingestion options

​Set Up the Ingestor

​2. Build Docker Image

​Docker Hub Setup (first-time users)

​Place data files on the client host

​Write the Dockerfile

​Build Docker Image

​3. Configure Kubernetes

​4. Deploy

​Run as non-root

​Verify Deployment

​Dataset Management Interface

​Best Practices

​Troubleshooting

​Next Steps

​Need Help?

Overview

Setup options

Declarative YAML (recommended)

1. Add the chart repo (one-time)

2. Stage your data on the cluster’s shared PVC

3. Write your `ingest.yaml`

4. Install once per dataset

Custom Python script (advanced)

Quick Setup

Steps

Detailed Setup

1. Configure a script

Install the ingestor package

Pick the dataset template for your task

High Level Script Structure

Customize the script

Needed for Tabular Data: Define Schema

Needed for Image Classification Data: Define Image Options

Needed for Object Detection Data: Define Image Options

Needed for Text Data: Define File Extension

Set CSV ingestion options

Set Up the Ingestor

2. Build Docker Image

Docker Hub Setup (first-time users)

Place data files on the client host

Write the Dockerfile

Build Docker Image

3. Configure Kubernetes

4. Deploy

Run as non-root

Verify Deployment

Dataset Management Interface

Best Practices

Troubleshooting

Next Steps

Need Help?