Container-Based Task Creation

A container task runs a Docker image on a GPU instance until the process exits. Use this approach for training jobs, batch inference, data processing, or any workload that runs to completion.

Container requirements

NVCT does not impose a server or health check requirement. The container only needs to:

• Perform its workload.
• Exit with code 0 on success, or a non-zero code on failure.

GPU drivers and CUDA libraries are available on the host. Use an image based on an appropriate CUDA base image for your workload.

Creating a container task

$
# Minimal example using CLI flags
$
nvcf-cli task create \
>
  --name my-training-job \
>
  --gpu H100 \
>
  --instance-type GPU.H100_1x \
>
  --image my-registry/training:latest
$
$
# With arguments, environment variables, and secrets
$
nvcf-cli task create \
>
  --name my-training-job \
>
  --gpu H100 \
>
  --instance-type GPU.H100_1x \
>
  --image my-registry/training:latest \
>
  --container-args "--epochs 10 --batch-size 32" \
>
  --container-env DATASET_PATH=/data/train \
>
  --container-env LOG_LEVEL=info \
>
  --secrets S3_ACCESS_KEY=... \
>
  --max-runtime PT4H \
>
  --result-strategy NONE
$
$
# From a JSON file (recommended for repeatable configurations)
$
nvcf-cli task create --input-file task.json

Example JSON configuration

1
{
2
  "name": "my-training-job",
3
  "description": "Fine-tuning run for Q4 experiment",
4
  "tags": ["training", "q4"],
5
  "gpuSpecification": {
6
    "gpu": "H100",
7
    "instanceType": "GPU.H100_1x"
8
  },
9
  "containerImage": "my-registry/training:latest",
10
  "containerArgs": "--epochs 10 --batch-size 32",
11
  "containerEnvironment": [
12
    {"key": "DATASET_PATH", "value": "/data/train"},
13
    {"key": "LOG_LEVEL", "value": "info"}
14
  ],
15
  "maxRuntimeDuration": "PT4H",
16
  "maxQueuedDuration": "PT72H",
17
  "resultHandlingStrategy": "NONE",
18
  "secrets": [
19
    {"name": "S3_ACCESS_KEY", "value": "..."},
20
    {"name": "S3_SECRET_KEY", "value": "..."}
21
  ]
22
}

Environment variables

The task system injects the following environment variables into the container:

GPU specification

Runtime limits

A task that exceeds maxRuntimeDuration moves to EXCEEDED_MAX_RUNTIME_DURATION status. A task that is not scheduled within maxQueuedDuration moves to EXCEEDED_MAX_QUEUED_DURATION status.

maxRuntimeDuration has no default or maximum: omit it and the task runs with no time limit.

Secrets

Secrets are delivered to the container as a JSON file at /var/secrets/secrets.json. The file has the form:

1
{
2
  "SECRET_NAME": "secret-value",
3
  "ANOTHER_SECRET": "another-value"
4
}

Provide secrets via --secrets NAME=value on the CLI or as a secrets array in the JSON file. Secret values are stored encrypted and are not returned by default in task detail responses.

To rotate secrets on a running task, use update-secrets. The file at /var/secrets/secrets.json is refreshed automatically with the new values:

$
nvcf-cli task update-secrets --secrets NEW_KEY=new-value

Model and resource artifacts

Attach model or resource artifacts to a task using the --models and --resources flags (format: name:version:uri) or via the JSON models and resources arrays. The task system downloads and mounts them before the container starts:

• Models are available at /config/models/{modelName}
• Resources are available at /config/resources/{resourceName}

Progress file

The progress file is the primary mechanism for signaling task progress and completion. The task system reads it to determine whether the container is alive and when to transition the task to COMPLETED.

The container must write a JSON object to NVCT_PROGRESS_FILE_PATH:

1
{
2
  "taskId": "579ad430-34b9-4a6e-9537-a060db4a9e6c",
3
  "percentComplete": 42,
4
  "name": "checkpoint-step-1000",
5
  "metadata": {
6
    "step": 1000,
7
    "loss": 0.312
8
  },
9
  "lastUpdatedAt": "2025-01-02T15:04:05.999999999Z"
10
}

Field requirements:

Heartbeat: If lastUpdatedAt falls more than 5 minutes behind the current time and percentComplete is not 100, the task moves to ERRORED.

Completion: Set percentComplete to 100 when the workload finishes. That write transitions the task to COMPLETED. A container that exits without writing 100 will not reach COMPLETED.

Intermediate results: For checkpoints or partial outputs, write percentComplete between 1 and 99 with a name for each checkpoint. When using UPLOAD, each checkpoint name must be unique and you must not write to a result directory after updating name to a new value, since the system reads resultsDir to upload files by name. With NONE, there is no write-ordering constraint on the results directory.

Atomic writes: Write to a temporary file and rename it over NVCT_PROGRESS_FILE_PATH to avoid the task system reading a partial file. In Python: os.rename(tmp_path, progress_path).

Result handling

With NONE, the container is responsible for delivering its own outputs — for example, writing to a volume or pushing to external storage using credentials from task secrets. The task system does not upload anything but still tracks progress through the progress file.

After the task completes, list any result names the container reported:

$
nvcf-cli task results

Monitoring a task

$
# Check status and progress
$
nvcf-cli task get
$
$
# Stream lifecycle events
$
nvcf-cli task events

A task get response includes percentComplete and healthInfo (the GPU type, instance type, and any error message from the platform) in addition to the lifecycle status.

To stop a running or queued task:

$
# Stop execution gracefully; task moves to CANCELED
$
nvcf-cli task cancel
$
$
# Remove the task and its secrets; allows the container to finish within
$
# terminationGracePeriodDuration before the instance is terminated
$
nvcf-cli task delete