NovaPodWorkload Playbook
GPU for LoRA fine-tuning
Use this flow when you want cheap iteration speed first, then scale once the run is stable. The goal is to reduce failed long runs by validating data, tokenizer, and trainer settings early.
Before you launch
- - Dataset shard and tokenizer are prepared before launch.
- - Start with one GPU and short sequence length for smoke test.
- - Disk budget includes checkpoints, logs, and merged adapters.
- - Keep an API key ready if the run is managed by an agent script.
Execution steps
STEP 01
Start from a LoRA-friendly template
Pick an Unsloth or PyTorch template in Deploy so CUDA and trainer dependencies are preconfigured.
STEP 02
Run a 5-10 minute validation epoch
Verify loss decreases, checkpoints write correctly, and output format matches your downstream inference.
STEP 03
Tune memory and throughput
Adjust batch size, gradient accumulation, and sequence length to fit VRAM without OOM restarts.
STEP 04
Scale only after stable baseline
Move to larger VRAM or multiple GPUs only when a single-GPU run is reproducible end to end.
Prove the LoRA run on one GPU first
LoRA fine-tuning on a rented GPU fails most often because the dataset, tokenizer, and sequence length were never checked on a short run. Start on one card. A 24 GB GPU is enough for a small adapter smoke test. An H100 or H200 is for the run you already trust, not for the first time you see the loss curve. Launch an Unsloth or PyTorch template from the Templates page so CUDA and the trainer are already installed.
Copy the dataset and tokenizer onto the pod before you start the trainer. A five to ten minute run should write a checkpoint, show loss moving, and produce a sample that matches the format you will serve later. If the checkpoint directory is empty, fix the path before you buy another hour. Disk has to hold the base weights, the adapter, and the logs. A full disk looks like a random trainer crash.
Fit the batch to the card, then scale
Out-of-memory on the first long run means the batch or the sequence length does not fit. Lower the batch, raise gradient accumulation, and keep the same seed so you can tell a code change from a hardware change. Only after one GPU finishes a reproducible epoch should you move to more VRAM or more than one GPU. Multi-GPU is a second experiment. It is not a way to skip the broken config.
Read the live ask and the median on that GPU's price page before a multi-hour job. An ask far under the median can be a bargain or a host that will not stay up. Save checkpoints often enough that a dead pod costs minutes, not the whole run. Export the adapter, copy it off the pod, and stop the rental. The per-second clock does not know that training finished.
What to record if the run fails
Keep the template name, GPU name, sequence length, batch size, and the last twenty lines of the trainer log. That is the useful support note. Saying the GPU is broken, without those fields, sends you back to the same smoke test. If SSH itself fails, the pod may already be stopped. Open the rental and copy the current host.
When the adapter is good, serve it with the vLLM playbook on a card that can hold the base model plus the adapter. Do not leave the training pod running as an inference server unless you have compared that ask with a serving template. Training images and serving images are different boots from Templates.
Pre-warmed Container Cache & 1s Crypto Settlement
Base CUDA runtimes and foundational weights are distributed across verified high-bandwidth nodes. Boot instances in under 15 seconds with zero KYC and pay with USDT, USDC, SOL, or ETH.
Budget and reliability tips
- - Compare ask versus median before long jobs; outliers can be cheap but unstable.
- - Use restartable checkpoints so spot-like interruptions are survivable.
- - Stop idle rentals immediately after adapter export to avoid overnight burn.