Cloud GPU for LLM serving with vLLM

Pick a GPU before you open the template

vLLM serving on NovaPod starts with VRAM, not with the lowest ask. A 7B or 8B model in 16-bit weights often fits a 24 GB card if the context stays short. A 70B-class model, or a long context with a large batch, needs a datacenter card such as H100 or H200. Open the GPU list, read the memory on the card, then open that card's price page and compare the live ask with the median.

If the ask is far below the median, use it for a smoke test only. Serve a few prompts, watch whether the process stays up, and then decide whether to keep the rental. The hourly number on the marketplace is the rate billed per second while the pod is running. Stopping the pod stops that clock.

Launch the vLLM template and check one request

From Templates, start the vLLM image instead of installing CUDA by hand. The first boot should expose an OpenAI-compatible HTTP route. Send one short chat completion with max tokens set low, and confirm the JSON shape, the model name, and a timeout you can live with. If the route never opens, the usual causes are the pod still pulling the image, a port that is not the one in the template notes, or a model path that does not exist on disk.

Keep the first test at concurrency 1. Raise batch size only after a single request returns. If you see out-of-memory errors, drop the context length or move to the next VRAM class on the marketplace. Do not debug a 70B model on a 24 GB card and then open a support ticket about CUDA. The card was the wrong size.

Move traffic only after the smoke test

When the single-request test is stable, raise concurrency in small steps and watch latency, not just tokens per second. A cheap ask that queues for thirty seconds is more expensive than a higher ask that answers immediately, if your users are waiting. Write down the GPU name, the ask, and the context length that passed. That note is what you paste if you later ask support why a pod died.

Point production DNS or your app config at the pod only after that note exists. Keep a second, smaller model ready if the primary card disappears from the market. Spot-style offers can end. Checkpoints are not the issue for serving, but the model weights must already be on the image or on the disk you attached, or the next boot starts with a download.

Shut the pod off when the endpoint is no longer receiving traffic. Per-second billing still charges an idle server. The Templates page is the place to relaunch the same image later. The H100 SXM and H200 price pages are the place to see whether the card you tested is still at a sensible ask.