Most people spin up a cloud GPU, pick whatever instance looks reasonable, and end up either underprovisioning or massively overpaying. The real skill is knowing your memory requirements before you provision, matching your workload type to the right GPU class, and setting things up so you're not bleeding money on idle compute. Once those pieces click, running AI workloads in the cloud gets a lot more predictable. if i edit the first paragraph.
TLDR:
PyTorch and TensorFlow both run on CUDA by default, so most AI workloads are tied to that programming model. Switching stacks means rewriting a substantial amount of code, which is why cloud GPU instances across nearly every provider use the same hardware family. The A100 vs H100 comparison is the one that matters most for picking between training and inference tiers.
A cloud GPU gives you on-demand access to high-performance graphics processing units hosted in a remote data center, billed by the hour or second. Instead of buying a $10,000+ GPU outright, you rent exactly what you need, run your workload, and stop paying the moment you're done.
For AI workloads, this matters because GPUs handle the parallel matrix operations that power model training and inference far faster than CPUs can. A task that takes days on a CPU often completes in hours on a GPU.
There are three situations where cloud GPUs make the most practical sense:
Cloud GPUs are the right call for most AI teams, but two scenarios push toward dedicated hardware: continuous 24/7 inference at high utilization, where hourly rates can exceed the cost of owned hardware over the same period, and sub-millisecond latency requirements where network round-trips matter. For everything else — spot usage, experimentation, variable load — cloud rental vs buying almost always wins on cost and flexibility.
GPU memory is the single most common bottleneck when running AI workloads in the cloud. Get it wrong and your job crashes mid-run, or you end up paying for a GPU you're only half-using.
The rule of thumb is straightforward: your model weights, activations, and optimizer states all need to fit in VRAM simultaneously during training. A 7B parameter model in fp16 requires roughly 14GB of VRAM just for weights, and that number climbs fast once you factor in gradients and optimizer state. For a detailed breakdown of how memory scales with model size and precision, Modal's VRAM fine-tuning guide is a useful reference.
Different workloads have very different memory profiles, and choosing the right GPU tier starts with understanding where your job falls.
One practical check before provisioning: run a quick memory estimate using tools like Hugging Face's model memory calculator or the nvidia-smi output from a smaller test run. Undershooting VRAM leads to OOM errors; overshooting means paying for idle memory you never touch.
Training and inference have fundamentally different hardware requirements, and picking the wrong configuration wastes money fast.
Training runs are memory-intensive and compute-bound. You need high VRAM, fast interconnects between GPUs, and consistent throughput over hours or days. Fine-tuning a 7B parameter model on a single H100 80GB can take anywhere from 30 minutes to several hours depending on dataset size, while pre-training larger models often requires multi-GPU setups with NVLink or InfiniBand to keep inter-GPU communication from becoming the bottleneck.
Inference workloads look completely different. Latency matters more than raw throughput, and many production inference jobs run efficiently on smaller, cheaper GPUs. A model that required an H100 to train might serve requests perfectly well on an L4 or L40S at a fraction of the cost.
Here is a practical breakdown of how to think about configuration by task type:
| workload type | key requirement | recommended gpu class | avoid |
|---|---|---|---|
| LLM pre-training | max VRAM + fast interconnect | H100 SXM (NVLink) | single small GPU |
| fine-tuning (7B-13B) | 40-80GB VRAM | H100 PCIe or A100 | T4, L4 |
| inference (batch) | throughput per dollar | L40S, A100 | H100 (overkill) |
| inference (real-time) | low latency | L4, T4, A10G | multi-GPU setups |
| diffusion / image gen | VRAM + memory bandwidth | A100, L40S | CPU-only |
| experimentation / dev | cost minimization | T4, L4 | H100 |
One often-overlooked factor is memory bandwidth versus raw compute. An H100 SXM specifications include around 3.35 TB/s of memory bandwidth, which matters enormously for transformer inference where memory reads dominate the compute cycle. For smaller batch sizes, that bandwidth goes underused, meaning you pay for capacity you never touch.
Start by profiling your workload before provisioning. Tools like nvidia-smi, PyTorch's built-in profiler, and Weights and Biases can tell you whether you are compute-bound, memory-bound, or simply bottlenecked on data loading, each of which points to a different hardware solution.
Picking the right provider shapes everything: cost, GPU availability, setup time, and how well your workload actually runs. The market has consolidated around a few major players, but they serve very different use cases.
There are three broad categories worth comparing:
Cost per GPU-hour gets most of the attention, but a few other factors matter just as much in practice:
Matching provider type to workload type is the right starting point before comparing specific GPU prices.
Getting a GPU instance running takes less than 10 minutes on most providers. The process follows the same general pattern regardless of which cloud you choose.
Start by creating an account on your chosen provider and adding a payment method. Most providers require identity verification before granting access to GPU instances, and some require you to request quota increases before you can spin up high-end hardware like H100s. Plan for a 24 to 48 hour wait if you need quota approval.
Once your account is ready, the setup flow looks like this:
After launching, you connect via SSH using the key pair you set up during provisioning. From there, verify your GPU is visible by running nvidia-smi, which shows GPU utilization, VRAM usage, and driver version.
A quick sanity check before starting any job:
nvidia-smi
python3 -c "import torch; print(torch.cuda.is_available())"
Both commands should return clean output confirming your GPU is accessible and your ML framework can see it. Once confirmed, you are ready to run training scripts, launch inference servers, or start a Jupyter notebook session.
Most providers give you at least three ways in: direct SSH, a browser-based terminal, and VSCode Remote SSH. The right choice depends on your workflow, but the underlying access pattern is consistent across all of them.
SSH is the foundation. Once your instance is running, connect using the key pair you configured during setup:
ssh -i ~/.ssh/your_key ubuntu@<instance_ip>
For Jupyter notebooks, port forwarding bridges the remote session to your local browser:
ssh -L 8888:localhost:8888 -i ~/.ssh/your_key ubuntu@<instance_ip>
Start Jupyter on the remote machine, then open localhost:8888 in your local browser. Your notebook runs against the GPU with no local compute involved.
VSCode's Remote SSH extension connects to your instance and mounts the remote filesystem as if it were local. Install the extension, add your instance IP to your SSH config file, and click "Connect to Host." From there, you get full IDE features including file browsing, an integrated terminal, and debugger support, all running against the remote GPU environment.
For file access outside of an active SSH session, SSHFS lets you mount your instance directory directly to your local machine:
sshfs -i ~/.ssh/your_key ubuntu@<instance_ip>:/home/ubuntu ~/remote-gpu
Once mounted, you can read and write files through your local file manager or any editor, without manually copying files back and forth. When you're done, unmount with fusermount -u ~/remote-gpu on Linux or through Finder on macOS.
Once your instance is running, confirming that the GPU is visible and properly configured is the first thing you should do before running any workloads.
The quickest check is nvidia-smi, which reports driver version, CUDA version, GPU model, memory, and current utilization in a single output. If the command returns an error, the driver is either missing or incorrectly installed. On most cloud providers, GPU instances come with drivers pre-installed, but it is always worth verifying.
CUDA version mismatches are one of the most common causes of failed AI workloads. After confirming the driver, check that your CUDA version aligns with your framework.
For PyTorch, run the following in Python:
import torch
print(torch.cuda.is_available())
print(torch.cuda.get_device_name(0))
For TensorFlow:
import tensorflow as tf
print(tf.config.list_physical_devices('GPU'))
Both should return your GPU name or a non-empty device list. A False or empty result means PyTorch or TensorFlow cannot see the GPU, which usually points to a CUDA or cuDNN version mismatch, not a hardware problem.
nvidia-smi first and confirm the driver is loaded. If it fails entirely, the driver package may need reinstalling via your cloud provider's documentation.Once your instance is running, you have two main paths for setting up your AI environment: pull a pre-built container image or install dependencies manually.
Pre-built containers are almost always faster. Most providers offer CUDA-compatible Docker images with PyTorch or TensorFlow pre-installed, and pulling one takes seconds compared to resolving dependency conflicts from scratch. For most AI workloads, starting with an official PyTorch CUDA image from Docker Hub or the NGC catalog gets you running immediately.
Custom installs make sense when you need specific library versions or have dependencies that conflict with the base image. Conda handles this well because it manages CUDA toolkit versions independently of the system driver:
Or if you prefer pip over Conda:
pip install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu121
Conda and pip both work. Conda is more reliable when you need to pin a specific CUDA toolkit version; pip is faster and sufficient when the instance already has the right driver installed.
conda create -n myenv python=3.10
conda activate myenv
conda install pytorch torchvision torchaudio pytorch-cuda
After setup, confirm your GPU is actually accessible before running any workload. A quick check in Python catches driver mismatches or misconfigured environments early:
import torch
print(torch.cuda.is_available())
print(torch.cuda.get_device_name(0))
If cuda.is_available() returns False, the most common culprits are a CUDA version mismatch between PyTorch and the installed driver, or a container that was not launched with GPU passthrough flags. On Docker, you need to pass --gpus all at runtime, otherwise the container has no visibility into the host GPU.
By default, whatever you store on a cloud GPU instance's local disk disappears the moment that instance is terminated. This catches people off guard more often than you'd expect, especially when a training run finishes and the instance auto-shuts down.
The key distinction is between ephemeral storage and persistent volumes. Ephemeral storage is fast, tied directly to the instance, and gone when it stops. Persistent volumes are network-attached, survive instance deletion, and can be detached and reattached to new instances. Most providers let you attach a persistent volume at launch or after the fact.
Structure your data based on how frequently it needs to be read and how replaceable it is:
Snapshots capture your instance's full disk state at a point in time. The right moment to snapshot is right after you've installed dependencies and confirmed your environment works, before starting any long job. If something goes wrong mid-run, you can restore to a known-good state instead of repeating setup.
One cost consideration: snapshots and persistent volumes are billed separately from compute, and data transfer between storage and compute can add fees depending on the provider. Keeping your dataset storage in the same region as your compute instances avoids most of those charges.
Three billing models structure the market, and each suits a different usage pattern. The advertised GPU-hour rate is rarely your actual cost.
On-demand pricing charges you by the hour or second with no commitment. You pay more per unit than other options, but you get full flexibility. Per-second billing makes a real difference for short jobs since a 20-minute training run shouldn't cost a full hour.
Spot instance availability pulls from spare capacity and can cut costs by 50 to 80% compared to on-demand rates. The tradeoff is interruption risk: the provider can reclaim your instance with little warning, so spot instances only make sense for workloads that checkpoint regularly and can resume cleanly.
Reserved capacity requires committing to a specific GPU type for a defined period, typically one month to one year, in exchange for a meaningful discount. If your usage is predictable and continuous, reserved pricing often wins on total spend, but you're paying whether you run jobs or not.
A few line items frequently catch people off guard beyond the base compute rate:
The real number to track is total cost per completed job, not cost per GPU-hour. That framing forces you to account for idle time, storage, and transfer fees together.
GPU utilization across AI and ML workloads averaged just 5%, meaning most provisioned capacity sits idle while billing continues. For most teams, fixing utilization matters more than negotiating a better hourly rate.
Rightsizing is the highest-impact starting point. Teams routinely over-provision because picking the wrong GPU feels risky, so they default to the biggest available option. Run a short profiling job first, check nvidia-smi for actual memory and compute consumption, then match your instance to what your workload genuinely needs. Dropping from an 80GB GPU to a 40GB GPU on a job that uses 22GB can cut your compute cost roughly in half.
Beyond rightsizing, a few habits compound savings considerably:
With your environment configured and GPU confirmed, the remaining steps are about moving data in, running the job, and getting results out.
Upload your dataset using rsync, which handles large file transfers more reliably than scp and supports resuming interrupted transfers:
rsync -avz --progress ./data/ ubuntu@<instance_ip>:/home/ubuntu/data/
Once your data is in place, avoid running long training jobs directly in your SSH session. If the connection drops, the process dies. Use tmux or screen to create a persistent session that keeps running regardless of your connection state:
tmux new -s training
python train.py --data /home/ubuntu/data --epochs 10 --checkpoint-dir /home/ubuntu/checkpoints
Detach with Ctrl+B D. Reconnect later with tmux attach -t training. The job keeps running without you.
Two sources cover most of what you need to watch during a long run:
nvidia-smi dmon -s u streams GPU utilization and memory usage every second. If utilization is consistently below 60%, your bottleneck is likely data loading, not compute.python train.py ... 2>&1 | tee training.logTools like Weights and Biases give you richer real-time dashboards if you want metrics accessible outside the instance without SSHing back in.
If a spot instance gets preempted, your last saved checkpoint is your recovery point. Resume from it by pointing your script to the most recent checkpoint file instead of restarting from scratch.
When the run finishes, pull your outputs back locally:
Here is a minimal PyTorch pattern for saving and resuming checkpoints:
# Save a checkpoint at the end of each epoch
torch.save({ 'epoch': epoch, 'model_state_dict': model.state_dict(), 'optimizer_state_dict': optimizer.state_dict(), 'loss': loss,
}, f'/home/ubuntu/checkpoints/ckpt_epoch_{epoch}.pt') # Resume: load and restore state
checkpoint = torch.load('/home/ubuntu/checkpoints/ckpt_epoch_5.pt')
model.load_state_dict(checkpoint['model_state_dict'])
optimizer.load_state_dict(checkpoint['optimizer_state_dict'])
start_epoch = checkpoint['epoch'] + 1
rsync -avz ubuntu@<instance_ip>:/home/ubuntu/checkpoints/ ./local-checkpoints/
Stop the instance immediately after. That single habit prevents the idle billing that accumulates fastest after a job wraps.
Thunder Compute is built around one core idea: developers shouldn't spend time fighting infrastructure to run AI workloads. GPU instances launch in seconds with pre-built environments for PyTorch, Ollama, Comfy UI, and other common workflows, so you spend time computing, not configuring.
The VS Code extension removes manual SSH setup entirely. One click connects you to a persistent cloud environment directly from your editor. Snapshots preserve that environment, including notebooks, installed packages, and datasets, so stopping an instance doesn't mean losing your setup. When you come back, everything is exactly where you left it.
GPU swapping is where the flexibility shows up in practice. Start prototyping on an RTX A6000 at $0.35/hr (there are proven ways to cut cloud GPU costs during this phase), then switch to an H100 for a heavier training run without tearing down your environment. The hardware changes; your configured workspace doesn't. Pay-per-minute billing keeps costs proportional to actual compute time, not rounded up to the nearest hour.
For teams comparing options, H100 instances start at $1.38/hr, with on-demand rates running 80% below AWS across comparable GPU tiers. Training, fine-tuning, and inference workloads all run through the same persistent environment, following you across hardware tiers as your workload requirements change.
Most of what makes cloud GPUs expensive is avoidable. Over-provisioned instances, idle compute, and missed shutdowns add up fast, but they're all fixable with a bit of upfront planning. Pick the GPU your workload actually needs, checkpoint regularly if you're on spot instances, and track total job cost instead of hourly rate. Thunder Compute is worth a look if you want pay-per-minute billing and GPU swapping without rebuilding your environment each time.
The fastest path is choosing a provider that ships pre-built environments for PyTorch, Ollama, or your framework of choice, so your first command is a training script, not a dependency install. On Thunder Compute, GPU instances launch in seconds with pre-configured templates, and the VS Code extension connects you to a persistent environment in one click without any manual SSH configuration.
Start by estimating your VRAM requirements: a 7B parameter model in fp16 needs roughly 14GB just for weights, which fits comfortably on an RTX A6000 at $0.35/hr, while fine-tuning models in the 13B to 70B range pushes you into H100 territory. Run nvidia-smi on a small test job first to check actual memory consumption before committing to a more expensive tier.
For active development and experimentation, Thunder Compute's H100 pricing starts at $1.38/hr against AWS on-demand rates that run well above $10/hr per GPU, a difference that adds up fast on iterative workloads. AWS makes sense when you need deep integration with managed services like SageMaker; for straightforward training and inference jobs where you want to stay in VS Code and swap hardware as needs change, the cost gap is hard to ignore.
Attach a persistent volume at launch to store datasets and model checkpoints separately from the instance disk, and snapshot your environment after installing dependencies so you can restore it without repeating setup. On Thunder Compute, persistent storage and snapshots are built into the workflow, so your notebooks, packages, and data remain intact across sessions regardless of which GPU tier you are running on.
Persistent storage bills continuously even when your instance is stopped, egress fees apply when pulling large model artifacts out of a provider's network, and idle instances after a job completes are the single most avoidable cost. Track total cost per completed job, not cost per GPU-hour, to get an accurate picture that accounts for all three.
