We watched two rented GPUs for 24 hours: 21 thermal throttles, and throughput didn't move

A one-time check tells you what a GPU looks like when you get it. It can't tell you what happens at 3 a.m., eighteen hours into a rental, when the room is warmer and the card has been working all day. In our H100 study, the problems we found (power limits set below NVIDIA's range) were there from the start. So we asked the other question: does a rented GPU change while you use it?

What we did. We rented an RTX 4090 and an A40 on RunPod Secure Cloud and served Qwen3-8B on each with vLLM for 24 hours, under a steady load that kept both GPUs busy essentially all the time (6 and 3 requests in flight, prompts of about 700 words, 256 output tokens each). Our open-source checker ran next to it in watch mode, sampling every GPU once a minute and reporting to TokenWatt Monitor, which emails on every change.

The RTX 4090 ran into its thermal limits 21 times

Most of the day the 4090 sat at 64–69 °C. Every so often, about one five-minute window in six, its temperature spiked to 77–82 °C. Seventeen times the driver's software thermal slowdown kicked in, and four times the card's own hardware thermal slowdown did, which is the GPU protecting itself rather than following a setting. Each episode lasted about a minute. During them the SM clock dipped from about 2,630 MHz to as low as 2,355 MHz (−10%), then recovered.

The spikes came and went with no change in the load we were sending, a pattern that points to cooling in the room or the chassis, not to the card. We can't see the host's airflow from inside a container, so that part is an inference.

The A40, in the same cloud and just as busy, never came close: 65 °C at the hottest moment of the day, no thermal events, and no change in its power limit.

Throughput didn't notice

Here is the part that changed how we think about alerts. Over 144 ten-minute windows the 4090 delivered 294 to 298 output tokens per second. The worst ten minutes of the day were 0.7% below the average, and latency held at 5.2 s median, 5.9 s at the worst p99. A one-minute, 10% clock dip inside a ten-minute window is simply too small to see in served tokens.

So a thermal throttle is worth knowing about, but it isn't automatically a problem. What would matter is the same signal sustained: throttling for minutes at a time, clocks collapsing to half, or a power limit lowered mid-rental. None of that happened here, on either card, over 24 hours.

We're changing Monitor accordingly: brief software throttles become warnings collected into a daily summary, while hardware thermal slowdowns, sustained throttling and power-limit changes still alert immediately, with the duration and the clock drop in the email.

What this means if you rent GPUs

• Check once at the start, then watch. The expensive surprises in our tests were set before we ever started (an H100 at 40% of its normal clock, NVL cards capped at 310 W). A two-minute check catches those: pip install tokenwatt-check && tokenwatt-check (what it checks: GPU health check guide).
• Consumer cards in datacenters run close to their thermal limits. The 4090 is built for a desktop case. Under steady load in a dense server it touched its limits about once an hour. That didn't cost throughput at our load, but it leaves less margin when a card is pushed to its limit.
• Judge throttling by its effect. Count tokens served, not throttle flags. A card that throttles for a minute an hour and serves within 1% of its average is healthy; one whose throughput drops for an hour is not.

Limits

One card of each model, one platform (RunPod Secure Cloud), 24 hours, one model and one steady load. The 4090 drew 306 W of its 450 W limit with this request mix; larger batches would draw more power and heat. We measured board power, not wall power. The cause of the temperature spikes is our inference from the pattern.

Method: how we measure. Watch your own GPUs: TokenWatt Monitor, free during early access.