Last week brought a milestone the whole Pebble team is excited about. At a Plug and Play event at San Jose State University, AMD's Ted Marena gave a talk on enterprise AI — and spent a few minutes of it on us.
He walked the room through a proof-of-concept our team has been running with AMD and MiTAC Computing for the past couple of months: putting Pebble Sonar on AMD Instinct™ MI350X GPUs to see how much more useful work the same hardware could do.
This post is the longer version of what he shared.
“A very clever, very interesting product, where they actually utilize AI and have our GPUs consume less power — but the performance per watt is increased.”
Power is becoming the real ceiling on AI
If you ask the operators running today's frontier AI models what's limiting them, the answer is increasingly the same: not money, not chips — power.
GPUs are abundant and get faster every generation. But the electricity needed to feed them is much harder to scale than the silicon itself. Utility upgrades take years. On-prem facilities hit their rated power limits long before they run out of rack space. Even new colo leases now often start with a power-availability conversation, not a square-footage one.
What makes this worse is that most clusters today run their GPUs at hardware-default power caps — settings engineered for peak thermal safety, not for peak inference efficiency. Significant capacity is being burned as heat instead of being converted into tokens.
That gap is what Pebble Sonar exists to close.
What Pebble Sonar does
Pebble Sonar is a continuous profiling and optimization layer that runs alongside the inference stack. It watches the live workload — power draw, GPU utilization, clock frequencies, token throughput — and builds a real-time model of where each joule is actually being spent.
From that, it does two things at once:
- Tunes the hardware. Sets per-GPU power caps and clock targets at the operating point where the chip converts the most energy into useful work.
- Tunes the serving engine. Adjusts how vLLM packs work into each forward pass — batch sizes, concurrency, replica configuration — in coordination with the hardware settings.
Both layers treated as one joint problem, not two independent knobs.
For the AMD proof-of-concept, we wanted to see what would happen on the MI350X — AMD's newest 1,000 W flagship — across two very different workloads.
01 Llama 3.1 70B — an efficiency breakthrough
Setup: 8 × AMD Instinct MI350X. Llama 3.1 70B Instruct in FP8. vLLM as the serving engine. 512 concurrent requests, three trials of 1,200 seconds each.
We ran two configurations against identical workload parameters:
- Run A (baseline): GPUs at AMD's 1,000 W hardware default.
- Run B (Sonar): Each GPU capped at 825 W — about 82.5% of rated TDP.
The result was counter-intuitive. Lowering the power cap nearly doubled throughput.
| Metric | Run A · 1,000 W | Run B · 825 W | Change |
|---|---|---|---|
| Power cap per GPU | 1,000 W | 825 W | −17.5% |
| Total tokens / sec | 862 | 1,608 | +86.5% |
| Tokens / watt | 0.148 | 0.270 | +82.4% |
The reason it works is in the thermals. At 1,000 W, the MI350X chips were operating in a thermally constrained regime: clock frequencies fluctuated as the hardware managed its own heat, memory bandwidth contention crept in, and vLLM lost the ability to pack work into forward passes predictably — queuing jitter, occupancy loss, and wasted compute cycles.
At 825 W, the thermal instability resolved. Clocks held steady. The serving engine could finally schedule cleanly.
Put another way: lowering the power cap didn't slow the GPUs down. It stopped them from slowing themselves down.
02 Llama 3.1 405B — scaling within fixed power
The second test pushed the same idea in a different direction. What if the constraint isn't per-GPU efficiency, but a hard cap on the entire rig?
Setup: Llama 3.1 405B in FP8 — one of the largest open AI models available. vLLM 0.6.4. ShareGPT workload at 1,024-request concurrency. Total rig power strictly enforced at 6.000 kW across every run.
| Metric (P99) | Run A · 6 GPU, no Sonar | Run B · 6 GPU + Sonar | Run C · 8 GPU + Sonar |
|---|---|---|---|
| Total tokens / sec | 4,582 | 4,391 | 5,866 |
| Power draw | 5,908 W | 5,005 W | 6,002 W |
| Tokens / watt | 0.740 | 0.781 | 0.849 |
| vs. baseline | — | +5.5% tok/W | +28.0% tok/s · +14.7% tok/W |
Run B revealed the hidden slack. With Sonar tuning the existing six GPUs, the rig pulled 5,005 W instead of 5,908 W — a 15% reduction in power — while giving up only ~4% of throughput. That gap was the energy the default configuration had been quietly converting into heat rather than tokens.
Run C cashed the slack in. We brought two additional GPUs online — stranded hardware that the previous power envelope couldn't accommodate — and Sonar redistributed the recovered budget across the larger fleet, holding total rig power pinned to the 6 kW ceiling.
Throughput climbed 28%. The power bill didn't move. Latency held within measurement noise (TTFT P99 was 9.94 s vs. 9.80 s at baseline), and the GPU hotspot stayed well inside MI350X operating limits.
The pattern underneath both experiments
Read individually, each experiment is interesting. Read together, they trace the same insight from two angles:
The default operating point isn't the optimal one. Hardware-default power caps are engineered for thermal safety, not for workload efficiency. The optimal point is almost always somewhere below the ceiling.
The waste is invisible by default. No standard tooling in vLLM, ROCm, or cluster managers tells you where on the efficiency curve a workload is actually sitting.
The recoverable headroom is significant. ~15% of a power budget reclaimed in the 405B run; a near-2× throughput swing on the 70B run.
These aren't exotic numbers. They're what happens when a workload that was running at factory defaults is tuned to the operating point it actually wants.
“It's been great to work with them.”
What's next
We're continuing the work with AMD and MiTAC Computing on broader MI350X deployments, and extending the same approach to other accelerators in our customers' clusters.
Pebble's second product, Pebble Flex, takes this further. It lets a GPU cluster participate in utility flexibility programs — automatically reducing and restoring power on grid signals without breaking workload SLAs — turning stranded grid capacity into deployable AI capacity. Ted alluded to it on stage:
“…they have a utility solution where they're able to throttle the energy back, still having SLA agreements for your workloads, but being able to significantly reduce the power — and that allows the user to actually buy power at a lower rate.”
That's where we're headed: power not as a fixed constraint, but as a dynamic resource your AI infrastructure can negotiate with.
Thank-yous and further reading
Huge thanks to AMD and MiTAC Computing for the hardware, the partnership, and the time on stage at the Plug and Play event. We're grateful for the collaboration — and we're just getting started.
The full methodology, per-trial measurements, and operating conditions for both experiments are available in our Pebble Sonar case-study collection, or by request at tech@gopebble.com.