Tuning · 2026-10-01 · Checked 2026-10-10
Power-limit your card, keep your speed
Token generation reads active weights from VRAM for every new token. GPU cores wait for memory during this process. Generation speed tracks memory bandwidth (GB/s) more than power or clock speed.
Prompt processing is compute work. It slows down more when you cut the power.
Check your current limits
nvidia-smi -q -d POWER
This shows the current, default, minimum and maximum power limit of each card.
Set a lower limit
sudo nvidia-smi -pm 1
sudo nvidia-smi -i 0 -pl 280
The first line turns on persistence mode, the second caps GPU 0 at 280 W. Use -i 1 for the second card. Start around 70 to 80 % of the default limit, then measure.
Measure before and after
llama-bench -m model.gguf -ngl 99 -p 512 -n 128
Compare tg128 (generation) and pp512 (prompt) at each limit. A wall power meter tells you what you save in W. Keep the limit that gives up little tg128 for a clear drop in watts.
Make it survive a reboot
The limit resets at boot. A small systemd unit sets it again (create it with sudo vi /etc/systemd/system/gpu-power-limit.service):
[Unit]
Description=GPU power limit
After=nvidia-persistenced.service
[Service]
Type=oneshot
ExecStart=/usr/bin/nvidia-smi -i 0 -pl 280
[Install]
WantedBy=multi-user.target
Then sudo systemctl enable --now gpu-power-limit.service.
AMD cards
rocm-smi --showpower
sudo rocm-smi -d 0 --setpoweroverdrive 250
Same idea: lower the cap, run the benchmark, keep what works.
Questions
- Does lowering the power limit slow generation down?
- Far less than expected. Token generation tracks memory bandwidth more than power, so 70 to 80% of the default limit often costs very little generation speed. Prompt processing is compute work and does slow down more.
- How do I set a power limit and make it survive a reboot?
sudo nvidia-smi -pm 1thensudo nvidia-smi -i 0 -pl 280, plus a small systemd unit that reapplies the limit at boot. Measuretg128andpp512with llama-bench before and after.- Which number should I optimise?
tg128, the generation speed: it is the number people feel. Keep the limit that gives up littletg128for a clear drop in watts.