Tuning · 2026-10-01 · Checked 2026-10-10

Power-limit your card, keep your speed

Token generation reads active weights from VRAM for every new token. GPU cores wait for memory during this process. Generation speed tracks memory bandwidth (GB/s) more than power or clock speed.

Prompt processing is compute work. It slows down more when you cut the power.

Check your current limits

nvidia-smi -q -d POWER

This shows the current, default, minimum and maximum power limit of each card.

Set a lower limit

sudo nvidia-smi -pm 1
sudo nvidia-smi -i 0 -pl 280

The first line turns on persistence mode, the second caps GPU 0 at 280 W. Use -i 1 for the second card. Start around 70 to 80 % of the default limit, then measure.

Measure before and after

llama-bench -m model.gguf -ngl 99 -p 512 -n 128

Compare tg128 (generation) and pp512 (prompt) at each limit. A wall power meter tells you what you save in W. Keep the limit that gives up little tg128 for a clear drop in watts.

Make it survive a reboot

The limit resets at boot. A small systemd unit sets it again (create it with sudo vi /etc/systemd/system/gpu-power-limit.service):

[Unit]
Description=GPU power limit
After=nvidia-persistenced.service

[Service]
Type=oneshot
ExecStart=/usr/bin/nvidia-smi -i 0 -pl 280

[Install]
WantedBy=multi-user.target

Then sudo systemctl enable --now gpu-power-limit.service.

AMD cards

rocm-smi --showpower
sudo rocm-smi -d 0 --setpoweroverdrive 250

Same idea: lower the cap, run the benchmark, keep what works.

Questions

Does lowering the power limit slow generation down?
Far less than expected. Token generation tracks memory bandwidth more than power, so 70 to 80% of the default limit often costs very little generation speed. Prompt processing is compute work and does slow down more.
How do I set a power limit and make it survive a reboot?
sudo nvidia-smi -pm 1 then sudo nvidia-smi -i 0 -pl 280, plus a small systemd unit that reapplies the limit at boot. Measure tg128 and pp512 with llama-bench before and after.
Which number should I optimise?
tg128, the generation speed: it is the number people feel. Keep the limit that gives up little tg128 for a clear drop in watts.