Idle Watts Audit: Halve Your Lab's Draw

Your Server Is Doing Nothing, Loudly

You set up a box to serve a few containers. It sits at 4% CPU all day. The wall meter says 45 watts. That is a lot of electricity for a machine that is mostly staring at the wall.

Here is the claim of this post: most of that idle draw does not come from the CPU model. It comes from a handful of blockers. PCIe power management is off. One device keeps the CPU package out of its deep sleep states. Disks spin 24 hours a day. A discrete GPU can idle at 10 to 20 watts. A container polls something every second. Powertop and a wall meter find most of them in one evening.

A caveat up front. “Half” is a typical result on a misconfigured consumer box, and it is not a promise. A desktop-class board with a spinning-disk array and an old GPU has a lot of room. An enterprise server with a BMC, redundant PSUs, and fans has a much higher floor, and tuning will move it less. Any numbers below are illustrative ranges, not our measurements. Your hardware will land somewhere else.

For how to measure with a plug-in meter, see the earlier Kill A Watt power tour. For turning watts into a bill, see the power math post. This post is the part in between: what to change.

The Method: One Change, One Measurement

Everyone skips this part, then cannot tell which tweak helped. Do it like this.

1. Boot the box and let it sit with your normal containers running for 10 minutes.
2. Read the wall meter. Write the number down. This is the baseline.
3. Change one thing.
4. Wait a few minutes and read the meter again.
5. Keep the change if it helps. Revert it if it breaks something or does nothing.

Software estimates are not a substitute. Powertop can estimate power on laptops with a battery. On a desktop or server it mostly shows you wakeups and tunables, not watts. The wall meter is the truth. Turbostat can report package power from RAPL counters on supported CPUs, which is good for comparing changes, but it does not see the disks, fans, or PSU losses.

Step 1: See What the CPU Is Actually Doing

An idle CPU should spend most of its time in deep package C-states. A package C-state means the whole chip, including cache and memory controller, is napping. The deeper the state (PC2, PC3, PC6, PC7 and beyond on Intel), the less it draws. If the package never gets past PC2, something is holding it up.

Turbostat ships with the kernel tools. Run it as root:

sudo turbostat --quiet --Summary --show Busy%,CPU%c6,Pkg%pc2,Pkg%pc6,PkgWatt --interval 10

The columns:

• Busy% is how much of the time the CPUs ran.
• CPU%c6 is residency in the core C6 state.
• Pkg%pc2 and Pkg%pc6 are package-level residency from hardware counters.
• PkgWatt is watts for the whole package.

Which package states exist depends on the CPU. If a column is missing, run turbostat --list to see what your chip reports. The pattern you want on a quiet box: low Busy%, high Pkg%pc6 or deeper. The pattern that means a blocker: low Busy%, Pkg%pc2 near 100, and nothing deeper. On AMD chips turbostat usually shows no package C-state columns at all, so lean on the wall meter and CPU%c6 there.

Powertop shows the same story interactively. Install it, run sudo powertop, and use Tab to move to the Idle stats page. The package C-state residency is listed there. The Overview tab lists what wakes the CPU and how often, and the Tunables tab lists settings marked Bad or Good.

sudo powertop
# Tab / Shift+Tab switches between Overview, Idle stats,
# Frequency stats, Device stats, Tunables

Do not run --auto-tune yet. Look first. If you apply everything at once, you learn nothing about which blocker was the real one.

Step 2: Check ASPM, the Usual Suspect

ASPM (Active State Power Management) lets PCIe links drop into low-power states when idle. If one link on the path stays awake, the package often cannot enter a deep C-state. Check the current policy:

cat /sys/module/pcie_aspm/parameters/policy
# [default] performance powersave powersupersave

The bracketed word is active. The default policy uses what the firmware set up. Now look at each device:

sudo lspci -vv | grep -E '^[0-9a-f]{2}:|LnkCap:|LnkCtl:' | grep -E '^[0-9a-f]{2}:|ASPM'

You get a device line, then a LnkCap line saying what the device supports (ASPM L0s L1) and a LnkCtl line saying what is enabled (ASPM Disabled, ASPM L1 Enabled). A device that supports L1 but shows Disabled is a candidate. A device whose LnkCap shows no ASPM support at all is a hardware blocker.

Turning ASPM on

Try the cheap options first.

1. In BIOS, look for ASPM, PCIe power saving, or “Native ASPM” settings. Enable them. Also look for “PCH” or “DMI” link power states. Our BIOS tweaks post covers the C-state side of this.
2. If the firmware hands control to the OS, switch the kernel policy at runtime:

echo powersave | sudo tee /sys/module/pcie_aspm/parameters/policy

The other policy names are performance, powersupersave (the most aggressive), and default. Watch for dropped devices, link errors in dmesg, and NIC resets.

On some boards the firmware does not give the OS control of ASPM. Then the write to policy fails with “Operation not permitted”, and dmesg says the OS has no ASPM control. The kernel has a pcie_aspm=force boot option for that case. The kernel documentation describes it as enabling ASPM “even on devices that claim not to support it,” with the warning that forcing ASPM on may cause system lockups. Treat that as a last resort. Test it with the box on your desk, not in a closet two floors away, and keep a way to boot without it.

The cheap-hardware blockers

When ASPM is on and the package still sits in PC2, check what is attached. The repeat offenders in home labs:

• Cheap Realtek NICs. Some Realtek PCIe NIC and driver combinations misbehave with L1 and L1 substates. Reports vary by chip revision and driver, so test your own. An Intel NIC is a well-trodden fix.
• ASM1166, JMB585 and similar add-in SATA controllers. Some of these cards keep a link awake or do not support ASPM cleanly. Reports vary by card and firmware.
• A discrete GPU. An add-in card idles at several watts on its own and can block the package from sleeping. If the box is headless, pull it. If you need it for transcoding, an iGPU does that for far less idle power.
• Cheap PCIe switches or riser cards. These can sit in the middle of the link path.

The test is simple. Pull one device at a time, boot, and run turbostat for a few minutes. If Pkg%pc6 jumps, you found it.

Step 3: CPU Frequency Policy

Modern CPUs mostly pick their own frequency. Your lever is a hint. On Intel with the intel_pstate driver and on AMD with amd-pstate in active mode, the hint is the Energy Performance Preference (EPP). It lives in each policy directory:

cat /sys/devices/system/cpu/cpu0/cpufreq/scaling_driver
cat /sys/devices/system/cpu/cpu0/cpufreq/scaling_governor
cat /sys/devices/system/cpu/cpu0/cpufreq/energy_performance_available_preferences
cat /sys/devices/system/cpu/cpu0/cpufreq/energy_performance_preference

The available values are typically default performance balance_performance balance_power power. The kernel docs say the default value stands for whatever the platform firmware set. In intel_pstate active mode, the powersave governor passes your EPP value to the hardware, which makes its frequency choices less performance-focused. The performance governor forces EPP to performance and rejects any other value you write. Which one a distro ships depends on its kernel build options, so check yours rather than assuming.

EPP only takes your value under the powersave governor, so switch to it first, then set EPP on all CPUs:

echo powersave | sudo tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor
echo balance_power | sudo tee /sys/devices/system/cpu/cpu*/cpufreq/energy_performance_preference

The kernel docs recommend the same hint on all CPUs (or pinning tasks to CPUs), because tasks migrate between them. The glob handles that.

How much does this save at idle? Often little. An idle core already drops into a deep C-state regardless of EPP. EPP matters more under light load, where it decides how eagerly the chip boosts. If your box has a wall meter reading that does not change, leave it and move on. Do not spend an hour here.

Step 4: Disks

Spinning drives draw a few watts each when idle (typically 3 to 8 W per drive, depending on size and model). Four of them idle at the wall for 24 hours cost more than a lot of CPU tuning will recover. Two choices:

• Spin them down when nothing touches them.
• Move the hot data off them. Put containers, databases, and logs on an SSD, and keep the HDDs for media and backups that are touched a few times a day.

hdparm standby timers

sudo hdparm -C /dev/sdX          # active/idle, standby, or sleeping
sudo hdparm -S 242 /dev/sdX      # standby after 1 hour
sudo hdparm -y /dev/sdX          # force standby now, to test

The -S encoding is odd. Per the hdparm man page: 0 disables the timer, 1 to 240 are multiples of 5 seconds (5 seconds to 20 minutes), and 241 to 251 are units of 30 minutes (30 minutes to 5.5 hours). So -S 242 is 1 hour and -S 120 is 10 minutes.

Run hdparm -y, wait, then hdparm -C. If the drive says standby, the drive supports it. Then find out what wakes it up. Usual culprits are a container writing logs to the array, a SMART poll, a file indexer, or a share that something scans. The drive spinning back up every 10 minutes is worse than never spinning down, because spin-up is hard on the motor.

hd-idle for the awkward cases

USB-attached disks often ignore the ATA timer. hd-idle watches /proc/diskstats and sends the stop command after an idle period:

sudo hd-idle -i 600

Per its README, -i is idle time in seconds and, run bare, it spins down SCSI-layer disks (USB and similar) after 10 minutes. For SATA disks add -c ata so it uses the ATA command, as hdparm does (that flag is in the maintained adelolmo rewrite, not the old SourceForge version). The README warns that hd-idle does not work alongside disk monitoring tools such as smartmontools, and says to disable them. If you need SMART monitoring on those drives, use the hdparm -S timer instead where the drive accepts it.

SATA link power

The SATA link itself can sleep. The kernel exposes a policy per host:

cat /sys/class/scsi_host/host*/link_power_management_policy
echo med_power_with_dipm | sudo tee /sys/class/scsi_host/host*/link_power_management_policy

The documented values are max_performance, medium_power, med_power_with_dipm, and min_power. The med_power_with_dipm value is medium_power plus device-initiated power management, matching the Intel Rapid Storage settings Windows uses. min_power saves the most and costs the most latency. Some drives and controllers misbehave with it, so start with med_power_with_dipm. Powertop’s auto-tune writes a policy here too.

Step 5: NIC and USB Tweaks

Ethernet

Energy-Efficient Ethernet (EEE) lets the link sleep between packets. Check and toggle it with ethtool:

sudo ethtool --show-eee eth0
sudo ethtool --set-eee eth0 eee on

(Replace eth0 with your interface name from ip link.) The saving is small, and some NIC and switch pairs flap or add latency with EEE on. If you see link drops, turn it off and live with the extra half watt.

Wake-on-LAN is a different matter. If you never wake the box over the network, you can turn it off with sudo ethtool -s eth0 wol d. If you do use it, remember that powertop’s auto-tune may flip it. Check sudo ethtool eth0 | grep Wake-on after a reboot.

USB autosuspend

USB devices can suspend when idle. The per-device switch is in sysfs:

cat /sys/bus/usb/devices/1-1/power/control   # "auto" or "on"
echo on | sudo tee /sys/bus/usb/devices/1-1/power/control

Writing on disables autosuspend for that device. This matters because autosuspend breaks things that need to stay alive. A keyboard drops its first keystroke. A UPS monitored over USB through NUT can lose contact. A Zigbee or Z-Wave stick can stop receiving. Those failures show up at 2 AM, not while you are watching. Keep a short allowlist of devices that must stay on.

Step 6: Make It Stick Without Breaking It

Powertop’s --auto-tune sets every tunable to its “good” setting in one shot. It resets at reboot, so the usual approach is a systemd oneshot. Check where your distro puts the binary with command -v powertop and adjust the path.

[Unit]
Description=Apply powertop tunings
After=multi-user.target

[Service]
Type=oneshot
ExecStart=/usr/sbin/powertop --auto-tune
ExecStartPost=/usr/local/sbin/usb-keep-awake.sh
RemainAfterExit=yes

[Install]
WantedBy=multi-user.target

The second command re-pins the devices that must not suspend. Match them by vendor and product ID (find yours with lsusb). The IDs below are placeholders.

#!/bin/bash
# vendor:product pairs that must never autosuspend (placeholders)
KEEP="0000:0000 1111:2222"
for dev in /sys/bus/usb/devices/*; do
  [ -f "$dev/idVendor" ] || continue
  id="$(cat "$dev/idVendor"):$(cat "$dev/idProduct")"
  case " $KEEP " in
    *" $id "*) echo on > "$dev/power/control" ;;
  esac
done

Enable it:

sudo chmod +x /usr/local/sbin/usb-keep-awake.sh
sudo systemctl daemon-reload
sudo systemctl enable --now powertop-tune.service

Before you enable it for good, run sudo powertop --auto-tune-dump once. That option needs powertop 2.16 or newer (released August 2026), and many distro packages still ship 2.15 or older, which reject it. On an older version, read the Tunables tab instead. Per the man page, the option prints the commands auto-tune would run, instead of running them. Read them. This is where you spot the NIC or USB change you did not want.

A word on the “apply everything” approach. Auto-tune is the forklift here: it moves the whole pallet, including the couch you wanted to keep. Use it as a starting point. If something breaks, find the offending line in the dump and exclude it. Reboot after enabling and test every device that matters: network, UPS, USB sticks, disk spin-up.

Step 7: Containers That Never Sleep

Software can pin the CPU awake as well as hardware. A container that wakes 1,000 times a second keeps the cores out of deep C-states even at 1% CPU. Total CPU use hides this, so it will not show up in a normal graph.

Powertop’s Overview tab lists processes, interrupts, and kernel timers with an Events/s column. It sorts by estimated power and CPU time, not by wakeups, so scan the Events/s column for high numbers. For a text-only view, pidstat from the sysstat package reports context switches per process:

LC_ALL=C pidstat -w 5 1 | grep '^Average' | sort -k4 -nr | head

The cswch/s column counts voluntary context switches per second, and nvcswch/s counts involuntary ones. A process with hundreds per second while “idle” is polling. Map the PID to a container with docker top <container> or by reading /proc/<pid>/cgroup.

Common sources:

• Healthchecks with a tiny interval. A healthcheck that runs every second spawns a process (plus a shell with CMD-SHELL) and often a network call, every second. Docker’s default interval is 30 seconds, so someone tightened it. Loosen it:

services:
  app:
    image: example/app:latest
    healthcheck:
      test: ["CMD", "curl", "-fsS", "http://localhost:8080/health"]
      interval: 60s
      timeout: 5s
      retries: 3

• Apps with a tight poll loop. Some apps check a queue, a file, or an API on a short timer. Look for an interval setting in the config, and raise it.
• Tools that scan constantly. Media indexers, file watchers, and metrics exporters scraping every second all add wakeups. Scrape every 30 to 60 seconds unless you have a reason not to.
• Timers in the kernel. If Powertop shows a kernel timer or a driver at the top, you are back in hardware territory.

Fix the worst offender, then rerun turbostat. A single busy-polling container can be the difference between PC2 and PC6, and that shows up on the meter.

What a Reasonable Result Looks Like

Here are typical ranges people report, and not our measurements. A consumer desktop board used as a server, with ASPM off and a GPU installed, often idles in the 40 to 60 W range. After enabling ASPM, removing the GPU, spinning down disks, and calming a few containers, reports of 15 to 25 W are common. That is the “half” in the title. A mini PC or laptop-class chip starts lower and has less to gain. A rack server with a BMC, dual PSUs, and fans may only drop by 10 to 20 percent.

Cost depends on your rate. At an example rate of $0.15 per kWh, 25 W saved over a year is about 219 kWh, or roughly $33. That is real money but not life-changing. The bigger win is often heat and noise, which matter in a closet.

Work in this order:

1. Baseline at the wall.
2. Check package C-states with turbostat or powertop.
3. Fix ASPM in BIOS, then the kernel policy.
4. Find the blocker device by removal.
5. Spin down disks and set the SATA link policy.
6. Tame containers.
7. Persist the tunings with the systemd unit, keeping a USB allowlist.

Your 2 AM self will appreciate a box that stays up and sips power at the same time.

Common Questions

Does powertop --auto-tune survive a reboot?

No. Powertop applies its settings to the running system, and they reset at boot. To keep them, run powertop --auto-tune from a systemd oneshot service at startup. Add a script that re-pins any USB device that must not suspend, since auto-tune can set those devices to autosuspend.

Is it safe to use pcie_aspm=force?

It is risky, so use it only as a last resort. The kernel documentation warns that forcing ASPM on may cause system lockups. Try the BIOS options and the runtime policy first. If you force it, test with physical access to the machine and keep a boot entry without the option.

Why does my CPU never reach deep C-states even when idle?

A device or a process is holding the package awake. The usual hardware causes are a PCIe device with ASPM off, a discrete GPU, or a cheap NIC or SATA card. The usual software cause is a container waking the CPU hundreds of times per second. Check turbostat package residency, then remove devices one at a time.

Will spinning down disks shorten their life?

Yes, spinning down disks shortens their life if they spin up and down many times a day. Each spin-up adds wear to the motor. A long timeout, such as one hour with hdparm -S 242, keeps spin-ups to a few per day. If something wakes the disk every few minutes, fix that source instead.

Related Reading

• Kill-A-Watt Tour of Every Box in the Rack
• Power Math: Watts, Heat, and Your Electric Bill
• BIOS/UEFI Tweaks for Headless Servers
• SMART Disk Monitoring with smartmontools
• NUT: Network UPS Tools for Home Labs That Don’t Want to Crash