The Blue Screen Pointed at the Wrong Suspect
My mini PC started dying with a blue screen full of corruption. Linux has a BSOD now: the kernel’s DRM panic screen (CONFIG_DRM_PANIC=y in the CachyOS kernel) paints one when the machine goes down. I read the last lines of the journal, saw a GPU warning, and blamed the GPU.
The GPU was innocent. The last log line lied. The real failure was the NVMe drive falling off the PCIe bus when the kernel tried to wake it from a deep power state. Once the root filesystem vanishes, the journal has nowhere to write the actual error, so you only see whatever got logged just before the lights went out.
Three kernel flags have kept the machine stable since. A replacement drive with a heatsink is the real fix, and it is on the way. Here is the trail, dead ends included.
Full example: Clone the working files at github.com/KingPin/sumguy-examples/tree/main/linux/nvme-falls-off-pcie-bus
The Setup
The machine is a small AMD Ryzen 8000-series mini PC, about 135 mm on each side, running CachyOS with systemd-boot. The boot drive was a budget DRAM-less 500 GB Gen4 NVMe drive with a Silicon Motion controller, no heatsink, running at PCIe Gen4 x4. The case has a black plastic bottom cover full of vent holes. The CPU blower sits on the opposite side of the board and does not touch the SSDs. Remember that cover, because it matters later.
The first symptom showed up before the first crash. A heavy gaming session lagged badly, with swap thrashing and the drive working flat out. About twelve minutes after I closed the game, the machine died. Most likely the SSD was thermal throttling under that load. I can’t prove it, so I’m calling it likely and moving on.
Over the next few hours I got at least four or five panics. The boot timeline looked like this:
boot 1 ~35 seconds crashboot 2 54 minutes crash (panic)boot 3 17 seconds crash (panic), new kernelboot 4 2h43m clean poweroff, first boot with the fixBefore all this, the previous boot had run for more than 17 hours without a hiccup. Keep that in mind, because it kills one of the suspects.
Suspect One: The GPU Driver
The last kernel lines before the first crash were amdgpu warnings:
amdgpu 0000:c5:00.0: Mode Validation Warning: Viewport size failed validationThey landed right as KDE’s Wayland session was setting display modes, and then the journal just stopped. It looked like a confession. It wasn’t.
The same warning appeared in the 17-hour boot, during out-of-memory events, and that boot never crashed. The journal also held no panic, oops, or watchdog message for the crashing boot. A warning that shows up in healthy boots is noise, not evidence. The ACPI BIOS errors that print on every boot are the same kind of noise.
Suspect Two: A Bad Kernel
Next I blamed the kernel. I ran a full system upgrade, which moved the kernel from 7.3.0 to 7.3.4 along with a Mesa and Plasma bump. Gotcha number one: the boot after the upgrade was still on the old kernel, because the bootloader had not picked up the new one yet. That boot panicked.
The next boot used the new kernel and panicked 17 seconds in, right as the login screen started. Same crash, two kernel versions, same point in boot. That is useful data. A bug introduced in one kernel version would not show up on the other, so this pointed at hardware or firmware.
Turn On pstore Before You Need It
I had a crash and no panic message. The journal could not record the death because the thing it writes to was dying.
This is what pstore is for. It stashes panic logs somewhere that survives a reboot, such as EFI firmware variable storage, and systemd-pstore.service copies them into /var/lib/systemd/pstore/ on the next boot. The CachyOS kernel ships with CONFIG_EFI_VARS_PSTORE=y and CONFIG_EFI_VARS_PSTORE_DEFAULT_DISABLE=y, so EFI pstore is off by default. My /var/lib/systemd/pstore/ directory existed and was empty, which is what a disabled feature looks like.
The fix is one kernel parameter:
efi_pstore.pstore_disable=0Then enable the service:
sudo systemctl enable systemd-pstore.servicels /var/lib/systemd/pstore/Do this today, on every machine you care about. Your 2 AM self will thank you. I did it after the first few crashes, which is the wrong order.
The Clue in the Journal
One boot started with this from systemd-journald:
system.journal corrupted or uncleanly shut down, Device or resource busyThe previous boot had reached the shutdown target normally. A clean shutdown that still corrupts the journal points at storage, not the GPU. I also saw two NVMe lines in the kernel log:
block nvme0n1: No UUID available providing old NGUIDOptimal transfer size 33553920 bytes not a multiple of preferred minimum block size (4096 bytes)Both are harmless quirks of this drive. The NGUID line prints on every boot, healthy ones included. journalctl with -b -1 and -b 0 side by side is the fastest way to tell the difference.
The Real Cause: D3cold and 0xffffffff
I dropped into the emergency shell and read the kernel ring buffer. There it was:
Unable to change power state from D3cold to D0CSTS=0xffffffff, PCI_STATUS=0xffffHere is what those lines mean. D3cold is the deepest PCIe device power state. The kernel puts idle devices there to save power. D0 is fully awake. The kernel asked the NVMe controller to wake up and the controller never answered.
CSTS is the NVMe controller status register. When a PCIe read to a device that is not there comes back, the bus returns all ones. A register that reads 0xffffffff and a PCI status of 0xffff mean one thing: the controller is not answering PCIe reads. The device has dropped off the bus. Not slow, not erroring, gone. It happened about 35 seconds into boot, which matches the “dies near the login screen” pattern. The GPU warnings were just the last thing logged before the root filesystem disappeared.
Another symptom made the story confusing. fsck.ext4 crashed with SIGSEGV, which looks exactly like filesystem corruption. It was not corruption. The drive was dropping off the bus in the middle of reads, and fsck choked on the failures. Once the drive stayed online, the filesystem checked out fine.
So the firm finding is the wake-up failure. The link to heat is a hypothesis: the controller appears to lock up during the power-state change when it is hot. That fits a DRAM-less budget drive with no heatsink in a small case that had just been through a gaming session with heavy swap. It is the likely trigger, not a proven one.
The Fix: Three Kernel Flags
If the drive dies when it changes power states, stop it from changing power states. Three kernel parameters do that:
nvme_core.default_ps_max_latency_us=0pcie_aspm=offpcie_port_pm=offWhat each one does:
nvme_core.default_ps_max_latency_us=0disables NVMe Autonomous Power State Transitions (APST), so the drive stays in its full-power state instead of napping on its own.pcie_aspm=offtells the kernel to keep its hands off Active State Power Management, the link-level power saving. Read the kernel docs closely: the kernel leaves ASPM exactly as the firmware set it up. If your BIOS already enabled ASPM on a link, it stays enabled. The flag only stops the kernel from turning on extra link states by itself.pcie_port_pm=offdisables power management of the PCIe ports, so they never go down to D3cold.
The price is higher idle power draw and a little more heat. This is the opposite of the advice in cutting idle watts, where you turn ASPM on to save power. Both are correct. Power saving is the right default. Turning it off is the right call only when a drive is actively falling off the bus. A forklift is a bad commute vehicle, but it is the right tool when you have to move the couch.
The Gotcha That Would Have Brought the Crashes Back
I applied the flags by editing the systemd-boot entries by hand. That works until the next kernel update, because sdboot-manage (from the systemd-boot-manager package) regenerates the entries every time a kernel package updates. It builds them from /etc/sdboot-manage.conf, and mine still held the old options:
LINUX_OPTIONS="nowatchdog quiet splash"The next kernel update would have rewritten the entries and silently dropped the fix. The crashes would have returned with no obvious reason. I would have blamed the new kernel. The correct line puts everything in the config:
LINUX_OPTIONS="nowatchdog quiet splash nvme_core.default_ps_max_latency_us=0 pcie_aspm=off pcie_port_pm=off efi_pstore.pstore_disable=0"Then regenerate the entries:
sudo sdboot-manage genOn a distro that uses GRUB, the same flags go in GRUB_CMDLINE_LINUX_DEFAULT in /etc/default/grub, followed by sudo grub-mkconfig -o /boot/grub/grub.cfg. The config file differs, the trap is the same: put the fix where the regeneration step reads from.
Verify It Stuck
Do not trust a config file. Trust the running kernel:
cat /proc/cmdlinecat /sys/module/nvme_core/parameters/default_ps_max_latency_usThe second command should print 0. To check APST from the drive’s side, nvme get-feature /dev/nvme0 -f 0x0c from nvme-cli reads the APST feature (feature ID 0x0c). Then hunt for errors:
journalctl -b 0 | grep -ciE 'nvme.*(timeout|reset|error)'I wanted 0, and I got it. The first boot with the flags ran almost three hours and ended in a clean power-off. The kernel log showed the controller initializing normally, ext4 mounted read-write, and no D3cold errors.
What SMART Said
The drive passed its overall health check with zero media errors. That is the part that makes this failure mode so annoying: the flash was fine, and the controller or firmware was the problem. This failure mode can look exactly like a dead drive.
The interesting numbers were in the temperature counters. The “Warning Comp. Temperature Time” and “Critical Comp. Temperature Time” fields showed six-digit numbers. NVMe reports those counters in minutes, not hours, so divide by 60 before you panic. Even divided by 60, that came to thousands of hours above the drive’s thermal thresholds, which is an ugly history for a drive without a heatsink. See NVMe SMART for how to read the rest of that output.
sudo smartctl -a /dev/nvme0 | grep -i 'temperature time'Picking the Replacement
Flags are a workaround. I ordered a 2 TB Gen4 drive with a heatsink pre-attached. Two things about the listing are worth passing on.
First, the spec sheet said PCI Express Gen3 next to a rating of 7000 MB/s sequential read. Those two numbers cannot both be true, because Gen3 x4 tops out around 3500 MB/s in practice. The 7000 MB/s rating says Gen4, and I will confirm the real link speed after install. Always check the controller and the real interface, not the copy-pasted line item.
Second, its rated operating maximum was 70°C, which is lower than the old drive’s warning threshold of 82.8°C. A heatsink gives you better cooling and less headroom at the same time, so you still need to watch temperatures. See NVMe heatsinks for what actually cools. A bare drive in a tiny case runs hot under sustained load.
The heatsink adds about 10 mm of height, so measure the clearance to the cover before you order. A ball of modeling clay on the slot area, a gently screwed-on cover, and a ruler give you the number. I’ll cover the migration from the old drive in a separate post.
Where Does the Second M.2 Slot Live?
My first assumption, based on lspci, was one SSD slot. Wrong. lspci lists only devices that are present, and an empty M.2 slot does not appear. Review photos answered what software could not. With the board flipped over, only the CPU cooler sat on the far side, so every slot had to be on the RAM side. Screw posts about 80 mm from each connector confirmed two 2280 slots: one flat on the board, one stacked above the Wi-Fi card.
The heatsinked drive goes in the flat slot, because the stacked one would trap heat against the Wi-Fi card. After install, check the link you got:
cat /sys/class/nvme/nvme*/device/current_link_speedcat /sys/class/nvme/nvme*/device/current_link_widthYou want 16.0 GT/s PCIe and a width of 4 for a Gen4 x4 link.
Cooling a Mini PC With a Plastic Lid
Most mini-PC cooling advice says to stick a thermal pad between the SSD and the metal bottom plate, so the case becomes a heatsink. My case has a plastic bottom cover. Plastic does not move heat, so the pad does nothing.
The CPU blower is on the other side of the board, so the SSDs get only passive convection through the vents. A drive with its own heatsink is the right answer for this case. The vents face down, so keep the machine off carpet and beds.
An interim fix was a USB fan beside the case, aimed at the vents. It does not blow into the case, but it moves the warm air away. The drive read 40 to 41°C afterward, though I measured under a different load, so that is not a fair before-and-after.
The longer plan is a 120 mm USB fan with rubber feet, laid flat under the case, blowing up through the vents. Push beats pull here. Pushing sends cool room air straight onto the SSDs and RAM and feeds the CPU blower’s intake. Pulling draws air from every gap in the case, and most of it bypasses the drives. Pushing blows dust in, so put a thin filter on the intake. A 120 mm fan covers the vent area and stays inside the case edges. Pick a model with speed control. If you want a number instead of my opinion, run the same heavy disk load in both directions and compare temperatures after ten minutes.
A Temperature Alert You Can Copy
A hot drive that nobody notices is how this story started. This script sends a desktop notification when any NVMe drive reaches a limit. It reads sensors straight from /sys/class/hwmon, finds NVMe devices by name instead of by hwmonN number (the number can change between boots), and needs no root.
#!/usr/bin/env bash# Desktop alert when any NVMe composite temp reaches LIMIT (°C).# Re-alerts every REPEAT seconds while it stays hot.LIMIT=${LIMIT:-70}INTERVAL=${INTERVAL:-10}REPEAT=${REPEAT:-300}declare -A lastwhile :; do for h in /sys/class/hwmon/hwmon*; do [[ $(<"$h/name") == nvme ]] || continue t=$(( $(<"$h/temp1_input") / 1000 )) dev=$(basename "$(readlink -f "$h/device")") now=$(date +%s) if (( t >= LIMIT )); then if (( now - ${last[$dev]:-0} >= REPEAT )); then echo "$dev at ${t}°C (limit ${LIMIT}°C)" notify-send -u critical -a "NVMe temp" "$dev at ${t}°C" "Limit is ${LIMIT}°C" last[$dev]=$now fi else unset "last[$dev]" fi done [[ $ONCE ]] && exit 0 sleep "$INTERVAL"doneSave it as ~/.local/bin/nvme-temp-alert and make it executable. Then add a systemd user unit:
[Unit]Description=Desktop alert when NVMe temperature is highAfter=graphical-session.targetPartOf=graphical-session.target
[Service]ExecStart=%h/.local/bin/nvme-temp-alertRestart=on-failure
[Install]WantedBy=graphical-session.targetPut the unit at ~/.config/systemd/user/nvme-temp-alert.service, then enable it:
systemctl --user daemon-reloadsystemctl --user enable --now nvme-temp-alert.serviceI tested it with LIMIT=30, which fired a notification at 41°C, and with the default 70, which stayed silent. The alert is critical priority, repeats every five minutes while the drive stays hot, and resets when it cools. To change the limit, run systemctl --user edit nvme-temp-alert and add Environment=LIMIT=65 under [Service]. For a quick look without any service, watch -n2 sensors works if you have lm_sensors installed.
Lessons From a Very Bad Weekend
- The last log line lies. If storage dies, the journal cannot record the real error. Distrust whatever came last.
- The same crash on two kernel versions rules out a bug specific to one version. It points at hardware, firmware, or a long-standing driver issue.
- A crashing
fsckcan be a symptom of a vanishing drive, not a corrupt filesystem. Check dmesg before you reformat anything. CSTS=0xffffffffmeans the device is gone from the bus. Treat it as a hardware or firmware event.- Power management is a classic trigger for budget NVMe drives on Linux. APST-off and ASPM-off workarounds are widely cited for budget NVMe drives on Linux, and they are still workarounds.
- Workarounds can vanish silently. If your boot entries are generated, fix the generator’s config, not the output.
- Turn on pstore before you need it. A disabled feature does not log anything.
- Spec sheets are sometimes wrong. A Gen3 label next to a 7000 MB/s rating is almost certainly a typo, not a miracle.
lspcicannot see an empty M.2 slot. Photos and screw-post spacing can.- A plastic bottom cover means thermal pads to the lid are pointless. Airflow and a heatsink are what work.
The darkly funny part: that drive had logged 82.6 TB of writes on a 500 GB disk and a pile of unsafe shutdowns, and it was blamed last. The GPU took the heat for a crime the SSD committed.
Common Questions
Does disabling NVMe APST hurt SSD lifespan?
No. Disabling APST with nvme_core.default_ps_max_latency_us=0 keeps the drive in its full-power state, which raises idle power and heat slightly. It adds no writes. Sustained heat is a reliability risk for any drive, so pair the flag with a heatsink or airflow on small cases.
How do I know if my NVMe is dropping off the PCIe bus?
Look for Unable to change power state from D3cold to D0 or CSTS=0xffffffff in dmesg or the journal. Both mean the controller stopped answering. Other signs are a filesystem that goes read-only, fsck crashing, and a journal that reports corruption after a clean shutdown. If the journal is gone, enable pstore to capture the panic.
Do NVMe power-saving kernel flags increase power usage?
Yes. pcie_port_pm=off and nvme_core.default_ps_max_latency_us=0 keep the PCIe ports and the drive awake when they would otherwise idle, and pcie_aspm=off stops the kernel from enabling extra link power states, so idle watts go up. Measure the difference with a wall meter instead of guessing. Use these flags only on a machine with a drive that is actually dropping off the bus.
Can I remove pcie_aspm=off after replacing the SSD?
Yes, and you should test it. Wait for a week of stable running on the new drive, then remove one flag at a time, a few days apart. Do not reuse the old drive in the machine, because the flags stay needed while it is installed. If crashes return, put the last flag back.