Linux 7.3's Quiet Fix for Firmware That Swallows Whole Systems

One Stuck Firmware Call Used to Take Everything Down

A new safeguard in Linux 7.3 stops a class of bugs that could silently grind enterprise workloads to a halt: a single hung EFI runtime service call no longer needs to be waited on forever.

Before this change, the kernel would sit inside a firmware call and never return. The caller held the efi_runtime_lock for the duration. Every subsequent process that needed to talk to EFI — NVRAM writes, variable storage through efivarfs, ACPI PRM handlers, even the simple task of setting the wakeup time — blocked behind that same lock. User-space processes piled up uninterruptibly on the semaphore. The only externally visible symptom was a workqueue lockup message, repeated at 127 seconds, 157 seconds, 188 seconds and beyond, until some external monitoring system finally rebooted the host.

The firmware was never coming back. But the kernel had no way of knowing that.

The efi_runtime_lock is not a peripheral concern. It serializes every EFI runtime call across the entire system. When one call hangs inside the firmware, the lock never releases, and everything that depends on EFI — the TPM, NVRAM, the power-management path, the boot variable infrastructure — backs up behind it. Containers, initramfs scripts, logrotate, systemd timers, and any user-space tool that reads or writes EFI variables all block. The system appears healthy to most monitoring agents because the CPU is not spinning; it is simply stuck. A dashboard sees green lights while the machine is already dead.

The Bug Was Found on a NVIDIA Grace Server

Breno Leitao, a Debian developer and kernel engineer at Meta, hit the problem running on NVIDIA Grace hardware. He sent patches through to enforce a timeout on EFI runtime service completions, and the changes landed in the EFI feature merge for Linux 7.3.

In the patch series cover letter, Leitao laid out exactly what was happening. The PC and link register were stuck inside EFI firmware memory — the firmware simply never returned. The worker stayed wedged across multiple workqueue lockup reports until an outside system forced a reboot. The kernel itself could not recover.

This is not a rare edge case. EFI runtime services are called constantly during normal operation. Any vendor with sloppy firmware — and there are many — can trigger a condition where a call hangs. Before this fix, the only option was to reboot and hope the next call happened to succeed. In a fleet environment, that meant dozens of servers potentially stuck in the same state with no clear diagnostic signal.

The NVIDIA Grace platform compounds the problem because it runs a non-x86 architecture using ARM-based superchips. Firmware maturity on newer server silicon is still uneven, and the Grace architecture introduces its own initialization path through EFI runtime calls that differ from traditional x86 deployments. A bug that might surface as a footnote on commodity x86 hardware becomes a fleet-impacting failure on a platform where uptime is measured in revenue impact rather than inconvenience.

The Timeout Is 120 Seconds. That Is Intentional.

The timeout threshold is set at 120 seconds. Leitao described it as longer than any plausible legitimate call into EFI runtime services. No normal firmware routine should take two minutes to complete. If it does, something is wrong, and the kernel should stop pretending otherwise.

When the timeout fires, the firmware is declared wedged. The kernel breaks out of the stuck call, drops the efi_runtime_lock, and returns an error to whoever made the original request. Subsequent EFI operations are no longer blocked behind the original hung worker. User-space resumes. The system stays up.

The implementation works by wrapping the EFI call in a timed wait. The kernel starts a timer before invoking the firmware routine. If the routine has not completed before the timer expires, the kernel treats it as hung and aborts. The exact mechanism does not forcibly unmap the firmware — the firmware is still executing, possibly in a bad state — but the kernel severs its own dependency on the result and continues. This is a containment strategy, not a repair.

That distinction matters. The timeout does not prevent the firmware bug from recurring. A subsequent EFI call on the same code path may also hang. What the timeout does prevent is a single hung call from cascading into a full-system freeze. It converts a hard hang into a recoverable error with a defined timeout window.

The Real Win Is Diagnostic Clarity

The more important outcome may not be the recovery itself but the signal it provides to engineers. Before this change, a workqueue lockup could mean anything. A memory leak. A deadlock in kernel code. A scheduler starvation. A hardware fault. On a fleet scale, the same symptom could be attributed to dozens of unrelated root causes.

Now, when a system hits this timeout, dmesg produces an unambiguous EFI firmware-at-fault message. Operators know exactly where to look. Vendors get a clearer bug report. And the engineer on call at 3 a.m. does not need to spend an hour ruling out half the kernel subsystems before deciding to reboot.

This kind of diagnostic precision matters most at scale. A single server hanging is an inconvenience. A hundred servers all silently wedged on the same firmware bug across a data center is an incident. With the timeout in place, the inciting event is logged with enough specificity that post-mortem analysis can begin immediately rather than after a day of digging through kernel call stacks. The error message itself becomes evidence.

The second-order effect extends to observability tooling. Prometheus exporters, Datadog checks, and internal alerting pipelines that already watch for workqueue lockups can now correlate the timeout event with the EFI firmware error and automatically classify the incident. That classification, in turn, routes the ticket to the firmware vendor instead of consuming an on-call engineer’s time.

Why This Matters Beyond Linux Enthusiasts

EFI firmware bugs are not a theoretical problem. They are a production risk for any organization running Linux on heterogeneous hardware. Server vendors ship firmware updates at their own pace. Some platforms carry buggy implementations that only surface under specific conditions. The kernel has traditionally absorbed these failures gracefully, which sounds like a feature until it means your workload is silently dead and you cannot tell why.

Meta’s involvement here is notable. Leitao works at the intersection of upstream kernel development and production infrastructure at scale. The NVIDIA Grace server that exposed the bug is not a consumer device — it is infrastructure-grade hardware where uptime matters and recovery time is measured in lost revenue, not inconvenience. The fact that the fix originated from someone working inside a cloud provider’s kernel team, rather than a firmware vendor, underscores a broader pattern: the kernel is absorbing bugs that should never have reached production firmware in the first place, and timeout safeguards are a pragmatic stopgap.

The 120-second timeout is conservative by design. It will rarely trigger on healthy systems. But when it does, it turns a mystery stall into a known failure mode with a clear origin. That is the kind of change that does not make headlines but quietly improves the reliability of every enterprise Linux deployment running 7.3 or later.

Firmware vendors have a long history of shipping runtime services that assume the kernel will wait indefinitely. This timeout forces a reckoning. It makes the cost of a hung firmware call measurable instead of invisible. Over time, that visibility should push vendors toward fixing the root cause rather than relying on the kernel to absorb the failure. Until then, the 120-second gate keeps systems running long enough for engineers to notice, diagnose, and act — instead of discovering a dead server only when the customer complains.

The firmware is still broken. The kernel is just smarter about pretending it is not.