The release of Linux RT-Preempt Kernel 6.12 marks another significant step in enhancing real-time performance for critical applications. This version introduces improved low-latency scheduling, deterministic behavior, and better IRQ handling, providing a more stable foundation for industrial automation, robotics, and other real-time workloads.
With enhanced IRQ threading, latency spikes are significantly reduced, even under high-load scenarios. The improved preemptibility ensures consistent response times for real-time applications, making the system more predictable. CPU isolation techniques have also been refined, minimizing jitter and improving determinism.
Additionally, real-time scheduling policies have been optimized, increasing efficiency for time-sensitive operations.
However, some areas in the kernel still pose potential risks for latency spikes during real-time operations. For instance, the seccomp() syscall triggers a call to the tlb_flush_all() function, which invalides all TLBs on all CPUs.
On an x86 machine with an SMP configuration, this function relies on a special APIC mechanism to broadcast IPIs to all CPUs in a single instruction. The problem? There’s no check against an isolated CPU, which should not be disturbed especially if an RT application is performing constant read/write operations. The IPI results in a CPU interrupt, introducing a latency spike.
Here's a excerpt of function call path:
tlb_flush_all(void) -> ... -> x2apic_send_IPI_allbutself() -> .. -> __wrmsr()At the end, there is only one instruction doing the work:
static __always_inline void __wrmsr(unsigned int msr, u32 low, u32 high)
{
asm volatile("1: wrmsr\n"
"2:\n"
_ASM_EXTABLE_TYPE(1b, 2b, EX_TYPE_WRMSR)
: : "c" (msr), "a"(low), "d" (high) : "memory");
}Fortunately, there's a workaround: pass the no_ipi_broadcast=1 argument to the kernel command line. A simple fix, but not an obvious one! This problem became apparent when we noticed unexpected high latency spikes when performing basic operations such as ssh'ing to the remote machine running an RT application.
.
Looking Ahead
Beyond these considerations, Linux RT-Preempt 6.12 continues to push the boundaries of real-time performance, making it an excellent choice for embedded systems and time-critical applications. As adoption grows, its impact will likely extend further into robotics, telecommunications, and automotive systems.
Looking forward to sharing insights and experiences on its implementation in various real-time environments!