The work done on real-time Linux has benefitted the open-source OS for years, but it was only this week that Linus Torvalds admitted its last piece into the mainline kernel. Exactly what took so long?

After 20 years, Real-Time Linux (PREEMPT_RT) is finally – finally – in the mainline kernel. Linus Torvalds blessed the code while he was at Open Source Summit Europe. […] The real-time Linux code is now baked into all Linux distros as of the forthcoming Linux 6.12 kernel. This means Linux will soon start appearing in more mission-critical devices and industrial hardware. But it took its sweet time getting here. An RTOS is a specialized operating system designed to handle time-critical tasks with precision and reliability. Unlike general-purpose operating systems like Windows or macOS, an RTOS is built to respond to events and process data within strict time constraints, often measured in milliseconds or microseconds. As Steven Rostedt, a prominent real-time Linux developer and Google engineer, put it, “Real-time is the fastest worst-case scenario.” He means that the essential characteristic of an RTOS is its deterministic behavior. An RTOS guarantees that critical tasks will be completed within specified deadlines. […]

So, why is Real-Time Linux only now completely blessed in the kernel? “We actually would not push something up unless we thought it was ready,” Rostedt explained. “Almost everything was usually rewritten at least three times before it went into mainline because we had such a high bar for what would go in.” In addition, the path to the mainline wasn’t just about technical challenges. Politics and perception also played a role. “In the beginning, we couldn’t even mention real-time,” Rostedt recalled. “Everyone said, ‘Oh, we don’t care about real-time.’” Another problem was money. For many years funding for real-time Linux was erratic. In 2015, the Linux Foundation established the Real-Time Linux (RTL) collaborative project to coordinate efforts around mainlining PREEMPT_RT.

The final hurdle for full integration was reworking the kernel’s print_k function, a critical debugging tool dating back to 1991. Torvalds was particularly protective of print_k --He wrote the original code and still uses it for debugging. However, print_k also puts a hard delay in a Linux program whenever it’s called. That kind of slowdown is unacceptable in real-time systems. Rostedt explained: “Print_k has a thousand hacks to handle a thousand different situations. Whenever we modified print_k to do something, it would break one of these cases. The thing about print_k that’s great about debugging is you can know exactly where you were when a process crashed. When I would be hammering the system really, really hard, and the latency was mostly around maybe 30 microseconds, and then suddenly it would jump to five milliseconds.” That delay was the print_k message. After much work, many heated discussions, and several rejected proposals, a compromise was reached earlier this year. Torvalds is happy, the real-time Linux developers are happy, print_K users are happy, and, at long last, real-time Linux is real.

    • @sir_pronounEnglish
      -7•2 months ago

      Look what I just did:

      "In a real-time kernel, the trouble with using print_k (or similar logging functions) often revolves around potential disruptions to real-time performance. Here are some key issues:

      1. Blocking Behavior: print_k may block if the output buffer is full, leading to unpredictable delays in real-time tasks.

      2. Interrupt Context: Using logging functions within interrupt handlers can lead to priority inversion, causing lower-priority tasks to block higher-priority ones.

      3. Latency: Printing can introduce significant latency, which is detrimental in real-time systems that require deterministic timing.

      4. Context Switching: Frequent logging can increase context switching overhead, impacting overall system performance.

      5. Overhead: The computational overhead of formatting and outputting strings can interfere with time-sensitive operations.

      For these reasons, it’s typically recommended to use alternative methods, such as circular buffers for logging, or to minimize logging in real-time contexts.

      "

      • Match!!English
        4•2 months ago

        This is exactly the information I already had before based purely off the name print_k

        • melroy
          6•2 months ago

          And is this already where it goes wrong with people trying to use “AI”…

      • melroy
        4•2 months ago

        I got this back from ChatGPT (most likely false info!):

        ChatGPT response (you have been warned)

        The compromise for integrating PREEMPT_RT into the Linux mainline kernel, including the handling of printk, required several changes and concessions over time. These compromises made it possible to finally integrate real-time (RT) capabilities while maintaining the overall philosophy and structure of the Linux kernel. Key Compromises Made:

        1. Soft-Real-Time Focus:

          One of the biggest compromises was accepting that Linux would focus on soft real-time capabilities instead of hard real-time guarantees. This was crucial for widespread acceptance in the mainline kernel, as hard real-time guarantees were too difficult to meet without significant changes that would have disrupted the kernel’s general-purpose use cases. PREEMPT_RT was designed to offer deterministic behavior with reduced latencies, but it doesn’t provide the strict guarantees of traditional hard real-time systems like VxWorks or QNX.

        2. printk Latency Handling:

          Non-blocking printk: One compromise involved updating printk to avoid long blocking during logging operations. It was changed to be more asynchronous, reducing the impact on real-time scheduling. Deferred printing: Another approach was to defer the actual printing of log messages to avoid introducing large latencies in time-critical paths. The goal was to prevent printk from stalling critical tasks, especially those with real-time priority.

        3. Voluntary Preemption Points:

          Voluntary preemption was introduced, allowing kernel code paths to insert preemption points that allow the scheduler to preempt running tasks, improving latency. However, this does not guarantee immediate preemption, which is another compromise compared to true hard real-time systems. These preemption points had to be carefully placed to balance performance and responsiveness without destabilizing the general-purpose kernel.

        4. Threaded Interrupts:

          Another significant compromise was the conversion of hard IRQs (interrupts) into threaded interrupts. While this allows real-time tasks to take precedence over hardware interrupts (which would traditionally have a higher priority), it involves some overhead and performance trade-offs. Not all interrupts could be threaded easily, and this change required reworking many drivers to ensure that they were compatible with threaded interrupts.

        5. Preemptible Spinlocks:

          Spinlocks in the kernel traditionally prevent preemption. To enable real-time preemption, spinlocks were made preemptible, allowing real-time tasks to preempt even code holding spinlocks. This change wasn’t trivial, as it involved significant reworking of kernel locking mechanisms to ensure the system remained stable and avoided deadlocks, without degrading performance in non-real-time scenarios.