Real-Time on Linux
Christopher Besch • 30th June 2024
.

Real-Time on Linux

What's Real-Time?

Nancy Grace Roman Space Telescope's
ACS (Attitude Control System)

  • Correct output
  • Within deadline

So Real-Time means really fast...right?

fast != deterministic

  • Classic Linux goal:
    • through-put, efficiency [1]
    • possibly unbounded latency [3]
  • Real-Time OS:
    • deterministically bounded worst case latency
  • Linux with PREEMPT_RT:
    • no unbounded latency [8]
    • some throughput degradation [9]

User-Space: Real-Time POSIX.4 [2]

Default: SCHED_OTHER

Output: SCHED_OTHER
  • Waste time on "low" prio task

Real-Time: SCHED_FIFO/SCHED_RR

Requires root or high enough rtprio in limits.conf [3]

Output: SCHED_FIFO/SCHED_RR [3]
  • Focus on high prio task
  • Danger: lock up CPU with high prio task
    • Consider RT throttling

Priority Inversion

Output: Priority Inversion [4]
  • Medium prio task could run very long [3]
  • Real problem for Mars Pathfinder [5]

Priority Inheritance

Output: Priority Inheritance [4]
  • Implemented with rt_mutex [6]

Kernel-Space: PREEMPT_RT

Scheduler Preemption Model

Kernel Config

Scheduler Preemption Model [7]

  • CONFIG_PREEMPT_NONE
    • No preemption in kernel (except entering userspace)
  • CONFIG_PREEMPT_VOLUNTARY
    • might_sleep in kernel
  • CONFIG_PREEMPT
    • kernel preemptible unless preemption disabled
  • CONFIG_PREEMPT_RT
    • fully preemptible unless in raw_spinlock_t or IRQ

raw_spinlock_t

include/linux/spinlock_api_smp.h

Why disable preemption?

raw_spinlock_t

raw_spinlock_t without preempt_disable

PREEMPT_RT Solution [6]

rt_mutex

  • Lower throughput:
    • many context switches
    • mutex slow for short critical sections

When spinlock_t doesn't spin

include/linux/spinlock_types.h

include/linux/rtmutex.h

Interrupts Request Handler (IRQ)

Threaded IRQ

  • No preemption in NMI, hardirq, softirq context [8]
  • Threaded IRQ have RT prio 50 by default [7]

Hardware Considerations

  • System management interrupts (SMI) [1]
  • Simultaneous multithreading (SMT) [1]
  • Dynamic frequency scaling [1]
  • Much more...
    e.g., ARM Cortex-R

What we've Covered

  • User-Space:
    • SCHED_FIFO/SCHED_RR
    • PTHREAD_PRIO_INHERIT
  • Kernel-Space: PREEMPT_RT
    • rt_mutex instead of raw_spinlock_t
    • Threaded irq handlers

What we've Left Out

  • User-Space Memory handling
  • Setting up PREEMPT_RT
    See: chris-besch.com/articles/riscv_rt
  • Benchmarking: e.g., Cyclictest
  • Linux Limitations: mainly provability
  • Linux/PREEMPT_RT alternatives
    e.g., VxWorks (actually used by Mars Pathfinder [5])

Sources

Sources

Sources

Sources