Setting Up Real Time Operation on GNU/Linux Systems
From Bennet
This page is intended to give a set of guidelines for a realtime system. The primary target is Audio and the secondary is Graphics. It will walk step by step through a confirmation process, using standard tools to interrogate the system, so that you can confirm that everything is in order for realtime low latency operation.
As of 2008-02-20, this document concerns itself with linux-2.6.23.11-rt14
Contents |
[edit] Kernel
I've documented my own build instructions in Patching and Building the Linux Kernel, to help some lost souls.
Kernel Configuration:
Enable the tickless kernel feature zcat /proc/config.gz|grep -i CONFIG_NO_HZ
Enable the High Resolution Timers
Enable Complete Real Time Premption Mode
Set Timer Frequency to 1000
[edit] File System
Never consider something like ReiserFS 3. A good FS would be XFS or ext3. No matter what filesystem you have, there should not be dropouts, but some filesystems are inherently more slow. Also, do not have more than a 75% full HD, as the drive is reportedly slower. In any case, no matter what, there should never be XRUNS related to any of this, but the more CPU you use, the sluggier the interaction will be.
ReiserFS 3 (RFS) is good for filesystems with lots of small files. It is (historically) awful at management of smaller numbers of large (e.g. audio) files and historically not reliably low-latency.
A ReiserFS v3 filesystem is very secure, but not good for audio.
XFS is reputed best, but not tested much by the GNU/Linux audio crowd
The audio section of the HD should be mounted with noatime, as it is almost the biggest most important contribution. noatime has generally been necessary to get the best performance for any filesystem that does a lot of reads
When setting up JACK, a shared memory device is set up, usually on /dev/shm. This is absolutely crucial. To confirm that JACK uses shared memory and not the actual HD, look at the output from JACK when launching it.
The see the file systems on partitions, issue the "mount" command.
I am not too lazy to write an academic paper. I just do not have time for writing. Thus, I need to go to the essay writing service to get custom papers, which are composed by the best writing experts. I understand that a lot of people cannot become successful without writing assistance.
[edit] PCI
The Peripheral Interconnect Bus (PCI) is a shared bus and suffers from this, of course. PCI needs to die, but in the meantime, we can adjust the pci_latency_timer with setpci. The problems with shared bus architecture are f.ex dropouts at the hardware DMA level. This problem is an unfixable problem, but the impact is adjustable with the setpci tool.
Audio devices are especially susceptible to PCI latency issues because they generally have small data buffers and need their audio data delivered to them on time to avoid buffer underruns.
Increase the PCI latency of all devices to a value of 180 (devices normally default to 32 or 64. Then set the PCI bus latency of the latency sensitive devices to the maximum setting, 248. This allows the soundcard to burst relatively huge chunks of data across the bus in one go, allowing it to maximize its use of the bus and avoid buffer underrun conditions. At the same time, other devices can also transmit data in chunks that are small enough not to hog the bus, but large enough to use the bus efficiently.
First, find you soundcards' PCI ID with "lspci", then, as super user set it, f.ex:
export SOUND_CARD_PCI_ID=03:07.0
#allowing medio long bursts for all PCI devices setpci -v -d *:* latency_timer=b0 #..then maximize latency timers for audio setpci -v -s $SOUND_CARD_PCI_ID latency_timer=ff
Also, turn off "PCI burst" in the BIOS.
[edit] Hardware Priority
The current hardware interrupt system is fubar. Without going into further detail right now, the highest available priority in the system is IRQ9. Check reference.
Here's the order of interrupt priority on a non-APIC machine:
0, 1, 8, 9, 10, 11, 12, 13, 14, 15, 3, 4, 5, 6, 7
To verify you've got a non-APIC system, issue
cat /proc/interrupts
Non-APIC will list XT-PIC or XT-PIC-XT here, while APIC systems will list APIC something. Check reference.
The soundcard needs the highest hardware priority (interrupt) in the system.
The means that you should put it on IRQ9.
Confirm with "cat /proc/interrupts".
Here's my output:
CPU0
0: 520311190 XT-PIC-XT timer
1: 12 XT-PIC-XT i8042
2: 0 XT-PIC-XT cascade
6: 4 XT-PIC-XT floppy
7: 0 XT-PIC-XT uhci_hcd:usb2, uhci_hcd:usb3
8: 0 XT-PIC-XT rtc
9: 12297953 XT-PIC-XT hdsp
10: 36907708 XT-PIC-XT uhci_hcd:usb1, uhci_hcd:usb4, eth0, eth1, radeon@pci:0000:01:00.0
14: 477112 XT-PIC-XT libata
15: 3311 XT-PIC-XT libata
NMI: 0
ERR: 0
HDSP at IRQ9 is my RME MultiFace soundcard. It can be a real damn pain to get your soundcard to appear at IRQ9. If you don't have a CoreBoot BIOS, then you probably need to try every damn PCI slot to see if it shows up on IRQ9. It is possible but it takes determination with current crap BIOS'es.
It goes without saying that IRQ9 should exclusively be for the soundcard. Don't share this IRQ9 with anything else.
[edit] Software Priority
The linux kernel implements something which is known as POSIX 1.e. This is an extension specification to POSIX for realtime capabilities that can be granted to processes. When a process is granted realtime capability, it is known as a SCHED_FIFO process.
The applications that needs to run realtime would need SCHED_FIFO. A system such as PAM (See POSIX 1.e Realtime Capability with PAM) can grant certain processes these rights, when they ask for them.
When a process is SCHED_FIFO, it has priority over any other process in the system.
The realtime process also has a priority ranking from 1 to 99. It is common to give a watchdog a priority of 99, the highest priority. A watchdog can kill processes which behaves badly and uses too much system resources.
You need JACK and the audio interrupt to be at a higher priority than all the other IRQ threads. Pass -P84 to JACK and set the IRQ 9 process to priority 85. This means that the IRQ of you soundcard at 85 is higher than jack at 84.
You should use the script from RNC to set the processes. Also look at .
As super user, you run it, like this:
rtirq.sh start
You should also confirm that your processes are correctly set up.
You can use "ps -Leo rtprio,cmd,pid" to list all RT processes along with their RT priority.
Here's the relevant parts from my output:
RTPRIO CMD PID
99 [posix_cpu_timer] 3
50 [softirq-high/0] 4
50 [softirq-timer/0] 5
50 [softirq-net-tx/] 6
50 [softirq-net-rx/] 7
50 [softirq-block/0] 8
50 [softirq-tasklet] 9
50 [softirq-sched/0] 10
50 [softirq-hrtimer] 11
50 [softirq-rcu/0] 12
1 [events/0] 14
50 [IRQ-6] 735
50 [IRQ-14] 749
50 [IRQ-15] 750
79 [IRQ-10] 785
80 [IRQ-7] 793
50 [IRQ-12] 874
75 [IRQ-1] 875
85 [IRQ-9] 1980
90 [IRQ-8] 2629
99 ./das_watchdog 2644
1 ./das_watchdog 2644
84 jackd -v -R -P84 -p256 -t49 2741
83 /usr/local/bin/xmms2d --sta 2751
83 patchage 2786
83 jackminimix 2888
83 jack.plumbing 2924
This is after I run the rtirq script. I'm not sure why IRQ-8 is given priority 90 and I'll look into this.
Also, one article mentions that jackd also runs threads besides its main loop. Namely a watchdog at priority 10 above its main loop (80 in this example), and all clients’ process() threads will run with a priority 1 smaller than the main loop (69). I'll look into this..
[edit] Watchdog
It is wise to run a general watchdog with the highest priority in the system to keep an eye on misbehaving processes. A watchdog watches realtime processes that uses too much time, and deal with them appropriately. An example application of this type is das watchdog, which temporarily disables realtime priority when the computer normally would lock up.
Give this watchdog a RTPRIO (Realtime Priority) of 99.
[edit] RAM
In my How to Buy a Computer HOWTO, I advice to get atleast 2Gio of RAM for a well responsive system and I advice to get good memory with a CAS latency of 2.
Be sure that your system is not paging. You can use iostat to confirm this. If your system is paging under normal use, it's a clear indication that you need more RAM.
[edit] Network
Linux Ethernet driver tuning begins by selecting the best Ethernet driver. For example, the HS20 blades recommended driver is the tg3-3.43b driver (or later), which can be found at the Web site for Broadcom Ethernet NIC Driver Downloads. The following shell commands have been used to tune the Linux kernel Ethernet driver: /sbin/ifconfig eth0 txqueuelen 2000 /sbin/ifconfig eth1 txqueuelen 2000 ethtool -s eth0 autoneg off speed 1000 duplex full ethtool -A eth0 autoneg off rx on tx on ethtool -C eth0 adaptive-rx off adaptive-tx off rx- usecs 20 rx-frames 5 tx-usecs 60 tx-frames 11 ethtool -G eth0 rx 511 rx-jumbo 255 tx 511 Linux kernel tuning uses the following commands: echo 1048576 > /proc/sys/net/core/rmem_max echo 1048576 > /proc/sys/net/core/rmem_default echo 1048576 > /proc/sys/net/core/wmem_max echo 1048576 > /proc/sys/net/core/wmem_default echo 10000000 > /proc/sys/net/core/optmem_max echo 262143 262143 262143 > /proc/sys/net/ipv4/tcp_rmem echo 262143 262143 262143 > /proc/sys/net/ipv4/tcp_wmem echo 8388608 8388608 8388608 > /proc/sys/net/ipv4/tcp_mem echo 400 > /proc/sys/net/unix/max_dgram_qlen echo 400 > /proc/sys/net/core/message_burst echo 2800 > /proc/sys/net/core/mod_cong echo 1000 > /proc/sys/net/core/lo_cong echo 200 > /proc/sys/net/core/no_cong echo 2900 > /proc/sys/net/core/no_cong_thresh echo 3000 > /proc/sys/net/core/netdev_max_backlog
[edit] Kernel
Certain state in the kernel must be set for it to be considered a realtime capable kernel.
To confirm this state:
- zcat /proc/config.gz | grep CONFIG_PREEMPT_RT
CONFIG_PREEMPT_RT=y
If you don't have that /proc/config.gz file, the one who built the kernel is ignorant.
[edit] Latency Testing
[edit] Known Issues
First off, disable any battery monitoring utilities as still it is not safe for realtime performance. This is a linux kernel issue and is not fixed, atm.
[edit] ALSA and OSS Applications
People still write applications that access ALSA directly. To get them into the JACK system, your best bet is running pulseaudio as a JACK client. This will force all ALSA applications to go through pulseaudio, which is then a JACK client. Applications that access the ALSA hardware layer directly will require a ALSA library rewrite and is not done yet. Look at the pulseaudio wiki and this site:
http://linuxlive.joshuacorps.org/?p=57
There is a ALSA JACK plugin, but this has a terrible track record and is to be avoided.
For applications that access OSS, there is the OSS Proxy:
http://userweb.kernel.org/~tj/ossp/
I've not played with this myself, but I've heard great things about it.
I've used oss2jack for a couple of years, but it has quite a few bugs and is now not developed anymore. There was a project called oss2pulse, which is also dead now, so your best bet is the OSS Proxy.
[edit] Other Resources
This wiki plans to be the end of your journey, but is far from it.
Check out the fugal and the lad wiki, too
There is also some old info on the
Improving GNU/Linux Sound Latency OUTDATED, but still useful
Setting Real Time Scheduler Priorities
[edit] Trivia
If you compile JACK with --enable-preemption-check, it will do the gettimeofday() calls at the right time.
With X11, enable PCI_burst, disable PCI_retry. This helps ensure the graphics card does not use too many PCI bus cycles, stalling the sound card.
