8.7. Pthread priority and scheduling
By default, each Pthread within a process has its own a dynamic or floating priority on AIX. This mechanism ensures that all Pthreads on a system have an opportunity to run and not be locked out by other Pthreads. As individual Pthreads consume CPU time, they are charged. Pthreads that are not ready to run, and that are not given a processor upon which they consume cycles, are not charged. Over time, CPU intensive Pthreads use more CPU cycles, are charged more, and their priority degrades. Pthreads that run only briefly before being preempted or relinquishing control of the processor tend to retain a fairly constant priority. The net result is that I/O intensive tasks stay near a constant priority, and computational tasks become lower in priority. The I/O intensive tasks usually run briefly. The computational tasks run for most or all of their allotted processor time. This allows all processes on the system to be responsive, ensures that every task is treated fairly with respect to CPU utilization, and prevents long-running tasks from hogging the system at the expense of others.
It is also possible to fix the priority of a Pthread at a specific value. This facility allows processes with Pthreads that must run at a certain response level to accomplish tasks in a predicable manner. While a normal process can fix the priority of a Pthread up to a certain level, at high priority levels the root user authority is required.
8.7.1. Thread models in AIX
Pthreads (user threads) are mapped to underlying kernel threads. A thread model refers to the way this mapping is done. A light-weight process (LWP) refers to a kernel thread.
There are three possible thread models, corresponding to three different ways to map Pthreads to kernel threads (see Figure 8-4 on page 323).
Figure 8-4. Thread models
The M:1 model[15]
[15] This was a thread model on AIX Version 3.2 when DCE is installed.
Also known as the library thread scheduling, it refers to the case where threads share a pool of available LWPs.
In the M:1 model, all user threads run on one VP and are linked to exactly one LWP; the mapping is handled by a POSIX thread library scheduler. All user threads programming facilities are completely handled by the library. This model can be used on any system, especially on traditional single-threaded systems.
The 1:1 model[16]
[16] This was the thread model on AIX Version 4.1 and 4.2.
The 1:1 model is sometimes referred to as bound thread scheduling. In this model, each user thread is bound to a VP and linked to exactly one kernel thread (LWP). The VP is not necessarily bound to a real CPU (unless binding to a processor was done). Each VP can be thought of as a virtual CPU available for executing user code and system calls. A thread that is bound to a VP is said to have system scope, because it is directly scheduled with all the other kernel threads by the kernel scheduler. Most of the user threads programming facilities are directly handled by the kernel threads.
The M:N model
In the M:N model (this is also known as multiplexed thread scheduling), several user threads can share the same virtual processor or the same pool of VPs. A thread that is not bound to a VP is said to be a local or process scope, because it is not directly scheduled with all the other threads by the kernel scheduler. The Pthreads library will handle the scheduling of user threads to the VP and then the kernel will schedule the associated kernel thread. This is the most efficient and most complex thread model; the user threads programming facilities are shared between the threads library and the kernel threads.
The mapping of user threads to kernel threads is done using virtual processors. A virtual processor (VP) is a library entity that is usually implicit. For a user thread, the virtual processor behaves as a CPU for a kernel thread. In the library, the virtual processor is a kernel thread or a structure bound to a kernel thread, that is, user threads sit on top of virtual processors, which are themselves on top of kernel threads.
Note
The M:N model is the default thread model on AIX, beginning with Version 4.3.
Comparing bound and multiplexed threads
Bound thread scheduling differs from multiplexed thread scheduling in the following important ways:
A bound thread is permanently associated to its kernel thread, hence, it is exempt from the intermediate level of scheduling provided by the POSIX threads library used under the M:1 and M:N thread models.
A bound thread executes exactly when its associated kernel thread is scheduled by the kernel scheduler.
A bound thread’s scheduling policy is related to the underlying kernel thread. In AIX, only kernel threads with root authority can use a fixed-priority scheduling policy.
Multiplexed thread scheduling has the following important characteristic: A multiplexed thread is subject to two levels of scheduling. First, the thread is assigned to a kernel thread and preempted by a POSIX thread library scheduler. Second, the kernel scheduler assigns the LWPs to processors and then preempts them.
8.7.2. Scheduling Pthreads
A kernel component, called scheduler, is used to share processor cycles among running processes. The kernel scheduler coordinates ready-to-run tasks of varying priorities, running at least 100 times every second to reevaluate tasks on the run queue and allow processes to run based on the scheduling policy.
Starting with Version 4.3, AIX supports Pthread scheduling at the process level. The user scheduler is provided by the Pthreads library and supports the M:N thread model on AIX. The Pthreads library scheduler has a pool of one or more virtual processors (VPs) upon which a user-space Pthread may run. You can think of a VP as a kernel-space thread. This mapping of the user thread to kernel thread provides more effective use of kernel resources for processes that create many threads, not all of which will be running at any given moment. For example, in a producer-consumer thread pair, we will often see only one task running while the other waits to send or receive data. The Pthread library scheduler itself is also created as a Pthread in a process and runs on its own VP.
The Pthreads library allows the programmer to control the execution scheduling of the Pthreads. The control can be performed in two different ways:
By setting scheduling attributes when creating a Pthread.
By dynamically changing the scheduling attributes of a created Pthread.
A Pthread has three scheduling parameters:
| Scope | The contention scope of a Pthread is defined by the thread model used in the Pthreads library. |
| Policy | The scheduling policy of a Pthread defines how the scheduler treats the Pthread once it gains control of the CPU. |
| Priority | The scheduling priority of a Pthread defines the relative importance of the work being done by each Pthread. |
The scheduling parameters can be set before the Pthread’s creation or during the Pthread’s execution. In general, controlling the scheduling parameters of Pthreads is important only for Pthreads that are compute-intensive. Thus, the Pthreads library provides default values that are sufficient for most cases.
Controlling the scheduling of a Pthread is often a complicated task. Because the scheduler can handle all system or process-wide Pthreads, depending on the scope context, the scheduling parameters of a Pthread can interact with those of all other Pthreads in the process and in the other processes on the system.
Scheduling policies
AIX also provides support for a number of scheduling policies. Each policy has certain characteristics that will benefit a specific workload, but not all policies are suitable for every task. We can create an application that assigns different policies to different Pthreads, tuning our application to make optimum use of the system. AIX provides the following policies at the system, or kernel, level:
SCHED_FIFO
SCHED_RR
SCHED_OTHER
SCHED_FIFO2
SCHED_FIFO3
As long as a process is using process-level scope (that is, the M:N thread model is in effect), the process may modify the scheduling policy of the Pthreads within that process. If the process is using system-scope (the 1:1 model), then the root user authority is required to modify the scheduling or priority aspects of the Pthread. Root privileges are also required to modify the policy of a Pthread outside the scope of the process, or to set the priority of a Pthread or process to a fixed value.
SCHED_FIFO
SCHED_FIFO is First-In/First-Out scheduling. A Pthread with fixed priority and SCHED_FIFO scheduling will be allowed to run to completion unless it:
It yields the processor voluntarily.
It blocks because something it needs is not available.
A kernel thread with a higher priority is available to run.
Running Pthreads under a FIFO policy is discouraged because it prevents the system from balancing resource utilization in a way that lets all tasks run.
SCHED_RR
SCHED_RR is round-robin scheduling. Pthreads using this policy will receive a time slice (time slicing is a mechanism that ensures that every Pthread is allowed time to execute by preempting running Pthreads at fixed intervals) of the CPU. Only Pthreads with a fixed priority can take advantage of round-robin scheduling.
As each Pthread completes its allotted time on the processor, it is placed at the end of the queue. The Pthread at the front of the queue is allowed to run. This circular sharing of the processor ensures that systems that require a given number of Pthreads to execute on a regular basis will not starve any task.
SCHED_OTHER
SCHED_OTHER is the default scheduling policy used by AIX. Unlike first-in/first-out and round-robin, the SCHED_OTHER policy is used to manage Pthreads that have dynamic priority. In this policy the scheduler runs 100 times per second, and at each interval charges the Pthread(s) running on a processor(s) for its use of the CPU. Floating priorities are computed by using the Pthread priority, the CPU usage, and various tunable kernel parameters.
By default, a Pthread is created with a priority of 40 plus its nice value. The default nice value is 20 and can be changed within a relative range of plus or minus 20 using the nice command. This results in a default priority for a newly created Pthread of 60. If the Pthread has control of a processor when the scheduler next runs, the Pthread is charged and its priority is recalculated. The formula used to recalculate Pthread priority is:
recalculated_priority = P _nice + (C * R / 32)
Where:
| P_nice | 40 + NICE. |
| C (recent CPU usage) | Old CPU Usage * D / 32 (default D is 16 and ranges from 0-32). |
| R / 32 | The CPU penalty factor, where R is a kernel tunable and ranges from 0-32. |
The values for D and R are tunable. D defaults to 16 and may range from 0 to 32; R also defaults to 16 and ranges in value from 0 to 32.
SCHED_FIFO2
SCHED_FIFO2 is similar to SCHED_FIFO, but allows a Pthread that has slept for a short period of time to be added to the front of the run queue when it awakened. The time period is a kernel parameter that can be modified. This affinity limit is adjusted using the schedtune command:
/usr/samples/kernel/schedtune -a N
Where N is the number of clock ticks.
SCHED_FIFO3
SCHED_FIFO3 will always place an awakened Pthread at the front of the queue, regardless of how long it was asleep. While this is similar to FIFO2, we should remember that when a FIFO3 Pthread is on a run queue, the parameters of the queue are modified to prevent a FIFO2 Pthread from being put at the head of the queue.
Setting Pthread scheduling policies
Scheduling policies are inherited from the process that created the Pthread. If a different scheduling policy is required, the pthread_attr_setinheritsched() routine can be used to set the thread attribute prior to Pthread creation. Again, this applies to Pthreads in a process that uses the M:N model:
pthread_attr_setinheritsched(pthread_attr_t *thread_attr, int value)
Where:
| thread_attr | Thread attribute structure or object |
| Value | Either PTHREAD_INHERIT_SCHED or PTHREAD_EXPLICIT_SCHED |
If the Pthread is already created, the pthread_setschedparam() routine can be used to modify a Pthread’s scheduling policy. The scheduling change is not realized until the next time the Pthread is put on the run queue. That is, if a Pthread is currently running and a call is made to pthread_setschedparam(), the change will not become effective until the Pthread has consumed its time slice:
pthread_setschedparam(pthread_t thread_id, int policy, *param)
Where:
| thread_id | Thread ID to operate on. |
| policy | Scheduling policy, for example, SCHED_RR or SCHED_FIFO. |
| param | This structure contains a policy field but is ignored. |
8.7.3. Scheduling limitations
Thread scope affects how the underlying kernel thread’s scheduling will behave. For system scope threads, there exists one kernel thread for every user thread, that is, pthread. Modifying a system scope thread’s scheduling policy modifies the underlying kernel thread’s scheduling policy; thus, system scope threads are limited to SCHED_OTHER unless the process has root privileges. Also, in the case of system scope threads, the other scheduling policies (FIFO and round-robin) require a fixed priority and root is the only user who can modify a thread’s priority.
As stated earlier, process scope threads are threads that have M user threads to N kernel threads. The scheduling policy for process scope threads is handled at the library level and does not affect the underlying kernel thread(s). At the process level, the scheduling policy affects how the ready-to-run Pthreads are placed in the run queue. The scheduling policy of the underlying kernel thread (which implements the virtual processor, or VP) is managed by the kernel.