This chapter discusses mutex locks, conditions variables and read-write locks. Table 13.1 summarizes the synchronization functions that are available in the POSIX:THR Extension. Each synchronization mechanism provides an initialization function and a function for destroying the object. The mutex locks and condition variables allow static initialization. All three types of synchronization have associated attribute objects, but we work only with synchronization objects that have the default attributes.

A mutex is a special variable that can be either in the locked state or the unlocked state. If the mutex is locked, it has a distinguished thread that holds or owns the mutex. If no thread holds the mutex, we say the mutex is unlocked, free or available. The mutex also has a queue for the threads that are waiting to hold the mutex. The order in which the threads in the mutex queue obtain the mutex is determined by the thread-scheduling policy, but POSIX does not require that any particular policy be implemented.

When the mutex is free and a thread attempts to acquire the mutex, that thread obtains the mutex and is not blocked. It is convenient to think of this case as first causing the thread to enter the queue and then automatically removing it from the queue and giving it the mutex.

The mutex or mutex lock is the simplest and most efficient thread synchronization mechanism. Programs use mutex locks to preserve critical sections and to obtain exclusive access to resources. A mutex is meant to be held for short periods of time. Mutex functions are not thread cancellation points and are not interrupted by signals. A thread that waits for a mutex is not logically interruptible except by termination of the process, termination of a thread with pthread_exit (from a signal handler), or asynchronous cancellation (which is normally not used).

Mutex locks are ideal for making changes to data structures in which the state of the data structure is temporarily inconsistent, as when updating pointers in a shared linked list. These locks are designed to be held for a short time. Use condition variables to synchronize on events of indefinite duration such as waiting for input.

SYNOPSIS

   #include <pthread.h>

   int pthread_mutex_init(pthread_mutex_t *restrict mutex,
                          const pthread_mutexattr_t *restrict attr);
   pthread_mutex_t mutex = PTHREAD_MUTEX_INITIALIZER;
                                                                     POSIX:THR

pthread_mutex_t mylock = PTHREAD_MUTEX_INITIALIZER;

int error;
pthread_mutex_t mylock;

if (error = pthread_mutex_init(&mylock, NULL))
   fprintf(stderr, "Failed to initialize mylock:%s
", strerror(error));

SYNOPSIS

   #include <pthread.h>

   int pthread_mutex_destroy(pthread_mutex_t *mutex);
                                                           POSIX:THR

pthread_mutex_t mylock;

if (error = pthread_mutex_destroy(&mylock))
   fprintf(stderr, "Failed to destroy mylock:%s
", strerror(error));

SYNOPSIS

  #include <pthread.h>

  int pthread_mutex_lock(pthread_mutex_t *mutex);
  int pthread_mutex_trylock(pthread_mutex_t *mutex);
  int pthread_mutex_unlock(pthread_mutex_t *mutex);
                                                           POSIX:THR

pthread_mutex_t mylock = PTHREAD_MUTEX_INITIALIZER;

pthread_mutex_lock(&mylock);
    /*  critical section */
pthread_mutex_unlock(&mylock);

#include <pthread.h>
static int count = 0;
static pthread_mutex_t  countlock = PTHREAD_MUTEX_INITIALIZER;

int increment(void) {                  /* increment the counter */
   int error;
   if (error = pthread_mutex_lock(&countlock))
      return error;
   count++;
   return pthread_mutex_unlock(&countlock);
}

int decrement(void) {                 /* decrement the counter */
    int error;
    if (error = pthread_mutex_lock(&countlock))
       return error;
    count--;
    return pthread_mutex_unlock(&countlock);
}

int getcount(int *countp) {           /* retrieve the counter */
    int error;
    if (error = pthread_mutex_lock(&countlock))
       return error;
    *countp = count;
    return pthread_mutex_unlock(&countlock);
}

#include <pthread.h>
#include <stdlib.h>

int randsafe(double *ranp) {
    static pthread_mutex_t lock = PTHREAD_MUTEX_INITIALIZER;
    int error;

    if (error = pthread_mutex_lock(&lock))
       return error;
    *ranp = (rand() + 0.5)/(RAND_MAX + 1.0);
    return pthread_mutex_unlock(&lock);
}

#include <pthread.h>
static int doneflag = 0;
static pthread_mutex_t donelock = PTHREAD_MUTEX_INITIALIZER;

int getdone(int *flag) {                   /* get the flag */
    int error;
    if (error = pthread_mutex_lock(&donelock))
       return error;
    *flag = doneflag;
    return pthread_mutex_unlock(&donelock);
}

int setdone(void) {                        /* set the flag */
    int error;
    if (error = pthread_mutex_lock(&donelock))
       return error;
    doneflag = 1;
    return pthread_mutex_unlock(&donelock);
}

void docommand(void);

int error = 0;
int done = 0;

while(!done && !error) {
   docommand();
   error = getdone(&done);
}

#include <pthread.h>
static int globalerror = 0;
static pthread_mutex_t errorlock = PTHREAD_MUTEX_INITIALIZER;

int geterror(int *error) {                             /* get the error flag */
    int terror;
    if (terror = pthread_mutex_lock(&errorlock))
       return terror;
    *error = globalerror;
    return pthread_mutex_unlock(&errorlock);
}

int seterror(int error) {         /* globalerror set to error if first error */
    int terror;
    if (!error)            /* it wasn't an error, so don't change globalerror */
       return error;
    if (terror = pthread_mutex_lock(&errorlock))         /* couldn't get lock */
       return terror;
    if (!globalerror)
       globalerror = error;
    terror = pthread_mutex_unlock(&errorlock);
    return terror? terror: error;
}

#include <pthread.h>
#include "globalerror.h"

static int count = 0;
static double sum = 0.0;
static pthread_mutex_t  sumlock = PTHREAD_MUTEX_INITIALIZER;

int add(double x) {                                          /* add x to sum */
    int error;
    if (error = pthread_mutex_lock(&sumlock))
       return seterror(error);
    sum += x;
    count++;
    error = pthread_mutex_unlock(&sumlock);
    return seterror(error);
}

int getsum(double *sump) {                                     /* return sum */
    int error;
    if (error = pthread_mutex_lock(&sumlock))
       return seterror(error);
    *sump = sum;
    error = pthread_mutex_unlock(&sumlock);
    return seterror(error);
}

int getcountandsum(int *countp, double *sump) {      /* return count and sum */
   int error;
   if (error = pthread_mutex_lock(&sumlock))
      return seterror(error);
   *countp = count;
   *sump = sum;
   error = pthread_mutex_unlock(&sumlock);
   return seterror(error);
}

#include <math.h>
#include <stdlib.h>
#include <time.h>
#include "doneflag.h"
#include "globalerror.h"
#include "randsafe.h"
#include "sharedsum.h"
#define TEN_MILLION 10000000L

/* ARGSUSED */
void *computethread(void *arg1) {             /* compute a random partial sum */
   int error;
   int localdone = 0;
   struct timespec sleeptime;
   double val;

   sleeptime.tv_sec = 0;
   sleeptime.tv_nsec = TEN_MILLION;                                  /* 10 ms */

   while (!localdone) {
       if (error = randsafe(&val)) /* get a random number between 0.0 and 1.0 */
           break;
       if (error = add(sin(val)))
           break;
       if (error = getdone(&localdone))
           break;
       nanosleep(&sleeptime, NULL);                   /* let other threads in */
   }
   seterror(error);
   return NULL;
}

#include <math.h>
#include <pthread.h>
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <unistd.h>
#include "computethread.h"
#include "doneflag.h"
#include "globalerror.h"
#include "sharedsum.h"

int showresults(void);

int main(int argc, char *argv[]) {
    int error;
    int i;
    int numthreads;
    int sleeptime;
    pthread_t *tids;

    if (argc != 3) {    /* pass number threads and sleeptime on command line */
        fprintf(stderr, "Usage: %s numthreads sleeptime
", argv[0]);
        return 1;
    }

    numthreads = atoi(argv[1]);      /* allocate an array for the thread ids */
    sleeptime = atoi(argv[2]);
    if ((tids = (pthread_t *)calloc(numthreads, sizeof(pthread_t))) == NULL) {
        perror("Failed to allocate space for thread IDs");
        return 1;
    }
    for (i = 0; i < numthreads; i++)     /* create numthreads computethreads */
        if (error =  pthread_create(tids + i, NULL, computethread, NULL)) {
            fprintf(stderr, "Failed to start thread %d:%s
", i, strerror(error));
            return 1;
        }
    sleep(sleeptime);                      /* give them some time to compute */
    if (error = setdone()) {  /* tell the computethreads to quit */
        fprintf(stderr, "Failed to set done:%s
", strerror(error));
        return 1;
    }
    for (i = 0; i < numthreads; i++)     /* make sure that they are all done */
        if (error = pthread_join(tids[i], NULL)) {
            fprintf(stderr, "Failed to join thread %d:%s
", i, strerror(error));
            return 1;
        }
    if (showresults())
        return 1;
    return 0;
}

#include <math.h>
#include <stdio.h>
#include <string.h>
#include "globalerror.h"
#include "sharedsum.h"

int showresults(void) {
   double average;
   double calculated;
   int count;
   double err;
   int error;
   int gerror;
   double perr;
   double sum;

   if (((error = getcountandsum(&count, &sum)) != 0) ||
       ((error = geterror(&gerror)) != 0)) {                  /* get results */
      fprintf(stderr, "Failed to get results: %s
", strerror(error));
      return -1;
   }
   if (gerror) {          /* an error occurred in compute thread computation */
      fprintf(stderr, "Failed to compute sum: %s
", strerror(gerror));
       return -1;
   }
   if (count == 0)
      printf("No values were summed.
");
   else {
      calculated = 1.0 - cos(1.0);
      average = sum/count;
      err = average - calculated;
      perr = 100.0*err/calculated;
      printf("The sum is %f and the count is %d
", sum, count);
      printf("The average is %f and error is %f or %f%%
", average, err, perr);
   }
   return 0;
}

#include <errno.h>
#include <pthread.h>
static pthread_mutex_t listlock = PTHREAD_MUTEX_INITIALIZER;

int accessdata_r(void) {  /* return nonnegative traversal key if successful */
   int error;
   int key;
   if (error = pthread_mutex_lock(&listlock)) {        /* no mutex, give up */
      errno = error;
      return -1;
   }
   key = accessdata();
   if (key == -1) {
      error = errno;
      pthread_mutex_unlock(&listlock);
      errno = error;
      return -1;
   }
   if (error = pthread_mutex_unlock(&listlock)) {
      errno = error;
      return -1;
   }
   return key;
}

int adddata_r(data_t data) {        /* allocate a node on list to hold data */
   int error;
   if (error = pthread_mutex_lock(&listlock)) {        /* no mutex, give up */
      errno = error;
      return -1;
   }
   if (adddata(data) == -1) {
      error = errno;
      pthread_mutex_unlock(&listlock);
      errno = error;
      return -1;
   }
   if (error = pthread_mutex_unlock(&listlock)) {
      errno = error;
      return -1;
   }
   return 0;
}

int getdata_r(int key, data_t *datap) {             /* retrieve node by key */
   int error;
   if (error = pthread_mutex_lock(&listlock)) {        /* no mutex, give up */
      errno = error;
      return -1;
   }
   if (getdata(key, datap) == -1) {
      error = errno;
      pthread_mutex_unlock(&listlock);
      errno = error;
      return -1;
   }
   if (error = pthread_mutex_unlock(&listlock)) {
      errno = error;
      return -1;
   }
   return 0;
}

int freekey_r(int key) {                                    /* free the key */
   int error;
   if (error = pthread_mutex_lock(&listlock)) {        /* no mutex, give up */
      errno = error;
      return -1;
   }
   if (freekey(key) == -1) {
      error = errno;
      pthread_mutex_unlock(&listlock);
      errno = error;
      return -1;
   }
   if (error = pthread_mutex_unlock(&listlock)) {
      errno = error;
      return -1;
   }
   return 0;
}

If a mutex isn’t statically initialized, the program must call pthread_mutex_init before using any of the other mutex functions. For programs that have a well-defined initialization phase before they create additional threads, the main thread can perform this initialization. Not all problems fit this structure. Care must be taken to call pthread_mutex_init before any thread accesses a mutex, but having each thread initialize the mutex doesn’t work either. The effect of calling pthread_mutex_init for a mutex that has already been initialized is not defined.

The notion of single initialization is so important that POSIX provides the pthread_once function to ensure these semantics. The once_control parameter must be statically initialized with PTHREAD_ONCE_INIT. The init_routine is called the first time pthread_once is called with a given once_control, and init_routine is not called on subsequent calls. When a thread returns from pthread_once without error, the init_routine has been completed by some thread.

SYNOPSIS

  #include <pthread.h>

  int pthread_once(pthread_once_t *once_control,
                   void (*init_routine)(void));
  pthread_once_t once_control = PTHREAD_ONCE_INIT;
                                                               POSIX:THR

If successful, pthread_once returns 0. If unsuccessful, pthread_once returns a nonzero error code. No mandatory errors are defined for pthread_once.

Program 13.10 uses pthread_once to implement an initialization function printinitmutex. Notice that var isn’t protected by a mutex because it will be changed only once by printinitonce, and that modification occurs before any caller returns from printinitonce.

#include <pthread.h>
#include <stdio.h>

static pthread_once_t initonce = PTHREAD_ONCE_INIT;
int var;

static void initialization(void) {
   var = 1;
   printf("The variable was initialized to %d
", var);
}

int printinitonce(void) {        /* call initialization at most once */
   return pthread_once(&initonce, initialization);
}

The initialization function of printinitonce has no parameters, making it hard to initialize var to something other than a fixed value. Program 13.11 shows an alternative implementation of at-most-once initialization that uses a statically initialized mutex. The printinitmutex function performs the initialization and printing at most once regardless of how many different variables or values are passed. If successful, printinitmutex returns 0. If unsuccessful, printinitmutex returns a nonzero error code. The mutex in printinitmutex is declared in the function so that it is accessible only inside the function. Giving the mutex static storage class guarantees that the same mutex is used every time the function is called.

#include <pthread.h>
#include <stdio.h>

int printinitmutex(int *var, int value) {
   static int done = 0;
   static pthread_mutex_t lock = PTHREAD_MUTEX_INITIALIZER;
   int error;
   if (error = pthread_mutex_lock(&lock))
      return error;
   if (!done) {
      *var = value;
      printf("The variable was initialized to %d
", value);
      done = 1;
   }
   return pthread_mutex_unlock(&lock);
}

int whichiteration = -1;

void *thisthread(void *) {
   int i;
   for (i = 0; i < 100; i++)
      if (dostuff())
         printinitmutex(&whichiteration, i);
}

The testandsetonce function of Program 13.12 atomically sets an internal variable to 1 and returns the previous value of the internal variable in its ovalue parameter. The first call to testandsetonce initializes done to 0, sets *ovalue to 0 and sets done to 1. Subsequent calls set *ovalue to 1. The mutex ensures that no two threads have ovalue set to 0. If successful, testandsetonce returns 0. If unsuccessful, testandsetonce returns a nonzero error code.

#include <pthread.h>

int testandsetonce(int *ovalue) {
   static int done = 0;
   static pthread_mutex_t lock = PTHREAD_MUTEX_INITIALIZER;
   int error;
   if (error = pthread_mutex_lock(&lock))
      return error;
   *ovalue = done;
   done = 1;
   return pthread_mutex_unlock(&lock);
}

int error;
int oflag;
int var;

error = testandsetonce(&oflag);
if (!error && !oflag) {
   var = 1;
   printf("The variable has been initialized to 1
");
}
var++;

The strategies discussed in this section guarantee at-most-once execution. They do not guarantee that code has been executed at least once. At-least-once semantics are important for initialization. For example, suppose that you choose to use pthread_mutex_init rather than the static initializer to initialize a mutex. You need both at-most-once and at-least-once semantics. In other words, you need to perform an operation such as initialization exactly once. Sometimes the structure of the program ensures that this is the case—a main thread performs all necessary initialization before creating any threads. In other situations, each thread must call initialization when it starts executing, or each function must call the initialization before accessing the mutex. In these cases, you will need to use at-most-once strategies in conjunction with the calls.

Consider the problem of having a thread wait until some arbitrary condition is satisfied. For concreteness, assume that two variables, x and y, are shared by multiple threads. We want a thread to wait until x and y are equal. A typical incorrect busy-waiting solution is

while (x != y) ;

Having a thread use busy waiting is particularly troublesome. Depending on how the threads are scheduled, the thread doing the busy waiting may prevent other threads from ever using the CPU, in which case x and y never change. Also, access to shared variables should always be protected.

Here is the correct strategy for non-busy waiting for the predicate x==y to become true.

The mutex must be held until a test determines whether to suspend the thread. Holding the mutex prevents the condition x==y from changing between the test and the suspension of the thread. The mutex needs to be unlocked while the thread is suspended so that other threads can access x and y. The strategy assumes that the code protects all other access to the shared variables x and y with the mutex.

Applications manipulate mutex queues through well-defined system library functions such as pthread_mutex_lock and pthread_mutex_unlock. These functions are not sufficient to implement (in a simple manner) the queue manipulations required here. We need a new data type, one associated with a queue of processes waiting for an arbitrary condition such as x==y to become true. Such a data type is called a condition variable.

A classical condition variable is associated with a particular condition. In contrast, POSIX condition variables provide an atomic waiting mechanism but are not associated with particular conditions.

The function pthread_cond_wait takes a condition variable and a mutex as parameters. It atomically suspends the calling thread and unlocks the mutex. It can be thought of as placing the thread in a queue of threads waiting to be notified of a change in a condition. The function returns with the mutex reacquired when the thread receives a notification. The thread must test the condition again before proceeding.

pthread_mutex_lock(&m);
while (x != y)
   pthread_cond_wait(&v, &m);
/* modify x or y if necessary */
pthread_mutex_unlock(&m);

The function pthread_cond_wait should be called only by a thread that owns the mutex, and the thread owns the mutex again when the function returns. The suspended thread has the illusion of uninterrupted mutex ownership because it owns the mutex before the call to pthread_cond_wait and owns the mutex when pthread_cond_wait returns. In reality, the mutex can be acquired by other threads during the suspension.

A thread that modifies x or y can call pthread_cond_signal to notify other threads of the change. The pthread_cond_signal function takes a condition variable as a parameter and attempts to wake up at least one of the threads waiting in the corresponding queue. Since the blocked thread cannot return from pthread_cond_wait without owning the mutex, pthread_cond_signal has the effect of moving the thread from the condition variable queue to the mutex queue.

pthread_mutex_lock(&m);
x++;
pthread_cond_signal(&v);
pthread_mutex_unlock(&m);

In Example 13.14, the caller holds the mutex while calling pthread_cond_signal. POSIX does not require this to be the case, and the caller could have unlocked the mutex before signaling. In programs that have threads of different priorities, holding the mutex while signaling can prevent lower priority threads from acquiring the mutex and executing before a higher-priority thread is awakened.

Several threads may use the same condition variables to wait on different predicates. The waiting threads must verify that the predicate is satisfied when they return from the wait. The threads that modify x or y do not need to know what conditions are being waited for; they just need to know which condition variable is being used.

SYNOPSIS

  #include <pthread.h>

  int pthread_cond_init(pthread_cond_t *restrict cond,
                        const pthread_condattr_t *restrict attr);
  pthread_cont_t cond = PTHREAD_COND_INITIALIZER;
                                                                      POSIX:THR

pthread_cond_t barrier;
int error;

if (error = pthread_cond_init(&barrier, NULL));
   fprintf(stderr, "Failed to initialize barrier:%s
", strerror(error));

SYNOPSIS

  #include <pthread.h>

  int pthread_cond_destroy(pthread_cond_t *cond);
                                                          POSIX:THR

pthread_cond_t tcond;

if (error = pthread_cond_destroy(&tcond))
   fprintf(stderr, "Failed to destroy tcond:%s
", strerror(error));

SYNOPSIS

  #include <pthread.h>

  int pthread_cond_timedwait(pthread_cond_t *restrict cond,
                        pthread_mutex_t *restrict mutex,
                        const struct timespec *restrict abstime);
  int pthread_cond_wait(pthread_cond_t *restrict cond,
                        pthread_mutex_t *restrict mutex);
                                                                    POSIX:THR

pthread_mutex_lock(&mutex);
while (a < b)
    pthread_cond_wait(&cond, &mutex);
pthread_mutex_unlock(&mutex);

SYNOPSIS

  #include <pthread.h>

  int pthread_cond_broadcast(pthread_cond_t *cond);
  int pthread_cond_signal(pthread_cond_t *cond);
                                                           POSIX:THR

static pthread_mutex_t m = PTHREAD_MUTEX_INITIALIZER;
static pthread_cond_t v = PTHREAD_COND_INITIALIZER;

pthread_mutex_lock(&m);
while (!test_condition())                      /* get resource */
   pthread_cond_wait(&v, &m);
    /* do critical section, possibly changing test_condition() */
pthread_cond_signal(&v);          /* inform another thread */
pthread_mutex_unlock(&m);
                                      /* do other stuff */

#include <errno.h>
#include <pthread.h>

static pthread_cond_t bcond = PTHREAD_COND_INITIALIZER;
static pthread_mutex_t bmutex = PTHREAD_MUTEX_INITIALIZER;
static int count = 0;
static int limit = 0;

int initbarrier(int n) {              /* initialize the barrier to be size n */
   int error;

   if (error = pthread_mutex_lock(&bmutex))        /* couldn't lock, give up */
      return error;
   if (limit != 0) {                 /* barrier can only be initialized once */
      pthread_mutex_unlock(&bmutex);
      return EINVAL;
   }
   limit = n;
   return pthread_mutex_unlock(&bmutex);
}

int waitbarrier(void) {    /* wait at the barrier until all n threads arrive */
   int berror = 0;
   int error;

   if (error = pthread_mutex_lock(&bmutex))        /* couldn't lock, give up */
      return error;
   if (limit <=  0) {                       /* make sure barrier initialized */
      pthread_mutex_unlock(&bmutex);
      return EINVAL;
   }
   count++;
   while ((count < limit) && !berror)
      berror =  pthread_cond_wait(&bcond, &bmutex);
   if (!berror)
      berror = pthread_cond_broadcast(&bcond);           /* wake up everyone */
   error = pthread_mutex_unlock(&bmutex);
   if (berror)
      return berror;
   return error;
}

All threads in a process share the process signal handlers, but each thread has its own signal mask. The interaction of threads with signals involves several complications because threads can operate asynchronously with signals. Table 13.2 summarizes the three types of signals and their corresponding methods of delivery.

Signals such as SIGFPE (floating-point exception) are synchronous to the thread that caused them (i.e., they are always generated at the same point in the thread’s execution). Other signals are asynchronous because they are not generated at a predictable time nor are they associated with a particular thread. If several threads have an asynchronous signal unblocked, the thread runtime system selects one of them to handle the signal. Signals can also be directed to a particular thread with pthread_kill.

SYNOPSIS

  #include <signal.h>
  #include <pthread.h>

  int pthread_kill(pthread_t thread, int sig);
                                                        POSIX:THR

if (pthread_kill(pthread_self(), SIGKILL))
   fprintf(stderr, "Failed to commit suicide
");

SYNOPSIS

  #include <pthread.h>
  #include <signal.h>

  int pthread_sigmask(int how, const sigset_t *restrict set,
                      sigset_t *restrict oset);
                                                                     POSIX:THR

#include <errno.h>
#include <pthread.h>
#include <signal.h>
#include <stdio.h>
#include "doneflag.h"
#include "globalerror.h"

static int signalnum = 0;

/* ARGSUSED */
static void *signalthread(void *arg) {    /* dedicated to handling signalnum */
   int error;
   sigset_t intmask;
   struct sched_param param;
   int policy;
   int sig;

   if (error = pthread_getschedparam(pthread_self(), &policy, &param)) {
      seterror(error);
      return NULL;
   }
   fprintf(stderr, "Signal thread entered with policy %d and priority %d
",
              policy,  param.sched_priority);
   if ((sigemptyset(&intmask) == -1) ||
       (sigaddset(&intmask, signalnum) == -1) ||
       (sigwait(&intmask, &sig) == -1))
      seterror(errno);
   else
      seterror(setdone());
   return NULL;
}

int signalthreadinit(int signo) {
   int error;
   pthread_attr_t highprio;
   struct sched_param param;
   int policy;
   sigset_t set;
   pthread_t sighandid;

   signalnum = signo;                                    /* block the signal */
   if ((sigemptyset(&set) == -1) || (sigaddset(&set, signalnum) == -1) ||
      (sigprocmask(SIG_BLOCK, &set, NULL) == -1))
      return errno;
   if ( (error = pthread_attr_init(&highprio)) ||    /* with higher priority */
        (error = pthread_attr_getschedparam(&highprio, &param)) ||
        (error = pthread_attr_getschedpolicy(&highprio, &policy)) )
      return error;
   if (param.sched_priority < sched_get_priority_max(policy)) {
      param.sched_priority++;
      if (error = pthread_attr_setschedparam(&highprio, &param))
         return error;
   } else
     fprintf(stderr, "Warning, cannot increase priority of signal thread.
");
   if (error = pthread_create(&sighandid, &highprio, signalthread, NULL))
      return error;
   return 0;
}

#include <math.h>
#include <pthread.h>
#include <signal.h>
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <unistd.h>
#include "computethread.h"
#include "globalerror.h"
#include "sharedsum.h"
#include "signalthread.h"

int showresults(void);

int main(int argc, char *argv[]) {
   int error;
   int i;
   int numthreads;
   pthread_t *tids;

   if (argc != 2) {                   /* pass number threads on command line */
      fprintf(stderr, "Usage: %s numthreads
", argv[0]);
      return 1;
   }
   if (error = signalthreadinit(SIGUSR1)) {          /* set up signal thread */
      fprintf(stderr, "Failed to set up signal thread: %s
", strerror(error));
      return 1;
   }
   numthreads = atoi(argv[1]);
   if ((tids = (pthread_t *)calloc(numthreads, sizeof(pthread_t))) == NULL) {
      perror("Failed to allocate space for thread IDs");
      return 1;
   }
   for (i = 0; i < numthreads; i++)      /* create numthreads computethreads */
      if (error =  pthread_create(tids+ i, NULL, computethread, NULL)) {
         fprintf(stderr, "Failed to start thread %d: %s
", i,
                 strerror(error));
         return 1;
      }
   fprintf(stderr, "Send SIGUSR1(%d) signal to proc %ld to stop calculation
",
                   SIGUSR1, (long)getpid());
   for (i = 0; i < numthreads; i++)    /* wait for computethreads to be done */
      if (error = pthread_join(tids[i], NULL)) {
         fprintf(stderr, "Failed to join thread %d: %s
", i, strerror(error));
         return 1;
      }
   if (showresults())
      return 1;
   return 0;
}

The reader-writer problem refers to a situation in which a resource allows two types of access (reading and writing). One type of access must be granted exclusively (e.g., writing), but the other type may be shared (e.g., reading). For example, any number of processes can read from the same file without difficulty, but only one process should modify the file at a time.

Two common strategies for handling reader-writer synchronization are called strong reader synchronization and strong writer synchronization. Strong reader synchronization always gives preference to readers, granting access to readers as long as a writer is not currently writing. Strong writer synchronization always gives preference to writers, delaying readers until all waiting or active writers complete. An airline reservation system would use strong writer preference, since readers need the most up-to-date information. On the other hand, a library reference database might want to give readers preference.

POSIX provides read-write locks that allow multiple readers to acquire a lock, provided that a writer does not hold the lock. POSIX states that it is up to the implementation whether to allow a reader to acquire a lock if writers are blocked on the lock.

POSIX read-write locks are represented by variables of type pthread_rwlock_t. Programs must initialize pthread_rwlock_t variables before using them for synchronization by calling pthread_rwlock_init. The rwlock parameter is a pointer to a read-write lock. Pass NULL for the attr parameter of pthread_rwlock_init to initialize a read-write lock with the default attributes. Otherwise, first create and initialize a read-write lock attribute object in a manner similar to that used for thread attribute objects.

SYNOPSIS

  #include <pthread.h>

  int pthread_rwlock_init(pthread_rwlock_t *restrict rwlock,
                        const pthread_rwlockattr_t *restrict attr);
                                                                  POSIX:THR

If successful, pthread_rwlock_init returns 0. If unsuccessful, it returns a nonzero error code. The following table lists the mandatory errors for pthread_rwlock_init.

The pthread_rwlock_destroy function destroys the read-write lock referenced by its parameter. The rwlock parameter is a pointer to a read-write lock. A pthread_rwlock_t variable that has been destroyed with pthread_rwlock_destroy can be reinitialized with pthread_rwlock_init.

SYNOPSIS

  #include <pthread.h>

  int pthread_rwlock_destroy(pthread_rwlock_t *rwlock);
                                                             POSIX:THR

If successful, pthread_rwlock_destroy returns 0. If unsuccessful, it returns a nonzero error code. No mandatory errors are defined for pthread_rwlock_destroy.

The pthread_rwlock_rdlock and pthread_rwlock_tryrdlock functions allow a thread to acquire a read-write lock for reading. The pthread_rwlock_wrlock and pthread_rwlock_trywrlock functions allow a thread to acquire a read-write lock for writing. The pthread_rwlock_rdlock and pthread_rwlock_wrlock functions block until the lock is available, whereas pthread_rwlock_tryrdlock and pthread_rwlock_trywrlock return immediately. The pthread_rwlock_unlock function causes the lock to be released. These functions require that a pointer to the lock be passed as a parameter.

SYNOPSIS

  #include <pthread.h>

  int pthread_rwlock_rdlock(pthread_rwlock_t *rwlock);
  int pthread_rwlock_tryrdlock(pthread_rwlock_t *rwlock);
  int pthread_rwlock_wrlock(pthread_rwlock_t *rwlock);
  int pthread_rwlock_trywrlock(pthread_rwlock_t *rwlock);
  int pthread_rwlock_unlock(pthread_rwlock_t *rwlock);
                                                                    POSIX:THR

If successful, these functions return 0. If unsuccessful, these functions return a nonzero error code. The pthread_rwlock_tryrdlock and pthread_rwlock_trywrlock functions return EBUSY if the lock could not be acquired because it was already held.

Program 13.16 uses read-write locks to implement a thread-safe wrapper for the list object of Program 2.7. The listlib.c module should be included in this file, and its functions should be qualified with the static attribute. Program 13.16 includes an initialize_r function to initialize the read-write lock, since no static initialization is available. This function uses pthread_once to make sure that the read-write lock is initialized only one time.

#include <errno.h>
#include <pthread.h>

static pthread_rwlock_t listlock;
static int lockiniterror = 0;
static pthread_once_t lockisinitialized = PTHREAD_ONCE_INIT;

static void ilock(void) {
   lockiniterror = pthread_rwlock_init(&listlock, NULL);
}

int initialize_r(void) {    /* must be called at least once before using list */
   if (pthread_once(&lockisinitialized, ilock))
      lockiniterror = EINVAL;
   return lockiniterror;
}

int accessdata_r(void) {               /* get a nonnegative key if successful */
   int error;
   int errorkey = 0;
   int key;
   if (error = pthread_rwlock_wrlock(&listlock)) {  /* no write lock, give up */
      errno = error;
      return -1;
   }
   key = accessdata();
   if (key == -1) {
      errorkey = errno;
      pthread_rwlock_unlock(&listlock);
      errno = errorkey;
      return -1;
   }
   if (error = pthread_rwlock_unlock(&listlock)) {
      errno = error;
      return -1;
   }
   return key;
}

int adddata_r(data_t data) {          /* allocate a node on list to hold data */
   int error;
   if (error = pthread_rwlock_wrlock(&listlock)) { /* no writer lock, give up */
      errno = error;
      return -1;
   }
   if (adddata(data) == -1) {
      error = errno;
      pthread_rwlock_unlock(&listlock);
      errno = error;
      return -1;
   }
   if (error = pthread_rwlock_unlock(&listlock)) {
      errno = error;
      return -1;
   }
   return 0;
}

int getdata_r(int key, data_t *datap) {               /* retrieve node by key */
   int error;
   if (error = pthread_rwlock_rdlock(&listlock)) { /* no reader lock, give up */
      errno = error;
      return -1;
   }
   if (getdata(key, datap) == -1) {
      error = errno;
      pthread_rwlock_unlock(&listlock);
      errno = error;
      return -1;
   }
   if (error = pthread_rwlock_unlock(&listlock)) {
      errno = error;
      return -1;
   }
   return 0;
}

int freekey_r(int key) {                                      /* free the key */
   int error;
   if (error = pthread_rwlock_wrlock(&listlock)) {
      errno = error;
      return -1;
   }
   if (freekey(key) == -1) {
      error = errno;
      pthread_rwlock_unlock(&listlock);
      errno = error;
      return -1;
   }
   if (error = pthread_rwlock_unlock(&listlock)) {
      errno = error;
      return -1;
   }
   return 0;
}

Unfortunately, POSIX lists strerror as one of the few functions that are not thread-safe. Often, this is not a problem since often the main thread is the only thread that prints error messages. If you need to use strerror concurrently in a program, you will need to protect it with mutex locks. Neither perror nor strerror is async-signal safe. One way to solve both the thread-safety and async-signal-safety problems is to encapsulate the synchronization in a wrapper, as shown in Program 13.17.

The perror_r and strerror_r functions are both thread-safe and async-signal safe. They use a mutex to prevent concurrent access to the static buffer used by strerror. The perror function is also protected by the same mutex to prevent concurrent execution of strerror and perror. All signals are blocked before the mutex is locked. If this were not done and a signal were caught with the mutex locked, a call to one of these from inside the signal handler would deadlock.

Programs that use synchronization constructs have the potential for deadlocks that may not be detected by implementations of the POSIX base standard. For example, suppose that a thread executes pthread_mutex_lock on a mutex that it already holds (from a previously successful pthread_mutex_lock). The POSIX base standard states that pthread_mutex_lock may fail and return EDEADLK under such circumstances, but the standard does not require the function to do so. POSIX takes the position that implementations of the base standard are not required to sacrifice efficiency to protect programmers from their own bad programming. Several extensions to POSIX allow more extensive error checking and deadlock detection.

Another type of problem arises when a thread that holds a lock encounters an error. You must take care to release the lock before returning from the thread, or other threads might be blocked.

Threads with priorities can also complicate matters. A famous example occurred in the Mars Pathfinder mission. The Pathfinder executed a “flawless” Martian landing on July 4, 1997, and began gathering and transmitting large quantities of scientific data to Earth [34]. A few days after landing, the spacecraft started experiencing total system resets, each of which delayed data collection by a day. Several accounts of the underlying causes and the resolution of the problem have appeared, starting with a keynote address at the IEEE Real-Time Systems Symposium on Dec. 3, 1997, by David Wilner, Chief Technical Officer of Wind River [61].

#include <errno.h>
#include <pthread.h>
#include <signal.h>
#include <stdio.h>
#include <string.h>

static pthread_mutex_t lock = PTHREAD_MUTEX_INITIALIZER;

int strerror_r(int errnum, char *strerrbuf, size_t buflen) {
   char *buf;
   int error1;
   int error2;
   int error3;
   sigset_t maskblock;
   sigset_t maskold;

   if ((sigfillset(&maskblock)== -1) ||
       (sigprocmask(SIG_SETMASK, &maskblock, &maskold) == -1))
      return errno;
   if (error1 = pthread_mutex_lock(&lock)) {
      (void)sigprocmask(SIG_SETMASK, &maskold, NULL);
      return error1;
   }
   buf = strerror(errnum);
   if (strlen(buf) >= buflen)
      error1 = ERANGE;
   else
      (void *)strcpy(strerrbuf, buf);
   error2 = pthread_mutex_unlock(&lock);
   error3 = sigprocmask(SIG_SETMASK, &maskold, NULL);
   return error1 ? error1 : (error2 ? error2 : error3);
}

int perror_r(const char *s) {
   int error1;
   int error2;
   sigset_t maskblock;
   sigset_t maskold;

   if ((sigfillset(&maskblock) == -1) ||
       (sigprocmask(SIG_SETMASK, &maskblock, &maskold) == -1))
      return errno;
   if (error1 = pthread_mutex_lock(&lock)) {
      (void)sigprocmask(SIG_SETMASK, &maskold, NULL);
      return error1;
   }
   perror(s);
   error1 = pthread_mutex_unlock(&lock);
   error2 = sigprocmask(SIG_SETMASK, &maskold, NULL);
   return error1 ? error1 : error2;
}

The Mars Pathfinder flaw was found to be a priority inversion on a mutex [105]. A thread whose job was gathering meteorological data ran periodically at low priority. This thread would acquire the mutex for the data bus to publish its data. A periodic high-priority information thread also acquired the mutex, and occasionally it would block, waiting for the low-priority thread to release the mutex. Each of these threads needed the mutex only for a short time, so on the surface there could be no problem. Unfortunately, a long-running, medium-priority communication thread occasionally preempted the low-priority thread while the low-priority thread held the mutex, causing the high-priority thread to be delayed for a long time.

A second aspect of the problem was the system reaction to the error. The system expected the periodic high-priority thread to regularly use the data bus. A watchdog timer thread would notice if the data bus was not being used, assume that a serious problem had occurred, and initiate a system reboot. The high-priority thread should have been blocked only for a short time when the low-priority thread held the mutex. In this case, the high-priority thread was blocked for a long time because the low-priority thread held the mutex and the long-running, medium-priority thread had preempted it.

A third aspect was the test and debugging of the code. The Mars Pathfinder system had debugging code that could be turned on to run real-time diagnostics. The software team used an identical setup in the lab to run in debug mode (since they didn’t want to debug on Mars). After 18 hours, the laboratory version reproduced the problem, and the engineers were able to devise a patch. Glenn Reeves [93], leader of the Mars Pathfinder software team, was quoted as saying “We strongly believe in the ’test what you fly and fly what you test’ philosophy.” The same ideas apply here on Earth too. At a minimum, you should always think about instrumenting code with test and debugging functions that can be turned on or off by conditional compilation. When possible, allow debugging functions to be turned on dynamically at runtime.

A final aspect of this story is timing. In some ways, the Mars Pathfinder was a victim of its own success. The software team did extensive testing within the parameters of the mission. They actually saw the system reset problem once or twice during testing, but did not track it down. The reset problem was exacerbated by high data rates that caused the medium-priority communication thread to run longer than expected. Prelaunch testing was limited to “best case” high data rates. In the words of Glenn Reeves, “We did not expect nor test the ’better than we could have ever imagined’ case.” Threaded programs should never rely on quirks of timing to work—they must work under all possible timings.

Reimplement the barrier of Program 13.13 so that it supports multiple barriers. One possible approach is to use an array or a linked list of barriers. Explore different designs with respect to synchronization. Is it better to use a single bmutex lock and bcond condition variable to synchronize all the barriers, or should each barrier get its own synchronization? Why?

Most operating systems books spend some time on synchronization and use of standard synchronization mechanisms such as mutex locks, condition variables and read-write locks. The review article “Concepts and notations for concurrent programming,” by Andrews and Schneider [3] gives an excellent overview of much of the classical work on synchronization. “Interrupts as threads” by Kleiman and Eykholt [63] discusses some interesting aspects of the interaction of threads and interrupts in the kernel. An extensive review of monitors can be found in “Monitor classification,” by Buhr et al. [17]. The signal and wait operations of monitors are higher-level implementations of the mutex-conditional variable combination. The Solaris Multithreaded Programming Guide [109], while dealing primarily with Solaris threads, contains some interesting examples of synchronization. Finally, the article “Schedule-conscious synchronization” by Kontothanassis et al. [65] discusses implementation of mutex locks, read-write locks and barriers in a multiprocessor environment.