This chapter discusses the various debugging and tracing capabilities provided by ITDS 5.2. These facilities provide a directory administrator greater insight into what on side of the directory any given time.

The process of installing and configuring ITDS, for various reasons, is not always error free. The directory administrator can encounter problems with basic installation of the server, configuring various server components, or the server might fail to start for no obvious reason.

Debugging is the process of finding the cause of the problem using various tools and techniques and eliminating them. Due of these inherent problems, ITDS provides administrators various command line options, tools and detailed log files that help the user find the cause of the problem.

The following sections describe how to debug configuration problems, directory server errors, directory server debug modes, and DB2 debug logs.

The first thing that is to be done after installation of the product is the configuration. If this fails then there is no way you can go but to resolve the issue. The very basic steps towards making the IBM Tivoli Directory Server up and running are:

The above basic tasks are performed by using the configuration tools provided by the directory server. These are:

While the configuration of Admin DN and password is fairly straightforward and is less error prone, the database configuration may not be that easy. Generally, the only reasons the configuration of the Admin DN fails are if the IBM Directory configuration file (<ldapinstalldir>/etc/ibmslapd.conf) has had the permissions accidentally changed, or if the user enters an invalid DN. If the database configuration fails, the following sources can be checked to find the cause of the failure:

All of the configuration programs are either started from a console command line prompt (ldapcfg, ldapucfg) or open a background console (ldapxcfg). As the database configuration progresses, status messages (and limited error messages) are displayed in the associated console window. If a problem occurs, the user should copy these messages to the system clipboard and then save them in a file for the support teams.

If the error is a direct error from DB2, then DB2 often creates message/error files in the /tmp directory (on UNIX platforms). If the user has a database configuration problem on UNIX systems, they need to examine all of the files in the /tmp directory that were created around the time of the attempted configuration. On Windows systems, examine any DB2 error logs in the directory named for the instance you were trying to configure under the DB2 install directory under. For example, if your were trying to create the default ldapdb2 instance and database, and if your DB2 was installed in D:sqllib, then you need to examine the files in the D:sqllibldapdb2 directory if it exists. Especially look for and examine the db2diag.log file in that directory.

IBM Directory logs most configuration errors in the file ldacfg.out. On UNIX platforms, this file can be found in the /tmp directory. On Windows platforms, this file is created in the root directory of the drive you ran configuration from.

If the above sources are not sufficient for determining the cause of the problem, we can resort to advanced debugging. In advanced debugging we set two environment variables and collect the relevant logs as explained below:

Set this environment variable to any non-empty value, for example:

On UNIX platforms, use export JAVA_DEBUG=1. This causes certain Java debug information built into the code to be displayed on stdout (the console). The best practice is to redirect the output to a file and then analyses the same.

Set this environment variable to any non-empty value. For example:

On UNIX platforms, use export LDAP_DBG=1. This causes a debug file to be created for the IBM support and development teams. The file name that is created is dbg.log. It is created in /var/ldap/ directory.

On the Windows NT and Windows 2000 platforms, dbg.log is created in the <ldapinstalldir> var directory.

When a problem occurs that appears to be related to the IBM Directory Server, you should first check the following files for error messages. The default locations of these files are /var/ldap in case of Unix and <ldapinstalldir>var in case of windows.

You can change the location of these files by modifying the appropriate settings from the Web administration tool or by directly editing the ibmslapd.conf file.

For more information on how to handle the above log files, please go through Chapter 17, “Monitoring IBM Tivoli Directory Server” on page 547.

If the above listed log files do not provide enough information about the problem, you can run the IBM Tivoli Directory Server in a special debug mode that generates very detailed information. An ibmslapd trace provides a list of the SQL commands issued to the DB2 database. These commands can help you identify operations that are taking a long time to complete.

Here is a snippet of the server debug trace:

The above statements show that the time taken to do the SQLExecutes of the statement with a handle of 20001 is 423 micro seconds and that the time required by the SQLFetch for the same SQL statement is 26 micro seconds.

In some cases, the log files you have collected seem to provide that some problem had occurred with the directory server, but fail to hint at the probable cause. The debug trace would help in such situations, as it would give more detailed steps of the server operations. The best example would be, suppose due to some error, the database starts up in configuration mode. The logs will not provide you the problem cause. They will just hint that there was a problem with the server. So how do you get to the root of the problem? Of course yes, by running the server in debug mode. That helps out to know the exact cause of the server starting up in configuration mode. You just correct the problem, pointed to, by the trace, and there you are, you’re server is up and running once again.

The server executable ibmslapd must be run from a command prompt to enable debug output. The syntax is as follows:

The above means to turn (on) the ldap trace and keep a buffer of 20 Million lines. that is, out of the total trace information only the last 20 Million lines would be kept in the buffer and not more than that. This is handy sometimes, in situations where you want to reduce the size of the debug file to be analyses and are sure that you would stop the server instantly when the problem occurs. Let us see what the above commands mean. First We will look into what ldtrc means for us.

This command is used for enabling/disabling the trace options for ibmslapd. In the sense that you would need to use this command to turn on the trace options for allowing ibmslapd to run in debug mode. What debug level ibmslapd runs at is a later issue. This command is just a preparatory step towards enabling the debug mode of ibmslapd.

ldtrc usage:

The further details for the above shall not be given here and can be found in the description of ldaptrace in one of the following sections.

Once you have turned on the trace options using ldtrc, we can go ahead with collecting the server debug information by running ibmslapd in debug mode. The server is run in debug mode by specifying a bitmask. The bitmask helps the server in deciding the set of operations it is supposed to run with extensive tracing. Like if you have replication related issues and want to debug the replication operations of the server, you can turn on just the flags pertaining to replication (1024) that is, specify this debug bitmask while starting the server. A value of 65535 for the bitmask indicates that the maximum debug output should be generated. Table 18-1 describes the various flags you can set in the bitmask.

Now depending upon the type of operations you want to debug, just form the relevant bitmask by ORing the individual bitmasks and pass the consolidated bitmask to ibmslapd, during its startup.

The trace output will be directed to the standard output. It is recommended that you redirect the same to a file for analyzing later, as the console buffer might not be sufficient to retain all the information on the screen. The file redirection can be achieved by following any of the given steps below.

For Windows:

For Unix:

Running a trace on several operations will definitely result in slow performance, so remember to turn the trace off when you are finished using it by the following command:

It is not always feasible to have the server run in debug mode for all times especially in a production environment, where high performance is always a mandate. In such circumstances, it is recommended to go for dynamic tracing, as will be seen in the next section.

Sometimes it is observed that there are problems with the directory server, after running the directory server for a long time that is, say of the order of eight hours. The log files indicate that there is some problem, but it is not clear. However, it is essential to get to know the problem, because that’s impacting business. What do we do in such cases? It is not feasible to run the server in debug mode for say 8 hours and then analyze the tons of data that this will generate, taking a lot more time in debugging the issue. In some cases, we aren’t even sure that the problem would get generated after taking all the efforts of running the server in debug mode, in a planned downtime. Such situations are common and in such circumstances, it is better to go for dynamic tracing of the server. That would not impact the server’s performance, during the period when we know that the problem will not occur and also the amount of trace generated would be comparatively much smaller to amount generated using the static tracing. The utility being used for dynamic tracing is known as ldaptrace. Let us study this tool in more depth.

The administration tracing utility, ldaptrace, is used to dynamically activate or deactivate tracing of the Directory Server. This extended operation can also be used to set the message level and specify the name of the file, where the output is written. If LDAP trace facility (ldtrc) options are requested, they must be preceded by --.

To display syntax help for ldaptrace, type:

Here is a synopsis of what sort of parameters ldaptrace would take:

The options are:

A default keyring file that is, ldapkey.kdb, and the associated password stash file that is, ldapkey.sth, are installed in the /lib directory under LDAPHOME, where LDAPHOME is the path to the installed LDAP support. LDAPHOME varies by operating system platform:

See IBM Directory C-Client SDK Programming Reference for more information about default key database files, and default Certificate Authorities. This document can be found at:

If a keyring database file cannot be located, a “hard-coded” set of default trusted certificate authority roots is used. The key database file typically contains one or more certificates of certificate authorities (CAs) that are trusted by the client. These types of X.509 certificates are also known as trusted roots. For more information on managing an SSL or TLS key database, see “SSL/TLS support” on page 455.

This parameter effectively enables the -Z switch.

Now let us see some examples for using the ldaptrace utility.

To turn the ldtrc facility on and start the server trace with a 2M trace buffer, issue the command:

To stop the server trace, issue the command:

To turn off the ldtrc facility, issue the command:

To collect the trace over SSL, issue the command:

Thus using ldaptrace we can debug the server dynamically. Consequently we can maintain the performance of the server, by not running it in debug mode all the time and by taking the desired debug information as and when necessary.

The main db2 log file you need to check in case of errors is db2diag.log. This file is mainly meant for the support people and the developers to analyze.

In case of UNIX systems, it is located in <Directory containing your database>/sqllib/db2dump directory. In windows it is located in x:sqllib<instance_name> directory where x is the drive where DB2 is installed.

You can control the level of detailed information that is written to the db2diag.log file by using the DIAGLEVEL configuration parameter and the DLFM_LOG_LEVEL registry value.

Determines the severity of DB2 diagnostic information recorded in the db2diag.log error log file. Valid values are from 1–4. 1 denotes that a minimal amount of information is to be recorded, and 4 denotes that the maximum amount of information is to be recorded. The default setting is 3. You can increase the amount of error information recorded using the following command: db2 update dbm cfg using DIAGLEVEL 4. This setting should be changed only at the request of IBM service or development for debugging purposes.

Determines the severity of DLFM diagnostic information recorded in the db2diag.log error log file. Its default setting is LOG_ERR. You can increase the amount of error information recorded using the following command:

In summary: