Diagnostic tests
This appendix describes some procedures and methods that are used for performing problem determination, troubleshooting the status of the components, and acquiring diagnostic information. Enabling traces for the components is done only at the request of IBM Support.
This appendix includes the following topics:
B.1 Diagnostic messages overview
Diagnostic messages are written to separate places, depending on the component for which diagnostic information is being collected. Look for messages in the SYSLOG or the running started tasks. CFZCIM, IZUANG1, CPOSERV, and IZUSVR1 started tasks (STC) are often the first places to check.
Part of the CIM server STC that records a RACF profile issue for the MELVIN ID is shown in Figure B-1.
|
1 IAT6140 JOB ORIGIN FROM GROUP=ANYLOCAL, DSP=SR , DEVICE=STC , 0000
13:20:29 ---- IAT6853 THE CURRENT DATE IS THURSDAY, 03 AUG 2017 ----
13:20:29 IEF695I START CFZCIM WITH JOBNAME CFZCIM IS ASSIGNED TO USER CFZSRV , GROUP CFZSRVGP
13:20:29 CFZ12580I: CIM server running eligible for zIIP.
13:20:29 CFZ10025I: The CIM server is listening on HTTP port 5988.
13:20:29 CFZ10028I: The CIM server is listening on the local connection socket.
13:20:29 CFZ10030I: Started CIM Server version 2.14.2.
13:20:29 CFZ12532I: The CIM server successfully registered to ARM using element name CFZ_SRV_SC80 .
13:20:43 CFZ13007W: Request UserID TOBIASdoesnt have READ permission to profile CIMSERV CL(WBEM).
13:20:43 ICH408I USER(TOBIAS ) GROUP(SYS1 ) NAME(TOBIAS )
13:20:43 CIMSERV CL(WBEM )
13:20:43 INSUFFICIENT ACCESS AUTHORITY
13:20:43 ACCESS INTENT(READ ) ACCESS ALLOWED(NONE )
|
Figure B-1 RACF violation
B.2 CIM server tracing
Logging and tracing troubleshooting tools are provided by CIM. These tools are defined in the CIM server configuration properties.
Use the UNIX System Services cimconfig command or the MVS Modify command to enable tracing and logging for the CIM server. For more information, see z/OS Common Information Model User's Guide, SC34-2671.
You need CONTROL access to the CIMSERV profile in the WBEM class to run
these commands.
these commands.
By default, tracing is written to memory and included in a memory dump; this default can be changed by using the traceFacility setting. IBM Support often asks you to modify the Loglevel, traceComponents, and traceLevel settings.
B.2.1 Log levels
You can choose between the following log levels:
•INFORMATION : The default level, which provides informational messages
•WARNING : Returns log messages for warnings, and severe and unrecoverable errors
•SEVERE : Returns log messages for severe and unrecoverable errors
•FATAL : Returns log messages only for unrecoverable errors
•TRACE : Returns all log messages and all trace messages
B.2.2 Trace components
The traceComponents parameter specifies the components that you want to trace. Although you can isolate a trace to a specific CIM component, “All” is suggested.
For more information about the available options, see the “Tracing” section of z/OS Common Information Model User's Guide, SC34-2671.
B.2.3 Trace level
The traceLevel parameter turns tracing on and off and specifies the trace level. The following trace levels are available:
•0 : Tracing is off
•1 : Severe errors
•2 : Warning level error messages (default)
•3 : Inter-function logic flow, medium data detail
•4 : High data detail
•5 : High data detail, method enter, and exit
B.2.4 Example of tracing
In this section, we describe a tracing example in which we perform the following actions:
1. Specify a trace level of 5 for High data detail (traceLevel).
2. Specify a trace for all components (traceComponents).
3. Run the trace.
4. Write the trace to the default trace file /tmp/cimserver.trc. This file can be changed by using the traceFilePath parameter.
To set tracing dynamically, run the commands that are shown in Example B-1 from the UNIX System Services shell. You can also save these commands in a UNIX System Services shell script for faster setup.
Example B-1 Cimconfig commands to enable tracing
cimconfig -c -s traceFacility=File
cimconfig -c -s traceFilePath=/tmp/cimserver.trc
cimconfig -c -s traceLevel=5
cimconfig -c -s logLevel=TRACE
cimconfig -c -s traceComponents=All
You can also use MVS console commands, as shown in Example B-2.
Example B-2 MVS commands to enable tracing
F CFZCIM,APPL=CONFIG,TRACEFACILITY=FILE:
F CFZCIM,APPL=CONFIG,TRACEFILEPATH=/tmp/cimserver.trc
F CFZCIM,APPL=CONFIG,TRACELEVEL=5
F CFZCIM,APPL=CONFIG,LOGLEVEL=TRACE
F CFZCIM,APPL=CONFIG,TRACECOMPONENTS=ALL
Trace data is written in ASCII format to the specified file, which can be viewed by using ISPF with the View ASCII (VA) option. This data also can be downloaded to a workstation in bin format and viewed with a text editor.
The trace records use the following format:
<Seconds after 1970>s-<micro seconds>us:<Component Name> [<ProcessID:ThreadID:File name: Line Number>]: <detailed information>
For more information, see the OpenPegasus Tracing User Guide.
These changes are in effect while the current CIM server is running. To make a persistent change, use the -p option instead of the -c option with cimconfig. For the MVS command, you use the PLANNED parameter.
To turn off the traces, run the commands that are shown in Example B-3 from the UNIX System Services shell. You can also save these commands in a UNIX System Services shell script for faster setup.
Example B-3 Cimconfig commands to turn off tracing
cimconfig -c -s traceLevel=1
cimconfig -c -s logLevel=INFORMATION
Leave the default for traceComponents as ALL. The MVS console commands that are used to update for the next IPL are shown in Example B-4.
Example B-4 MVS commands to turn off tracing
F CFZCIM,APPL=CONFIG,TRACELEVEL=1,PLANNED
F CFZCIM,APPL=CONFIG,LOGLEVEL=INFORMATION,PLANNED
B.2.5 OSBASE
You also might need to perform an OSBASE trace, which traces CIM instrumentation. This trace is set by the OSBASE_TRACE CIM environment variable in the /etc/wbem/cimserver.env file.
The default setting is 0 and can go up to 4, which provides the most detail. This trace is written to the location that is specified by the OSBASE_TRACE_FILE= variable.
This trace can also be changed dynamically by running the following MVS commands:
F CFZCIM,APPL=ENV,OSBASE_TRACE=4
B.2.6 Caveats
Running a trace does result in some unwanted effects. Depending on what is being traced, you might see increased processing time for the CIM server.
Along with increased processing time, data is written to files that can grow large (how large is proportionate to the type and length of tracing).
Typical trace requests are run for short periods and then reset to the default settings. Space and processor usage should not be an issue. Ensure that the traces are disabled.
B.3 Application server
The WebSphere Application Server Liberty Profile is the application server that we use in our example. Tracing can be controlled by MVS commands to the Liberty Server address space or by modifying the bootstrap.template file.
B.3.1 z/OS modify command
z/OSMF tracing can be enabled by running the following MVS MODIFY command:
f server-name,logging='trace_specification'
Where server-name is the z/OSMF server, which is IZUSVR1 by default, and trace_specification is the level of tracing to be used.
The following examples show the use of this command:
•To enable all tracing, run the following command:
f izusvr1,logging='*=warning:com.ibm.zoszmf.util.data.*=all'
•To trace security, run the following command:
F IZUSVR1,LOGGING='com.ibm.ws.security.*=all'
•To reset the trace specification, run the following command:
f izusvr1,logging='reset'
B.3.2 Bootstrap updates
Trace specifications can be set in the file /var/zosmf/configuration/local_override.cfg. The first time that you want to add or change some configuration settings, you must create this file. After the started task IZUSVR1 is restarted, the new configuration is loaded in the /var/zosmf/configuration/active_configuration.cfg file.
The following example shows the default setting:
LOGGING='*=warning:com.ibm.zoszmf.*=info:com.ibm.zoszmf.environment.ui=finer'
To enable security tracing, modify or create the local_override.cfg file, as shown in Example B-5.
Example B-5 local_override.cfg file
LOGGING='com.ibm.ws.logging.trace.specification=*=audit:com.ibm.zoszmf.*=all:com .ibm.ws.security.*=all=enabled:com.ibm.ws.security.*=all=enabled'
After the trace is captured, remove the additions because this change is persistent.
B.3.3 Tracing and logging output
Message and traces are written to the following file:
IZU_DATA_DIR/logs/zosmfServer/logs
The trace.log data set contains z/OSMF-related trace messages. The message.log contains non-z/OSMF trace messages (for WebSphere Application Liberty profile). These messages are created as a result of enabling tracing, as described in B.3, “Application server” on page 551.
B.3.4 First Failure Data Capture
WebSphere Application Server uses the First Failure Data Capture (FFDC) facility to instantly collect information about the events and errors that lead up to the failure. The captured data can then be used to analyze the problem.
After a maximum number of days, these files are automatically deleted from your system. These logs are stored in IZU_DATA_DIR/logs/zosmfServer/logs/ffdc.
FFDC files for errors are created in this directory.
B.3.5 Dumps
You can create Java dumps, memory dumps, and display the server’s status by using the MODIFY console command. The Liberty server supports requests for SVC dumps and transaction dumps by using the following commands:
•MODIFY jobname,svcdump
•MODIFY jobname,tdump
Server commands can also be used to issue dumps and check the server’s status.
When the WebSphere Application Server Liberty profile is used to create Java dumps, memory dumps, and the server's status, export the following environment variables:
•WLP_INSTALL_DIR="IZU_CODE_ROOT/wlp"
•WLP_OUTPUT_DIR="IZU_DATA_DIR/logs"
•WLP_USER_DIR="IZU_CONFIG_DIR"
Then, you can run the following UNIX System Services commands:
•IZU_CODE_ROOT/wlp/bin/server javadump zosmfServer
This command creates a snapshot of the JVM and archived output goes to IZU_DATA_DIR/logs.
•IZU_CODE_ROOT/wlp/bin/server dump zosmfServer
This command creates a snapshot of the server and archived output goes to IZU_DATA_DIR/logs.
•IZU_CODE_ROOT/wlp/bin/server status zosmfServer
This command checks if the server is running.
For more information about the WebSphere Application Server Liberty profile trace and logging functions, see the Liberty:Logging and Trace page of IBM Knowledge Center.
For more information about WebSphere Application Server Liberty profile administration through the command line, see the Administering Liberty from the command line page of IBM Knowledge Center.
B.4 z/OSMF
Do not change the default trace and log settings unless you are instructed by IBM Support to change them. Certain trace and log settings can cause performance degradation on
the system.
the system.
B.4.1 z/OSMF logs
Two types of logs are available for z/OSMF: runtime logs and configuration logs.
Runtime logs
The runtime logs contain the messages for all plug-ins. WLM Software Management, Performance Management, and so on, include the messages that are written here, unless they are written elsewhere.
z/OSMF creates the log files in the /var/zosmf/data/logs directory by default. z/OSMF names the log files IZUGn.log, where n is a number 0 - 9.
z/OSMF creates log files in a “cascading” manner. The most current log file is always named IZUG0.log. When this log file reaches the limit of 2 MB, it is saved as IZUG1.log and z/OSMF begins writing to a new IZUG0.log file.
When the IZUG0.log file is again full, z/OSMF saves it as IZUG1.log after renaming the existing IZUG1.log file to IZUG2.log. z/OSMF continues this process, saving each log file under the next available name, up to a maximum of 10 log files. After 10 log files are reached, z/OSMF discards the oldest log file whenever a new log file is created.
The z/OSMF runtime log files are written in ASCII format, and UNIX System Services tools, such as vi, emacs, and grep, display the text correctly. ISPF Browse and Edit also work (you select the Edit ASCII or View ASCII option), as shown in Figure B-2.
|
VIEW /WTSCPLX8/var/zosmf/data/logs/IZUG4.log Columns 00001 00072
Command ===> Scroll ===> CSR
****** ***************************** Top of Data ******************************
000001 2017-08-14T20:02:48.977Z|0000002C|com.ibm.zoszmf.util.data.DataRegistry|
000002 FINE:Found correct permissions and owner for file /var/zosmf/data/logs/I
000003 [tx000000000000000F:*izubootstrap*]
000004 2017-08-14T20:02:48.978Z|0000002C|com.ibm.zoszmf.util.data.DataRegistry|
000005 FINE:Analyzing file /var/zosmf/data/logs/IZUG5.log.
000006 [tx000000000000000F:*izubootstrap*]
000007 2017-08-14T20:02:48.980Z|0000002C|com.ibm.zoszmf.util.data.DataRegistry|
000008 FINE:Found correct permissions and owner for file /var/zosmf/data/logs/I
000009 [tx000000000000000F:*izubootstrap*]
000010 2017-08-14T20:02:48.980Z|0000002C|com.ibm.zoszmf.util.data.DataRegistry|
000011 FINE:Analyzing file /var/zosmf/data/logs/IZUG6.log.
000012 [tx000000000000000F:*izubootstrap*]
000013 2017-08-14T20:02:48.981Z|0000002C|com.ibm.zoszmf.util.data.DataRegistry|
000014 FINE:Found correct permissions and owner for file /var/zosmf/data/logs/I
000015 [tx000000000000000F:*izubootstrap*]
|
Figure B-2 ISPF View of a log file
If a problem occurs with the product logs directory (for example, the directory is not writable or z/OSMF encounters an error on initialization), z/OSMF writes its log data in the Application Server Servant log instead.
B.5 Common Event Adapter
Common Event Adapter (CEA) issues are diagnosed by using MVS MODIFY commands. For more information about displaying the CEA, see MVS System Command, SA38-0666.
For example, F CEA,D,P displays the active CEA settings that are based on PARMLIB, as shown in Figure B-3.
|
CEA0023I COMMON EVENT ADAPTER 805
STATUS: ACTIVE-MINIMUM CLIENTS: 0 INTERNAL: 0
CEA = (ZM)
SNAPSHOT = Y
HLQLONG = HLQ = CEA
BRANCH = 180 COUNTRYCODE = 000
CAPTURE RANGE FOR SLIP DUMPS:
LOGREC = 01:00:00 LOGRECSUMMARY= 04:00:00
OPERLOG = 01:00:00
CAPTURE RANGE FOR ABEND DUMPS:
LOGREC = 01:00:00 LOGRECSUMMARY= 04:00:00
OPERLOG = 01:00:00
CAPTURE RANGE FOR CONSOLE DUMPS:
LOGREC = 01:00:00 LOGRECSUMMARY= 04:00:00
OPERLOG = 01:00:00
SMS STORAGE CLASS = DEFAULT
TSOASMGR:
RECONSESSIONS = 0 RECONTIME = 00:00:00
MAXSESSIONS = 50 MAXSESSPERUSER= 10
|
Figure B-3 F CEA,D,P output
If you want to change CEA settings, you can update a CEAPRMxx member dynamically by running F CEA,CEA=xx, as shown in Example B-6.
Example B-6 Modify CEA
f cea,cea=00
CEA0502I CEA PARMLIB PROCESSING COMPLETE.
B.5.1 CEA tracing
CEA REXX tracing is available and should be used only under the direction of IBM Support. You can turn on this tracing facility by running F CEA,DIAG,REXXDEBUG=ON. The trace is written to a data set with the HLQ qualifier according to your PARMLIB setting. It looks similar to CEA.REXXDBG.CEACDMPC.RC6C95F7.X6F1AE7A.TC3.
B.5.2 CEA Component Trace
You can create a CEA CTRACE by completing the following steps:
1. Issue the following command on the console to see whether CTRACE is active:
D TRACE,COMP=SYSCEA
The output should look similar to the output that is shown in Example B-7.
Example B-7 Display trace output
SYSCEA ON 0016M
ASIDS *NONE*
JOBNAMES *NONE*
OPTIONS ALL
WRITER *NONE*
2. The OPTIONS setting should be set to ALL. If it is not, or if the CTRACE is not ON for SYSCEA, enable the CTRACE by running the following command:
TRACE CT,ON,COMP=SYSCEA
Then, to turn on the ALL option, run the following command:
nn,OPTIONS=(ALL),END
3. Collect the trace by taking a dump of OMVS and CEA and the OMVS data spaces by running the following commands:
DUMP COMM=('cea and omvs')
R xx,JOBNAME=(cea procname,OMVS),CONT (procname is probably CEA)
R xx,DSPNAME=('OMVS'.*),CONT
R xx,SDATA=(ALLNUC,PSA,GRSQ,SUM,CSA,LPA,LSQA,RGN,SWA,SQA,TRT),END
4. You can view the output in IPCS by starting the dump and then running the
following command:
following command:
‘IP CTRACE COMP(SYSCEA) FULL’
B.6 System REXX
As with CEA, System REXX information is best obtained through MVS MODIFY commands. These commands also are described in MVS System Command, SA38-0666. To display the general status of the server, run the F AXR,SYSREXX,STATUS command, which should produce the output that is shown in Figure B-4.
|
AXR0200I SYSREXX STATUS DISPLAY 817
SYSTEM REXX STARTED AT 18.03.19 ON 08/03/2017
PARMLIB MEMBERS: AXR00
CPF: @ (SYSTEM) AXRUSER: AXR
TIMEINT: 30 TMP: NOT ENABLED
SUBSYSTEM: AXR TSO=YES ENABLED
REQUESTS QUEUED: 0 ACCEPTING NEW WORK
REXX WORKER TASKS: ACTIVE: 0 TOTAL: 4
IDLE: 4 MAX: 32
ASYNC: 0 SYNC: 0
UNTIMED: 0
TSO SERVER SPACES: ACTIVE: 0 TOTAL: 1
IDLE: 1 MAX: 8
ASYNC: 0 SYNC: 0
UNTIMED: 0
|
Figure B-4 F AXR output
B.7 RMF
Errors that are encountered when you use the Monitoring Desktops (RMF) should be written to the SYSLOG, GPMSERVE STC, or GPM4CIM STCs. However, if not enough information is available, you can turn on more debugging options.
In the GPMSRVxx parmlib member that you use, set DEBUG_LEVEL(3) to get the most messages. You can run F GPMSERVE,OPTIONS to display the current options. You must restart GPMSERVE to display the options.
Comments in the GPMSERVE proclib member that tell you how to turn on the gpmserve trace (such as TRACEON). You must change the SYSPRINT and SYSOUT options in the GPMSERVE proc to SYSOUT=x.
To turn on the trace, run F GPMSERVE,TRACEON and then, re-create the issue. The trace output goes to the SYSOUT, as specified in the GPMSERVE proc. After the trace is complete, run F GPMSERVE,TRACEOFF to stop the trace.
For the GPM4CIM procedures, you can modify the applicable /etc/gpm/gpm4_.cfg file in the same manner as GPMSERVE. Modify commands produce similar results. For example, to turn on a trace for the AIX version of GPM4CIM, run F GPM4CIM.GPM4A,TRACEON.
..................Content has been hidden....................
You can't read the all page of ebook, please click here login for view all page.