© The Author(s), under exclusive license to APress Media, LLC, part of Springer Nature 2022
S. Banik, V. ZimmerFirmware Developmenthttps://doi.org/10.1007/978-1-4842-7974-8_4

4. System Firmware Debugging

Subrata Banik1   and Vincent Zimmer2
(1)
Bangalore, Karnataka, India
(2)
Issaquah, WA, USA
 

“Programming allows you to think about thinking, and while debugging you learn learning.”

—Nicholas Negroponte

If system firmware development is an art, then debugging that firmware is a fine art. The debugging process primarily depends on a thorough understanding of the platform capabilities (for both the SoC and board design), system architecture, and system boot state. Additionally, requires an appropriate debug methodology to deliver a solution. Embedded systems are highly customized to address the market needs. As the smartphone, tablet, personal computer, and household robot markets expand, silicon and platform designers have a challenge to provide ample hardware capabilities to debug the platform at the many different stages of the product life cycle. In addition, most embedded systems have a limited dedicated debug capability due to their small form factor (SFF) and fewer hardware revisions possible between prototype and mass production (MP).

System failures are expected at any phase of the product development cycle and even after the product launch while the device is in use. Firmware, being the closest possible entity to the underlying SoC and hardware, is responsible for diagnosing a defect and providing a solution if possible. System firmware running into those embedded platforms has its own debug architecture and methodology that needs an adaptations based on the target hardware. For example, the system firmware debug methodology will undergo changes from the early stages of product development (proto, engineering validation test [EVT]) in an open case or bench environment to the advanced/final stage (design validation test [DVT] to platform MP) in a closed case.

The system firmware needs to inherit a variety of debugging methodologies to overcome such dynamics to the platform hardware design and provide SoC architecture-agnostic solutions. This functionality may need to be implemented at the firmware architecture level to ensure easier migration of system firmware without any visible impact on platform debug capabilities.

At a high level, these debug capabilities can be divided into two categories.

Hardware-assisted debugging: The key reason to use hardware-assisted debugging methods is the nature of the defects, such as if the detects are seen early in the boot phase, where software-based debugging is not feasible, or there is a need to access a CPU register in a multithreaded environment where attempting to enable software-based debugging might result in unpredictable system behavior.

This hardware-assisted debugging can be further divided into the following subcategories:
  • Generic debugging: The kind of hardware debugger is attached neither with any SoC or CPU architecture nor with any OxM-specific hardware. These debug aids are very generic and can be applicable based on the hardware interface.

  • SoC/CPU-based debugging: The hardware debugger used in this process is very specific to the target CPU or SoC architecture. No matter which target SoC or CPU you want to debug and which debugger you choose to use, there will always be some common features that every debugger will offer. For instance, every debugger will provide ways to do the following:
    • Connect to the target hardware

    • Download the software programming on the host system

    • Allow start, stop, and step through program execution

    • Dump memory and register contents

  • ODM/OEM hardware-based debugging: This approach is independent of the target CPU or SoC architecture. This method will rely more on utilizing the common hardware capabilities that OxM hardware is proving to access the underlying hardware resource. For example, Closed Case Debug (CCD) is a widely used method on all Chrome platforms irrespective of the underlying SoC architecture like ARM, AMD, or Intel. Typically, hardware vendors use the common practice methods across CPU designs while creating such common hardware debuggers.

Software-based debugging: The idea is to be able to debug firmware without any additional cost or hardware-based tools. The most widely used debug method in the system firmware development model is to utilize the software-based methods or firmware’s own capabilities to debug the defects. Firmware development, debugging, and providing resolution toward the coding issues has always been one of the most time-consuming aspects while working in a product development cycle. Debugging is sometimes a complex affair due to multilayered communication between various boot firmware, the host CPU, and the underlying IP firmware communication in a multithreaded environment. The most common method that developers or debug engineers are using to monitor the execution flow is printf. This approach may be useful in most cases where the developer is highly aware of all layers of the system firmware stack or when the problem is known to a single code block or module in question. Keep track of all the different types of problems or unexpected dependencies with other components in the firmware stack; the system firmware should be equipped with different debug methods. This software-based firmware debugging can be further divided into subcategories as follows:
  • Traditional breakpoints

  • I/O-based checkpoints

  • Serial messages or serial buffers

  • Preboot environments

  • Runtime debugging
    • ACPI debug message or ACPI buffer

    • Windows Debugger (WinDbg)

    • GNU Debugger (GDB)

This chapter will present an in-depth overview of system firmware debug techniques on embedded systems for the x86 and ARM architectures. The topics discussed in this chapter will identify the debug methodology used for different microcontrollers as part of the hardware block. Additionally, it will present the different ways to debug coding issues and show debugging techniques for complex issues such as cache eviction, finding the local variable value in a multithreaded environment, and techniques to help bring different components of the embedded system stack to maturity.

Figure 4-1 illustrates a typical ARM-based platform block diagram with possible hardware components associated with it. The idea here is to show the different debug capabilities that each hardware block is permitting. For example, the system firmware running on the ARM CPU can use a serial UART to debug the CPU, and the embedded controller (EC) ROM can be used to debug the EC interfaces such as the battery information, sensor values, key sequence, etc. The physical security chip, TPM, can also have its own serial console to debug those private registers and have special access to the CPU and EC SPINOR as well. A hardware-based debugger can be used over Serial Wire Debug Port (SW-DP) to allow accessing CPU registers and memory blocks; similarly, a native USB debug interface can be used to route CPU, EC, and TPM debug interfaces to avoid dedicated debug interface needs on hardware. Other devices such as storage, panel, and audio codec firmware need special driver-based access methods that can be done using higher-level debuggers like the Windows Debugger (WinDbg) or special debug kernel drivers for Linux-like OSs.

A block diagram depicts starts from S W D P to A R M with the panel, audio codes, U S B on the right, memory, storage, spinor on the left, and T P M, embedded controller with keyboard divided to battery and sensors, and U A R T on the bottom. Three color blocks are labeled software-based, kernel driver, and hardware-assisted debug.

Figure 4-1

Debug view of an ARM-based platform hardware

Let’s start with a detailed analysis of each debugging method and its debug aid in this section.

Hardware-Assisted Debugging

As mentioned, the most advanced form of system firmware debugging is utilizing the hardware capability. Typically, this form of debugging is a combination of the hardware interface exposed as part of the CPU architecture and a connector on board, enabling the SoC or CPU capability using the native firmware and high-level software. In many cases, this debugging method requires a debugger and a cross-development environment. Based on the complexity of the problem, which requires a debug aid, a debug engineer might decide to use this approach over traditional ones. Every SoC vendor has their own hardware debugger and associated software that can be used while debugging the target hardware. The major consideration point in this approach is the cost: the cost involved to purchase the hardware debugger, the amount of debug signals needed to get routed on the board layout to allow hardware debugging, and the purchasing cost of a software license to allow cross-development.

In this section, we will discuss the widely used hardware-based debug aids in the system firmware development and debugging process. As explained, all the hardware-assisted debug tools can be divided into three categories.
  • Generic debugging

  • SoC/CPU architecture-specific debugging

  • OxM hardware-specific debugging

Generic Debugging

This method involves investing a higher cost into purchasing oscilloscopes like hardware equipment along with other tools. The only consideration point is that the cost involved here is a generic investment, so the same hardware can be used across different embedded systems.

Oscilloscope

A traditional misconception around debugging the embedded system is that it always relies on the hardware rework to bring out various probe points and attach them with an oscilloscope. Debugging with an oscilloscope is not a scalable solutions due to various reasons.
  • It requires dedicated rework to attach the probe points to monitor the signal. The test points might be present at different sides of the board, which makes it difficult to handle a reworked board efficiently.

  • Debugging with oscilloscope has a limited scope; hence, oscilloscope users are also equipped with other hardware instruments like the following:
    • Digital voltmeter

    • Logic analyzer

    • Protocol analyzer

  • The main purpose of the oscilloscope in early embedded system development is to discover signal anomalies. Typically, the correct expectation from hardware validation is to probe around the design and to get a sense of whether any anomalies exist.

Figure 4-2 provides a debugging scenario where oscilloscope-based debugging is useful for embedded system development cycles. Prior to communicating with onboard third-party components like TPM, a touchpad, etc., using a standard firmware routine, if hardware compliance ensures the device is meeting its power/initialization sequence guideline, would help to avoid any anomalies throughout the embedded system lifespan.

A snapshot of immediate detection of interruption while connecting to a third party. It depicts a grid with vertical and horizontal lines at the center. The threshold line labeled interrupt is getting detected immediately. On the right, mode and hold off, auto untriggered roll, normal, holdoff time, holdoff percentage of record, and so on.

Figure 4-2

Verifying signal integrity between Chip-Select (CS) and Interrupt while communicating with an I/O Device.

Benefits: Figure 4-2 is from a real-life problem that occurs early during the boot phase where an oscilloscope is used to verify if the endpoint hardware attached to the SoC is able to send the acknowledgment upon receiving the CS signal.

Protocol Analyzers

It’s the fact that Modern embedded system designs are getting complicated. Devices belonging to the advanced technology families like PCIe, USB, NVMe, SATA, I2C, and SPI are getting attached to the motherboard design (as shown in Figure 4-1). The protocol analyzers are the answer to performing the test solution for these computer buses and network communication standards. The protocol analyzer is an indispensable tool for embedded engineers.

A protocol analyzer works by capturing the data across the communication bus in the embedded system and then displaying it using GUI tools. With the help of a protocol analyzer, hardware engineers can design an embedded system, while firmware engineers can develop any new firmware module for these hardware interfaces listed earlier, and validation engineers can test the hardware product.

A protocol analyzer is a combination of dedicated hardware and software tools. The hardware captures the data, and the software displays the captured data. The hardware block typically needs three-way communication such as the following:
  • Connecting to DUT: An interface that is attached to the device under test to capture the data.

  • Connecting to HOST: To show the captured data in real time, it needs another interface to the HOST CPU. Engineers are using the HOST machine to detect the anomalies if any are in the bus communication.

  • Input interface: This is an optional interface to attach the external devices like USB or Ethernet, which emulate attaching the device directly to the DUT.

Let’s take a look at some popular protocol analyzers used during the system firmware development and debug phase:
  • USB protocol analyzer: The most commonly used communication protocol in the computer machines is the USB protocol. The USB protocol analyzer is also referred to as a USB sniffer; it is a connection between the host computer and the DUT to capture and decode raw bus data and event information in a human-readable format. This information is useful to identify the bus errors.

  • I2C protocol analyzer: In embedded system design, there are more devices that are getting attached with the I2C protocol due to its low power and simple bus communication. The I2C protocol analyzer can be used to debug any communication issue where the slave device address is not known or sees a timeout-related error.

  • SPI protocol analyzer: This is another commonly used hardware interface in embedded systems where the SPI protocol analyzer can be used to connect multiple slave devices.

  • PCIe protocol Analyzer: A PCIe bus is the default de facto industry standard for any high-speed communication between the CPU and motherboard component. Each PCIe specification has its own criteria in terms of speed, operating voltage, etc., to meet the PCIe compliance test for certification. This tool is used to monitor and interpret data transferred over a PCIe bus and generate error reports.

Benefits: The protocol analyzer provides output in the form of a report that covers the error type, recommendations, and directly captured data format. Traditional test tools relied on oscilloscopes for doing such compliance tests, which need manual effort and are eventually time-consuming. To reduce the product development cost and to meet a faster time to market (TTM), a protocol analyzer is the only logical solution. See Figure 4-3.

A window screen depicts the menu bar, toolbar, and two panels. The left panel is labeled U S B device view, and the right panel depicts S e q, type, request, I O, En dots, device object, I R P, status, buffer snippet, and size. Below are two panels. 1, additional information labeled U S B description. 2, buffer label decode of data packets.

Figure 4-3

USB protocol analyzer to trace USB bulk transfer

SoC-Specific Debugging

For embedded system developers and security researchers, a SoC/CPU-based debugger is the minimal requirement while debugging. This method is capable of providing access to microprocessor registers and system memory while the system is operational. There are different SoC debug interfaces being explored to allow debugging on different hardware phases efficiently. For example, during the early development cycle, because the hardware is bare metal, the platform remains open chassis, which makes debugging comparatively easier. A more sophisticated form factor design at the later phase of the product cycle makes the debugging harder. Hence, SoC vendors are trying to improve the debuggability and validate and test the platform’s scope even for the product at the final MP stage efficiently.

This section will provide an idea about all the possible hardware debuggers and their control methods across popular CPU architectures.

This debugging process on embedded systems is referred to as cross-debugging. Cross-debugging in an embedded system is a development model that involves two different computing machines working together. The target hardware, which is intended to get debugged, is not supposed to have any debug tools installed on it. For this reason, in the cross-debugging model, the development host system is where all the required debug tools are installed. The software running on a host development system that provides access to the target hardware using the standard user interface might belong to a different architecture. In a nutshell, the debugger is a combination of hardware and software tools that work together to provide a user interface. It lets developers harness the benefits of the underlying CPU, GPU, and/or APU with a single program.

Hardware Interface

In a cross-debugging model, a debug communication channel between the host and target needs to be established on the target hardware device. Figure 4-4 shows a typical debug setup between the host and target hardware.

An image of a hardware set-up. The custom H W is connected to the host system by the U S B, and H W from the other end is connected to the D U T through debug ports.

Figure 4-4

Typical hardware-based debug setup

The debug port is the interface between the host and the DUT debug access port. For embedded systems, the de facto standard for hardware-based debugging and accessing the hardware registers is the Joint Test Action Group (JTAG). The IEEE 549.1 standard defines a “Standard Test Access Port and Boundary-Scan Architecture” for Test Access Port (TAP) used for testing printed circuit boards (PCBs). This standard is commonly referred to as the JTAG debug interface. Since its origin, the JTAG debug interface has become a widely used interface for debugging the system firmware. Figure 4-5 shows a JTAG debug probe connected to a CPU. The JTAG interface allows access to the various systems on chip (SoC) test access ports (TAPs) like CPU and PCH. The JTAG protocol provides a serial interface to add to a chip device. The host system running the debug tools can use the serial link to reach those TAPs to access memory and registers that are running on the target hardware chip logic.

Today most of the embedded devices are equipped with a JTAG port to support early hardware debugging and firmware development. The JTAG-based debug port doesn’t require any special firmware programming to access the TAPs; hence, this mechanism can be used while debugging early CPU reset issues and early platform boot stages like SEC and PEI for UEFI and bootblock and romstage in coreboot.

A photograph of a debug port with a circuit board connects through wires.

Figure 4-5

Debugger attached to the JTAG debug port

Depending on the specification of JTAG, typically this interface supports four PINs as follows:
  • TMS: Test Mode Select

  • TCK: Test Clock

  • TDI: Test Data In

  • TDO: Test Data Out

In modern embedded systems, every device is equipped with multicore CPU architecture. Hardware-based debugging is absolutely necessary to debug CPU features such as SGX, VMX, etc. In such a multicore boot environment, hardware debuggers should be capable of halting all possible cores using a single JTAG scan chain. Figure 4-6 shows a daisy-chained technique that CPU designers can use where the output of the one core is acting as the input to the next core.

A circuit diagram depicts T M S, T C K, and T D I connects to three boxes labeled T M S, T C K, T D I, Core 1, and T D O. Each box is labeled core 1, 2, and 3. The results to T D O.

Figure 4-6

Daisy-chained JTAG interface for multicore CPU architecture

Different SoC vendors are utilizing this JTAG interface to create their own hardware debug tool.
  • The Intel architecture has with different hardware debuggers for open-case and closed-case debugging using the JTAG interface.

  • ARM architecture processors come with JTAG support as well. Sometimes it supports another debug interface with a lower PIN count such as serial wire debug (SWD).

eXtended Debug Port
The traditional way to do hardware-assisted debugging on Intel platforms is to use a proprietary 60-pin connector known as eXtended Debug Port (XDP), an extension of the JTAG specification. XDP communications are based on the physical connectivity assumption that the host machine running the debugging tools is a closed case but the target hardware under test is an open case. The XDP pod sits between those two layers, as shown in Figure 4-7. The host machine running the debug tools is connected via a USB interface. The debugging tool workflow passes through the proprietary USB protocol to the XDP pod, where the XDP pod is designed to translate the host tool workflow into JTAG probe mode. The DUT side of the pod is directly connected to a specific debug port on the motherboard. The debug port has access to all TAPs that are available in the SoC, CPU, and PCH. This method of hardware debugging is expected to expose more debug signals on the motherboard or silicon products; hence, these XDP transports are primarily used in open cases or during early product development.

A set of two flow charts open and close case debugging. 1, HOST machine with debugging tools to X D P, then D U T, S O C, P C H with taps and C P U, and C P U with C P U TAP. 2, HOST machine with debugging tools to D C I, then D U T, S O C, P C H with E x l bridge and taps, and C P U with C P U tap.

Figure 4-7

JTAG-based open and closed case debugging

Direct Connect Interface

Over time, more sophisticated devices and smaller form factors have challenged the SoC side to have a simpler debug hardware interface with the same capabilities. Also, open-case debugging may not always be the scalable solution for the product development life cycle. Direct Connect Interface (DCI) is the solution for such problems where the assumption is that both sides of the debugger are now enclosed systems. On the host side, it still uses the same connection as XDP, but on the DUT side, it connects to the JTAG interface using a new transport layer named Embedded DFx Interface (Exl). Exl works as a bridge behind the USB controller, which is responsible for passing the debugging tool workflow to the target via the proprietary USB protocol. In this mode of debugging, the control and data pass through the Exl bridge to gain access to JTAG and probe mode.

The primary goal here is to allow debugging closed-case OxM platforms like sealed tablets, smartphones, laptops, etc., where debuggers don’t need to access to the XDP header on the motherboard. Figure 4-7 provides the high-level architectural difference in closed-case compared to open-case debugging.

Serial Wire Debug
On embedded systems the JTAG interface is the default standard for attaching debuggers. The major drawback in this protocol is the higher number of signals, which may not be possible for smaller and compact form-factor hardware. To solve this problem on microcontrollers with low pin counts, an alternative debug interface was created known as Serial Wire Debug (SWD). SWD uses only two wires, a clock wire and a data wire. The connector pins are as follows:
  • SWDCLK: Serial Wire Debug Clock signal sent by the host.

  • SWDIO: Serial Wire Debug Input Output is a bidirectional signal used to carry data between the host and debug port. The data sent by the host is getting sampled at the rising edge and sampled by the debug port (DP) during the falling edge of the SWDCLK signal.

Figure 4-8 provides an architecture overview of the SWD interface and access mechanism.

A block diagram flow from left to right, S W D, D A P is divided into two S W D P, J T A G D P. It flows to Access ports and is divided into cores 1, 2, and 3.

Figure 4-8

SWD architectural overview in multicore CPU environment

Unlike the JTAG interface, which uses a daisy-chain topology to connect multiple debug components, the SWD interface uses a bus called Debug Access Port (DAP). The external debug interface connects to the DAP through a DP. There are three different debug ports available to access the DAP.
  • JTAG Debug Port (JTAG-DP): This is similar to the JTAG interface and protocol to access the access ports (APs).

  • Serial Wire Debug Port (SW-DP): This uses the SWD protocol to access the AP.

  • Serial Wire/JTAG Debug Port (SWJ-DP): This allows external debuggers to attach to SoC using either JTAG or SWD DP. It provides a mechanism to select between the JTAG and SWD interfaces. It allows an easy migration between the JTAG and SWD interfaces where SWDIO and SWCLK can be overlaid on the JTAG TMS and TCK pins.

Multiple access ports can be attached to the DAP. This APs can further access different debug components, for an example:
  • JTAG access port (JTAG-AP): This allows access to JTAG equipped cores.

  • Memory access port (MEM-AP): This provides access to system memory, bus-based debugging, and device registers such as AMBA Advanced High-performance Bus Access Port (AHB-AP) or AMBA AXI Access Port (AXI-AP) or AMBA Advanced Peripheral Bus Access Port (APB-AP)

Software Interface

After understanding the different hardware interfaces to connect hardware debuggers between the host system and target device for debugging, it’s time to take a look at different debugging tools provided by various CPU vendors while accessing the debug ports to get into system memory or registers.

The first step in this cross-debugging setup is to have the required debugging tools installed on the host machine. Typically, every SoC vendor provides the flexibility of installing the debugging tools on all leading operating systems. Debugging the system firmware requires access to various different debug components like CPU registers, device registers, system memory and local variables, etc. A cross-debugging session with an integrated development environment (IDE) would make it very simple.

On AMD platforms the debugging is done through CodeXL, whereas on the Intel architecture, it’s the Intel System Debugger, and on the ARM-based platform, the ARM debugger as part of ARM Development Studio is used for debugging the embedded systems.

CodeXL

CodeXL is the comprehensive tool suite used on AMD-based platforms to access the CPU, GPU, and APUs with a single program. It includes powerful GPU/CPU/APU debugging and CPU and GPU profiling as well. It works as a stand-alone application on both Windows and Linux OS.

After downloading the installer package on the host system and installing the package, developers can start using CodeX. The CodeXL GUI window should appear as Figure 4-9 with debug explorer view notes.

A window titled no project loaded with code X L debug mode. It depicts a menu bar and a toolbar. Three panels are exhibited. The left panel depicts code X L explorer and output. Center panel titled welcome page with debug, profile, build and analyze, recent, and samples. The right panels exhibit memory, and no data is available.

Figure 4-9

CodeXL debug mode: no project loaded

The CodeXL debugger will allow developers to access the runtime behavior of the target hardware based on the control programming buttons while debugging.

A window displays a bug icon, timer, graph with search icon, pause button labeled host plus G P U debug, stop button, square shape button, A P I. Below that, Code X L explorer and start debugging F 5 is labeled.

These controls are as follows (left to right): Debug Mode, Switch to Profile Mode for GPU, Analyze Mode, Start, Pause and Stop Debugging, Step In, Step Over, and Step Out.

In order to perform source code based debugging, you need to connect the target device and load the debug symbols to map the program running in the target device memory to its original source file. Figure 4-10 shows the source code view after starting the debugger program as described earlier and then hit the Break button to interrupt it program execution to inspect the current execution state (i.e., memory view, register view etc.).

An image depicts a window labeled as CodeXL Teapot with a menu bar and toolbar. It has two panels named as VirtioMmioDevice.c and memory. VirtioMmioDevice.c has programming of 32 lines, and the memory panel has graphic object details. No data is available there below the memory panel. The window has four sub-panels at the bottom.

Figure 4-10

CodeXL source code view

Intel System Debugger

The Intel System Debugger is the GUI-based system software debugger to allow access to the system state, processor registers, platform device registers, and system memory via a JTAG-based hardware interface. The debugger GUI provides complete control over the debugging process by allowing the basic functions such as stepping in, stepping through, and displaying memory by clicking the menu toolbar button. The GUI also supports source code debugging after loading the symbols files of the same program running into the target hardware memory.

Figure 4-11 shows the options to connect the target hardware debugger after loading the Intel System Debugger. After successfully connecting with the target hardware, the debugger command will modify from xdb_D> to xdb_R> unless the developer uses the following control programmer buttons to pause the execution on the target system:

An image illustrates the tool bar with connect icon, disconnect, loading option with rectangular box, right arrow, four vertical dots, left arrow, four vertical dots, rightwards triangle, two vertical bar, arrow inside the parentheses, arrow outside the parentheses, arrow goes out from parentheses, arrow inside the parentheses with subscript i.

These controls are as follows (left to right): Connect, Disconnect, Load/Unload the debug symbols, Reset the target system, Start, Pause and Stop debugging, Step In, Step Over, and Step Out.

A window titled intel R debugger legacy. It depicts a menu bar and toolbar. On the center, the dialog box titled connect to target with the hardware probe and target platform. On the bottom, debugger commands are depicted.

Figure 4-11

Intel System Debugger: connecting the debugger

The Intel System Debugger also allows source code–level debugging for any bootloader, even coreboot, which is an open source firmware project. To start debugging coreboot with the Intel System Debugger, developers need to load the symbol files manually. Figure 4-12 shows the default loading process by selecting File ➤ Load/Unload Symbol File after halting the target.

A window titled intel R debugger legacy. It depicts a menu bar and toolbar. On the center, the dialog box titled load and unload symbol file, load symbol file is enabled. It exhibits the symbol file and offset. On the bottom, the console view with debugger commands is depicted.

Figure 4-12

Intel System Debugger: loading the symbol files

After loading the necessary .debug files, you can start debugging coreboot. Figure 4-13 shows how to debug coreboot using the Intel System Debugger where developers can make use of the source code viewer to view the assembly code, CPU registers, local variables, memory dump, and access to the global descriptor table (GDT). Developers can make use of the debug console or directly override the registers or memory values.

A window depicts a menu bar, toolbar, and three panels. On the left, it depicts the trial, address, opcode, and sources. On the right, it is titled registers and illustrates register and value. The bottom depicts console view, locals, memory, processor-specific register, and G D T.

Figure 4-13

Intel System Debugger: source code debugging

Arm Debugger
Arm Debugger is capable of providing a GUI-based environment that allows users to debug the complex SoC bringing-up scenarios and debug multicore environments like symmetric (SMP) and asymmetric (AMP) and also heterogeneous systems. Figure 4-14 provides a high-level overview of ARM DS-5, a powerful development toolkit with an IDE for ARM-based processors, an ARM compiler, support for a simulation model for software development without the target hardware, streamlined tools for analyzing software performance, JTAG debug, and trace support.

A block diagram depicts the different shapes of blocks. It labeled compiler, eclipse, I D E, debugger, streamline, device configuration database, simulation, and hardware debug.

Figure 4-14

ARM-DS5: development studio

All these debugging tools are the same in terms of the underlying capabilities such as the register access, memory access, etc. Here is a list of common semantics used by different debuggers irrespective of CPU architecture:

Debugger Features

Description

Connection type

All debuggers will give you the option to connect to the target using different methods, for example single-core and multicore access using the JTAG daisy-chain method and SWD star topology. The idea is to halt all available cores using a single command.

Loading program

Most debugging sessions are focused on debugging some type of program. This process might involve loading a program into the target device, or the target is already loaded with a program while debugging connects or loads the same program on the host side to allow source code debugging. In most system firmware debugging processes, the SPINOR already has the program preloaded, and in the debugging process either the SPINOR is mapped into system memory or the system firmware is responsible for copying them into memory.

Reset target

Being able to reset the target hardware is the minimum expectation from a hardware debugger. Allowing the target to reset will help to restore the target to a known working state. The CPU architecture-specific part is the reset mechanism, which is different across different SoCs.

Run control

Most debuggers provide run control options such as start, pause or stop, step in, step through, and step out. These options will impact the state of system registers, memory, local variables, etc.

Breakpoints

Almost all debuggers are capable of setting breakpoints. There are two types of breakpoints that developers can use while debugging.

Hardware breakpoints: Special hardware registers are used to create logic that halt the CPU execution.

Software breakpoints: It’s much easier to create software breakpoint by adding an assembly instruction.

Implementing breakpoints on embedded systems is CPU architecture specific.

Watcher

Also referred to as watchpoints, this is a feature that many debuggers provide to set a watchpoint on a particular memory value or I/O port value. Execution on the target system will autobreak upon hitting the watchpoint either on memory or on I/O watched addresses.

Semihosting

Some custom debuggers provide this option where the target hardware can make use of I/O facilities on the host machine. For example, a program running on the target machine will use the host system console out to redirect the console message. This feature is useful while doing remote debugging where all required outputs are coming into single units applicable on the host system.

Registers, memory (system memory), special bus like PCI, AMBA, etc.

These allow access toward all possible CPUs, PCH registers, system memory, and special bus architecture. Depending on access points, the debugger will allow you to view and modify the system memory and registers.

Low-power mode debugging

Debuggers are equipped with a special mode when the OS has put the system into lower power mode and all cores are in power-down mode. The other low-power, always-on microcontroller APs can be used to monitor limited device registers without impacting the device’s operational state.

OxM-Secific Debugging

The major drawbacks in SoC-specific debugging are the cost of hardware debuggers, usage of proprietary software tools (in many cases, available only under nondisclosure agreements), and lack of applicability of these in cross-architecture debugging. To solve this problem, many ODM/OEMs have come up with more generic approaches that can be used for hardware-based debugging even on cross-architecture platforms. This section provides an overview of a few low-cost, handheld debug tools.

AMIDebug Rx

System firmware developers have been relying on checkpoint cards to debug early boot stages where the serial console is not available. This debug method is tightly coupled with open-case debugging where a PCI-based card is attached to the motherboard. AMIDebug Rx is designed as a replacement for the PCI port 80 POST checkout card and makes port 80–based debugging a scalable solution on closed-case devices as well.

AMIDebug Rx is built around the debug port feature on USB 2.0 EHCI controllers. To enable this mode of debugging, system firmware is needed to program USB 2.0 controller PCI configuration space and implement base address register (BAR) address space for communication. Typically, system firmware has a native USB 2.0 debug driver that uses the “USB debug port” to transmit the checkpoint data on the device.

XHCI Debug Capability

The XHCI debug capability (Dbc) is an open specification part of the XHCI host controller that allows low-level system firmware debugging over USB without any additional cost. Figure 4-15 shows the Dbc interface connecting two systems; one system is the debug host and another is the DUT as the debug target. After the Dbc is initialized, it will present the device target as a debug device through a debug USB port. This method can be useful to replace the proprietary UART implementation on different motherboard designs.

A diagram depicts a computer labeled host system titled debug host connects laptop labeled D U T titled debug target with D b c.

Figure 4-15

Dbc connection between debug host and target

Figure 4-16 shows an example of the Dbc software architecture, which is completely independent of the XHCI interface that is typically developed by system software for other USB device-class communication.

A set of two block diagrams. Set 1 labeled debug host, U S B debug applications flow to U S B debug class driver, bus driver, x H C I HOST controller driver, then x H C I P1, P2, and P3. Set 2 labeled debug targets. System debugs flow to debug capability driver, then x H C I P1, P2. P3 of debug host is linked to P1 of debug target.

Figure 4-16

Dbc debug software stack

The USB debug application is running as part of the debug host. The debug host provides a USB debug capability class driver that will communicate with the device target after the debug device is enumerated. At a high level, the debug device can expose all its debugging capabilities as part of the debug driver. The debug capability driver is expected to be loaded immediately after power-on to let the system firmware debug process use this method.

Closed Case Debug

Legacy Chrome OS devices were using a custom debug header known as Servo to access the CPU and EC serial console, SPINOR, etc., in a generic way across cross-architecture platforms. Newer Chrome OS devices have introduced a multipurpose secure microcontroller, referred to as H1 and running an embedded OS called Cr50. The debugging method using Cr50 is called closed-case debug, which replaces the need to have a dedicated servo header to allow access to the CPU, EC UART, and SPI interface on the device under test.

The Chrome OS devices and H1 microcontroller are communicating using a custom USB Type-C cable called SuzyQ. The debug architecture has been built around the USB Type-C specification. Figure 4-17 shows the debugger architecture and communication flow.

A diagram depicts a rectangular box filled with rows of eleven boxes. Below that, on the left, two rectangles labeled C P U U A R T, and E C U A R T connect to H 1 with AP U A R T, E C U A R T, S P S, 1 2 C, and others. And it connects to U S B dash C, then a vertical arrow points to the upper rectangular box.

Figure 4-17

CCD accessing the CPU and EC UART

To put the Cr50 into the debug mode, the SuzyQ cable needs to connect to Chrome OS devices, and users need to specify the physical presence. The H1 includes two pins that can detect the debug accessory signature on the CC1 and CC2 pins. After detection, Cr50 enables a USB full-speed USB 2.0 slave interface that connects to the SBU pins on the USB-C connector. Cr50 makes several USB endpoints available to the host to communicate with the consoles. For example, Figure 4-17 shows the access of the CPU and EC UART. In addition, Cr50 also allows access to the H1 console.

Software-Assisted Debugging

The most cost-effective debug method in the system firmware development model is utilizing the software and firmware’s own capabilities without being dependent on hardware debuggers. Adapting hardware debuggers has its own difficulties during later stages of SoC and/or product development where many CPU interfaces are required to be disabled by default; hence, it needs a special firmware image to enable all the required debug interfaces. In many cases, the timing-related issues are not possible to replicate with a hardware debugger attached due to an induced delay in the debug workflow between the host to the target access points. For such reasons, firmware developers need to rely on their traditional debugging techniques and skill sets to identify a defect and provide a solution.

This section will highlight a few known good debugging methods that developers are using on embedded systems and that are even applicable across architectures.

Traditional Breakpoint

Traditional breakpoints are the most common debugging technique being used developers to track the code flow. While debugging the unknown defects, every developer is trying to get ahold of the code flow without adding any new piece of code. There are different ways to introduce breakpoints.

Usage

Method to Apply a Breakpoint

In assembly files

The most difficult code block in system firmware to debug is the assembly instructions. The most common usage of the breakpoint in assembly code is jmp.

The syntax for JMP is JMP <label>. The level specifies an address to which the code will jump upon execution. In this case, . specifies the current address; hence, this special symbol works as an infinite loop unless the developer overrides the program counter (PC) or instruction pointer (IP).

In C Files

It’s much easier to apply the breakpoint in a C-based programming file. Developers can either choose to generate a break on the CPU or execute an infinite loop.

void CpuBreakpoint (void)

{

  __asm__ __volatile__ ("int $3");

}

void CpuDeadLoop (void)

{

  volatile uint32_t  Index;

  for (Index = 0; Index == 0;);

}

A debugger may be used to skip past the loop and continue the code execution if needed.

I/O-Based Checkpoint

Another traditional and popular debug technique for system firmware debugging without impacting much of the program execution flow is POST codes, also known as progress codes. The I/O ports on the X86 platform, 0x80 and 0x81, are used for such debugging while checkpoint cards can be either a PCI add-in card or an onboard LED display or rely on EC to sample the POST codes at regular intervals. Every BIOS vendor has their own predefined error codes or POST code implementation to identify the underlying system firmware block if executed.

Usage

Method to Apply an I/O Checkpoint

In Assembly, C files

Implementing the I/O checkpoints depends on the underlying assembly instructions due to an out operation to write into the legacy port. Put any byte value intended to write into CONFIG_POST_IO_PORT(Port 0x80) into AL.

movb    $value, %al;

outb    %al, $CONFIG_POST_IO_PORT

In ASL files

This method helps debug runtime communication issues between the OS and firmware layer. For example: performing sleep state transitions (S3 to S0 and vice-versa). While debugging, ACPI source code (ASL) writes into the debug port (address 80h) using ACPI operating region. Here is the code snippet to illustrate this operation.

OperationRegion(PRT0,SystemIO,0x80,1)

Field(PRT0,WordAcc,Lock,Preserve)

{

  P80B, 8

}

Method(D80H,1,Serialized)

{

  If(LEqual(Arg0,0))

  {

    Store(Arg0, P80T)    // Write into the Port 80h

  }

}

For hybrid system firmware development models with integration of closed-source binary blobs (FSP), it’s important to clearly understand the postcode debugging model for FSP.

FSP outputs 16-bit postcode to indicate which API and inside that which module is getting executed. The postcode is in the following format:

Bit Range

Description

Bit 15 : Bit 12 (X)

Indicate the phase/API under which the code is executing

Bit 5 : Bit 18 (Y)

Indicate the module

Bit 7 (ZZ bit 7)

Reserved for error

Bit 6 : Bit 0 (ZZ)

Individual codes

Figure 4-18 represents the 16-bit postcode usage model in FSP.

A block diagram depicts a rectangular box divided into three X, Y, and Z Z. X connects F S P A P I of 4 BITS. Y connects Module A P I of 4 BITS. ZZ connects individual codes.

Figure 4-18

FSP Postcode block diagram

Serial Message or Serial Buffer

The most trusted debugging method on embedded systems is using printf and redirecting the output over the serial UART of the processor. Based on the motherboard designer, there could be different hardware interfaces that the system firmware needs to provide while making use of serial UART. Here are some examples:
  • Legacy CPU UART ports: On an X86 platform, this makes use of 0x3F8 or 0x2F8 legacy ports to get serial consoles. System firmware (in this case coreboot) uses CONFIG_DRIVERS_UART_8250IO and 8-bit I/O-based serial drivers to set up the console and read and write operations.

  • PCH UART ports: Another alternative for serial console debugging is using MMIO-based UART controllers on modern chipsets. System firmware (in this case coreboot) uses CONFIG_DRIVERS_UART_8250MEM_32, which is 32-bit MMIO access for setting up the console for debugging.

  • UART over SuperI/O: Many motherboards designed with SuperI/O controllers also provide access to serial ports for debugging. The developer needs to select the applicable driver for the SuperI/O controller. For example, the delta lake OCP platform is using Aspeed SuperI/O and hence enabling the corresponding driver by selecting CONFIG_SUPERIO_ASPEED_AST2400 Kconfig (for coreboot as the underlying boot firmware).

While developers are comfortable using the serial console for debugging and redirecting various messages to the serial console using printf, they don’t even realize the problem that this debugging technique often causes. Many times developers complain that the issue is seen with only the release build but not with the debug binary. They are ignorant about an unseen delay due to redirecting those debug message characters into a serial port using native serial UART driver implementation. In a single-threaded environment, calling a serial write method would eventually pause the execution until every intended debug character is successfully transmitted to the serial console. As all microcontrollers inside the SoC are running their own firmware, such a delay might be useful for restoring their state into a good working condition compared to a release binary, without any delay in execution.

The best method to overcome this problem is to rely on a serial buffer rather than a serial message. System firmware would create a reserved memory during POST, and a native serial write implementation would write into that memory rather than writing to the serial port. One would argue that it is a waste of system resources if the system hangs prior to fetching this serial buffer. The counter argument is that using some basic hardware debugger functionality to retrieve this memory is the best solution to reduce the gap between the debug and release system firmware binaries while debugging a defect.

Preboot Environment

All modern system firmware is equipped with a basic preboot environment to provide minimum access to CPU registers, system memory, and chipset registers prior to booting to the OS such as EFI Shell, Embedded Boot Loader (EBL), u-root console, depthcharge console, etc. The preboot environment serves as an important debug vehicle to narrow down an issue between the system firmware and OS. It provides open access to all chipset registers without connecting any hardware debugger prior to boot to OS; hence, developers can dump the required registers in the firmware space to ensure the recommended hardware registers have been programmed correctly by system firmware drivers.

ACPI Debug

The majority of modern operating systems are adhering to the Advanced Configuration and Power Interface (ACPI) specification. This forces the adaptation responsibility on the underlying system firmware as well. The ACPI specification has introduced a whole new language called the ACPI Source Language (ASL) to implement the required communication in the firmware layer. Debugging this layer is more challenging because of its usage model. For example, the ASL implementation doesn’t conform with native C-based serial libraries that typically get used across firmware debugging. The communication is at runtime level and hence can’t make use of boot services data or protocols. Developers need various different techniques to debug ACPI source code.

Method

Details

Serial console

The system firmware needs to implement a whole new serial I/O function in ASL so that the runtime firmware drivers can make use of it while debugging. coreboot has a unique implementation called APRT and ADBG in UEFI to redirect any debug string to serial.

Serial buffer

This method overcomes the limitation of the serial console implementation due to its serialized implementation and might cause anomalies while loading the kernel driver. The system firmware reserves runtime memory so that both system firmware and OS ACPI driver can access it.

Debug using the kernel

The ACPI Component Architecture (CA), Linux ACPI core, and ACPI drivers can combine to generate the debug output. Kernel developers can enable the CONFIG_ACPI_DEBUG Kconfig to start getting more advanced options for acpi debug.

For example, use the following command-line argument to enable all acpi debug output:

acpi.debug_layer=0xffffffff acpi.debug_level=0x2

Windows Debugger

Windows Debugger (WinDbg) is a free, powerful debugger available as part of Windows OS to allow debugging the kernel, user mode drivers, and applications. It also provides the provision to load and analyze the crash dump. WinDbg is a host-based software debugger and is capable of working in different modes based on developers’ needs. In stand-alone mode, WinDbg can be used to debug an application or kernel driver after loading either the executable or the driver source files.

Figure 4-19 shows an example of how to use WinDbg to debug an application after importing the debug symbols (PDB) from the Microsoft debug server. WinDbg is also a useful tool to debug the DUT using USB, a network, or a serial interface. This method is used frequently to debug early kernel hang or blue screen of death (BSOD), a normal phenomenon on Windows devices. Figure 4-20 shows the required configuration changes both at the DUT and host sides to enable kernel debugging over USB.

A program coding page depicts Forty-seven lines of program codes. The first line reads Microsoft of R windows debugger version 10 dot 0 dot 10586 dot 567 X 86. The second line reads copyright of c Microsoft corporation dot all rights reserved.

Figure 4-19

Debugging Windows applications using WinDbg

Follow these steps on the DUT side to configure for debugging:
  1. 1.

    Press Windows Key+R to open the Run box.

     
  2. 2.

    Type msconfig to enable the debug configuration.

     
  3. 3.

    Select the Boot tab and then select “Advanced options.”

     
  4. 4.

    Select the Debug check box.

     
  5. 5.

    Select the Debug Port options, either COM and USB. In the case of a USB interface, specify the USB target name. See Figure 4-20.

     

An image depicts two sets of panels Device target and the Host system. The device target exhibits two dialog boxes. Box one depicts boot is enabled. Box two is titled Boot advanced options. Host system titled winDbg 10 dot 0 dot 10586 dot 567 x 86. It depicts a kernel debugging dialog box and exhibits U S B is enabled.

Figure 4-20

Kernel debug mode using WinDbg

Follow these steps on the Host side to configure for debugging:
  1. 1.

    Open WinDgb.exe.

     
  2. 2.

    Go to the File menu and select Kernel Debugging.

     
  3. 3.

    Specify the same hardware interface and name if the target debug port is USB.

     

Now reboot the DUT to start kernel debugging; developers will be able to see the kernel modules are getting loaded using the host system debug console.

GNU Debugger

The GNU Project Debugger (GDB) is a command-line debug tool that runs on Linux and other Unix-like embedded operating systems. Figure 4-21 shows a GDB session.

An image depicts seventeen command lines. It starts with g d b help, a List of classes of commands. Command 1, aliases dash aliases of other commands. 2, breakpoints making program stop at certain points. 3, data examining the data and so on.

Figure 4-21

GDB: command-line interface

GDB offers an extensive set of features including debugging other software or executable files, accessing the register contents to help optimize the program, inspecting the usage of variables and altering those values at runtime, stopping the program execution and performing step operations, adding breakpoints, analyzing the crash, and allowing remote debugging between DUT and host systems using either a network or a serial interface. Independent system firmware modules are being developed using Assembly, C, Go, and Rust-based languages and can make most use of GDB during the early development stages to optimize the development cost and increase the product quality.

Summary

This chapter provided detailed information about the popular hardware and software debuggers and their usage on cross-architecture platforms while debugging a defect. It also provided some guidelines and tips on how to identify a problem and choose the right debugging method between hardware-assist and software-assist after considering the cost factor, the stage of product, and the criticality of the defect. Based on my experience, during critical debugging, debug engineers spend the most time trying to figure out the best method to attack the problem.

This chapter was a good starting point for developers and embedded system engineers to understand the underlying architecture of various hardware debuggers like JTAG and SWD and their interfaces across different CPU architectures. This information might be useful while migrating the system firmware development across various SoC architectures.

..................Content has been hidden....................

You can't read the all page of ebook, please click here login for view all page.
Reset