Throughput

Speed is essential. With 10 or 100 or 1,000 boxes the speed of testing, configuring, and provisioning each box is not a significant factor. But with 1,000,000?

Take a single hypothetical assembly line that runs for 24 hours a day where each box takes only 5 minutes to manufacture. One million boxes means 3,472 days or slightly over 9.5 years. Each additional second for a single box adds almost 12 days to the length of the production run. Consider that even with 100 such manufacturing lines, it is still over one month of non-stop, around-the-clock, no-down-time factory effort.

There are no more dreaded words than “line down.” For a firmware engineer, a line down issue means travel to the factory floor on the next available flight, no sleep, and hourly executive management status reports. Why? Because any delay means idle people and production equipment, delayed product launches, delayed delivery schedules, and delayed sales.

For many companies, the manufacturing environment is still Microsoft’s MSDOS, or one of the various alternatives (PC-DOS, DR-DOS). Companies (and engineers!) are risk-averse: they don’t like to change what has been working. Considerable expertise has been developed over time. Special-purpose tools have been created for a company’s specific needs. Manufacturing-line servers that manage the whole process often have a fragile relationship with those tools. So, why break what works?

The trouble is that Microsoft hasn’t updated MS-DOS in over 20 years (version 6.22, 1994), and it and the related tool chains are moribund. Many times it is hard to even find them, and precious floppy disks are hoarded. Even more significant: it is hard to find engineers who understand the unique APIs and batch file syntax of DOS and the quirks of the 16-bit C compiler included with Visual Studio 1.52 (we are now at Visual Studio 14.0!). Python? Forget about it! More than 4GB (much less 640KB)? Unlikely!

Many alternatives have been proposed, from Windows PE to various versions of Linux, but the substantial hardware requirements for these environments makes it likely that the environment itself will be configured improperly.

The UEFI Shell provides a unique environment for testing the boxes at an early stage because:

The UEFI Shell provides a complete, Posix-compliant C library and supports dozens of toolchains, including every major version of Microsoft’s Visual Studio and GCC. This allows manufacturing tools to be migrated easily from the current manufacturing environment to the UEFI Shell.

Also, the UEFI Shell supports multiple file systems, including FAT12/16/32 and El Torito (used on optical media), and there are multiple sources for UEFI support of NTFS, EX4 and more.

Finally, the UEFI Shell shares many development principles with UEFI itself, which allows engineers who are an expert in one to also develop code for the other.

Manufacturing Test Tools

Testing plays a big part in the length of time that a box stays on the manufacturing line. The raw box may have defects. These defects may appear consistently or sporadically. The earlier in the manufacturing process these defects are detected, the less time is wasted on that box.

Some test failures only occur on some boxes. The testing phase also detects patterns among failing boxes, such as shared faulty components, shared malfunctioning assembly equipment, or shared suppliers. Then steps can be taken to correct the problems, possibly through something as simple as a modified configuration, a corrected process, or a firmly inserted cable.

Before the manufacturing line started, there was a golden system. Poked and prodded by technicians and firmware engineers, it passed every test thrown at it. The thousands or tens of thousands, or hundreds of thousands that follow must have the same level of quality. Will the other boxes maintain this level?

After the box has been put through circuit testing, making sure that there is continuity for the signals, the new box is put through tests to verify that it functions as expected. Since the UEFI Shell has access to the UEFI networking stack, a simple script downloads these tests from a network server on the factory floor, executes them, and then sends the log back to the network server. The network server quickly scans the log for errors and, if detected, flags the box and alerts the supervisor.

Some of the tests require more time, or require special environments, or are less likely to occur (but expensive if they do). Rather than hold up all of the manufacturing lines, a sample of the boxes from each of the lines is set aside. In the initial runs, all units go through this more intensive testing. Later, during volume production, the sample size is reduced.

Some of these tests are long-run tests, consisting of repeating a single action over the course of thousands of reboots. For example, entering the standby or power-off state repeatedly to stress the hardware and the firmware. Some of these tests are environment tests, consisting of exposing the system for hours or days to high humidity, low humidity, high temperature, and low temperature. Some of these might be subsystem tests, such as memory, which are prone to failure because of slight variations in manufacturing lines, or component supplies.

The good news is that, if these failures can be isolated and diagnosed, many times a UEFI firmware-level fix can be implemented and then that UEFI firmware image can be fed back into the process. In addition, the update can be applied to those units that were diverted because of the issues found.

These benefits are not just reserved for the original equipment manufacturers. Other companies that provided integrated solutions, where they take a box and add specific peripherals, operating systems, components and applications can use the UEFI Shell to make sure that all of the systems they ship have the correct configuration by updating the firmware, checking the installed devices against a known-good list, configuring those devices, and provisioning the hard drive, all from the UEFI Shell.

Hardware Access with Manufacturing Tools

When writing tests during manufacturing, you need access to different levels of information or control. DOS gives you one level of control: INT 0x21 for the file system, and legacy BIOS gave you one more level of control: INT 0x13 for block I/ O (or INT 0x10 for video). Of course, a library like the C library could abstract this, but it could not give you more control.

So your application looks something like Figure 5.1:

If your application needed more control, it must have intimate knowledge of the file systems, partitions, buses, and devices. With UEFI Shell applications, however, applications have access to the same file system level abstractions and block I/O abstractions that DOS offers. But they also have access to so much more, as shown in Figure 5.2:

Through the use of the UEFI programming model, it is possible to enumerate all of the devices of a specific type in the system. UEFI provides a standard data structure, the device path, which describes exactly how a device is attached to the box. The device path can be walked all the way from the host bus through the block device to the file system and even to an individual file. This gives numerous options for a UEFI Shell application so that it can insert its test at exactly the right level of abstraction.

Using the right level of abstraction means that tests for a specific class of devices can be done without worrying about unnecessary details. This, in turn, means that the test will be more portable from one generation of boxes to the next generation of boxes. It also means it is more applicable, to other devices of the same type, but which are located on another similar bus. For example, many tests for IDE also work for SATA. Many tests for optical disks also work for SCSI.

Using the right abstraction level for access also means that the tests are less likely to destabilize the box (or subsystem in that box) under test. Since the box (or subsystem) is usually running many tests during the manufacturing process, it is important for them to be run one after the other to save time. This can only be done if the box (or subsystem, or device) is still in a working state after finishing the previous test. The worst case is that there must be either a physical intervention (that is, cable inserted or removed) with the box, or the box must be reset. While these are sometimes unavoidable, each one adds time to the manufacturing test. (Yes, ResetSystem() is disruptive and adds time.)

Using the right abstraction also allows third-party tests to be inserted into the manufacturing process. For example, each component vendor can provide tests of the condition of their individual component and even, in some cases, the UEFI-compatible driver that supports it. Since they use standard APIs provided by UEFI, they can be used over and over again, often with no more modification than changing a few command-line arguments. In addition, while they test their component, additional tests that use that component can be inserted in the same script.

Converting Manufacturing Tools

So, how do you convert tools to work in the UEFI Shell environment? Even though the UEFI Shell has many advantages, there is still work required to make the leap from the 16-bit, 1980’s MS-DOS environment tools.

The clock is ticking. Every month that you defer converting over to a more modern tool environment simply makes the job harder. It is harder to find a system that can host the operating system and compiler necessary to build the tools. It is harder to find engineers in your company who still remember how the tools are supposed to work. It is harder to find documentation that describes the process.

Here are some key steps:

Just one final note on converting manufacturing tools: accept the new interface! Sometimes there is a temptation to create some sort of “interface layer” that makes UEFI look like DOS or Linux or some other favorite environment. Resist the temptation. Don’t try to make UEFI look like Linux or DOS.

Conclusion

The UEFI Shell has a lot to offer during the manufacturing process:

Now is the time to make the upgrade to UEFI Shell-based manufacturing tools.