Roy’s Rule #1: Passing the Buck Is Expensive (or Carry Your Own Burden)

Experience has demonstrated that the later a problem is fixed, the harder and more expensive the fix becomes. Similarly, passing a task to the downstream user of a product can have a similar effect, and increase the likelihood of errors as well. For example, if a requirement author does not want to go through the effort of writing down the fine details of a specification and elects to let the reader of the document “figure it out themselves,” the requirement has just gotten considerably more expensive, and the chance of errors has gone up as well. Considering that a document (or in this case a requirement) will typically have one author and many readers, and the author knows the topic best and can provide the details more efficiently than anyone else whereas the readers will have less expertise and will take considerably more time to generate the same information, then letting the readers derive the details for themselves is a bad business decision at best, and could be a problematic one as well since the reader is more likely to make a mistake when deriving that information. Supposing that a requirement author could write down the details of one requirement in five minutes whereas a reader might take ten minutes to figure it out on their own, and if there are ten readers who will use that requirement, then the author’s failure to provide the detailed information has increased the cost of that requirement by a factor of 20. If that figure is multiplied by the number of incomplete requirements in a document (which is likely to be significant given the attitude of the author), then it quickly becomes clear that the simple act of delegating the details to the reader can be a very costly proposition, and that does not even include the cost and time associated with fixing errors that may occur.

It is tempting for a requirements author to leave some details to the readers to figure out, normally as a way to avoid spending time on something that the author may feel the readers can figure out on their own. In other cases, the author does not realize that their knowledge or experience is not common or universal. As described earlier, however, delegating the details in this way can result in wasted time and effort, delays due to mistakes in figuring out the requirement details, delays and errors when the requirements are used for verification (even more if the verification engineer misinterprets the requirement and creates erroneous test cases), and expensive hardware fixes if the mistaken interpretations are not caught early and find their way into hardware. DO-254 lists completeness as one of the objectives of validation, which means that a derived requirement that does not provide all relevant details is insufficient and should be corrected.

Another area where there is a strong temptation to delegate work is writing comments in HDL source code. Again, the reason is typically that a designer feels (with good intention) that some level of passing effort to the reader is harmless and will save time and effort on their part, not realizing that it will multiply the time and effort expended by anyone who reads the code, such as by a reviewer during a code review, when making changes later in the project, or when reusing the code at a later date in a new application. And as with requirements, poorly commented code will require that the reader interpret and figure out the functionality, which means there are now opportunities for misinterpretations and outright mistakes, which can result in more serious delays and costs if the mistakes find their way into the final hardware. Well- commented code will often have more lines of comments than lines of code. While this seems unnecessary to some people, anyone who has had to review or work on code that was commented less than this can testify that delegating the details to the reader is not the way to save time and effort.

Roy’s Rule #2: Predictability May Be Boring in People But It Is Goodness in Electronics

Deterministic operation, where there are no variations or uncertainty in how a system operates each time it operates, is one of those behavioral traits that can be immensely boring when exhibited by a person, but highly attractive in an electronic system. A deterministic system will behave predictably under all operating conditions, and is essentially incapable of changing its behavior regardless of the inputs it receives from the rest of the system. When a system or circuit is designed to operate in this manner, it is inherently immune to outside influences, including abnormal inputs. However, it is not the deterministic behavior that creates this immunity; instead, the determinism and immunity are both characteristics of a type of system or circuit that controls or is independent of, rather than reacts to, its environment. Or alternatively, a system will exhibit deterministic behavior if its inputs define its data output but not its operation and behavior.

A very simple example of this concept is shown in Figure 6.1 through Figure 6.3. Figure 6.1 shows a very simple finite state machine (FSM) that implements a control interface for an analog to digital converter (ADC) device. It is typical of how control circuits are implemented, particularly for devices that provide feedback. In this case, the state machine holds in an idle state until it receives a convert trigger, at which point it increments to state 1 to start a conversion cycle in the ADC, and then increments to state 2 to wait for the ADC’s end of conversion (EOC) signal. When the EOC signal is received, the state machine switches to state 3 to assert the ADC read control signals, and then to state 4 to read the ADC’s output data before going back back to state 0 to await the next conversion trigger.

FIGURE 6.1 Example One—Typical FSM Control Circuit

FIGURE 6.2 Example Two—Deterministic Circuit

FIGURE 6.3 Example Three—More Deterministic Circuit

An examination of this implementation to identify potential failure modes will immediately zero in on state 2 because the state machine could latch in that state if the end of conversion signal does not arrive as expected. There are also three unused states (states five, six, and seven) that could become undefined states if not properly managed in both the source code and the synthesis tool.

An improvement on this circuit is shown in Figure 6.2. The circuit in Figure 6.2 employs a counter in place of a state machine. The counter is reset to zero by the conversion start trigger, and then increments from zero to its terminal count, where it then latches and awaits the next conversion trigger. The control signals for the ADC are generated by decoding the counts. This circuit is semi-deterministic in that it does not have an equivalent to state 2 in the previous example, nor does it contain unused states, but it does include a trigger input that resets the counter to zero (its starting state).

One significant difference between these two examples is that the second circuit, if using the same type of asynchronous ADC as the first circuit, will count through the ADC’s maximum or worst- case conversion period (taken from the data sheet) and then read the output data rather than wait for the EOC signal from the ADC, and thus operate the ADC in open-loop fashion. However, for this type of circuit a better selection would be a synchronous ADC in which the conversion and its output are controlled explicitly by the control signals generated by this circuit.

For an asynchronous ADC the use of count states to mark time ensures that the circuit will not hang up if the EOC fails to appear for whatever reason. Thus the effect of a failed ADC or interrupted EOC signal will at most result in null data, a missed sample, or invalid data, which can be managed at the system level, and will not affect the circuit’s operation in any way. This approach creates a distinct division between the circuit’s (and system’s) operational and data characteristics, in particular ensuring that the circuit will operate normally and deterministically regardless of whether the data or its source becomes corrupted or otherwise untenable.

The most deterministic, and thereby robust, of these example circuits is shown in Figure 6.3. This circuit improves on example two by removing the trigger input and turning the circuit into a free running counter that operates independently of all external signals (except the clock). In all other respects this circuit operates identically to example two.

This circuit approach works best in a master role where the counter establishes the periodicity or frame rate for the system, so it would most commonly be used at a higher level than shown in the example. In this type of system every function and circuit is operated and timed by decoding the counts generated by the central counter, ensuring that every aspect of the system is precisely timed with respect to overall operation and to each other. Systems designed with a central free running counter of this type will exhibit truly deterministic behavior where every node, register, and even data value can be precisely predicted and modeled at any point after the system starts operating.

Roy’s Rule #3: Use HDL to Design Logic, Not Behavior

Of the three levels of HDL design (behavioral, RTL, and structural), register- transfer level (RTL) is most compatible with the objectives and processes in DO-254. RTL designs are compatible with a pin level description of requirements and the respective pin level testing. Structural HDL, because of its very low level of design expression, will increase the cost of a program by requiring that the processes and methodologies in DO-254 be applied at a sub-functional level. Behavioral HDL, which has the advantage of allowing a PLD design to be expressed at a functional level, might save time and effort during the design phase of a program, but because the code is describing the behavior of the design rather than the hardware itself, the logic design is left to the design tools rather than the designer. Since the processes and algorithms in the design tools are not visible to the designer, there is no way other than through exhaustive testing to determine conclusively whether the circuits created by the tools are in fact the circuits that the designer intended to put into his design. In addition, the high level at which the design is coded results in larger elements, which can affect the elemental analysis of the design. These loose ends represent a large unknown in design assurance.

RTL coding provides a good compromise between the two coding methods. It allows the designer to have a high level of control over the implementation of the design while keeping the size and complexity of the elements at a level equivalent to that of a circuit card, which is the level for which DO-254 was written. The added advantage of RTL coding is that its lower level of design expression allows the design to avoid many of the pitfalls that can be encountered during synthesis when the tools’ problematic reduction features are engaged. An example of this phenomenon is discussed in the second example circuit in the Design Execution portion of this chapter.

Roy’s Rule #4: Make Your Circuit Bulletproof Even If No One Is Shooting at It

This should really be obvious for engineers who appreciate the seriousness of level A design. It is not enough to design a circuit that works; a conscientious designer who focuses on safety and reliability will do their utmost to make sure that their circuits are bulletproof and will not malfunction or misbehave for any foreseeable operating condition (this phrase from the FARs should be familiar by now). An example of this rule is discussed in the first example circuit in the Design Execution portion of this chapter.

Roy’s Rule #5: Use Top-Down Design or Prepare to Go Bottom-Up

Top-down design, where a system or function is first defined at the topmost level and then decomposed downward to the lowest levels of the design, is the only recommended methodology for defining a system’s functionality and design. Bottom- up design, which defines a system or design from the bottommost elements and then attempts to work upward to define the system, is never recommended and normally results in curious designs and programmatic disasters.

However, the combination of top- down design and bottom- up implementation, where a hardware item is defined and designed from the top down and then assembled and tested from the bottom up, results in the best of both worlds.

Roy’s Rule #6: Find Your Own Failure Modes

There are two places where this rule should be applied: first, finding all possible and potential failure modes of a design; and second, finding and knowing our own failure modes, or in other words, understand ourselves and our issues so that we can prevent ourselves from sabotaging our own work. The first is actually the easiest since designs are generally straightforward and can be analyzed to consider all possible sources of failure. The second, on the other hand, can be complex and difficult due to the unpredictability of human nature and experience. However, understanding ourselves to the point where we can recognize our limitations, biases, and eccentricities and how they affect our work will allow us to mitigate their negative effects.

Roy’s Rule #7: Never Assume

An assumption is a decision that is based on a lack of knowledge—being fully informed means that assumptions are unnecessary. Some assumptions can be based on incomplete rather than no knowledge, but even then it is still a lack of knowledge that leads to an assumption instead of an informed decision. Making an assumption can jeopardize all downstream decisions and actions, so the best course of action is to avoid assumptions and rely as much as possible on informed decisions.

Roy’s Rule #8: Do Not Ask for Trouble (Avoidance Is Better Than Mitigation)

This means that it is a better idea to avoid any kind of risky or questionable design feature than to put one in and then mitigate its effects. For example, one- hot state machines can be unreliable, so rather than design a one- hot state machine into a PLD and then find ways to mitigate their weaknesses, it is better to not use them at all and instead rely on an alternative circuit type that is more reliable. Intentionally introducing a weakness into a design is not compatible with the goals of safety critical design no matter how well the weaknesses are mitigated.

Roy’s Rule #9: DO-254 Is Our Friend

DO-254 is a collection of industry best practices. They can improve the reliability of a system and even reduce development costs if applied properly, so the processes in DO-254 should be embraced, not shunned or avoided. In fact, considering that DO-254 contains best practices, anyone creating level A hardware should already be complying with DO-254.

Roy’s Rule #10: Review Now or Pay Later

It is tempting to save time and money by skipping or skimping on peer reviews, or even by conducting low quality reviews. However, this will only cause errors to be overlooked, which allows them to propagate downstream and become more expensive to fix. It is far easier and less expensive to spend a little extra time and effort on a peer review to make sure the review is complete and thorough, than it is to fix any problems that get through because of an inadequate review.

Roy’s Rule #11: Just Deal with It

Level A design is often difficult and expensive. Rather than fight it, it is usually a lot easier in the long run to just accept it and deal with it. The same applies to complying with DO-254: the most expensive approach to DO-254 is to try to avoid it, and the more you try to avoid it the more expensive it gets.

Roy’s Rule #12: Ignore the Trees

Sometimes, especially when a project is behind and time is critical, the need to focus on the immediate task can make it hard to see the big picture. When making decisions, always remember to look at the impact of every decision on the long-term conduct of a project and not just on the short-term cost or benefit. What might work well, look attractive, or solve an immediate problem may not work well or could even cause problems in the long run.

Roy’s Rule #13: Require Requirements

As described elsewhere in this book, DO-254’s processes focus on functions and functionality, and the way functionality is defined is through requirements. And as described in the Requirements, Validation, and Verification chapters of this book, the number and quality of the requirements will have an enormous influence on the cost and effort of complying with DO-254. With requirements being as important as they are, it makes sense to invest a great deal of effort in producing high quality requirements. Experience in DO-254 has shown that requirements are consistently the single most influential data item when it comes to affecting the course of a development project, so to even consider shortchanging the requirements is inviting potential increases in development time, cost, and effort.

Requirements are not an appendage to a project, nor are they a documentation burden. They are a critical part of the design and verification processes, and it is not an exaggeration to state that the quality of the requirements will dictate the conduct of the entire project. So it behooves us to treat requirements with respect and to never try to save time and money by reducing the effort and time that is put into them.

Roy’s Rule #14: Have No Faith

It can be easy to have faith in our designs, but often the reality is that our faith is misplaced or is not realistic. In other words, believing something does not make it so. Faith- based certification is not an accepted approach to compliance with the FARs; the only acceptable approach is to support all decisions and claims with hard data, so we need to know, not believe.

Roy’s Rule #15: There Is No Hope

Hope is one of those terms or concepts that should never find its way into the business of Level A design. We should never hope that our designs are safe, we should only know it for a fact. If we find ourselves hoping for the best, then we probably have not done our job with diligence. In level A design there should be no hope, only certainty.

Example 6.1: Shift Registers

Shift registers are one of the simplest and most useful of the basic digital logic circuits. However, because of their simplicity and utility they often escape scrutiny and may be overlooked as a source of potential errors.

Figure 6.4 illustrates a simple three-stage shift register that shifts on the rising edge of a common clock in an FPGA. Upon examination of this circuit and application of “Roy’s Rules” number 6 (find your own failure modes) to it, there is a potential error mode or weakness that is based on the fundamental criteria of how shift registers work: the hold time of each register’s output must equal or exceed the following register’s input setup time with respect to the rising edge of its clock input. In other words, the shift register will no longer work properly if the timing between a register’s clock edge and input setup time, and the previous register’s output hold time, deviate from normal specifications. These deviations can result from a number of factors, including a reduction in the output hold time of the preceding register, an increase in the input setup time, or an unequal delay in the active edge of the clock. The most common of these is a delay in the clock edge due to routing delays, which causes the clock edge to reach each register at a different time (clock skew).

FIGURE 6.4 Simple Shift Register

FIGURE 6.5 Waveforms for a Properly Functioning Shift Register

FIGURE 6.6 Shift Register with Clock Skew in Second Stage

Figure 6.5 shows typical waveforms for a properly functioning shift register, where the rising edge of the clock arrives simultaneously at each of the three registers. As can be seen in the waveforms, each bit of the input data stream is shifted through the three register stages and appears at the output of the shift register.

In contrast to the normally functioning shift register, Figure 6.6 shows the waveforms for a shift register where the rising edge of the clock arrives late at the second of the three registers. Because the rising edge of the clock arrives later than the hold time of the first shift register stage, the second register will immediately shift through the data going into the first stage, causing each bit of data to shift into both the first and second stages at the same time.

Clock skew can be caused by a number of factors, such as any of the following:

1.  A failure to use a dedicated low-skew clock net in the device.

2.  Failure of the dedicated low-skew clock net to perform to its expectations (in actual devices these nets have been shown to occasionally still have enough skew to cause shift registers to malfunction).

3.  The need to use normal routing resources because there are too many clocks for the available clock nets.

4.  The place and route tool automatically removed the clock from the clock net because its assignment there conflicted with the tool’s placement rules.

5.  The clock could not be placed on the clock net because access to the net was determined by pin assignment, and the input clock signal was assigned to the wrong pin on the device.

Since the introduction of clock skew can have a number of causes, and some of those causes may be inadvertent or even unknown to the designer, the application of Roy’s Rule number 4 (make your circuits bulletproof even if no one is shooting at it) is a good way to prevent potential clock skew problems.

FIGURE 6.7 Shift Register with Controlled Delays

How does one make a shift register reliable? Since the problem stems from clock skew exceeding the hold time of the previous shift register stage, and assuming there is little that can be done about this skew, the logical alternative is to design a shift register that is impervious to clock skew (within realistic limits of course), and one way to do that is to force the hold time of the previous stage to exceed any expected clock skew.

Figure 6.7 illustrates one way to do this using synchronous logic. In this solution, extra registers that use the alternate phase of the clock are inserted between the shift register stages to implement controlled delays (as opposed to uncontrolled, which would be the case if asynchronous logic was added between stages) between stages of the shift register. While this may almost double the number of registers in the shift register, the cost of additional registers is insignificant in a register- rich FPGA.

Figure 6.8 shows the waveforms for this shift register. Because the newly inserted shift register stages operate off the alternate (falling) edge of the clock, the shift register will now operate correctly for any amount of clock skew up to the period from the rising to falling edge of the clock, which for a symmetrical clock will be about half the clock period.

As noted previously, the lesson from this example is not how to design a robust shift register, but rather that applying a high integrity design philosophy can predict and identify potential failure modes in even the most simple and unassuming circuits that normally escape scrutiny. Identifying and correcting these weaknesses can significantly reduce the potential for latent failures.

FIGURE 6.8 Waveforms for Shift Register with Controlled Delays

Example 6.2: Synthesis Tools

Circuit designers, particularly the less experienced ones, have a tendency to put too much trust in their design tools. The problem with this is that the tools may not entirely deserve that trust.

Figure 6.9 shows the VHDL code for a very simple clock divider that divides a source clock by three. It is simply a modulo-three counter that counts from zero to two and then back to zero, with count three being an unused count. There is no reset in this circuit because the synchronous device in which it is used needs its clock during reset to operate and initialize. The circuit diagram for the code is shown in Figure 6.10. Because the most significant bit of the counter is fed back to its synchronous clear input, both the two and three count values (binary values 10 and 11) will cause the counter to synchronously reset and restart its count sequence. Thus, if for any reason the counter reached count three (11 binary) it would immediately recover and start over.

Figure 6.11 is the logic diagram for the expected output from a synthesis tool. Its truth table shows that it should behave in the same manner as the original design.

Figure 6.12 is the logic diagram for the actual output of a synthesis tool that processed the VHDL for the clock divider, taken directly from the synthesized netlist output of the tool. A quick analysis of the synthesized output will reveal that it not only looks different than the original circuit, it also behaves differently in that it will latch in count three. This is a potential failure condition which, because the circuit generates a clock, could cause a system to lock up and fail. Hardware testing showed that the PLD would in fact lock up regularly on power up or during a power transient because the two registers in the circuit would randomly initialize to count three.

FIGURE 6.9 VHDL for a Simple Clock Divider

FIGURE 6.10 Logic Diagram for a Simple Clock Divider

FIGURE 6.11 Expected Output of Synthesis Tool

FIGURE 6.12 Actual Output of Synthesis Tool

A call to the synthesis tool manufacturer revealed that the tool was doing what it was designed to do—it recognized the circuit as a modulo three counter and intentionally changed the circuit’s topology to be as efficient as possible (eliminating the unused state reduced the size of the circuit by a logic cell or two). Unfortunately it did so at the expense of changing a fail-safe unused count into a weakness that could cause a system failure. The tool manufacturer also revealed that the problematic reduction algorithm could not be turned off or defeated in that model of the tool. Since there was no software “switch” in the design tool user interface to turn off that feature, there was no way for the designer to know that that algorithm even existed, let alone that it could cause problems.

The synthesis tools used for PLD logic reduction are another example of the capability/complexity dichotomy that was brought up in the discussion of device selection, where an increase in capability will often bring with it a corresponding increase in complexity that will normally increase the uncertainty associated with using it. Since the goal of design assurance is essentially to reduce uncertainty to manageable levels, any device, tool, design feature, or design method that can increase the potential for uncertainty is at odds with this goal and needs to be dealt with in a deliberate manner. In this case, since there is no alternative to using a synthesis tool when designing with PLDs, and since the synthesis tool is an integral part of the design process, some other means of mitigating this type of effect (this is just one example of who knows how many such features exist in these highly complex but capable tools) must be brought to bear to maintain some degree of control (and therefore certainty) over the implementation of the design.

Synthesis tools are quite adept at recognizing certain code constructs and design techniques and then applying their technical magic to improve on them, often without the knowledge (or consent) of the designer. In the vast majority of cases this assistance is useful, but when considered within the context of high reliability design, the rare instances of fault introduction like this example are still frequent enough to be of concern, and should be anticipated where possible and then mitigated through defensive design techniques. Thus, if a circuit has any potential to be selected by a tool to be “improved,” or if the application of Roy’s Rule number 6 indicates that the circuit topology has any potential for inherent failure modes, it should be designed in a way to be bulletproof or to minimize the chance that the tool will recognize it.

Common circuit topologies, such as counters and finite state machines, are the most likely to be modified by synthesis tools. The essence of this problem is that the synthesis tool is recognizing the circuit and acting on it in an attempt to optimize it, but it is doing it at the expense of function and robustness. In order to prevent the tools from recognizing and then modifying our circuits without resorting solely upon setting switches to pre- emptively set up the tools to disable optimization features that cannot be fully predicted, the approach contained in Roy’s Rule number 3 can be used, which is to design logic without the use of behavioral HDL, which leaves the actual logic design to the designer rather than relying upon the synthesis tool to do the design. This approach can work well with functions that are normally designed with behavioral HDL, such as finite state machines and complex math functions such as multipliers. However, for circuits such as this example, where the logic was already designed at the RTL level, another option is to design the circuits in ways that can minimize the chance that the tool will recognize it for what it is, or as an alternative, modify the circuit topology in a way that the circuit no longer behaves like its standard form.

Figure 6.13 is the logic diagram for a modified version of the divide by three counter that breaks the terminal count feedback path with a register, which can reduce the probability that a synthesis tool will recognize the circuit as being a modulo counter. The addition of the register also alters the behavior of the counter such that, while it still counts in the normal manner, it does not rely on asynchronously feeding back the terminal count.

FIGURE 6.13 Modulo Counter with Broken Feedback Path

When this circuit is processed by the synthesis tool using the same settings as the original design, the result is the netlist shown in Figure 6.14. It is immediately apparent that this circuit is identical to the expected synthesis result shown in Figure 6.11 except for the addition of the delay register in the feedback path. Analysis of its operation results in the truth table shown in Figure 6.14, showing that the circuit’s operation is also identical to the original design. Thus, with the addition of the register this counter was able to avoid the problematic feature of the synthesis tool and retain its original (and safe) functionality.

Note that some synthesis tool manufacturers have come to recognize this concern and have introduced tools with a “safe” option that bypasses the optimization features that can introduce fault modes. This is a very welcome feature for those who work with high reliability designs. However, even with this nice addition it still behooves designers to continue to practice safe and defensive design techniques.

This example illustrates the problems that can occur if designers are too willing to trust their tools. Trusting their tools is, in fact, the first impulse of most designers. One insidious reality of tool- induced failure modes is that most, if not all, of these failure modes are latent and will not be revealed through normal requirements-based verification. The designs will function as they were intended to and still meet their requirements, and so will pass all requirements-based tests. Finding these tool-induced failure modes requires additional robustness verification that targets the circuit types that can contain potential failure modes, such as finite state machines and counters.

FIGURE 6.14 Actual Synthesis Result for Modulo-3 Counter with Broken Feedback Path

When dealing with complex tools that cannot be completely understood it is important that the designer know as much as possible about the tools, but never to trust them. In addition, designers should disable as many of the advanced features as practical, and never assume that the tools are going to correctly implement the design’s intended functionality. Designers should also be aware of any inferred functionality in their designs, such as unused states in counters and state machines, and create verification standards that account for all conceivable tool-induced errors.