This article is slightly out of flow. I should have gone from the introduction to SDLC, Agile and then Test Process before arriving at Error, Defect and Failure.
But if you do not understand these three, it is hard to explain why testing matters. So I am pushing it forward anyway; SDLC can wait for the next article.
The relationship between error, defect and failure
At work, people usually compress everything into one sentence:
This bit is broken.
That is fine in daily conversation; everyone on the team still understands. But when studying or analysing an incident, “broken” may mean a person's incorrect action, something wrong in the code, or the incorrect behaviour a user has just seen. Put all three into one bucket and it becomes difficult to trace where the story began.
The rough flow is:
Some cause makes a person do something incorrectly. That mistake leaves a defect in a requirement, code, test case or configuration. Under the right conditions, the defect is triggered and the software fails. When the failure reaches production, there is an impact: lost money, lost data, lost customers or, worse, a safety effect.
This is a useful flow, not a one-to-one formula. A large failure may result from a pile of defects and organisational problems layered on top of one another.
Typical question: How are error, defect and failure related?
A human error may introduce a defect into a work product. When the defect is triggered under particular conditions, it may cause an observable failure and create an impact.
Root cause
Root cause is the underlying cause. If you deal with it, the chance of the same kind of problem recurring goes down.
Suppose a developer forgets to validate a negative amount. Writing “the developer forgot validation” in the bug report describes only the error. Ask a little further and you may discover that the requirement never defined the valid range, the team has no validation checklist, the codebase has no shared component and the review covered only the happy path.
Adding an if fixes the current defect. If the root cause stays in place, the next feature may run into exactly the same thing.
Not every bug needs five meetings of Root Cause Analysis. But for a serious bug, or one that keeps getting closed and growing back, fixing the surface is not enough.
Error
An error is a human mistake: misunderstanding, forgetting, entering the wrong thing, choosing the wrong thing or making a wrong decision. It is not yet an object sitting in source code. It is the action that produced an incorrect result.
In philosophical terms, an error has no shape. You cannot open the repository and point to a lump labelled “error”. What you see in the repository is the defect left behind by the error.
A requirement example
You pasted a template into the requirement and forgot to change the rule?
Yes, I only noticed when I read it again. It still had the old project's rule.
The forgetting to change it is the error. The incorrect requirement left behind is the defect.
A source-code example
You did not handle the case where the API returns
null?Damn, I thought it always had data.
The wrong assumption about the input data is the error. The code branch that does not handle null is the defect. Run that exact case and the app crashes: that is the failure.
A test-case example
This test set seems to be missing the lower boundary.
Oh no. I divided the partitions and forgot to take the boundary value.
Failing to apply the test technique is the error. The testware missing a necessary case is a defect in the testware. It does not create another product defect by itself, but it weakens the fence that is supposed to catch defects.
There are many reasons people make errors: lack of knowledge, vague requirements, tiredness, deadline pressure, awkward tools, or the teacher calling to say your child has fallen face-first into the edge of a table while you were working. So reducing error cannot be just telling people to “be more careful next time”.
Requirements need review, code needs unit tests and code review, and test cases need peer review. Eating and sleeping properly help too—joking, but not really.
Typical question: What is an error?
An error is a human action that produces an incorrect result.
Defect
A defect is the incorrect thing left in a work product. It may be in a requirement, design, source code, test case, deployment script or configuration.
The three examples above become:
| Error | Defect left behind |
|---|---|
| BA forgets to change a rule from a template | Requirement contains another project's business rule |
| Developer assumes the API always returns data | Code does not handle null |
| Tester forgets the lower boundary | Test set lacks an important boundary case |
The old version of this article distinguished bug as an error in code and defect as an error in documentation. There is no common convention for that split, so it should not be used as a definition.
In most teams, bug, defect and sometimes fault are used interchangeably. If a company has its own glossary, follow it. If it does not, say what you mean: a defect in the requirement, code or test case. That ends the argument.
Typical question: What is a defect?
A defect is an imperfection in a work product that may cause the software to fail when it is triggered.
Failure
A failure is when a component or system does not perform a required function while running.
Suppose code calculates an average as total amount divided by item count but does not handle quantity = 0:
- The code line missing validation is the defect.
- The user opens an empty order and the screen shows an error or crashes: that is the failure.
- The user cannot place the order and goes elsewhere: that is the impact.
A defect can sit quietly for a long time. If nobody creates an empty order, the branch does not run and the failure does not appear. That is why “production has been running for three months, so there must be no bugs left” proves very little. Maybe the right condition simply has not occurred.
Conversely, a failure cannot always be reduced to one line of code. A dead network, wrong configuration, failed disk, dirty data or services that understand one another differently can bring down a whole system.
Severity and priority
Once we talk about defects, sooner or later we must assess Severity and Priority.
- Severity describes how serious the impact of the defect or failure is.
- Priority describes how urgently it needs to be handled.
They often travel together, but they are not always equal.
A mobile app that runs out of memory and crashes after 20 hours has high Severity because it eventually crashes outright. But if the flow is rare, there is a workaround and the team must fix a payment failure first, its Priority may be lower.
Conversely, a campaign banner on the home page has a white background but remains clickable. Functionally, Severity is low. The campaign starts tomorrow morning and the partner's logo is sitting there exposed on the home page, so Priority may be the highest thing on the board.
Critical, High, Medium, Low or P0, P1, P2 mean different things in different projects. Do not learn one table online and carry it into every company. In one place P0 means fix it tonight; somewhere else P0 is simply the default priority for a smoke test.
Typical question: What is the difference between severity and priority?
Severity describes the impact of a defect. Priority describes how urgently it should be fixed. A severe defect may have a lower priority, and a minor defect may be urgent because of business timing.
Some classic failures
These cases are not here so we can conclude “if only they had testers”. Large systems certainly have checks. The problem is that several technical and organisational layers failed together, and then a defect found a straight road to real damage.
Therac-25
Therac-25 was a radiotherapy machine used in the 1980s. Six known accidents between 1985 and 1987 caused patients to receive excessive radiation, leading to serious injuries and deaths.
Nancy Leveson and Clark Turner's investigation did not reduce the story to a coder typing one wrong thing. It involved software design, a race condition, missing independent safety interlocks, confusing error messages, poor incident investigation and excessive faith in the software's safety.
For a system that can affect lives, a few extra happy paths are nowhere near enough. It also needs fail-safe design, hardware interlocks, hazard analysis and a response process for when incidents happen.
Patriot at Dhahran
On 25 February 1991, a Patriot system at Dhahran failed to track and intercept a Scud missile. The missile hit a barracks, killing 28 US service members.
The U.S. GAO report identified an error in the calculation of elapsed time as the system ran. A short test might show everything working, but after dozens of hours the error was large enough to shift target tracking.
This is a clear example of why the test environment and duration need to resemble real operation. Running a system for five minutes and declaring it stable for continuous operation is a little brave.
Knight Capital
On 1 August 2012, Knight Capital deployed incomplete software to its servers. Old code was activated and the system started sending abnormal orders to the market.
According to the SEC, over 45 minutes the system sent more than four million orders, traded over 397 million shares and caused the company losses of more than USD 460 million.
Again, this was not one bug. Deployment control was weak, testing did not stop it, risk limits were insufficient, and warning emails appeared but were not handled in time. When a defect gets through every layer of protection, 45 minutes is enough to nearly wipe out a company.
Closing
The short version is:
Root cause → Error → Defect → Failure → Impact
People make a mistake, the mistake leaves a defect, the defect meets the right conditions and creates a failure, and the damage caused by that failure is the impact. The real flow may be more complicated, but this frame makes bug reports, Root Cause Analysis and the distinction between Severity and Priority easier to follow.
The next article returns to the original flow: Software Development Life Cycle. We will see where requirements, design, code, testing and release appear, and where testing joins in.
