Delivering Defect-Free Software: FAQ

Copyright © 1997, 1998 Terence M. Colligan

1. What is "Delivering Defect-Free Software"?

Delivering Defect-Free Software is a discipline of programming and QA targeted at delivering software to the users with no defects or errors. Using a combination of generally accepted programming techniques and quality assurance techniques, Defect-Free Programming delivers software into production with so few defects that users of the software consider it to be defect-free. Delivering Defect-Free Software can be applied to any software -- embedded, client, server, PC, mainframe, or workstation -- and in any programming language.

2. Why try to be "defect-free"?

For three reasons -- reduced costs, increased productivity, and increased usability.

Costs are reduced both because your programmers don't have to spend time fixing bugs, and because the users of your software don't waste their time encountering, reporting, and getting around bugs.

Productivity of your development team will increase, because you will spend less time on maintenance and bug fixing. Your development staff will also spend less time interacting with support personnel and/or end users, because your software will work correctly.

Your software is more usable because it does what it is supposed to and what your users expect it to, thus improving your users' productivity. This results in reduced user costs.

3. Isn't it impossible to write "Defect-Free" code?

No, not at all, although it is a very commonly held belief. We have identified in several different companies a number of programmers who were already producing virtually defect-free results for their programs. Although these programmers are rare, we found enough of them to be certain that defect-free is indeed possible.

Our Defect-Free Programming is our system for using the techniques of the naturally defect-free programmers by traditional programmers.

One of the most interesting things that we discovered is that the belief that delivering defect-free systems is impossible contributes greatly to the failure to produce defect-free systems.

In other words, whether you believe that you can deliver defect-free software or not, you're right!

4. What language do you use?

We have traditionally programmed in C or C++, but very little of the methods we use for Delivering Defect-Free Software is C or C++ specific.

We (and you) can produce defect-free software in any language -- on any platform.

5. Why doesn't anyone else guarantee their code?

Smart-alec answer: Because they know how good it is?

Professional answer: since most programmers and programmer managers don't believe that defect-free code is possible, they aren't willing to make any such guarantees.

Most software doesn't have a guarantee. If you read the warrantees for the software, what is promised is that you will find some software on a diskette or CD-ROM, and that the diskette/CD-ROM will be readable. If it isn't, they'll send you another copy for you to try again.

6. How does your guarantee work?

When we write Defect-Free software for you, we strive to have you discover no bugs at all. In the unlikely event that you find a bug, we will fix the bug for free. This guarantee is good for as long as you use the software, and for as long as we are in business.

Obviously, we have a tremendous incentive to deliver defect-free systems, or we'll spend all of our time fixing old systems.

7. What if you go out of business?

You will still have the source code to any software we wrote, and you will be in a no worse position that you would without the guarantee.

Since our code will have a much lower defect rate, you will actually be in a better position with our code than with a traditional alternative, even if we are wiped out by a meteor shower!

8. What if we modify guaranteed Defect-Free code?

First, we only guarantee the code that we write. So even if you have taken our Defect-Free training, you are responsible for any defects you create.

If you find a defect in guaranteed Defect-Free code, we will look at the problem and determine what the cause is. If it is in code we wrote, we'll provide a fix for free. If the problem is in the code you wrote or modified, we'll charge you for the time to identify the problem code. If you wish, we will develop a fix for your code at our standard consulting rates.

9. How long have you been doing Defect-Free code?

We have been developing the process and experimenting with different approaches for about four years.

We have been fully committed to producing Defect-Free software for about one year.

Some members of our staff have been producing Defect-Free code for over 20 years.

10. Are all your programs "Defect-Free"?

Unfortunately not.

11. Why not?

Because for a long time, I didn't believe that Defect-Free was possible, so I made management decisions which had the effect of insuring that our software had bugs in it.

I made these decisions over the objections of the few natural Defect-Free programmers that we employed over the years.

12. What kind of mistakes did you make?

Most of them. Generally, I pushed for schedule and "good-enough" software because I thought it would take too long and/or cost too much to "do it right."

As I learned over time, we never had time to do it right, but we always had time to do it over again and to fix problems that made it out to the field. In retrospect, Every decision I made for delivery time over quality turned out to be wrong -- not only did we not make the delivery time, but we usually paid a high price in support and maintenance.

13. Isn't it too expensive to make "Defect-Free" code?

NO! All of our data so far shows Defect-Free code to be cheaper to develop. It is certainly much cheaper to maintain and support.

There is a book by Phil Crosby about quality in a manufacturing organizations called "Quality is Free." Although the details are certainly different for software development than for manufacturing, I think most of the ideas are applicable. For software development, quality is also free. Or, at least cheaper than the alternative!

Although the actual design, coding and testing may take longer in our process (an assumption that so far is false), there is a huge savings in bug fixing, maintenance and support. There is also a large time savings for the users of Defect-Free software.

Some data published by IBM about their systems indicate a strong correlation between productivity and low defect rates. In other words, the highest-quality systems are the cheapest to create!

14. Can you make an old program Defect-Free?

Yes, but it may involve a lot of work, depending upon how well implemented and how well tested the old program is. Some of the parts of our process are hard to apply to old programs, while others fit old programs well. You will definitely be able to make an old program more reliable, but you may not be able to make it Defect-Free.

In particular, our Defect-Free process starts with a good design, one targeted at creating a Defect-Free program. In most cases, the design of the old program will have other goals -- small size, fast operation. For some really old programs, the design goals may not be apparent at all. If the old program's design didn't result in a modular design, it will be hard to achieve the full Defect-Free state.

The fully automated QA testing part of our process can usually be added to an existing program without significant rewriting. The changes necessary to support fully automated testing can usually be fit into old programs, even those with a very poor and unmodular structure.

15. How does this compare to object-oriented programming?

Defect-Free development is orthogonal to object-oriented programming. They coexist -- you can have Defect-Free code that is not object-oriented and you can have object-oriented code that is not Defect-Free.

You can also have code that is both Defect-Free and object-oriented. There is a synergy in that good object-oriented code will be modular, which is an important part of Defect-Free design.

We prefer object-oriented programming ourselves, and use it in our new programming.

16. What is the process?

We have an nine step process. In brief, the steps are:

1. Believe Defect-Free Software is Possible.
2. Think Defect-Free Software is Important.
3. Commit to Producing Defect-Free Software.
4. Design Your Code for Simplicity and Reliability.
5. Trace Every Line of Code When Written.
6. Review Code by Programmer Peers.
7. Build Automated QA into Your Code.
8. Build and Test Your Code Daily.
9. Use Automated Checking Wherever Possible.

17. What languages does Defect-Free apply to?

So far, we have done most of our work in assembler, C and C++, but the process is completely independent of any programming language.

Our Defect-Free process can be used on Cobol, Java, Fortran, Forth, Visual Basic, Ada, Modula, or any other language in current use. It can even be used on programs written in assembly language.

18. What platforms does Defect-Free apply to?

So far, we have done most of our work on Intel platforms, DOS and various forms of Windows and Windows NT, but our process is completely independent of any computing platform.

Our Defect-Free process can be used on MVS, Unix, OS/2, Linux, embedded systems, CICS, browsers, anywhere.

19. What kind of programs does Defect-Free apply to?

It's hard to think of a program category where Defect-Free programming would not be a benefit. If you are writing programs that will only be used a few times and only by a few people and the program doesn't have to be good, Defect-Free may not apply.

Programs which go through multiple releases, such as commercial software products, can particularly benefit from Defect-Free programming.

Programs which have a strong need to be correct, such as financial/accounting systems, medical systems, or military/avionics software, will benefit greatly from Defect-Free Programming.

Also, any software with wide distribution, such as game software, will have lower support costs when created using our Defect-Free process.

20. Doesn't it take too long to produce Defect-Free?

There are more steps in our process than in typical software development, so the superficial answer would seem to be "yes." Our experience, however, suggests that delivering software with our Defect-Free process happens faster and for less cost than with traditional methods.

To some extent, the answer to this question depends upon your definition of when software is done -- if it's when the programmer(s) says it's done and passes it into production or QA, then yes, it may take longer using our Defect-Free process.

If you think software is done when your users are successfully using it to improve their lives, reduce their costs, etc., then the answer is "Absolutely Not!" Delivering Defect-Free Software may well be the fastest way to produce software.

21. Can you teach others to do Defect-Free code?

Sure. We have a variety of classes and custom consulting offerings designed to do just that.

22. How long does it take to be effective at Defect-Free programming?

That depends upon how good your current practices are and how much of a change the Defect-Free process represents for you. (Remember that there are naturally occurring Defect-Free programmers -- just not very many!)

For an individual to adopt all of the process and become effective at Defect-Free programming should take from three months to perhaps one year. During that time, their code will be continuously improving in quality.

For a project to become fully Defect-Free, the time frame should be similar to that of an individual. Generally, the code quality should be improving during this start-up period. However, there are some parts of the process, particularly automated QA, which require the development of some internal tool software. For this portion only, the net reliability won't increase until the tools are put into production use.

It will take an organization longer to become completely Defect-Free, perhaps one to three years. If your organization has a lot of old programs, the time frame will be longer still, at least for the old programs. The time frame is longer for an organization purely for the obvious reasons: training, turnover, staggered project start times, etc.

23. Just how Defect-Free can you achieve?

We believe that you can achieve a level where the users of your software never experience a bug. Most of the large (and small) systems built by our naturally Defect-Free programmers are at that level -- no reported bugs at all.

Since we have relatively recently formalized the Defect-Free process, we don't have enough elapsed time to make the 'totally defect-free' claim for the formal process. We have observed very significant reduction in error rates, in excess of twenty times fewer errors, and expect the rates to go to zero.

One interesting side effect is that our criterion for what constitutes a defect has gotten much more wide ranging. Lots of things that in the past would not have been treated as a defect, but rather "the way things are," are now recorded as a defect.

So we are seeing significantly falling defect rates, even as we are much tougher about reporting defects.

24. Whatever possessed you to try Defect-Free code?

Three things: a growing unease with the quality of the software we were producing, support costs that were too high, and the discovery of the naturally Defect-Free programmers.

Actually, the naturally Defect-Free programmers were around all along, and were even complaining about how bad everyone else's code was, but I dismissed their complaints because I didn't believe Defect-Free was possible.

Only after I was frustrated at our defect rates and support costs and delivery times did I actually look at the claims of the naturally Defect-Free programmers. When I discovered that Defect-Free is possible, we set about making it a process that all programmers would follow.

25. Do you still have to test your software?

Absolutely. Full and complete regression testing is one of the major parts of the process, and requires a QA function in your organization.

Full and complete 'unit testing' under the debugger is a second part of the process.

I would say that there is more testing by QA in our process, and much more testing by the programmers, when compared to traditional development methods.

26. Is it hard to modify an Defect-Free program?

Quite the opposite. Defect-Free programs are highly modular, so changes are easier to make.

In addition, the fully automated QA makes it easy to make changes without worrying about breaking some existing function of the program.

27. Do you still need a QA department?

Absolutely! Complete and thorough QA is one of the bedrocks our process is built on.

The job of the QA folks changes, though, when comparing the Defect-Free process with traditional uses of QA. Under our Defect-Free process, QA engineers use the fully automated QA system to perform regression tests, so they don't spend a lot of time following test scripts.

Under our Defect-Free process, the job of the QA engineer is primarily to create new tests which are added to the automated system.

This is much less tedious than the job of QA folks in the traditional development processes, which gives most Defect-Free QA engineers more job satisfaction.