Software-Quality Discussion List
Digest # 020


Tenberry Home
Software-Quality Home
Reducing Defects
Automated QA
Consulting
Site Index

============================================================
           Software-Quality Discussion List

      S O F T W A R E - Q U A L I T Y   D I G E S T

      "Cost-Effective Quality Techniques that Work"
============================================================
List Moderator:                      Supported by:
Terry Colligan                       Tenberry Software, Inc.
moderator@tenberry.com               http://www.tenberry.com
============================================================
June 30, 1998                        Digest # 020
============================================================

====IN THIS DIGEST=====

    ==== MODERATOR'S MESSAGE ====

    Where's the Quality? (Part II)


    ==== CONTINUING ====

    Re: Software-Quality Digest # 019
      Travis Siegel 

    RE: Software-Quality Digest # 019
      David Bennett 

    Comment on 18
      "Niall Hammond" 


    ===== NEW POST(S) =====

    A Question
      "Niall Hammond" 



==== MODERATOR'S MESSAGE ====

  Where's the Quality? (Part II)

  In our last issue, your intrepid moderator related a disastrous
  systems failure which darkened his vacation, and made him consider
  a career change to any field that didn't involve any computers at
  all.  I asked

  THE QUESTIONS...

  Where is(are) the quality failure(s) here?

    - The notebook?

      Possibly there is a heat-related problem here, but there is
      no evidence that this is the problem.  However, no useful
      diagnostics exist or were produced.

    - Win95?

      With no evidence at all, I suspect Win95 -- possibly because
      dumping on MS is in at the moment.  There are no useful diagnostics
      here either.

    - The modem?

      Travis Siegel thinks the modem is to blame.  There are no useful
      diagnostics provided for the modem.

    - AT&T?

      No evidence points here, but there is no diagnostic help here,
      either.

    - Email software?

      Nothing points to this component.  There are modest diagnostics
      in that time-out messages were eventually displayed.

    - Me for not having sufficient diagnostics?

      I haven't been able to find any diagnostics for this environment
      and this dial-up ISP problem.

    - Me for not having a backup available?

      Partially guilty.

    - Other...

      David Bennett suggests poor planning and insufficient preparation.
      I agree (these are pretty safe criticisms of me in any situation!)


  However, I think that all of these systems and products have poor
  quality, perhaps even very poor quality, -- because there is virtually
  no support for diagnosing a problem in a system that is virtually
  certain to have problems, due to it's complexity and the use of dial-
  up phone connections.

  Therefore I propose the following test for a quality system:

  Either the system must be drop-dead reliable (failures measured in
  the 1 per 100,000 hours of usage or better), OR, there must be useful
  diagnostics for identifying common failure modes and reporting
  progress.

  (Note: I'm not suggesting the above as the *only* test for a quality
  system, just one why to check for quality.)

  Comments?  Do your systems have self-diagnostics?  Debugging aids
  that are available to the end-user?  Why or why not?



==== CONTINUING ====

++++ New Post, New Topic ++++

From: Travis Siegel 
Subject: Re: Software-Quality Digest # 019

Hi Terry.  Your experience with your dial up problems while on vacation
is something I've experienced right here at home (in Delaware)  The
symptoms are similar, as after a fashion, data just simply stops
flowing.  (at least my direction)  I've had the same problem with
multiple providers, one one of them, the problem was reversed, as long
as I was downloading, everything went fine, but when I tried to upload
something (such as ftp a file) it hung.  I tried to get both isps to
cooperate together, (and with me) to find out what the difference was
in their setups, but as you can imagine, that was like asking the
government to spend our tax dollars intelligently.  (sorry, had to be
said)

But as far as I could tell, the problem only manifested itself when I
was using my 56K modem.  If I stepped down to a 14.4 modem, the
problems went away.  I can also say that these problems showed up in
dos, linux, and windows 3.1 (possibly the same problem as dos) I still
don't know what the problem was, but I honestly suspect their routers,
since they had identical modems on their systems, and only the routers
were different from one isp to the next.  To some degree, the problem
was also much more apparent when using the unix shell that both
providers provided, while being not quite as bad when using the ppp
connections.  (possibly they went through different routers, I never
did find out)

But I also found that neither isp knew anything about the problem, and
kept blaming it on my software even after I had them both reproduce it
on their end.  

I attribute it to a lack of knowledge in general with most isps these
days.  It used to be that in order to be an isp, you really had to know
your stuff, it wasn't an option not to, since you were likely the only
one in your area.  Nowadays, it's extremely easy (and fairly
inexpensive) for anyone who has a couple thousand dollars to play with
to start their own isp business.  As a result, the market has catered
to these type of system operators, and overall, I personally believe
the quality of service has dropped drastically.  Of course this doesn't
mean it's not a good thing, it has certainly made access a whole lot
more affordable, not to mention more widespread, so I guess there's
always a good side to such matters.  Anyway, I'd be extremely
interested in hearing others views on this topic as well.  Is quality
(because of quantity) lacking these days in isp services, or is it same
as always in your opinion (for those of us who have been online for
more than 8 years, I fear the answer is definitely the former option)

++++ Moderator Comment ++++

  The only problem is that I was using AT&T, who presumably has spent
  more than a few thousand dollars setting up their POPs.  I'll try
  a slower modem, though -- thanks for the suggestion.

  This seems to suggest a worse quality problem than even I imagined --
  although I can't see how the routers (or anything north of the modems)
  can tell that it's a 56KB modem.

  I certainly don't have these problems on my office dedicated 56KB line
  or on my cable modem at home, which is about 2000KB!



++++ New Post, Many Topics ++++

From: David Bennett 
Subject: RE: Software-Quality Digest # 019

Hi Terry

*   - Me for not having sufficient diagnostics?
*   - Me for not having a backup available?
*   - Other...
*   Should a high-quality system provide diagnostics?
*     One of the biggest frustrations to me was there was apparently
*   no way to diagnose the problem.  All of the software was designed
*   to hide complexity (and making a dial-up TCP/IP connection
*   involves a *lot* of complex things going on!)
*   If everything went well, hiding complexity would be fine.
*   Unfortunately, it didn't.
*
*   What errors did I make?

Your major error was that you did not practise risk management.

You engaged in what my son does: risk-taking behaviour.  There is an
assignment due in by 10am on Tuesday.  At 3am Tuesday morning we're
driving to the office to get another laser printer cartridge because he
first tried to print out the assignment at 1am, printed 2 pages and ran
out of toner.  Obviously no-one could predict that this would happen.
The fact that I suggested a "release candidate" printout the previous
day (and the day before that) just adds tension to the situation.

++++ Moderator Comment ++++

  Guilty, your honor (honour?) !

++++ End Moderator Comment ++++

You have a project with a "drop dead" deadline (like year 2000).  You
have to work back from there.  Get the laptop with 4 weeks to go.
Waste a week getting it set up.  Use if for 1 week, work and home, to
do your regular e-mail.

If it matters enough, order the machine with all the software already
setup by someone else.  If they don't deliver right on time, order a
second (and a third) machine from someone else (all different brands,
but don't cancel the first one).  Buy a second-hand machine from a
member of staff which is known to work.  Hire a machine too.

Also, set up an e-mail account on Hotmail and Yahoo.  For that, all you
need is a Web browser.

Also, get accounts on two different ISPs (or 3 or 4).

++++ Moderator Comment ++++

  I wasn't expecting there to be so many problems in well-known products
  that had been in the market for months or years.

  I was also hoping to not spend 3 weeks getting ready for a 10-day
  vacation.

++++ End Moderator Comment ++++

Isn't it wonderful?  I have this great machine called a
retrospectoscope.  With it, I can see exactly what you should have
done.  Now, would I have done better.....??

++++ Moderator Comment ++++

  Do you rent this machine out???  ;-)

++++ End Moderator Comment ++++

* ... re "ifdef soup"...
* For these kinds of situations, I much prefer either:
*
*     - Different modules (i.e, vary what's in the build, particularly
*     since you are much more likely to have differing build scripts/
*     methods on different systems.)  For example, a SYSWIN.C and
*     a SYSUNIX.C which define the same functions and data.

We originally did that.  It was a maintenance nightmare.  There was
about 70% common code, but the common code was spread across multiple
functions and over a period of 3 years they drifted apart to the point
where they were impossible to work on.

++++ Moderator Comment ++++

  Interestingly, we've done this on 4 systems.  On 3 of them, it has
  been a maintenance godsend.  On the fourth, it started out as a
  godsend, and slowly changed into a small problem.  The reason was, I
  think, that junior people who couldn't/didn't understand the
  organization eventually destroyed the separation.  (This was before
  we started doing code reviews.  Since we've been doing code reviews,
  the overall design seems to hold steady or even improve.)

++++ End Moderator Comment ++++

*   - Run-time checks to disable/select between alternatives.

Sorry, impossible.  We're talking about a situation where the DOS
version includes  and the Windows version has  and
the Unix version has .  The parts of the code from the
"other" versions won't compile.  If it compiles, it won't link.

++++ Moderator Comment ++++

  Exactly why we use separate modules!

++++ End Moderator Comment ++++

*   I've never felt that #if/#endif's are worth the trouble they always
*   seem to cause. It seems to me that programming and debugging are hard
*   enough without having to guess what code might be generated from the
*   source you are looking at.

I agree 100%, but if runtime selection is impossible and the
alternative is large chunks of duplicated code, sometimes you have no
choice.  At least it was only 3 modules (file I/O, console I/O and
"system" functions).  I recently tried to port a Unix system.  Of 30
source code files, at most 5 were reasonably portable.  The others all
had #ifdefs and Unix dependencies smeared right across them.  At least
we didn't do that!

*   David, I'm curious.  How large would your system be if you just
*   enabled all the tracing at compile time?

A reasonable question.  For 16 bit Windows the answer was: too big.
The system wouldn't build with all tracing enabled (the DSEG limit).
For 32-bit Windows, probably about 50% size overhead currently, and not
more than a few % speed if everything is switched off.  Plus, take one
look inside the EXE and you can read a total history of the internals
of the software.  We don't like to do that.

++++ Moderator Comment ++++

  You've raised this issue before, but I don't see why it should be
  so unless you use printf to create your debugging data, and are
  extremely verbose in your dumps.  We tend to make our trace dumps
  somewhat terse, so we can see more data at one time.

  In this case, I don't see any exposure to the internals when looking
  at our modules with a binary editor...

++++ End Moderator Comment ++++

*   I find your approach cleaner than "#ifdef-soup", and it appears to
*   work for you.  However, to my mind, it still has the same problem
*   that I can't look at source code and tell whether or not it is
*   present in the module.

I would have to say this is more of a theoretical than actual problem.
When the code doesn't actual change the logic at all, the eye just
passes right by it.  Sort of like funny-looking comments...

*   Do you have problems with people putting side-effects into your
*   trace code?

Yes, but it's rare and it usually is extremely easy to pick up on the
production builds and validation.  We might pick up 100 bugs on
tracing, asserts, etc including a handful of really hard ones, and
introduce 1 easy bug in the tracing while we're doing it.  We have
definitely had more and harder problems from using optimising compilers
than this technique has ever caused us.




++++ New Post, New Topic ++++

From: "Niall Hammond" 
Subject: Comment on 18

>>    Should there be a separate testing/quality department at all?
>>   (Or should this function be part of the development team?)

Most emphatically the testing/quality assurance should be always
undertaken by someone other than the developer. A methodology that the
developer considers as 'safe' may in fact not be such, also they can
very easily miss potential problems, especially those that arise from
failing to consider a relevant issue.

By having a second person look at the code for QA/Test purposes much is
to be gained in terms of having a 'second opinion'.

++++ Moderator Comment ++++

  I absolutely agree on the value of having someone other than
  the original developer look at a program!

++++ End Moderator Comment ++++

As to a separate department. Routine development will on most systems
with most languages, involve some consideration of potential faults and
problems. However it will not normally involve detailed consideration
of what can be very specialist areas of expertise. Take for instance
that problematic area for many C++ developers, C++ Exception Handling
to which we add, for Windows 32 developers, Structured Exception
Handling. These are areas in which even very experience implementation
staff can make mistakes that are difficult to detect and resolve.

By having a specialist QA/Test department the staff can spend more time
gaining the specialist knowledge required to effectively review these
matters. It is also, in my view, possible to go further and develop a
team where individual members have given areas of responsibility, be it
formally or informally. Formally would involve team members having the
responsibility to review in detail given areas or aspects of the code,
which may well though prove dull for many people (It also presents a
potential recruitment problem if staff feel their narrow knowledge is
making them less commercial). An informal approach would in fact be
almost unavoidable, as team members will consult with one another when
thy feel a peer can provide a more experience view than their own.

Mind you I am biased as I am employed in a QA/Test role!

++++ Moderator Comment ++++

  I'm certainly not suggesting that we not have QA/Test engineers!

  Rather, I've found that having a separate QA/Test department can lead
  to a "us versus them" environment that was not so effective as a
  project approach, wherein the developers and the QA engineers worked
  together as a team.

  My sample sizes are very small (3 companies with separate departments,
  2 with integrated teams), so my experiences may not scale, but:

  I found it much more effective to have QA/Test engineers whose focus/
  loyalty was to the project, rather than to the department.  There was
  no friction whatsoever between QA and development in the project team
  companies, and much better communication.

  What are other peoples experiences?  If Gerry Weinberg is still
  listening, you must have seen lots of companies -- do you see any
  difference in effectiveness based upon the structure of the QA/Test
  engineers?

++++ End Moderator Comment ++++



===== NEW POST(S) =====

++++ New Post, New Topic ++++

From: "Niall Hammond" 
Subject: A Question

Terry, I realise the following are not directly quality issues but they
are management issues that would I believe have a strong bearing on
quality, are they suitable points for the mailing?

Tenberry I note offer a guarantee to fix forever, for free, any bugs
found in their software.

Should this be something that other corporations adopt?

Also to aid the corporation in adopting it should the staff be paid a
bonus for each week they spend on writing fresh code, or truly
enhancing existing code, with the test department being paid the same
for testing fresh, or enhanced code.

So if a developer is paid $3,000 per month, they may receive an
additional $250 per week for each complete week spent writing fresh
code. The same for the test team (though with an extra zero
perhaps). For weeks when the developers code is having a bug fixed
and also weeks when that bug fix is being tested, no bonus would be
paid to either the developer or the tester that signed the code off.

This would reflect the fact that new functionality gave the corporation
added value and should soon discourage the developer and tester letting
questionable things slip by.


++++ Moderator Comment ++++

  A very interesting idea!

  First, I think that *anything* which improves the shipping result is
  fair game.  Further, I think that management issues often have the
  most leverage towards improving quality.  Most of Jerry Weinberg's
  posts and many of my comments or David Bennett's comments are
  management issues.

  Secondly, a clarification:  we only guarantee new software forever,
  not all of our software -- most of it was created in a way that made
  enough support problems to turn me into a quality convert, but was
  not created with our process, which we only formalized about two years
  ago.  It's very hard to retrofit defect-free into old code (or at
  least, into our old code! :)

  The good news is that we have done two medium projects using our
  defect-free methods, and there have been no defects reported so far!
  The projects are too small and don't have wide enough distribution to
  make a compelling example, but the early returns are good! :)

  So far, we hadn't considered using compensation as a tool for defect-
  free, other than rewarding the "good ones".  I see a big problem
  with your proposal:  what do you do with code written and tested by
  engineers who no longer work at your organization?  The scheme you
  propose will make it hard to find someone willing to fix problems in
  someone else's code, since they would take a financial hit as well as
  the normal unpleasantness.

**** Moderator question: *****

  Can you so directly reward defect free production?

  Would your engineers revolt?

=============================================================
The Software-Quality Digest is edited by:
Terry Colligan, Moderator.      mailto:moderator@tenberry.com

And published by:
Tenberry Software, Inc.               http://www.tenberry.com

Information about this list is at our web site,
maintained by Tenberry Software:
    http://www.tenberry.com/softqual

To post a message ==> mailto:software-quality@tenberry.com

To subscribe ==> mailto:software-quality-join@tenberry.com

To unsubscribe ==> mailto:software-quality-leave@tenberry.com

Suggestions and comments ==> mailto:moderator@tenberry.com

=============  End of Software-Quality Digest ===============

[Tenberry] * [Software-Quality] * [Zero Defects] * [Automated QA] * [Consulting] * [Site Map]
Last modified 1998.7.6. Your questions, comments, and feedback are welcome.