Software-Quality Discussion List
Digest # 014


Tenberry Home
Software-Quality Home
Reducing Defects
Automated QA
Consulting
Site Index

============================================================
           Software-Quality Discussion List

      S O F T W A R E - Q U A L I T Y   D I G E S T

      "Cost-Effective Quality Techniques that Work"
============================================================
List Moderator:                      Supported by:
Terry Colligan                       Tenberry Software, Inc.
moderator@tenberry.com               http://www.tenberry.com
============================================================
April 4, 1998                        Digest # 014
============================================================

====IN THIS DIGEST=====

    ==== MODERATOR'S MESSAGE ====


    ==== CONTINUING ====

    Re: # 013: saving megabytes
      Jerry Weinberg 

    Response to John McGrail
      David Bennett 

    Re: attempts at controversial statements
      John McGrail 

    OS testing ...
      markw@ncube.com (Mark Wiley)


    ===== NEW POST(S) =====

    Debug vs production builds; reverse-engineerability
      David Bennett 


    ==== BOOK REVIEW ====

    (none)   (What?! Again? ;-)



==== MODERATOR'S MESSAGE ====


  I was following a thread in the comp.software.testing newsgroup
  when something snapped and I was compelled to write the following
  comment.  I thought the readers of SQD might find it interesting:

  

  In response to a discussion started by Steve Watson, a number of
  posters have expressed opinions that I think are self-fulfilling
  and non-productive:

   - Everyone hates QA.

   - QA has low status in the company.

   - Marketing does not understand anything about QA.

   - Developement despises QA.

   - Management doesn't have a commitment to QA.

   - Management only cares about money.

  There was wide-spread agreement, and several folks suggested
  that the only way out of this is to quit and change to a
  different industry and/or profession.

  I make periodic presentations to local quality groups, and often
  here the same sentiments expressed during my talks.

  I suggest that the problem lies with us, the testing/QA/quality
  folks, not with management/development/marketing!  *We* made
  this mess, and *we* can (and should!) fix it.

  Let me explain...

  Management
  ----------

  I have been selling tools and technologies that I created to
  other PC-oriented software companies for more than 15 years (as
  Rational Systems, renamed a few years ago to Tenberry Software.)
  During that time I have met hundreds of upper management,
  development managers and marketing managers.

  Contrary to the myths above, *all* of these managers were
  bright, competent folks who wanted to do the right thing!
  (Well, OK, there was one CEO, but his company no longer exists!
  ;-) If you make any these managers chose between two
  alternatives, they will chose the more advantageous alternative,
  100% of the time!

  The problem is that, generally, the Quality Folks don't
  understand what management's job is.

  Top management's job to insure and increase profits!  This is
  done by increasing revenues or decreasing costs or, hopefully,
  both.  I've seen lots of complaints by Quality Folks that their
  management doesn't understand/value/support them.  I rarely see
  in this newsgroup or any other forum where Quality Folks hang
  out *any* discussion at all about how implementing some quality
  program reduces costs, and I've never seen a discussion of how
  quality can increase revenues!

  Why is this?

  Do Quality Folks not understand the idea of profit? -- quite the
  contrary, this group is way above average intelligence and can
  easily grasp these simple ideas.

  Do Quality Folks not care about the success of their
  organizations? -- quite the contrary!  In my experience, Quality
  Folks are more likely to care passionately about the success of
  their companies! They like to feed their families!

  Do Quality Folks not think much about profits?  I think this is
  a factor for many, but not all.

  Do Quality Folks not believe that quality saves money?  Bingo! I
  think this is most often the case, even though industry
  experience (and my own) and the few academic papers that address
  this all strongly point to a direct relationship between high
  quality and low cost!

  When I give talks suggesting that we should make very low defect
  rates an important goal, I am often strongly attacked by Quality
  Folks -- more so than from any other group.  I think this
  feeling is so strongly (although incorrectly) held for two
  reasons:

    1) Any change seems more expensive than the status quo, and it
       is, if you look on a short enough time horizon.

    2) People instinctively use their experience in the physical
       world, where there is some justification for higher quality
       costing more.  For example, a Mercedes is perceived as
       being higher quality than a Yugo.  It's certainly more
       expensive, and a lot of that additional costs comes from
       more, stronger and heavier metals and other materials.
       However, when we our experiences with cars to software
       development costs, the beliefs are misleading.

  Why do companies have QA departments and Quality Folks at all?
  Because someone in management believes that having them will
  either reduce costs or increase revenues!  If you don't believe
  this, think about how many companies have departments of Art
  History, Basketball or Classical Guitar?

  So, the management has committed funds to quality (otherwise you
  wouldn't have a job!), but most Quality Folks are still
  complaining because "Management doesn't understand/support/value
  quality".  Rather, I submit, it is that Quality Folks don't
  understand/support/value management! This complaining seems, I
  suspect, to management much like the complaining of artists that
  the government won't fund their work!


  What can you do?

  Rather than complain about how stupid or short-sighted your
  management is, develop an explanation of how your quality
  programs either reduce costs, for example, by reducing
  maintenance and customer support, or increase revenues, for
  example by increasing customer satisfaction and repeat business.
  If you can't make a case that your quality saves money or
  increases revenues, change your quality program until it does!


  Marketing
  ---------

  (Repeat the above diatribe, substituting "customer satisfaction"
  for "increased profits".)


  Development
  -----------

  (Repeat the above diatribe, substituting "reduced delivery
  times" and/or "more reliable schedules" for "increased profits"


  Why this is your job?
  ---------------------

  First reason:

  They don't teach software QA in any MBA program that I'm aware
  of!  Nor, unfortunately, is it taught in any computer science
  program!  You have a better chance of understanding their goals
  than they will of yours.

  Second reason:

  You can do something about it -- by explaining and recasting, if
  necessary, your programs -- you are in control.

  If you leave it to the others to resolve, you are defining
  yourself as a helpless victim, with nothing to do but to be
  frustrated and to complain.

  

  Comments, anyone?



==== CONTINUING ====

++++ New Post, New Topic ++++

Subject: Re: # 013: saving megabytes
From: Jerry Weinberg 

>
>++++ Moderator Comment ++++
>  I may never live down my "It's only a megabyte comment"!
>
>  As a defense, I would say that I don't believe or recommend
>  that you or anyone should spend a megabyte for quality, just
>  because you can.  I do believe and recommend that you should
>  spend an economically justifiable amount on code to aid
>  quality, and that could be
>
>  Rather I was trying to give an example that spending lots
>  (or what seems like lots) of resources on testing, logging,
>  and other quality-specific code might be a good economic
>  trade-off.  I chose a megabyte because I thought it would
>  both seem wantonly wasteful to people, while actually only
>  costing $3 to $10 per system.

I think the correct principle is, again, understandability.  If
you do unnecessary code optimization (for speed or space or simple
cuteness), the code becomes more difficult to relate to the
specifications, which in turn makes it easier to do wrong and
harder to test.  The result is less reliability.

"Optimized" code is also more difficult to modify, which leads to
less reliability in subsequent upgrades.

But, if you have a *real* requirement for space or speed (like
each megabyte reduces your market share by 20%), meet it, but look
first to your design, not your code.  You can seldom find large
space savings by tweaking code, though sometimes you can get huge
speed savings by analyzing performance and changing one place in
the code.  After all, there can't be more than one place that uses
over 50% of the time.

To avoid cuteness, give each developer some play time and a
playspace - not in the product - in which to experiment with "cool
coding tricks."

Jerry
website = http://www.geraldmweinberg.com
email = hardpretzel@earthlink.net

++++ Moderator Comment ++++

  While you make valid and useful comments (as usual! :) about
  optimization, I think you are counting too much on
  understandability.  Two points:

   - Code can easily be understandable, but wrong.  If the algorithm
     or assumptions are wrong/inappropriate, all the understanding
     in the world won't help.  (Think of a bubble sort of a hundred-
     million element file.)

   - While understandability is positively correlated with many
     positive attributes (maintainability, reliability, etc.),
     it's merely a correlation -- being understandable doesn't
     guarantee any of the other properties.

  I use the following principles:

    Most important:  meeting specifications & defect-free (in case
       defects weren't mentioned in specification.)

    Important:  understandability, maintainability, robustness,
       testability, consistency, locality of concepts

    Nice-to-have: speedy implementation, small size, fast
       execution.  (Although, as Gerry points out, these may well
       be part of the requirements.)



++++ New Post, New Topic ++++

From: David Bennett 
Subject: Response to John McGrail

Hi all!

Frankly, I was disappointed by John McGrail's response.  My
initial response was annoyance, and a desire to leap into action
(dare I say it, flaming action).  But then I had second thoughts.

I think John is raising valid issues, but they are issues
concerning the cellular structure of the lichen on the bark on the
trees.  He's the poor guy getting stuck with trying to do the
impossible, and not enjoying it much from the sound of things.

The difference is, I run this company.  When I say *must* I mean
"this better happen or serious questions related to future career
choices could arise".  More reasonably it means "please do this if
at all possible within reasonable economic and time constraints;
otherwise refer to your superior".

* >2. Any defect detected in QA testing *must* be incorporated
* >into a process of automated testing in such a way that the
* >chances of that same defect appearing and not being detected
* >in a future QA testing of that product is less than 1 in
* >1,000,000.
* I disagree with the word "*must*".  While it is a lofty
* goal, it is unachievable.  For one thing, the cost of
* automating certain tests can be prohibitive.  For example, a
* small company might not be able to afford all the X.10 equipment
* needed to automate a test that power-cycles several machines to
* verify a distributed database recovers properly from a brownout
* just as it is beginning the 2nd commit in a two phase commit.

If it was picked up in QA once, we can do it again.  If we can't,
we have serious questions to ask.

If we can't prove that our distributed database recovers properly
in certain scenarios we should say so.  With software, we always
have the option to fix it, or to document it (or both).  We escape
from the above problem by simply documenting this as unsupported
or undefined behaviour, and letting the customer decide whether
they still want to buy it.

I repeat, you cannot and *must* not specify something as working
and then fail to prove it in QA.  That's dishonest and
unprofessional.  It can also get you into trouble.

* While I agree with the last sentence, I disagree with "*must*"
* again for pretty much the same reasons as numbers 2 and 3.  It is
* a lofty goal.  But, time, money, and resources will prevent it
* from ever being achieved. Assuming there is software between the
* button and the bomb, how do you truly automate the test where
* pushing the right button in the control room triggers the nuclear
* bomb to take off?

Accept it as a challenge, rather than a self-fulfilling prophecy
of failure.

a) The last bit (the bomb) is obviously mechanical.  Use a
mechanical test harness (leave the pin in the grenade, or don't
put in the fissionable material, or something).

b) Simulate it (before you get negative about that too, remember
that any testing is better than no testing)

c) Change the spec.  Document the limits, what is tested and what
not and estimate the chance of failure.  I read somewhere an
estimate that not more than about 80% of American missile and 50%
of Russian missiles would work, and each side knew that.  50% of a
hell of a lot of missiles is still a hell of a lot of missiles.
Uncertainties in QA didn't stop the generals buying missiles.

I know this is getting boring, but I pretty well agree with Terry.

Regards
David Bennett
[ POWERflex Corporation     Developers of PFXplus ]
[ Tel:  +61-3-9888-5833     Fax:  +61-3-9888-5451 ]
[ E-mail: sales@pfxcorp.com   support@pfxcorp.com ]
[ Web home: www.pfxcorp.com   me: dmb@pfxcorp.com ]

++++ Moderator Comment ++++

  Actually, it seems like David and John don't disagree about
  much of anything except how to word the concepts!

  But, maybe not:



++++ New Post, Same Topic ++++

From: John McGrail 
Subject: Re: attempts at controversial statements

Hi,
>  First, let me apologize for even unintentionally sullying
>  your reputation!  It's obvious that you have a lot to offer!

No need to apologize, I didn't take it as my reputation being
sullied ;->  But it was an important point to get across because
you are correct that many people have the reject it as a goal if
it can never be achieved attitude. With just a little bit of word
changing, you can often get the rejecters to come on board.

>  Third, let me re-phrase my question:  Why are you so opposed
>  to David Bennett's use of the word "MUST" in stating his
>  software-quality goals/techniques?
>  What harm comes from striving for them?

But, David did not say he was stating his goals and techniques.
He just threw out some statements.  As a statement, I can't agree
that (for example) all fixed customer-reported bugs must be
automated - for one, the cost may outweigh the benefit gained by
automating the test.

I feel much happier and will work harder to achieve a goal stated
like:
   Where reasonable, automated test cases should be created
   for fixed customer bugs.
rather than:
   Automated test cases must be created for all fixed customer bugs.

In the latter statement, I have no control over my destiny and
could see myself automating bug fixes for the next 3 years.  But,
with the former statement, I have some control over the decision
about what does and doesn't get automated making me feel part of
the team, etc.  With the latter goal, getting some, but not all
bugs automated and I can see I've met the goal.  With the former,
I can't say the same until I've automated everything.

In general I'm hesitant whenever I see words like "must" attached
to an open-ended commitment.  For me, must is reserved for
well-known commitments.  For example:
    All smoke tests must be run against one-off CDs, and must
    pass, before shipping .
What I mean by open ended commitment vs well-known commitment is
... In project mgmt land, the amount of time to run the smoke
tests is probably well known.  And, the amount of one-offs needed
initially is probably also well known.  Thus you can compute with
reasonable accuracy how long it will take to ship once the master
CD is cut (assuming no problems with the smoke-tests).

But, "all fixed customer defects must be automated" can be a black
hole. Since it might take an hour to automate 1 bug and 3 weeks to
automate another and since more problems might get fixed at any
time, you can't accurately predict how long it will take to
complete.

John
=====---------------------------------------------------=====
John McGrail
American Internet Corporation - http://www.american.com/
mailto:jmcgrail@american.com
1 (781) 276-4568

++++ Moderator Comment ++++

  Well, I thought you, David and I agreed!  I still think we do
  on the principles.  However, I find your phrasing a bit too weak.
  When I used similar words in my company, I didn't get the same
  level of testing that I did (no visible difference in costs)
  that I did when I used words like David's.

  Whatever the words used, it should be clear:

    that failure to automate will be an element of performance
    reviews (although probably not the largest);

    that it's a management decision not to automate any particular
    test; (although QA recommendations will likely be followed) and

    that it's neither required nor desired to spend months
    automating a single test. (although it might be)



++++ New Post, New Topic ++++

From: markw@ncube.com (Mark Wiley)
Subject: OS testing ...

> From: John Cameron 
> Subject: Re: OS Testing
>
> >In order for the testing to have maximum validity, it must
> >run on the version of the OS we ship, rather than an
> >instrumented version with hooks for special testing. This
> >means that our testing relies on the execution of tests in
> >user space, which mimic aspects of the the applications
> >normally run on our system.
> >
>
> The embedded stuff we ship is the instrumented version so
> there is no difference between the code we debug and
> the code that drives our machines.  That has to be an
> advantage in assuring correctness.

Yes, it's best to ship what you test. In our case that precludes
shipping the fully instrumented version of the kernel, as the
space and time penalty would be too large (this is not a PC based
system and we just can't put in more memory without a board
change).

>
> >The above problems tend to be load sensitive and
> >intermittent in nature. So, I run different mixes of the
> >small test applications to generate a high load on the
> >system. That tends to flush out problems with interrupts
> >handling in the devices that are tested, as well as other
> >problems.
> >
>
> Are your test repeatable?  If not how do you tell if an
> error occurred?  Seems like it would be very easy to get
> a bad location on a disk or to mismap memory without
> noticing or to fire off some process at the wrong priority
> without noticing.  Are you relying on the results of the
> applications as your only feedback?   Hate to be a pest,
> but am always looking for new ideas.

The tests are fully automated and are as repeatable as they can be
(there are always some random timing fluctuations based on a
variety of factors).

I do not use the customer applications to test the system (SI does
that), I have specialized applications that focus on testing
various areas of the system (disk, TCP/IP, etc.). In saying that
they are applications, I only meant that they run in user, and not
kernel space.

You're not being a pest, I always like exchanging info with other
testers.

Markw



===== NEW POST(S) =====

++++ New Post, New Topic ++++

From: David Bennett 
Subject: Debug vs production builds; reverse-engineerability

Hi all!

Two small topics for thinking about.

1. Debug vs production builds
We do all development and unit testing with debug builds.  This is
C/C++, optimisation can make a real difference to performance (but
can introduce subtle bugs).  The debugger needs a symbol table and
is a real pain to use on optimised code.

QA is done on the optimised runtime, but many of the asserts have
been removed (see next item).  Just occasionally we find a
difference related to the difference in builds (maybe once or
twice a year).

Note: Microsoft does the same (retail vs checked builds).

Question to those who say develop with what your customers use:
you either have to ship a debuggable executable, or debug with
poor tools.  Which do you recommend?

2. Reverse-engineerability
We work in a competitive environment, and from time to time we
have become aware of people reverse-engineering our software.
This could be to break the user counting, or for other reasons.

The presence of large amounts of tracing and assertions makes this
much easier to do, which is one of the reasons we remove it.

Question to those who leave tracing and checking code in shipped
product: doesn't this worry you at all?

Regards
David Bennett
[ POWERflex Corporation     Developers of PFXplus ]
[ Tel:  +61-3-9888-5833     Fax:  +61-3-9888-5451 ]
[ E-mail: sales@pfxcorp.com   support@pfxcorp.com ]
[ Web home: www.pfxcorp.com   me: dmb@pfxcorp.com ]

++++ Moderator Comment ++++

  On debug versus production builds:

  Clearly, we aren't going to ship a debug version with symbol
  table and full source code, at least not unless we are in the
  selling source business.

  As we neared a release date, we changed the definition of
  'debug' build to slowly add all of the optimizations, one per
  day. By the time we are close to release, there is no difference
  between the optimizations of the 'debug' and the 'release'
  versions.

  This has the added benefit that if any optimization breaks some
  code, you have a smaller range of things to look at to discover
  what's wrong.

  Similarly, we leave the asserts in the code, but change the
  implementation of the assertion failure action.

  But I was mostly talking about the debugging traces, dumping
  commands, etc., not the compiler-generated stuff.  I see little
  benefit to removing this code from the release version.


  On Reverse-Engineering:

  Obviously you have to protect your trade secrets.  However:

    - I don't see why tracing support makes it easier to reverse
      engineer.

    - I especially don't see how or why asserts have *any* affect
      on reverse engineering.

    - My experience is that users are surprisingly smart and that
      the few who will reverse-engineer things are easily daunted.
      We've had more than one person reverse-engineer the guts of
      the switching code of our DOS extender -- just so that they
      could do a better job of reporting a bug.

  My advise is:  don't worry about it, except that you might want
  your assert() macro to not stringize the expression.



==== BOOK REVIEW ====

  (Next Digest, for sure!)

=============================================================
The Software-Quality Digest is edited by:
Terry Colligan, Moderator.      mailto:moderator@tenberry.com

And published by:
Tenberry Software, Inc.               http://www.tenberry.com

Information about this list is at our web site,
maintained by Tenberry Software:
    http://www.tenberry.com/softqual

To post a message ==> mailto:software-quality@tenberry.com

To subscribe ==> mailto:software-quality-join@tenberry.com

To unsubscribe ==> mailto:software-quality-leave@tenberry.com

Suggestions and comments ==> mailto:moderator@tenberry.com

=============  End of Software-Quality Digest ===============

[Tenberry] * [Software-Quality] * [Zero Defects] * [Automated QA] * [Consulting] * [Site Map]
Last modified 1998.4.7. Your questions, comments, and feedback are welcome.