Tenberry Can Automate Windows Software Quality Control!

We at Tenberry know how to develop and maintain reliable, high performance software that runs in every environment. Tenberry technology is run on millions of PCs every day. Many commercial PC products run on our technology across DOS and all Windows variants. Thousands of business products and almost all games, built by over ten thousand customers, utilize our technology for memory access & management, execution controls, and cross-operating systems support.

We try to use automated Windows QC tools:

When we started work on the Windows version of our InstantC development environment, we looked around for tools to use in the QC process we had planned. In our DOS product, we rely heavily on automated QC processes, where the machine tests the software and reports on failures automatically. In our DOS development, we became used to running our whole QC suite against every internal build. That way developers got immediate feedback helpful to them, and management got a good handle on the current state of the software.

We evaluated all of the Windows QC tools that were available, and chose the "best" product, which is one of the leaders in the Windows QC tools market. When our software was far enough along to start testing, we tried to use the purchased QC tool. We found three problems:

• Preparing each test took an hour or more. Since we were expecting to have 10,000 tests (our DOS product has almost 8,000), we were looking at five person years, just to get the same level of automated testing we already had in DOS!
• The tests weren't reproducible between machines. In fact, because of the low-level capture of bit maps, the tests often weren't reproducible on the same machine.
• The QC tool required a fair amount of learning to use effectively. In DOS, each developer was responsible for creating the QC tests for their change. In Windows, it seemed unlikely that the developers would be able to keep sufficient facility with the QC tool to create any tests. There was also the per-developer cost of the Windows QC tool, just to make things really painful.

We discover our experience is not unique:

While we were trying to automate our testing with the Windows QC tool, we happened to attend a meeting of NESQCF, the New England Quality Control Forum, a Boston group of QC and development professionals. At this meeting the speaker asked for a show of hands to indicate how many companies were doing automated QC. About 30 hands went up, representing almost every company there. I put my hand up, figuring we were trying in Windows, and were succeeding in DOS.

Then the speaker asked how many had at least 10% of their tests automated. Half the hands went down. I kept mine up, deciding that I was reporting on our DOS work, since the Windows project was just getting started.

Then the speaker, trying to make a point, asked how many had at least 20% of their tests automated. All hands but mine and one other's went down. The other company dropped out at 50%. Since we had over 7,000 tests and fewer than 60 were not automated, our percentage of tests automated was over 99%. (It turned out the one company at 50% also had a DOS product.)

To summarize, out of about 30 companies (25 Windows, 3 UNIX/X-Windows, 2 DOS) attending this NESQCF meeting,

• 14 had less than 10% of their QC automated (mostly Windows apps)
• 28 had less than 20% of their QC automated (mostly Windows apps)
• 1 had about 50% of their QC automated (a DOS app)
• 1 (Tenberry) had 99% of their QC automated (a DOS app)

Now, it's obvious that this isn't a statistically valid study. Nonetheless, the results are striking: Windows QC tools don't seem to help automate testing!

I was very troubled by the above results (and particularly what they meant to our own Windows QC plans.)

So, why don't Windows QC tools work to automate testing?

The short answer: it's too hard and Windows doesn't help.

A longer answer: the Windows facilities (called "hooks") that QC programs use to drive a program under test are too low level. What they see and can record is screen pixels and mouse movement messages -- and lots more, but all of it low level.

The problems include:

• The data is too low level -- it's in terms of low-level Windows messages and screen pixels.
• The data is machine specific -- many of the Windows messages contain pixel-level addresses, and the screen data varies with the type of color display. All data varies with where Windows decides to place your windows, available fonts, etc.
• The data is system specific -- if you record a test in Win3.1 and then try to run the test under Windows 95, the shapes, sizes, and pixel contents of menus, message boxes, dialog boxes, etc. will all be different.
• There is too much data -- each window image captured can take 250,000 to 750,000 bytes on a 256 color display. Each mouse movement results in hundreds, if not thousands of Windows messages.
• There is too much noise in the data -- blinking carets can cause different images to be recorded; often we don't care about what font is chosen, or the size of a dialog box, or the positioning of a message box.
• Much of the data is irrelevant -- so the person using the QC tool has to be smart enough to record enough data to verify the behavior, while not recording irrelevant data that will increase the likelihood of spurious mismatches.

We try a different approach:

Because of the difficulties and costs associated with our Windows testing, we decided to try a different approach. Instead of using a general Windows QC tool, we would try to build the necessary parts of a QC tool into our application. This is the approach we used for our DOS QC work.

The automated testing software consists of four parts:

• A recording subsystem -- added as part of the application. The recording subsystem is responsible for recording low-level messages and significant events in a human-readable event file.
• A playback subsystem -- added as part of the application. The playback subsystem is responsible for reading the event file and driving the application.
• A logging subsystem -- added as part of the application. The logging subsystem is responsible for recording, again in human-readable form, the significant states and windows of the application. The logging subsystem will most likely touch more of the application than the other two subsystems.
• An automated tester program. This separate program executes your program, feeding it each test recorded in turn, and comparing the created log files with master files.

This approach requires development work, to record the low-level messages and application synchronization points, to log the state of the application, and to provide a way for playing back the recorded messages.

In addition to logging low-level messages, we log state changes in our application, not just dumps of whole windows. Because we are logging logical information, we gain a lot of benefits:

• The log files are much smaller.
• The log files are written as text files, so that normal text tools (editors, difference programs, Version Control Systems, etc.) can work on them.
• Because we decide once what's important to log during development, the logging is all automatic. This makes it much easier for programmers and QC engineers to create tests.
• The log files are easily readable by both programmers and QC engineers.
• The log files show transitory errors that might not be caught by other means.

We set several programmers to work, and now (a year later), we are beginning to see the benefits.

Windows automated QC that actually works!

We have just finished the first pass through our QC suites trying to automate the tests using all of our new software. So far, the results are exciting -- we are automating more than 50% of our tests. As we continue to fill out the missing parts of the recording, playback and logging software, we expect to reach over 90% automated testing by the end of 1997.

We have discovered a number of small, transitory quirks and problems that we have fixed.

We also find that we can run exactly the same automated tests on Windows 3.1, Windows for Workgroups, Windows NT, and Windows 95.

Can anyone use these techniques?

Yes, we believe that these techniques are applicable to any application, although the details will differ somewhat for each application.

It does require changes to the software to be QC'd. In particular, you will need to add recording, playback and logging functionality to your software. You will also need some way to drive your software with the test streams.

How does an automated QC project work? Who does the work?

Because doing automated QC this way requires updating your software, it normally involves your development staff and ours working together. We can do all of the work, but some parts (particularly logging) are best done by, or at least working closely with, your development staff.

We can provide expertise and code that can be used to implement the recording and playback subsystems. We can also provide our AutoTest driver program to actually run the automated testing.

Automated Quality Control sounds fabulous -- how do we proceed?

To start your process towards truly automated QC, contact our sales department at the address below. You'll be glad you did!

Sales Department
Tenberry Software, Inc.
P.O.Box 20050, Fountain Hills, AZ 85269-0050, USA
1.480.767.8709 fax
1.480.767.8868 phone
sales@tenberry.com