Tenberry Can Automate Windows Software Quality Control!
We at Tenberry know how to develop and maintain reliable, high performance software that runs in every environment. Tenberry technology is run on millions of PCs every day. Many commercial PC products run on our technology across DOS and all Windows variants. Thousands of business products and almost all games, built by over ten thousand customers, utilize our technology for memory access & management, execution controls, and cross-operating systems support.
We try to use automated Windows QC tools:
When we started work on the Windows version of our InstantC development environment, we looked around for tools to use in the QC process we had planned. In our DOS product, we rely heavily on automated QC processes, where the machine tests the software and reports on failures automatically. In our DOS development, we became used to running our whole QC suite against every internal build. That way developers got immediate feedback helpful to them, and management got a good handle on the current state of the software.
We evaluated all of the Windows QC tools that were available, and chose the "best" product, which is one of the leaders in the Windows QC tools market. When our software was far enough along to start testing, we tried to use the purchased QC tool. We found three problems:
We discover our experience is not unique:
While we were trying to automate our testing with the Windows QC tool, we happened to attend a meeting of NESQCF, the New England Quality Control Forum, a Boston group of QC and development professionals. At this meeting the speaker asked for a show of hands to indicate how many companies were doing automated QC. About 30 hands went up, representing almost every company there. I put my hand up, figuring we were trying in Windows, and were succeeding in DOS.
Then the speaker asked how many had at least 10% of their tests automated. Half the hands went down. I kept mine up, deciding that I was reporting on our DOS work, since the Windows project was just getting started.
Then the speaker, trying to make a point, asked how many had at least 20% of their tests automated. All hands but mine and one other's went down. The other company dropped out at 50%. Since we had over 7,000 tests and fewer than 60 were not automated, our percentage of tests automated was over 99%. (It turned out the one company at 50% also had a DOS product.)
To summarize, out of about 30 companies (25 Windows, 3 UNIX/X-Windows, 2 DOS) attending this NESQCF meeting,
Now, it's obvious that this isn't a statistically valid study. Nonetheless, the results are striking: Windows QC tools don't seem to help automate testing!
I was very troubled by the above results (and particularly what they meant to our own Windows QC plans.)
So, why don't Windows QC tools work to automate testing?
The short answer: it's too hard and Windows doesn't help.
A longer answer: the Windows facilities (called "hooks") that QC programs use to drive a program under test are too low level. What they see and can record is screen pixels and mouse movement messages -- and lots more, but all of it low level.
The problems include:
We try a different approach:
Because of the difficulties and costs associated with our Windows testing, we decided to try a different approach. Instead of using a general Windows QC tool, we would try to build the necessary parts of a QC tool into our application. This is the approach we used for our DOS QC work.
The automated testing software consists of four parts:
This approach requires development work, to record the low-level messages and application synchronization points, to log the state of the application, and to provide a way for playing back the recorded messages.
In addition to logging low-level messages, we log state changes in our application, not just dumps of whole windows. Because we are logging logical information, we gain a lot of benefits:
We set several programmers to work, and now (a year later), we are beginning to see the benefits.
Windows automated QC that actually works!
We have just finished the first pass through our QC suites trying to automate the tests using all of our new software. So far, the results are exciting -- we are automating more than 50% of our tests. As we continue to fill out the missing parts of the recording, playback and logging software, we expect to reach over 90% automated testing by the end of 1997.
We have discovered a number of small, transitory quirks and problems that we have fixed.
We also find that we can run exactly the same automated tests on Windows 3.1, Windows for Workgroups, Windows NT, and Windows 95.
Can anyone use these techniques?
Yes, we believe that these techniques are applicable to any application, although the details will differ somewhat for each application.
It does require changes to the software to be QC'd. In particular, you will need to add recording, playback and logging functionality to your software. You will also need some way to drive your software with the test streams.
How does an automated QC project work? Who does the work?
Because doing automated QC this way requires updating your software, it normally involves your development staff and ours working together. We can do all of the work, but some parts (particularly logging) are best done by, or at least working closely with, your development staff.
We can provide expertise and code that can be used to implement the recording and playback subsystems. We can also provide our AutoTest driver program to actually run the automated testing.
Automated Quality Control sounds fabulous -- how do we proceed?
To start your process towards truly automated QC, contact our sales department at the address below. You'll be glad you did!