Originally Posted by Banana
Does the software testing have to be done by a human? Can't a script do the testing and hang at failurewith a notification to the developer or something? Heck, program the script to do random things or pattern a normal usage with some random factors with a log of action taken, no?
What Brad said ...
Originally Posted by Brad
Oh, heavens yes. Most big software firms (and any reputable small software firm) should be using automated testing (unit tests, integration tests, etc.) from top to bottom.
However, tests are only as intelligent as the person who writes the tests. They can test all sorts of use cases and extreme edge cases, but they can't prod something they don't know to check.
... is spot-on. Both places I've worked, we've had huge suites of automated tests. But, like Brad says, they can't prod for things they don't know to check. We have to keep the scripts up-to-date. Also, there are two kinds of testing, really. One is API testing, to make your routines are fairly bulletproof. That's easy enough to code through unit testing and scripts. The other is UI testing.
UI testing is a very challenging beast, and difficult to automate reliably in many situations. Either (a) your code has to expose information about the UI programatically, so a script can "parse" the UI, or (b) you need a UI testing suite that is intelligent enough to look at raw graphics data and "find" controls. Because even though your functions might be fairly bulletproof, sometimes only the right sequence of UI events will cause an error. Regressions can be checked using scripts (i.e. make sure an old UI bug doesn't appear again), but it's hard/impossible to program a UI testing script to exhaustively check all possible input events, combinations, and timings
that might lead to problems (hangs, races, deadlocks, crashes, and so on).
That's why truly exhaustive testing is, and always will be, essentially an impossible feat. Part of that's because an exhaustive search of all UI events and timings would be a problem so big in size that no (super)computer in the world could conquer it in reasonable time. So to make use of the limited testing resources provided, we have to pick and choose our battles---that is, we run the tests deemed most important and most comprehensive.
Most of the time it works. Sometimes it doesn't.