Software QA FYI - SQAFYI

One suite of automated tests: examining the unit/functional divide

By: Geoffrey Bache , Emily Bache

Extreme Programming (XP) as written [1] prescribes doing and automating both unit and functional testing. Our experiences lead us to believe that these two sorts of testing lie at two ends of a more or less continuous scale, and that it can be desirable to instead run a XP project with just one test suite, occupying the middle ground between unit and functional. We believe that this testing approach offers most of the advantages of a standard XP testing approach, in a simpler way. This report explains what we have done, and our theory as to why it works.

When we introduced XP at Carmen Systems, the worst problem with our development process was not our testing procedures being out of control. We already had automated testing, though not along the lines outlined by Beck, Jeffries et al [1, 2]. Following the advice to “Solve your worst problem first”, we began introducing other aspects of XP, expecting that at some point testing would become our “worst problem” and we would start needing separate unit and functional test suites. That never seemed to happen - we have been doing all the other XP practices in 2 projects for 18 months or so, and our style of automated testing has not only not become a problem, but in fact a great success that seems to fit very well with the rest of XP.

The automated tests we have are perhaps best explained as “pragmatic acceptance tests” - we run the system as closely as possible to the way the customer will run it, while being prepared to break it into subsystems in order to allow fast, easily automatable testing. The overall effect is that the tests are owned by the customer, while being just about fast enough to be run by the developer as part of the minute by minute code-build-test cycle.

What we have created is an application independent automatic testing framework written in Bourne

shell and Python. The framework allows you to create and store test cases in suites, runs them in parallel over a network, and reports results. For each test case the framework provides stored input data to the tested program via options or standard input redirects. As it runs, the tested program produces output as text or text-convertible files. When it has finished, the testing framework then compares (using UNIX “diff”) this output to versioncontrolled “standard” results. Any difference at all10 is treated as a failure. In addition, the framework measures the performance of the test, and if it strays outside pre-set limits, (for example if it takes too long to execute) this is also recorded as failure.

New tests are added by providing new input options and running the system once to record the standard behaviour against which future runs will be measured. This behaviour is carefully checked by the customer, so that s/he has confidence the test is correct. Once verified, the new test case (ie input and expected results) is checked into version control with the others.

Full article...

Other Resource

... to read more articles, visit

One suite of automated tests: examining the unit/functional divide