Automated testing is essential to guaranteeing quality in Java applications with complex GUIs. Examination of newly available test tools and harnesses illuminates the manner in which strategic choices regarding testing need to be related to the nature of the software being tested.
Testing Java GUIs
GUIs (Graphical User Interfaces) may be a dream for users, but can easily be a nightmare for testers. As Java development takes off, so does the need to test GUI applications written in Java. Below is a discussion of solutions we've adopted for testing Java GUI products within Sun Microsystems' WorkShop Products group.
Though automated test suites can never entirely eliminate the need for manual testing, they are invaluable as an efficient way to detect new problems quickly, make sure old problems remain fixed and verify that a package satisfies fundamental quality criteria prior to shipment.
For applications with command line interfaces, test automation can, as a rule, be accomplished readily with the help of I/O redirection, which feeds the application input from a file, rather than via the keyboard, and sends its output to one or more files rather than to the screen, in conjunction with a scripting language that makes it easy to perform initial setup, run individual tests and evaluate and report results.
Automated GUI testing, on the other hand, is as problematic as it is essential. Special tools are needed to simulate input, which ordinarily comes in through the graphical interface. Similarly, graphical output can be captured and analyzed only with special-purpose tools. For Java GUI applications, the need to use special testing tools is unusually problematic.
Because a major attraction of Java is its platform independence, testing Java applications (or applets) tends to require testing on multiple platforms. On the face of it, therefore, the tools used need to work on all targeted platforms.
There are two basic ways of creating the required sort of multi-platform Java GUI test tool. The first option, adding Java capability to a pre-existing non-Java tool, has the advantage of inheriting the strengths of the pre-existing tool. On the other hand, such a tool is not guaranteed to run on all platforms of interest in the future. The second option, of course, is to create, from scratch, a Java-based tool, which can be expected to migrate automatically to new platforms along with the application being tested.
JavaSTAR, a newly developed Java GUI test development tool, has the advantages one would expect from a tool written in Java. As a purely Java-oriented tool, JavaSTAR naturally is aware of AWT components, recognizing them in the same way regardless of platform.
Overall, JavaSTAR impressed us as being fundamentally the right kind of tool for the sort of testing we need to do. As it becomes better complemented by supporting tools, JavaSTAR promises to become the optimal solution for testing Java applets and applications.
Being able to construct a GUI test that runs without human intervention is a big step forward, but the reality of automated testing requires running collections of tests, with convenient mechanisms for controlling the set of tests to be run and the conditions under which the tests are run (such as the set of binaries
used). Consequently, having decided to use JavaSTAR to construct our
tests, our next challenge was to choose a harness to provide the necessary ability to manage the tests.
JavaTest, a harness written entirely in Java and provided by Sun with its Java API test suite - the JCK (Java Compatibility Kit) harness - from the outset presented two advantages. First, it had been in use for some time and thus was relatively mature. Second, we had already used it as a framework for a non-GUI benchmark test suite we had constructed, and we wanted to avoid a proliferation of harnesses which would complicate the job of running any arbitrary subset of our test suites.
JavaHarness is a general purpose harness for Java testing which will soon be made available by SunTest along with JavaSTAR and JavaScope, a Java coverage tool. Written in Java, JavaHarness reports subtest results, how golden file comparisons failed and provides filtering capabilities.
An important attraction of JavaHarness is that it enables us to use a single test harness to run all of our test suites. JavaHarness can function as a meta-harness as well as a harness, allowing it to start up a JavaTest test suite instead of, or in addition to, one or more JavaSTAR test suites. The main JavaHarness configuration file, JavaHarnessSetup.ini, can include an essentially unlimited number of sections, each containing specifications for running a particular test suite. For a JavaSTAR test suite, the specifications refer to a separate file containing a list of tests to be run. For a JavaTest test suite, the specifications point to the ".jtp" file containing the information JavaTest needs to set up and run a set of tests.
The all too familiar pattern with test harnesses is that each is developed with a particular sort of testing need in mind, limiting its usefulness for other sorts of testing. JavaHarness attempts to be more flexible, providing interfaces that allow users to readily tailor its capabilities to the particular kind of testing they wish to accomplish.
Though problematic, automated testing is essential to guarantee high quality in substantial Java applications with complex GUIs. Though tools to help automate Java GUI testing are still immature and doubtless will become much more attractive with future refinement, they have already developed to the point where they can provide significant benefit.
In choosing tools for Java GUI testing, it is important to consider whether testing will be confined solely to Java GUIs or whether there will be a mix of Java and non-Java user interfaces. For testing purely Java applications, tools written in Java are apt to be preferable due both to their direct awareness of AWT components and to the fact that their portability will automatically match that of the application being tested.
JavaSTAR and other SunTest testing tools can be downloaded from http://www.suntest.com.