HomeDigital EditionSys-Con RadioSearch Java Cd
Advanced Java AWT Book Reviews/Excerpts Client Server Corba Editorials Embedded Java Enterprise Java IDE's Industry Watch Integration Interviews Java Applet Java & Databases Java & Web Services Java Fundamentals Java Native Interface Java Servlets Java Beans J2ME Libraries .NET Object Orientation Observations/IMHO Product Reviews Scalability & Performance Security Server Side Source Code Straight Talking Swing Threads Using Java with others Wireless XML
 

Why Test Applets?
Why do we test an applet? What are we trying to show when we test? This article is addressed to the developer/tester, someone who is testing their own Applets. Nonetheless, all but a very few of the observations and techniques apply to the problem of testing someone else's code. The largest difference is probably the emotion upon finding a bug.

An Applet is a Java class. When we test it, we are trying to show that it is a good Applet; specifically, that it has certain good properties, and lacks certain bad properties. The important properties of an applet can be categorized under two broad headings: functionality and portability.

Functionality
We wrote the applet to do something; perhaps as simple as displaying bouncing text, perhaps as complex as querying a remote database for molecule structures and displaying those in 3D.

Our testing should find out if our applet class, together with the other classes that make up our Applet program, does what we wrote it to do. There is no general rule for this; functionality testing depends on the function our applet performs. The minimum standard for our class, that it extends the java.applet.Applet class, is checked by the Java compiler.

Portability
Because our class is an Applet, we can expect it to run in many different environments. This includes, as a minimum, the various Java-enabled browsers. It also may include various states of connectivity, including various link speeds and firewalls, and various security situations. Applet testing should check for portability among those environments, which may fail due to either non-portable usage or to bugs in the Java implementation. Non-portable Usage
There are some things in the Java language and libraries that are intended for use by application or system implementors and should not be used in applets. For example, direct file access is not generally a good thing to do in an Applet, and no portable Java program should directly access an AWT peer class. Language Implementation Choices
In some cases, the Java language specification (like any specification) gives freedom of choice to the implementor. One example is the thread scheduling policy; another is the legal size of file names, or the number of pixels on the screen. Language Implementation Bugs
There are some bugs in current Java implementations. We can expect these to be eradicated in time (the JDK 1.1 test suite has over six thousand tests), but in the meantime, for best portability, your applet should avoid those bugs. JavaSoft (http://java.sun.com) publishes a list of known bugs that may be helpful.

Testing
Testing is the art of showing that a program has certain properties by looking at the executable (in the case of Java, the class file or files).

Narrowly defined, testing is the art of running a program with some controlled input under some kind of observation scheme in order to show that the program has certain properties. With Java programs, it is possible instead to determine certain properties of a program without running it, by inspecting the class files. For example, if the class file contains no references to any methods of any classes of the Java.io package, then the program will not do I/O.

When possible, a static check is much simpler and more reliable than a dynamic test. However, there are many properties that cannot be checked statically; for example, does a program respond correctly to resize events?

Coverage An important concept from testing theory is the notion of coverage. Roughly speaking, coverage is a measure of how good your test is. The simplest coverage measure is statement coverage; if every statement in your program has been executed in a test run, you have achieved 100 percent statement coverage. Other common coverage measures are branch coverage and path coverage.

If you have access to a coverage measurement tool, you can use it to drive your testing effort by deciding on a specific coverage measure as a goal for your test plan.

Available coverage tools for Java include JavaScope from SunTest and "TCAT for Java" from Software Research, which work by instrumenting the source code for a class.

Test Strategy
A complete Applet testing strategy, then, requires checking for functionality and for portability. That checking can be done by dynamic testing or, in some cases, by static inspection.

Functionality Testing
It's probably best to separate functionality testing from portability testing because the focus of the activities differs. Functionality testing assumes that the environment is OK and focuses on the value added by your individual Applet. Portability assumes that the value added by your Applet is OK and focuses on possible problems with the interaction of your Applet and the environment.

While discussing functionality testing, I will generally discuss function testing. I presume that the special functionality that your applet provides is in the form of methods that are not part of the Applet interface. If this is not true, if all the interesting work your Applet does is in the implementation of methods from the Applet interface, then a GUI testing tool (described in this section), should suffice for your functionality testing.

There are several ways to do function testing:

  • by adding a main() method to your applet
  • by creating a test program to test the methods of your class
  • by using a GUI test tool to drive your applet
  • by creating a self-checking version of your class.
Adding a main() method
One simple way to test the functionality of your Applet is to make it into an application by adding a static main() method.

The simplest kind of main does all the testing itself, as shown in Listing 1.

This kind of test is simple to run and interpret and simple to repeat, but is at best somewhat inconvenient to write. It requires enumerating the test cases and expected results in the test driver, which may be a lot of code to add to your Applet. It also requires the main to create an instance of the tested class, which may be difficult.

The example shows a very simple kind of check on the result of a method: comparing it to a tabulated result. This is not the only kind of check that can be done. Other, less direct, checks on the result of a method are possible, or required, in many cases.

double r = cubeRoot(f);
// somewhat arbitrary margin for floating-point error
if (Math.abs(r*r*r-f) > 1.0e-20) {
error();
}

Component c = this;
c.enable();
if ( ! c.isEnabled()) {
error();
}

The main() added for testing purposes can be more general than just a list of test cases. It can be, for example, a simple command processor that drives the methods of the application and prints the results; this can be used to exercise your Applet from the command line.

Use a Test Generator
Rather than writing a main() to drive the methods of your class, you can use a test program generator to create the code. An example of such a test program generator, the only one I am aware of for Java programs, is JavaSpec, from my company, SunTest (http://www.suntest.com/).

A test program generator works by taking a specification of the required behavior of the program and of the data to use for testing, and using those specifications to generate a test program. JavaSpec, in particular, uses fragments of Java code as specifications. Essentially, JavaSpec automates the bookkeeping required to implement the test driver and the test scorecard, leaving to you the interesting work of creating the test data and the checking code. In addition, JavaSpec provides a framework for the structure of the test, helping you to create comprehensive test data sets and helping you with the structure of the checking code.

GUI Testing
The simplest way to test an applet is to click the mouse at it. This can be done by hand, or with a GUI testing tool. In either case, the hard part is not clicking, but checking.

Recall that the purpose of testing is to show that a program has certain properties. The utility of GUI testing depends strongly on what properties you are trying to show. If you are trying to show that a program obeys buttons pushes and displays data, then GUI testing is indispensable. If you are trying to show that a program updates a database or calculates a square root correctly, then GUI testing may not be the most direct method.

GUI testing is generally best for two kinds of testing:

  • Safety checking: checking that an Applet does not crash
  • Display: checking that the display is drawn correctly
Safety checking is done by driving the Applet through a sequence of actions, trying to exercise all of the states of the system.

GUI Testing Tools
Using a tool to automate the GUI test has several advantages: It is much easier to repeat a test exactly, and the test can be much more extensive. In addition, the GUI testing tool may provide convenient ways to check the display, making it much easier to check the results of the test actions.

If you do decide to use a GUI testing tool, you must choose between a platform-specific GUI testing tool and a Java GUI testing tool. A platform-specific GUI test tool, like XRunner or WinRunner from Mercury Interactive, QA Partner from Segue, QC/Replay from Centerline, SQA Robot from SQA, or CAPBAK/X from Software Research, tests at the level of native code. A Java GUI testing tool, like JavaSTAR from SunTest, tests at the level of the Java AWT.

The advantage of a platform-specific tool is that it can test the complete operation of your applet in a specific browser on a specific platform, including the interaction of your applet, the browser's Java implementation and the native window system. The disadvantage is that the tool and the test are tied to the specific platform and applet viewer or browser used.

A Java GUI testing tool has the advantage of producing portable, cross-platform tests and of itself being portable, so that the test can be created on any platform. Its disadvantage is that it relies on the correctness of the Java implementation.

SunTest's GUI testing tool, JavaSTAR, has the additional feature, which we think an advantage, that the scripting language is Java; the test scripts are Java programs that can be edited as required. Generally, the platform-specific tools have a proprietary scripting language for describing tests, which must be learned for effective use of the tool.

Self-checking Code
One special technique that may make it simpler to write a test is to write a self-checking version of the class you are testing by adding assertions to your code using a method like this:

private void assert(boolean cond, String msg) {
if ( ! cond) {
System.err.println("assert failed: " + msg);
errorCount++;
}
}

If you have instrumented your code like this, then every program run becomes a test run; however, it is hard to do comprehensive checking from within a class and you must still plan and create the test data.

Portability Testing
Once you are satisfied that your Applet does what you want it to do on one platform, you need to check that it does what you want it do on all Java platforms. This is the province of portability testing.

Portability has various aspects. Fundamentally, portability testing is a check that you have stayed within the parts of the language that will work the same in every environment, or that you have worked around the differences.

An Applet may run on a WebTV, on a multiprocessor Solaris workstation with a 24-bit framebuffer, on a PC with no sound card, or on a JavaStation running JavaOS. The goal of portability testing is to check that your Applet will perform its function in all these environments.

The most important resource for improving the portability of your Applet is Javasoft's "100% Pure Java" program [http://www.javasoft.com/100percent], which provides information, training, testing and certification aimed at improving the portability of Java programs.

A good portability test requires a sanity check, static checking and dynamic testing.

Sanity Check
Try your applet in various browsers on various platforms. Does it display? Try resizing the window; leaving the page and coming back; pushing a few buttons; typing a few characters. This is preliminary to a full test; it's like checking that a car has four fully-inflated tires before going for a test drive. You could call it the "tire-kicking" step.

Its purpose is to catch errors so fundamental that they would prevent the more detailed test from running, and also to catch errors so unexpected that they are not checked in the test plan.

Static Checking
Does your Applet use any unportable methods? Use the Java Purity Checker, the "100% Pure Java" static checker, to point out possible problem areas. The Java Purity Checker will also flag classes that may not run under a SecurityManager. You can obtain the Java Purity Checker from SunTest [http://www.suntest.com].

Dynamic Testing
Firewalls
If possible, set up your Applet on a server behind a firewall and see if it still works. Common problems include loading the Applet class, loading other required classes and loading other resources like sounds and images; in general, anything that requires a new network connection back to the Applet's server may run afoul of the firewall.

. If your Applet does any of these things, you should make sure that your firewall testing covers those parts of the program. For example, if your Applet plays a sound when a button is pushed, then make sure that your GUI test script includes a push of that button.

Specific Problem Areas Using the test planning and creation techniques described above, check that your applet doesn't run into these known problem areas.

Layout If your applet uses a layout manager, try resizing it; if it changes the contents of a Container, exercise that part of the code in various browsers.

Threads If your applet creates a thread or synchronizes, it may behave differently on different Java implementations, or even at different times of day. This class of problem is, unfortunately, almost impossible to check by testing; there are just too many program states to check, and too little control over what state the program is in. The best cure is prevention: think hard about multithread behavior.

For example, if your program uses synchronization locks, define a global lock order and obey the rule that locks must be acquired in order (no thread may acquire a lock unless it holds the preceding lock). This rule prevents deadlock.

Test Automation
No applet test plan is complete without a human in the loop, for sanity checking and to evaluate reported errors. Nonetheless, automated test tools may significantly enhance your applet testing. The benefits of static checking tools like Java Purity Checker has already been discussed; the advantages of dynamic testing tools include:

Repeatability
Automated tests are repeatable, which makes it possible to run the same tests on different platforms. It also makes it possible to repeat the test in a regression run after fixing bugs.

Portability
Testing tools that use and are written in Java are portable to any Java environment, and produce tests that are portable to any Java environment. This is a key advantage for portability testing.

Completeness
An automated test can range over more test data, and perform more result checks, than a manual test. Hence, an automated test can give a more thorough test than a manual test.

Conclusion
The strategy and techniques presented should give you a good start at developing a test strategy for your Applets. The most important point is to have a test strategy; know what you are looking for, know how you will measure it and know how you will know when you have measured it. That ensures that your test efforts are not wasted.

About the Author
Roger Hayes has been working at Sun Microsystems, on various projects in the field of programming and testing tools, since receiving this Ph.D. in Computer Science from the University of Arizona in 1989. He is currently part of the SunTest group, which develops Java testing tools.

	

Listing 1.

/** for testing purposes only. */
static void main(String[] args) {
        int errct = 0;
        MyClass ego = new MyClass();

        // check method1
        String[] arg1 = { "", "hello", "\u0000abc\uffff"};
        int[] expected1 = {0, 5, 5};
         
        for (int i = 0; i < arg1.length; i++) {
                int res = ego.method1(arg1[i]);
                if (res != expected1[i]) {
    System.out.println("problem: method1(\"" + arg1[i] + "\") -> " + res);
     errct++;
    }
            
    // check other methods as required
            
      if (errct > 0) {
      System.out.println("Result: failed");
      } else {
      System.out.println("Result: passed");
      }
    }
}


 

All Rights Reserved
Copyright ©  2004 SYS-CON Media, Inc.
  E-mail: [email protected]

Java and Java-based marks are trademarks or registered trademarks of Sun Microsystems, Inc. in the United States and other countries. SYS-CON Publications, Inc. is independent of Sun Microsystems, Inc.