In the past, if you wanted to book a holiday over the Internet you would browse a travel agent's Web site, select a holiday, and book it. If you were lucky, you'd be able to rent a car and possibly even check on the weather. Behind the scenes, the Web server - the computer and software that the Web site lives on - would be accessing a proprietary database, or would be communicating in a proprietary fashion with back-office systems to get information about your holiday.
This all changes with the use of Web services. A Web service is a system designed for the sole purpose of sitting somewhere on a network and servicing specific requests from clients. These clients aren't real people browsing the Internet from their desks; they're other computers.
What Are Web Services?
When Web services are used to book a holiday, you still browse your travel agent's Web site, but behind the scenes something different happens. Rather than accessing a proprietary database to get flight details, the Web server talks to a Web service somewhere else on the Internet. All this Web service does, day in and day out, is respond to requests for flight information from travel agents' Web sites. What is more, these Web services can publicize their existence and communicate with Web sites and other Web services in an open, public format. The travel agent's Web site accesses different Web services for flight, accommodation, and weather information.
Since the Web service is on the Internet and communicates in standard, published protocols, anybody can communicate with it. If you're running a conference and want people to attend, you can have flight information on your Web page. With five minutes of coding, you'll be able to let people view flight availability to your conference and book flights online, without leaving your site (see Figure 1).
Challenges of Testing Web Services
Web services are vital to a business's operation and need to be tested like any other important application. Like Web sites, they are highly visible and should be robust.
There are two broad types of Web services: those used in an intranet and those used on the Internet. Intranet Web services are used internally by organizations and are not exposed to the general public. An intranet Web service might be responsible for handling vacation requests from employees, for example. The company's intranet Web site would access this Web service and employees could request vacations. Managers could authorize the vacations and colleagues could check when other employees are on holiday. Human resources could then write a simple application in Visual Basic to make sure that help-desk employees don't all take their vacation at the same time. All this is possible without the user having detailed knowledge of how or where this information is kept.
Testing intranet and Internet Web services presents subtly different challenges. With an intranet Web service, an organization is likely to have control over who can access its Web service. Since it is on an internal network and only internal users have access to it, there is a theoretical maximum number of users, and organizations can make certain assumptions about security. Anybody can access an Internet Web service, which means that there are additional scalability and security considerations.
Another challenge in testing Web services is that they do not inherently display a user interface that can be tested. Although some development tools will build a Web page around a Web service, this is not part of the Web service itself. Visual Studio .NET, for example, will generate a page that allows users to invoke methods of Web services and view the XML returned, but this is inefficient for anything except the simplest of Web services. This lack of user interfaces means that Web services are hard to test manually, but are an ideal candidate for automated testing. Consequently, some programming skills are almost certainly necessary for Web service testers. A Web service is not the sort of application you can test by key-bashing.
Know Your Clients
It's important to know your clients when testing Web services. If you're expecting to have 100,000 clients who will only ever make one request a day, then the load is not too heavy (about one request a second). If, however, 50% of these requests will happen between 9 and 10 a.m., then that's a different story (about 15 requests a second). Another possibility is that you will only have 100 clients, but these clients will be making 10 requests a second, each and every second of the day.
You might expect, on average, to be able to service 100 requests a second, but what happens in the event of a massive peak? Will your Web service cope, slow down, or crash? Or will it simply refuse to service the 101st request? These are all things you need to know and test before you release your Web service.
You must also know how clients will be accessing Web services. It is likely that they will be making SOAP requests. In this case, you should make sure that your Web-testing tool supports this protocol. Some testing tools record scripts by browsing to the Web page representing the Web service and then recording the HTTP GET and POST requests that the browser makes. Although similar, this is not the same as using the SOAP requests that your clients will almost certainly be making.
What Service is Acceptable?
The next question is, "What is acceptable service?" Your service-level agreements might specify that you need to respond to 95% of requests within one second; in this case, you know what is acceptable. In any case, it's important to realize that a Web service is not the same as a Web site, so acceptable response times will vary. While it may be acceptable to keep response times for a Web site under 10 seconds, a Web site may be making 10 calls to different Web services, so the Web site will expect each Web service to return information within one second.
While you are carrying out your test, the tool you're using should be able to identify how the values below change as you increase the number of clients. These are all measurements of how the client experiences your Web service.
Time to connect: The time it takes to make a connection from the client to the Web service. This should be as low as possible.
Time to first byte: The time it takes for the client to start receiving data back from the Web service. If the Web service needs to do a lot of thinking for each request, then this time could be significant.
Time to last byte: The time it takes for the client to receive the last byte of information back from the service. If the service needs to return a large amount of data (if it is returning maps or images, for example), then this could be significant.
Although the absolute values of these metrics are important (you want to keep these numbers as low as possible), the way they change as you increase the load on the Web service is more important. Ideally, you want these metrics to remain constant. If they increase linearly, then you're in for a shock.
Suppose that with one user, the time to last byte is a blazingly fast 10 milliseconds, for 10 users it's one-tenth of a second and for 100 users it is one second. That's still reasonable. But for 1,000 users it's going to be 10 seconds. If your Web service needs to cope with 10,000 users, then requests are going to take more than a minute. That's clearly unacceptable.
It's more likely that you'll find that your Web service scales (i.e., the response time for requests remains constant) up to a certain number of virtual users, and then it stops scaling. To troubleshoot this, you need to keep an eye on the performance of the server on which the service is running. You need to know what is causing this change in behavior: Is the CPU saturated? Is the disk thrashing? Is the network card causing the bottleneck? These are all possible causes of performance problems.
Once your system hits a bottleneck, you need to know if there's a way around this problem. If you add another processor to the server, will it double its capacity? If you add another server, will that double its capacity? If you have a Web service that will scale in this way, then you know that you will be able to cope with extra demand by adding more and more hardware. If your Web service doesn't scale in this way, then it won't perform no matter how much money you spend on expensive hardware.
In Figure 2, "Good Web Service" scales well until about 120 requests per second (the time to last byte is constant), but then stops scaling. "Bad Web Service" scales poorly. As the number of requests per second increases, the responsiveness decreases linearly.
Testing Types
As with any other application, there are different kinds of testing you can carry out on Web services.
Proof-of-Concept Testing
Since Web services are a new type of software, you need to understand if the architecture you have chosen for your Web service is the correct one. There are many different choices to make - which tool vendor to use, which programming language, and which database back end, for example. If you can clarify and resolve these issues before you start developing, or early on in the development life cycle, then you will save a lot of time and money further down the line.
Because most of the questions you need to resolve will concern scalability issues (will the architecture we're using really cope with 1,000 simultaneous users?), a proof-of-concept test is normally a cut-down load test (see below). There's no need to run it on powerful hardware or get exact answers; your aim is to answer the question, "Am I going in the right direction?"
Functional Testing
This ensures that the functionality of the Web service is as expected. If the Web service divides two numbers, does it give the expected result? If you place a zero as the denominator, does it handle this correctly? Does your Web service implement security and authentication as it is meant to? Does your Web service support all the communications protocols it is meant to? Because your Web service can be accessed by clients whom you can't control, what happens if they make requests you aren't expecting? Bounds testing and error-checking are especially important.
Regression Testing
A regression test is normally a cut-down version of a functional test. Its aim is to ensure that the Web service is still working between builds or releases and it assumes that some area of functionality was working in the past and its job is to check that it still does. If your development team has changed the method that divides two numbers, does that method still work? Does the method that multiplies two numbers still work? Is the performance still acceptable? Since regression testing is, by its nature, a repetitive task, it must be automated. This is true for traditional applications and Web sites; it is even truer for Web services.
Load/Stress Testing
The aim of load/stress testing is to find how your Web service scales as the number of clients accessing it increases. You have carried out functional and regression testing, so you know that your Web service will cope with a single user. What you need to know now is how it will cope with 10, 100, or 1,000 users. If you double the number of users, do response times stay the same? If you double the number of servers running your Web service, does its capacity double? This testing simulates a very complex environment, so it must be automated.
To obtain objective results, it's important to carry out the testing in a controlled environment. If you want to know how your Web service responds as you simulate more and more users, then you must keep all other factors (hardware and networking, for example) constant. This means that you should not carry out a load test on a live system over the Internet. Apart from the problem that bandwidth would pose, this is not a controlled environment.
You might decide that you simply cannot carry out the test in-house. If you are writing a large-scale Web service that needs to cope with 10,000 requests a second, then the odds are that you don't have the necessary hardware in-house to do this. You will probably need to consider hiring a third-party consultant or using a third-party scalability lab to carry out this test.
The ultimate aim of load testing is to reassure you and confirm that your Web service will respond acceptably for up to x clients making y requests a second.
Monitoring
Once your Web service is live and being used by real clients, you will need to keep an eye on it. Is it still working? Are response times adequate? At what times of day is it busiest? It is essential to monitor the Web service.
Testing Makes Business Sense
Web services are becoming a major part of the Internet and corporate intranets, ideally ensuring fast access to a wealth of related information. At the same time, these services place increased demands on Web applications and architectures, often resulting in poor performance that causes users to become frustrated or abandon a Web site.
Automated testing allows IT departments and contractors writing Web services to determine design options and performance trade-offs during the project development stage. This enables developers to side-step problems that could turn away customers or alienate internal users on a live site. Developers can optimize performance for specific applications, and test new and updated Web services to make certain they meet performance criteria before they are implemented.
While testing Web services can be challenging, it's a vital part of ensuring that your Internet or intranet performance is in tune with your business requirements.
Author Bio
Neil Davidson is technical director of Red Gate Software Ltd. Red Gate's latest product line is the Advanced .NET Testing System (ANTS), which provides tools for load-testing and code profiling for XML, Web services and Web sites. Visit www.red-gate.com/ants.htm
for more details.
neil.davidson@red-gate.com
All Rights Reserved
Copyright © 2004 SYS-CON Media, Inc.
E-mail:
info@sys-con.com
