Is it me, or are the months flying past? It seems like only last week I was sitting down writing this column, hoping to bring a little happiness into your lives. This month, fortunately, I have a lot to tell you about the wonderful world of Java as seen through the eyes of a European CEO. But let's not get into the heavy stuff just yet. This column is like getting to work in the mornings: you don't just rush straight into it; you go and make a cup of coffee, you relax, you allow your heart rate to return to normal after the road-war you had to go through to get here. Hoping you don't meet that wee old lady who insisted on driving 25 mph in front of you the whole way to work. Hell, you deserve a rest. So if this column is coming to you on a Monday morning or some other stressful morning, then let's ease into it together, gently.
Stress is a funny thing. It affects us all in different ways. I'm one of the lucky ones that don't suffer from stress. There's never much point in losing sleep over a problem; it'll still be there in the morning so you may as well get a good night's rest! May sound a bit oversimplified, but let's face it: it's when you're panicky or stressful that you make those absolutely monumental blunders. Your decision-making process is a little cloudy and you can't see the forest for the trees. Life is a gas, and we should all learn not to take things so seriously. They're not that complicated. We merely shroud them in buzzwords and technical phrases to ensure that not everyone knows what we're talking about. It's how we protect our jobs. The IT industry isn't the only one that plays this game; everyone is at it. But should it be like this?
I guess not. But now that we're settled, let's start work....Well, let me start work. I have to write this column; you merely have to read it.
This was one of those months that could have proved extremely stressful if we had let things get out of hand. Have you ever built a system that you completely underestimated the success of? You never thought that particular class or method would be used quite as much as that. Well, this was only one of the things we had to contend with this month.
As you may remember, some months ago we launched the system we've been working on to provide a free Web-based e-mail and newsgroup system, www.LiquidInformation.net. The growth of the system was something that took us all aback. We were signing up 300 new users a day, with around 40% daily usage from all of them. The system was being well stress tested. As you know, it was running on one Sun Solaris station with Oracle8 as the back-end database.
For up to around 3,000 users the system ran very well. Response from the Web site was fast. Sending and receiving e-mail was reliable and secure. When we topped 4,000 users, however, a different story evolved. Last month, remember, I ranted about the joys the JAR file brought to us. Well, this month I have an equally joyous rant to bring to you.
We couldn't understand why suddenly everything started to grind to a halt. With around 4,000 users we were holding around 30,000 e-mails. It was one of those situations where in attempting to locate the crux of the problem we discovered so many little anomalies on the way. For example, our mail server was a Java application that listened for requests on port 25 and processed incoming mail, rejecting unknown users, etc. But the log files started throwing up a rather strange error, java.lang.OutOfMemory. We spent ages trying to get to the root of it.
At first we upped the memory the virtual machine was starting out with, but that fixed the symptom, not the problem. Sure enough, it came back...suspiciously quick. So our mail server was crashing within 10 minutes of operation with memory problems. What could it be? The log files pointed toward the readLine(...) method from the BufferedReader class. Maybe there's a mail server hell-bent on sending us a line of a mail message that isn't terminated around the 80-character mark. Maybe that's overflowing the memory for the String class. At this point it was the best lead we had, but the problem still persisted after we wrote our own readLine(...) method to return a line no greater than 80 characters. We noted, however, a number of mail servers that never respected the 80-character rule of thumb, so this did indeed need addressing.
But the memory problem was still there. What could be taking away all the memory in such a short space of time? A spammer. We were being flooded with return e-mails from AOL where someone had put the return address of one of the users in Liquid. Our system permits only 10 originating e-mails per user per day, so we knew he wasn't using Liquid to spam thousands of users, but he was using our system as his return e-mail.
What happened was we forgot to limit the number of concurrent sessions our e-mail server could handle. AOL got a little excited and opened up in excess of 100 sessions with our mail server in an attempt to download all these return e-mails. That's what it was. The virtual machine couldn't handle the sudden stress of having to spawn over 100 threads and do all the necessary string handling. As soon as we built a throttling mechanism into the system to handle at most only 20 or 30 sessions at once, the problem never raised its ugly head again.
Phew! Thank goodness that was sorted out. But it never really solved the problem on the Web site. A major slowdown was still being experienced.
Looking at the load on the processor, we could see that Oracle was soaking up the majority of the CPU cycles. But why? The total size of the database was only around 80 MB, and we understood Oracle could handle far greater sizes than this. Granted, one of our tables had over a million rows in it but still we thought Oracle could handle it.
We have our own in-house connection pool manager that does a whole host of housekeeping routines and ensures a safe and reliable database operation. This manager was reporting that each query was taking two to three seconds to complete. No wonder the Web site was grinding to a halt! So what was Oracle doing with its time?
First of all, we rationalized the table that had over a million rows and this indeed improved the situation. But things were still slow. We looked closely at all the settings Oracle had and nothing too untoward was popping up. Next we looked at our table design.
Guess what it was? We had forgotten to allocate a primary key to one of the main tables. Duh! Talk about kicking yourself in the butt and rolling your eyes. When we rebuilt the table with a primary key, what a difference it made! Speed was regained and operation was smooth.
This was a classic case of bugs or smaller issues not being a problem in a test or low-volume environment, but when deployed in a high-transaction environment it simply failed. We had taken the time to test all our classes and ensure all exceptions and errors were reported, etc., but neglected to look closely at the other parts of the system.
As the user base grew, the need for a second machine to support the overall system soon became apparent. We toyed with a number of different configurations. We looked at how the larger sites operate - Amazon, Dell and the Internet Movie databases. These are all high-volume sites and lots could be learned from them. Ironically, each of them employed completely different ways of handling large volumes of users, with no one solution shining through as the perfect way to handle it.
The first thing that had to be replaced was the Web server. The one we had employed was simply not up to the task. We had the Java Web Server running, handling all our servlet and static file requests. It was using more and more time to process requests. We also noticed it had a tendency to simply hang or freeze after a prolonged period of time. We put this down to more of the virtual machine as opposed to the Web server itself, but we couldn't confirm or deny this. The mail server was up equally as much and it didn't freeze or hang. A new Web server was required, one that supported servlets.
We looked at Netscape and seriously thought of deploying this solution. But two things put us off: the handling of servlets, which would have to be performed by a third-party vendor, and the cost. Which leads me to a wee aside.
Don't you think the Internet has spoiled us somewhat? With the quality of free software that's available, when a product that does charge comes along it really has to justify its existence. A number of months ago I talked about the real costs of owning a Sun station as opposed to a Linux box, and the feedback I got from you all on this suggested we weren't the only company to realize this. Which makes sense if you think about it. If it didn't, then Linux wouldn't be gathering the popularity it is at the moment. It makes it harder for companies to make money on the Internet if they're expected to give a certain amount of their crown jewels away in order to gain popularity. It's a worryingly unclear future for product companies, and one we here at n-ary are addressing sooner rather than later. I think we'll come back to this next month.
If I asked you to name the most popular free Web server, I'd bet $1,000 that 99% of you would shout back "Apache!" And it's Apache we turned to. We downloaded it, installed and configured the servlet module, and were up and serving requests within two hours. A truly remarkable piece of software, and for this I take my hat off to the Apache team and publicly congratulate them on a job well done.
Apache did as fine a job of serving the same level of requests as the Java Web Server, but didn't take up as many processing cycles. Which concerns me a little. We in the Java world are trying to promote the use of Java and get it into the high-transaction environments, and here we are, a leading Java servlet company, dropping a Java product in favor of a more robust, platform-independent solution. We ask forgiveness from the Java community for this act of Judas, but we had a job to do, and Apache was the man for the job. Nothing would have given us more pleasure than to proudly announce a complete Java solution, but sadly it wasn't to be. Our database is Oracle and our Web server is Apache. but all our code is Java!
Next month I'll take you into the wonderful world of databases for Linux as we build up our server farm. And I promise you, I can save you money in this field too. Our eyes were well and truly opened, and I wish to share this wondrous sight with you all. But more next month.
Last month we asked you on our main Web site (www.n-ary.com) if you felt Java applets were still slow to execute. The poll ended with a resounding 77% of you indicating that applets were still slow. This is a very interesting result and we can see that virtual machine developers still have a long way to go to improve the reputation of Java's speed. We'll watch this one carefully and possibly redo the poll in six months to see if the situation has improved.
This month we're asking if you've looked at deploying a Java servlet solution as opposed to a Microsoft ASP or CGI solution. The results are encouraging, with 75% of the votes showing a swing toward servlets. It's good to see Java servlets becoming a viable solution to you all. Next month I'll report the final findings.
I've run out of space for the book review. So please accept my apologies and I'll ensure it gets in next month. But I'd like to thank all of you that have signed up on the mailing list that accompanies this column. It's proving to be a much better way to discuss issues that are raised here, so if you're up for some stimulating conversation, then come along. See our main Web site for details.
On that note, I'll look forward to hearing from you, and prepare myself for the hazardous drive home. Fortunately for me, as we're situated in the lowlands of Scotland, rush hour consists of making sure you get past the farm before the farmer drives the cows along the single-track road for milking.
And you think you have problems!
About the Author
Alan Williamson is CEO of n-ary ltd. A Java consultancy company with offices in Scotland,
England and Australia, they specialize solely in Java at the server side. Alan is the author of two Java Servlet books and contributed to the 2.1 Servlet API. He can be reached at [email protected] (www.n-ary.com) and welcomes all suggestions and comments.