The Challenge of Java Integration with Legacy Assets
Organizations are developing a new class of electronic business applications, using Java to capture new business logic. In some cases these e-business applications ,have gone beyond just successful deployment to redefining the competitive landscape. Examples are package tracking, home banking, travel services, automobile purchasing, home shopping, customer support, billing, global securities trading and publishing. The driving forces behind these applications are increased competition in growing markets and the need to reduce costs in services and support. Because e-business is a significant part of the business image in competitive markets, demand for higher performance and reliability is continually increasing.
Successful e-business applications must meet demanding performance, scalability and reliability requirements that previously have been found only on the mainframe. A short list of required Java e-business server capabilities includes multithreading, load balancing, resource management, state persistence, connection pooling and fault-failover. Achieving any one of these capabilities is a large development cost.
JavaSoft, along with leading partners like IBM, are meeting the challenge of Java e-business requirements with Enterprise JavaBeans (EJB). The key capabilities of the EJB framework are:
A complete, well-defined set of interfaces is a major strength of Enterprise JavaBeans. The service interface limits the complexity that's visible to the application implementation while enabling the application to exploit the full power of the distributed object system. Server-side component services are fundamentally new because in the past, the integration of service-based capabilities always had to be custom crafted. Without a component framework such as EJB, there was no way of predicting - other than empirically - how the capabilities of services from different vendors could be integrated. The EJB server standardizes how the behavior of one Java service interface will change when another Java service interface is also being used. Business logic and different service interfaces become Plug and Play application components (beans!).
- E-business logic is packaged as a Java-Bean component that executes within an EJB server. The EJB server platform supplies thread management, state persistence and other enterprise capabilities to the JavaBean. The e-business application developer gains performance and scalability from the EJB server platform facilities.
- The EJB server platform supplies data access (JDBC), transactions (JTS), messaging (JMS), directory (JNDI) and other fundamental services as standard Java interfaces. The e-business application developer gains Plug and Play component software development productivity by using tools that support EJB service specifications. Component software developer productivity, previously available only to the visual JavaBean developer, can now be achieved by the server-side Java developer.
Enterprise JavaBeans is a complete platform for Java e-business applications. But what about the world's existing business logic? Can we afford to build new e-business applications that ignore existing production systems? The answer, of course, is "NO!"
The most critical challenge facing Java developers today is with integrating legacy data and applications. The world's information technology (IT) investment (sometimes referred to as legacy) is realized in mission-critical business logic that controls such critical systems as fund transfers, stock exchanges, transportation scheduling, payroll and the worldwide distribution of goods and services. It's estimated that there's over $5 trillion in IT assets that conduct the day-to-day business of the world's economy. In 1997 the U.S., Europe and Japan spent over $600 billion on the maintenance and upgrade of these IT assets.
The Enterprise JavaBean framework
The roadmap for legacy integration is inherent in the specification of the EJB platform. Access to legacy assets should be accomplished using Java platform services such as Java Database Connectivity for legacy data and Java Transaction Service for legacy transactional logic. By integrating through Java platform services, legacy business logic and data appear as native JavaBeans to the e-business application developer.
Let's explore the technical challenge of transforming legacy IT assets into Java services.
The Challenge of Transforming IT Assets into Java Services
E-business applications are inherently distributed and multitier. In a multitier distributed application, the logical architecture is typically three-tier architecture, with business logic separate from presentation and data access logic. Mid-tier Java business logic requires access to back-tier IT assets that:
Java developers are discovering that existing middleware is not designed to support hundreds or thousands of connections from distributed business logic servers to databases. For example, client/server middleware, such as ODBC, was designed to connect a single resource-rich desktop client to a database server. An entirely new kind of middleware is required that can manage large concurrent sessions loads of an e-business application server.
- Provide high performance throughout for both small (one to hundreds of rows) and large (thousands or millions of rows) amounts of data
- Sustain high performance throughout under peak concurrent user loads as well as trivial user loads
- Maintain high availability greater than 99.9%
- Support centralized management and administration using standard tools
- Use standard interfaces to reduce the cost of development and maintenance
A new breed of middleware must meet the performance requirements of e-business and it must also meet the rigid integration limitations of legacy IT assets. By introducing a new tier, a middleware tier between the e-business logic server and the legacy asset tier, we have a platform to add all the facilities that we need. A middleware tier can enhance the legacy asset by adding such facilities as load balancing, connection pooling, multithreading and distributed failover. I-Kinetics calls this middleware architectural approach the Service Broker.
A good Service Broker isn't just a pass through proxy for the Java service interface. What looks like a simple implementation of a Java service interface may require the Service Broker to coordinate the implementation of several features within the limitations of a legacy interface. For example, obtaining a connection to a legacy application may appear straightforward. However, if that connection is to be pooled, load-balanced and encrypted, and to support a distributed two-phase commit, then the Service Broker will have to implement most of the distributed connectivity management facilities for the legacy asset.
Adding new capabilities is a significant advantage of a Service Broker architecture. For example, I-Kinetics DataBroker implements the JDBC service with additional capabilities such as multithreaded data streaming, load balancing and connection pooling. You can expect Service Broker architectures to be able to offer the best JDBC 2.0 extension implementations for transaction resource management, batch updates and backward and forward scrolling result sets. Additionally, SNMP management, query caching, messaging and object persistence are advanced capabilities a Service Broker can offer across JDBC, JTS and JMS services.
The Service Broker transforms legacy assets
into standard Java services
The impact of using a Service Broker on the full life-cycle cost of the e-business application can be enormous. For legacy system integration projects, 40 to 70% or more of the development cost may be eliminated because:
Let's explore a practical example of accessing mainframe data from Java with several different approaches, including the Service Broker approach.
- A Service Broker transforms the legacy data structures into standard Java data formats.
- A Service Broker offers standard application service interfaces that are widely supported by tool vendors. A Java development tool can be used for the complete development life-cycle from prototyping to full deployment of the multitier e-business application.
- Complex capabilities such as multithreading, opportunistic concurrency, load balancing, connection pooling, partitioning, recoverability, failure isolation and administration are introduced by the Service Broker without any implmentation cost or complexity in the business logic.
Comparison of Different Mainframe Data Access Solutions
One of the most significant data access challenges facing e-business application developers today is accessing mainframe data. Access to mainframe data can create significant business opportunities, but only if a sufficient business case can be made.
- Lower life-cycle costs - You need to retain mainframe business logic while moving new e-business application load off the mainframe to meet growing demand.
- Lower operational costs - Equipment support and staff training costs of terminal user interfaces (i.e., green screens) isn't competitive with Web-based browsers.
- Increase revenue - You can deploy e-business services that directly increase your market share at the expense of your competitors.
There seems to be a befuddling array of choices when it comes to which solution to use to access mainframe assets. However, your requirements are timeless as they exist for any system deployed for a business revenue-critical enterprise environment. These requirements are reliability, performance, scalability, security, legacy support, multiplatform support, management and low development and maintenance costs. There are a multitude of solutions for integrating mainframe assets, most of which can be identified as being in one of three general categories:
IBM's Information Management System (IMS) provides On-Line Transaction Processing (OLTP) for much of the world's critical business and commercial service applications. In existence for over 25 years, only CICS surpasses IMS in managing the world's business-critical transactional data. IMS is an ideal testbed for comparing the capabilities of terminal emulation, gateway middleware and Service Broker middleware.
- Terminal emulation middleware
- Indirect access gateway middleware
- Direct access Service Broker middleware
The business logic for the IMS testbed consists of a common Java application that invokes each different middleware solution to extract 100 rows of IMS data. The Java application and middleware is hosted on a Solaris 2.6 Ultrasparc platform (e-business server). The IMS application and data are hosted on an IBM OS/390 mainframe. The Java code differs only in the call syntax required by the specific middleware interface. JDK 1.1.6 with the JIT enabled was used for all configurations.
The performance data shown in Table 1 consists of the average of a sample size of five measurement runs. Column 2 (Process Launch), is the amount of time needed to launch and initialize the business logic process. Column 3 (Connect) is the amount of time needed to establish a connection to IMS. Column 4 (Execute) is the amount of time needed to execute the transaction through the middleware and Column 5 (Extract) is the amount of time needed to extract the IMS data from the data block returned by the transaction query. Column 6 (Total Time) is the total clock time needed to execute all aspects of the Java application and middleware on the e-business server. The individual times of Process Launch, Connect, Execute and Extract account for over 90% of the Total Time. Column 7 is the total number of bytes transferred from the OS/390 mainframe tier to the e-business server. Column 8 is the amount of OS/390 processor CPU consumed to execute the transaction.
If your mainframe applications are mature and you just want to overhaul the interface, not add any new business logic, then you may want to consider using terminal emulation middleware. The terminal emulation (or screen scraper) approach doesn't require any installation of new software on the mainframe. However, terminal emulation has distinct disabilities when used as e-business middleware. The mainframe host must manage a virtual terminal session as well as add terminal management control code data in the data stream for each transaction. All this terminal emulation processing and control is wasteful as it goes unused by the mid-tier business logic.
For the IMS testbed results given in Table 1, the terminal emulation middleware consumed 85 msec - 280% more than the Gateway and 400% more than the Service Broker solution. In terms of capacity planning, terminal emulation will need 300% or more equivalent MIPS to service the same user load as a Service Broker solution. Additionally, any 3270 screen format change will require code maintenance of each affected screen-scraper script. You can expect high maintenance costs, low performance and increased load on the mainframe with a terminal emulation solution. At a current cost of $10,000 per mainframe MIP, terminal emulation is impractical for the high concurrent user loads sustained by e-business systems.
The Gateway middleware for accessing IMS was the IBM DB2 Connect ODBC driver on the Java platform connecting to DB2 on the mainframe. DB2 acts as a gateway by enabling a DB2 procedure to access IMS. Two different Java platform configurations were used:
The performance of the CGI/Gateway configuration is at a large disadvantage because of the process launch time of 200 msec. However, most of the process launch time can be almost completely eliminated by using FastCGI, which keeps the CGI process in memory.
- The Java application invoked a CGI/Perl script which in turned invoked the ODBC driver. The Perl script "extracted" the returned IMS data from the ODBC driver as an array of strings. The array wasn't passed back to the Java application and if it had, it would have resulted in a larger Extract time.
- The Java application used the Javasoft JDBC/ODBC bridge. The JDBC/ODBC bridge was invoked directly and data returned directly to the Java application using the same extraction logic as the Terminal Emulation and Service Broker middleware configuration.
The JDBC/ODBC Connect and Extract performance was better than CGI/Perl for Gateway middleware. In this case, Java string manipulation and foreign function call facilities matched Perl. Perl's string manipulation facilities are famous for being able to match or sometimes exceed C/C++ code performance. Both were equal in their use of mainframe resources as they both shared the ODBC/DB2 gateway to IMS. The developer productivity advantages of Common Gateway Interface (CGI) using Perl and other scripting languages has been matched and exceeded by the developer productivity found with Java servlets. JDK 1.2 performance improvements in string manipulation and JIT technology will further increase Java development and runtime advantages. Given the performance advantages of Java servlets over CGI, it becomes easy to understand why server-side Java is so quickly replacing CGI as a business logic platform.
The Service Broker configuration shown in Figure 3 consists of the I-Kinetics DataBroker and the Neon Systems Shadow Direct for IMS. DataBroker features a complete JDBC driver. The DataBroker is the Service Broker that directly interfaces to Shadow Direct - an installed service running under OS/390 that integrates directly with the IMS Transaction Manager.
The first immediate performance advantage realized with the Service Broker approach is the Connect time of 12 ms for the IMS testbed. The DataBroker automatically organizes all database connections into a connection pool. When a requested database connection is already available in a connection pool, it will be reused.
For e-business transactions with small Execute and Extract costs, eliminating fixed overhead Process Launch and Connect costs are essential to achieving high performance. For Insert/Update/Delete transactions with an Execute time of 10 ms or less, a Connect time of 300 ms will heavily tax the peak transactional throughput. Reducing Connect time with connection pooling can achieve significant increases in performance and scalability.
Other performance advantages realized with the Service Broker were with Execute and Extract performance. Because DataBroker is directly integrated with Shadow Direct and Shadow Direct directly accesses the IMS transaction manager data queues, the resulting Execute performance is very high. Additionally, with Shadow Direct compressing the data stream the amount of data transmitted (3,500 bytes) is dramatically smaller than with Terminal Emulation (11,700 bytes) and Gateway (7,600 bytes) middleware configurations.
The Java code for accessing IMS using the Service Broker is shown in Figure 4. IMS applications are accessed using a SQL-92-compliant stored procedure call, broker_ims, invoked using the JDBC statement interface method executeQuery. The broker_ims stored procedure call includes parameters that specify the IMS application to execute and result set data formatting. From this example, it is easy to grasp the power of using Java service interfaces, such as JDBC, to access legacy assets, such as IMS.
The Right Way: Transforming Legacy Assets into Java Services
The Service Broker architecture provides a path for the enterprise to scale up existing client/server applications to high-performance, high-capacity multitier e-business applications. The enterprise is protected against being locked into a single platform or vendor middleware solution. The enterprise IT infrastructure decision maker can depend on Java services, such as JDBC and JTS, to be essential standard elements of the Java and Enterprise JavaBean platform. The Service Broker provides a complete migration from current IT systems to future IT systems, overcoming the drawbacks of other integration strategies.
An important part of the Java service legacy integration strategy is for the
software industry to create Java Service Brokers for the world's $5 trillion of IT assets. You can expect the Java software vendor community to supply Service Brokers for a wide range of legacy assets. Currently, you can access almost any mainframe asset, such as ADABAS, CICS, DB2, IMS, IDMS and VSAM, as a JDBC data source with the I-Kinetics DataBroker. IBM has announced the access of their popular messaging middleware, MQSeries, as a JMS service. Of course, the leading transaction processing vendors, such as BEA and IBM, will be offering JTS for their transaction monitors.
A good Service Broker will offer high-performance and scalability features to complement a high-capacity Enterprise Java-Bean platform. Building your own
Service Broker is a difficult and expensive experience. Seek out and research existing Service Brokers for your current legacy integration needs. With Plug and Play
Service Brokers delivering JDBC, JTS
and JMS services, you can focus on the critical tasks of developing new e-business applications.
References and Resources:
About the Author
- "Solving the Data Inter-Operability Problem Using a Universal Data Access Broker," Mike Higgs & Bruce Cottman. IEEE Data Engineering Bulletin, September 1998.
- "Introducing DataBroker: Standards Based Data Access for the Enterprise," Mike Higgs, Netscape ViewSource 1997.
- "DataBroker Version 6: A Technical White Paper," I-Kinetics, September 1998.
- "Universal Data Access," Bruce Cottman. Java Developer's Journal, April, 1997.
- The Essential Distributed Objects Survival Guide, Robert Orfali, Dan Harkey and Jeri Edwards. John Wiley & Sons, Inc., 1996.
- Client/Server Programming with JAVA and CORBA, 2nd ed. Robert Orfali and Dan Harkey. John Wiley & Sons, Inc., 1998.
Bruce H. Cottman is the president of I-Kinetics, Inc. Since founding I-Kinetics in 1991, Bruce has focused on solutions for transforming legacy into high-performance component software for large distributed systems. He holds S.B. and S.M. degrees in physics from the Massachusetts Institute of Technology and a Ph.D. in physics from Rensselaer Polytechnic Institute. Bruce can be reached at [email protected], or at his Web site, www.i-kinetics.com.