Service-Oriented Integration: An Evolutionary Step for IT

Service-Oriented Integration: An Evolutionary Step for IT
Start small, and experiment

This article outlines a set of best practices for service-oriented integration (SOI) by reviewing the evolution of integration practices, applying those lessons to service-oriented architectures (SOA), and finally analyzing SOA and SOI with the specific technology set of Web services today and into the future.

Services and Integration: How Did We Get Here?
The advent of the information age has brought an increased emphasis on timely access to information. Use of the Internet to reach employees, customers, and business partners creates corresponding demands to deliver solutions through different kinds of interfaces in shorter time frames. These compressed time frames frequently eliminate some traditional options for satisfying new application requirements, such as migrating to a new, state-of- the-art packaged application; or creating a custom solution built in isolation from the existing enterprise systems.

Enterprise-level integration offers a potential solution. By leveraging existing assets and building out only the required new logic, integration can shorten time frames and costs while providing the new interface access. The ongoing issue has been the difficulty in actually achieving the potential. While there are many different ways to view mainstream integration practices, we think it is useful to divide it into three main categories:

Custom-coded projects: For consultants or IT staff, composing a custom solution is potentially viable on a small scale. Over time and scale, this approach has some obvious limitations. Many complex problems such as security, reliability, scalability, support for transactions and related semantics are solved and then maintained by each enterprise over and over again. Increasing complexity and decreased timelines are making this path risky. The savings in technology acquisition is being swamped by the cost of maintenance and time to delivery.

Messaging middleware and integration brokers: Products such as IBM's WebSphere Business Integration Message Broker or webMethods' Integration Platform take the form of central "hub-and-spoke" messaging systems that transform and route messages based on custom sets of rules. The brokers are often packaged as suites with process automation tools and interfaces to popular databases and applications. These products have typically been built around proprietary technologies that were designed to solve data synchronization problems on a massive scale, resulting in hugely complex products with unfamiliar tools. The out-of-the-box interfaces in these products often struggle with higher level forms of integration that involve business logic, leading to unexpected custom coding. These two factors result in products that are very expensive and require many successful projects to show a return on investment.

Application servers: J2EE implementations, Microsoft's .NET, and other platforms have been developed and marketed as robust and scalable platforms with standard sets of services such as security and transaction monitoring. Adapter frameworks and business process features allow a customer to build automated processes in addition to user-facing applications. A high degree of skill is needed to effectively leverage these features. Coupled with the need to design and code an entire solution from the ground up, achieving the best architectures and solutions still requires a substantial investment beyond technology acquisition.

Thus, many organizations find themselves with seemingly conflicting needs. On the one hand, incremental delivery of application functionality with a quick return on investment (ROI) must be provided in short time frames. On the other hand, a sustainable and manageable enterprise architecture must be developed that can support the business into the future. This is where service-oriented integration can provide some assistance (see Figure 1).

SOI Defined
In the most generic sense, a "service" can be defined as a piece of functionality that is transparently accessible over a network. As the underlying concept behind SOI, serviceoriented architecture brings some refinement to the broad notion of a service:

Components in an SOA ecosystem are often published: The clients that use them are not predefined. Theoretically, any client that understands the interface could utilize the functionality.

The notion of multiple clients extends to an expectation of interoperability with multiple technology mediums: As an example, the service can be created using J2EE, an initial client with .NET, and a future client in a new language altogether.

In SOA, a service is usually "discoverable:" This means there is some directory that potential clients can use to discover interface metadata and/or location.

Service-oriented technologies can be applied to many different problems, such as business-to-business data exchange, Web application development, and point-to- point application data exchange. The important architectural requirements for use-of-service technologies are not necessarily the same. Thus, there are two important points to keep in mind:

We use the term "integration" to refer to the process of making a set of disparate enterprise systems work together to support common purposes. While developing a user-facing application may leverage integration components, enterprise integration and application development are distinct in purpose, scale, and complexity.

Our interest is specifically in how to apply a service-oriented architecture to the complex problem of sustainable, enterprise-level integration. While we will argue that a benefit of SOI is an enterprise architecture that can be built out on a project basis, our recommendations are about developing the architecture as a whole over time.

With these basic definitions in mind, we can consider the possible benefits of SOI to the enterprise (see Figure 2).

SOI: Theoretical Benefits
A consistent theme in the section on traditional integration methods is that of cost. The cost can stem from long-term, highly skilled development (custom coding and application servers) or from initial capital expenditure for acquisition and deployment (integration brokers). At the core, SOI holds out the promise of reducing the cost of developing integration architecture:

As an integration approach, SOI promotes reuse of existing assets with new business applications such as Web selfservice, employee portals, and supplychain management.

By utilizing a services-based approach, SOI encourages architectures built around services that can be produced on a project-byproject basis and can support reuse over time by multiple clients constructed with a variety of technologies. This leads to earlier return on investment and potentially lower costs over time.

If built around the standards-based services technologies discussed in succeeding sections, there is a strong possibility that the skill sets and the infrastructures for security, management and the like will cost substantially less than those required by traditional integration approaches.

These benefits are only theoretical, however. Failure to recognize the potential pitfalls of service-oriented architectures can quickly lead to failure on a massive scale. This is where a set of best practices becomes important.

SOI: Best Practices
At least three best practices can be derived directly from our notion of service-oriented architectures:

A repository supports future development: A core tenet of SOI is that services can be developed independent of a particular client. In turn, at a future design time, an application or service developer will need to know what services are available, what they do, and how to interface with them. A repository, or directory of service metadata, provides a common place to search and analyze what work has already been done. This is essential to promoting reuse, which is one key component to cost reduction.

A directory service provides flexibility: Enterprise integration is a complex undertaking. The underlying hardware and software infrastructures are likely to change over time. Thus, at deployment time affording clients the ability to ask a directory where to find the required service adds a key element of flexibility to the SOI architecture. The directory can be something as simple as a configuration file or environment variable or as sophisticated as an enterprise directory server using UDDI or LDAP.

Client neutrality assists in cost reductions: Choose a service technology that is independent of the service's implementation medium. This enables both sets of developers, those writing clients and those creating the actual services, to leverage the technologies with which they are most familiar and/or those technologies most applicable to the job at hand. In the context of SOI specifically, we believe in three additional best practices:

Ensure proper granularity of the service: Because services can be developed semi-autonomously, a client developer should not be expected to have intimate knowledge of the underlying enterprise system. Such a requirement slows the project down and risks the client containing implicit dependencies on the service's underlying system. Thus, encapsulation of complexity increases productivity and long-term flexibility to change implementation.

Furthermore, network exchange can be a performance bottleneck due to the overhead of formatting data for the network and reformatting it in the original form on the other side. Thus, a service should take as few trips across the network as possible to accomplish an atomic task. In particular, the fact that it might be easy to wrap a traditional component with a "service interface" does not mean that is the right architectural decision.

Minimize the impact of additional client load: You cannot assume knowledge of where or how often a service will be used. Consider carefully whether a service can be productively used in an asynchronous fashion, meaning a client leaves a request on a queue and expects a response later. This architecture allows the service configuration to dictate processing load, not the number of clients. As another example, in many cases an integrated service can be "stateless," meaning only a single exchange of data is necessary to accomplish a task. The service's client application or process has more particular knowledge of its own purpose; that is often the more desirable place to store state information.

Audit service usage: Tracking the actual usage of services provides information about which services are still in use, how often, and by whom. In a loosely coupled system this is probably the only realistic mechanism available to understand service interdependencies. In addition, auditing permits legacy services and service versions to be retired, those which receive heavy usage to be optimally deployed, and so forth. Lack of information in these areas will quickly render the enterprise services architecture unmanageable.

Service-oriented integration can be pursued with a number of technologies. However, Web services are among the most intriguing due to the corresponding development of industry standards.

SOI and Web Services Web services standards provide some key building blocks for our new integration paradigm:

XML is the common, text-based lexical format for data exchange.

WSDL is an XML-based language for describing service interfaces in a manner that is neutral to client technologies.

UDDI allows Web services to advertise their existence.

The related SOAP standard provides standard protocol bindings.

The catch in all of this is that Web services are just a collection of new technology standards. By themselves, these technologies don't constitute a service-oriented architecture, but only enable it. There are still many issues that must be carefully considered in the context of integration:

Reliability and security: These features of message-based and some application platform solutions have not been directly addressed. For example, the use of Secure Sockets Layer (SSL) encryption can address basic security requirements, but this does not directly provide message integrity and application-level authentication and authorization capabilities. These issues make the maturation of standards such as WS-Reliability and WS-Security or their equivalents an essential piece of the Web services puzzle.

Transaction support: In many cases, this is an important element of integration architecture. Web services transaction support presents an interesting problem because the resources involved could include almost any kind of system that resides almost anywhere. Once again, there are emerging standards, such as WS-Transaction, that should address this shortcoming.

Management: Since a service is intended to offer a loosely-coupled solution, this by definition means it is published over a network to any number of clients in any number of environments. Thus, there is no clear-cut method to manage changes to the service over time. The WS-Management and related standards bear watching in this area.

Technology acquisition provides the most promising avenue for addressing these concerns. While careful analysis regarding compatibility with existing enterprise assets will be essential, very little benefit is likely to be derived from a custom implementation of these complex standards.

Applying Web Services to Integration Today
The benefits of the standards-based services paradigm to enterprise development are quite compelling for many departments that need to break the cycle of monolithic application development and find new avenues for extensibility and reuse of existing assets.

The initial architectural consideration is building out services that are client-neutral, abstracted interfaces to existing functionality. This can be applied today in existing projects such that an extensive adoption of SOI can be made more easily in the future. The ability of enterprise applications, integration tools, and adapters to encapsulate the systems they represent should form the basis of one key purchase criterion for each type of product.

As Web services standards for management, security, transactions, and reliability mature, more and more products currently used to support integration efforts will begin to incorporate and interoperate with them. This will allow organizations with existing integration infrastructures to use Web services in a complementary fashion. For those with an existing componentbased architecture, the maturation of these standards will permit adoption of the SOI paradigm as a lower-cost option around which to build out the next generation architecture.

In the short term, the best strategy on implementing Web services in support of SOI will be to start with small departmental projects. This will enable your organization to experiment with the SOI methodology presented in this article. In addition, it enables your team to gain experience with the various tools and technologies available today in support of the serviceoriented approach to integration. You'll find that the initial projects will be leveraged over and over again as you build out your enterprise-wide set of services in support of current and future business initiatives.

About the Authors
Bill DeForeest is an expert in the development of software that enables customers to implement integration solutions in host-intensive environments. He is responsible for requirement analysis and authoring for WRQ Verastream, as well as strategic and tactical planning. Previously, Bill was a software engineer for LECO Corporation, and was a system administrator with the Spokane Intercollegiate Research and Technology Institute. billde@wrq.com

Scott Rosenbloom is a strong proponent of service-oriented integration. He looks for ways to help customers address current integration needs while building a foundation for future initiatives. Scott helps shape the future of the WRQ Verastream product line. Prior to joining WRQ, Scott spent 10 years at IBM in software design, development, and consulting. scottr@wrq.com

Java and Java-based marks are trademarks or registered trademarks of Sun Microsystems, Inc. in the United States and other countries. SYS-CON Publications, Inc. is independent of Sun Microsystems, Inc.