Technical Forces Driving the Adoption of SOA
This chapter applies force field analysis to service-oriented architectures (SOAs). It starts with analyses of integration techniques that preceded SOAs. It then applies force field analysis to the enterprise service bus (ESB), which is often used in SOAs. Toward the end of the chapter, the analyses are combined into a force field analysis of SOAs using Web services. This analysis shows many driving forces for adopting an SOA. It also shows that, over time, many technical restraining forces will diminish and the remaining restraining forces will be typical business and design issues.
Adopting Standard, Enterprise-Wide Software
One early integration technique was for an organization to adopt enterprise-wide software. This worked sometimes. When it did, however, usually it was successful only for a short period. The obvious appeal of adopting standard software is that everyone uses the same software. This means that the entire organization uses the same data definitions, semantics, and formats for exchanging data. Often, this worked best for organizations that were small and were putting a new set of systems in place. Nevertheless, standardizing on systems software often runs into problems, too. There are long-term restraining forces, such as mergers and acquisitions, that can come into play. Even a new, small organization can acquire another organization that uses an entirely different system, and integration problems begin. Figure 6.1 provides the force field analysis for adopting standard, enterprise-wide software.
Figure 6.1 Force field analysis for adopting standard, enterprise-wide software.
This approach has a mergers and acquisitions restraining force for a similar reason as seen in trying to establish standard data element definitions in Chapter 5. The other organization can easily use different software. It is also common in larger organizations that some departments have different software needs. It is rare that you can find “one size fits all” software. Another downside is that adopting a complete set of software systems from a single vendor makes your organization dependent on that single vendor. As soon as you move away from that vendor’s products, you might be back into common integration issues. For organizations that have existing systems, adopting standard software can mean a mass conversion to the new software. This is often problematic and should be seen as a restraining force. Finally, it is often the case that the product doesn’t provide all the functionality that is needed.
Note that none of the restraining forces in this figure are shown in gray. This means that they will not diminish over time and will remain restraining forces for the foreseeable future.
Of course, every example has a counterexample. There are some industries where mergers and acquisitions are commonplace. You will see organizations in those industries adopting common, industry-wide software packages so that it will be easier for one organization to be acquired or merged with another organization. So, mergers and acquisitions can also be a driving force. This is represented in Figure 6.1 with a dashed line. Although I have not seen any empirical data on it, my experience is that this is the exception rather than the rule. That is the reason for the dashed line, because it is likely to apply to only some industries.
Adopting an Object Request Broker
The 1990s saw the introduction of object request brokers (ORBs). The two best known ORBs were the Object Management Group’s Common Object Request Broker Architecture (CORBA) specification and Microsoft’s Distributed Common Object Model (DCOM). (CORBA is still around in various forms. DCOM is now a part of Microsoft’s .NET.)
ORBs are middleware that are one way to exchange data among two or more systems. These systems could be from multiple vendors. In fact, an ORB could be one way to integrate systems from two organizations when a merger or acquisition occurred. An ORB hides the complexity of the communication between two or more systems. They provide a means for applications to communicate with each other. Figure 6.2 shows that, historically, providing interoperable, networked applications was a driving force for adopting an ORB.
Figure 6.2 Force field analysis for adopting an ORB.
The original specifications for CORBA and DCOM, however, dealt with how to get data from one place to another. There were no specific requirements for the format of the data transmitted in the messages. The restraining forces related to data for an ORB are:
Advances in industry standards such as XML mitigated all these restraining forces, which is why they are shown as gray arrows in Figure 6.2. In fact, using XML makes for a more flexible system because of the tagged record structure of XML.1 This also mitigated the restraining force related to the brittleness of fixed record formats.
The mergers and acquisitions restraining force diminishes since an ORB would be one way to deal with the multiple systems resulting from a merger or acquisition.
There was a perception in the industry that neither CORBA nor DCOM were widely adopted and that using one or the other or both was too complex for many programmers. Whether the perceived lack of industry adoption or inherent complexity was actually true is irrelevant at this point. These perceptions are seen as restraining forces. Web services have just the opposite perception—they are seen as easy to adopt widely by industry and easy for most programmers to use. Perception in this case might well be the reality.
The very nature of creating interoperable, networked resources means that there could be a negative impact on operational systems when requests come in through an ORB. Many operational systems have not been designed to receive indeterminate or unexpected processing requests. These requests sometimes can have a negative impact on the performance of those systems. So, the effect on operations systems can be a restraining force if up-to-the-moment processing of those requests is needed.
Adopting an Enterprise Data Warehouse
The story about C. R.’s business trip mentioned that his organization had, at one time, an enterprise data warehouse (EDW). An EDW is one of the oldest and most successful ways to integrate and consolidate data from multiple systems. Commonly, it involves extracting data from existing systems and loading it into a single, central location to form an EDW. Using an EDW can be complementary to using an ORB or Web services. The force field analysis for this approach is shown in Figure 6.3.
Figure 6.3 Force field analysis for adopting an EDW.
In this figure, the easier exchange of data as a driving force is replaced with easier access to enterprise-wide data. This data is loaded from existing systems using various techniques that extract, transform, and load (ETL) the data in the EDW. Using ETL techniques means there is usually less impact on operational systems because the extracts of data from these systems could be done at a time convenient for the operational system. This minimal impact on operational systems is a significant driving force. Easier access to enterprise-wide data also allows the use of business intelligence (BI)/analytics software to find patterns or new business opportunities based on a wealth of data that could be stored in an EDW.
Most of the restraining forces are issues with the semantics or meaning of the data and the standardization of data definitions. Not surprisingly, these issues are similar to those involved with attempts at adopting standard data elements when existing data definitions are different. In Figure 6.3, the semantic translation is added to show the need to transform data, which can itself be a restraining force. Over time, however, these restraining forces have become weaker for two reasons:
A subset of the software industry is devoted to the development of ETL software. This software generally simplifies the development of the data extractions from existing systems, any semantic translation or transformation, and the loading of the data into the EDW.
More industry standards have become available. Initially efforts were related to electronic data interchange (EDI) and more recently to Web services.
Additional restraining forces include problems related to what data to store in the EDW and the delay or latency in getting data into the EDW. The issue of what data should be stored in an EDW will likely always remain a restraining force. The strength of this restraining force will vary by organization. The delay or latency of data is the result of performing data extracts at times convenient to the operational systems.2 Consequently, the very latest data is not always available in the EDW. To some organizations, this is no problem. Others, however, may find this a significant restraining force.
Redundancy of data also can be seen as a restraining force. Whenever data exists in more than one location, it is possible that the data will have different values for various reasons. This could result, in part, from the latency of data mentioned earlier. For example, the value of an account balance may be updated by the operational system but not forwarded to the EDW until some later date. At a given point in time, you could see two different values for the same account when looking at the EDW and the operational system.
Data quality issues are potentially a restraining force, because much depends on the quality of the data available. If data to be stored in the EDW is lacking in quality, there are options available for improving its quality. Changes could be made to improve data quality at the time it is entered. For existing data, the quality could be improved at the source. If that is not possible, the ETL software used to load data could be used to improve the quality of the data. Sometimes this is called data cleansing. This, of course, assumes the quality can be improved in some way that lends itself to programming. Data quality is a significant topic and you are encouraged to study it further if this is potentially a restraining force for your organization.
Finally, the brittleness of fixed record exchanges is a maintenance issue. If the EDW is changed in some way, it could create a need to change some or all the ETL programs. Because of the nature of fixed record exchanges, there is always a chance that not all ETL programs would be updated and the wrong data would be extracted. As a result, the transform and load portion could fail or the wrong portion of the record could be inappropriately transformed and loaded into the EDW, resulting in essentially a corrupted EDW. This brittleness problem is being addressed by the tagged record structure of XML or name/value pairs (see page 27). The tagged structure significantly reduces the chance of corrupting the data in the EDW and also presents the opportunity to reduce maintenance costs related to ETL programs. So, as a restraining force, the brittleness of fixed records will be reduced. Many of the restraining forces will be reduced because of efforts related to industry-standard semantic vocabularies and Web services as represented by the gray arrows in Figure 6.3.
Adopting an Enterprise Service Bus
Often when integrating systems, there is also a need to propagate data among internal systems. For example, if a customer’s address is changed in one internal system, you would want that change to appear as soon as possible in other internal systems.
Propagating data changes, however, can lead to complexity because of the possible number of connections among internal systems. If each internal system were directly connected to the other internal systems shown at the bottom in Figure 2.1, you could have up to 10 possible connections. Of course, if you need to propagate an update, such as a customer address, to multiple systems, you could end up in the situation shown in Figure 6.4. In this situation, every system potentially may need to communicate with every other internal system.
Figure 6.4 Possible connections for internal systems.
Message Routers
A good solution to this problem is to add a message router to internal systems, as shown in Figure 6.5. Such routers have been around for some time. They are also known as application routers. There are various ways a router could know the other internal systems that need to receive a certain type of update. The individual internal systems would not need to know who receives such updates. As a result, the number of interconnections is reduced, as shown in Figure 6.5.
Figure 6.5 Interconnections when using a message router.
A message router usually needs to transform the data in some way to match the format of the data expected by the receiving system. Figure 6.6 shows examples of such transformations. Internal system A at the left is sending data in tagged XML format. Internal system B at the right expects a tagged XML format but expects the tags to be different. For example, instead of the tag <name> in system A, system B expects the data to be tagged with <customer>. The tags for phone and postal code data also are different. Finally, system C expects a fixed record format. This fixed format is shown at the bottom in Figure 6.6.
Figure 6.6 Example transformations needed with a message router.
Adapters
These transformations do not occur automatically. Some type of adapter is needed to transform the data in the messages. Adapters also need to transform instructions that may be needed for communication. Some example instructions are starting a transaction, ending a transaction, getting query results, and so on.
The use of Web services and the development of SOAs created a need for a router that worked with Web services and that had adapters for various existing systems. Those capabilities are provided by an ESB. The term bus reflects the analogy of a computer hardware bus—a common architecture in computer design that uses standard connections. A computer bus makes it easier to transfer data and instructions among a computer’s subsystems. Similarly, an ESB makes it easier to transfer data and instructions among various software systems: services, business processes, applications, legacy systems, BI/analytics software, and so on.
Going back to the analogy of the audio-video (AV) system, the receiver plays a role that is similar to an ESB. For instance, you can assemble an AV system without a receiver just as you can use Web services without an ESB. Nevertheless, a receiver gives you more options and usually facilitates having multiple components—especially if differing ages of the components require different connections (like adapters that connect existing software systems).
An ESB can play multiple roles. As a router, an ESB monitors, logs, and controls routing of messages among services and systems. An ESB’s adapters enable the transformation and conversion of protocols and messages. The adapters ensure that the message vocabulary used within the ESB is the organization’s standard semantic vocabulary. The delegation of routing, protocol conversion, and message transformation to an ESB gives services and other software systems a convenient way to easily plug into a bus system.
There is no standard feature list for an ESB. If you are considering an ESB, be sure that it provides all the features you need. Figure 6.7 depicts an ESB with adapters for existing software systems.
Figure 6.7 ESB with adapters for existing software systems.
The force field analysis for adopting an ESB is shown in Figure 6.8. In this figure, a driving force is consistent enterprise-wide data in all applications. This means that customer data, for example, would be the same no matter what system used or managed that data. The impact on operational systems is minimized since any one system only needs to communicate with the message router and not all the other internal systems.
Figure 6.8 Force field analysis for adopting an ESB.
Most of the restraining forces are the issues with the semantics or meaning of the data and the standardization of data definitions that have been discussed earlier. Message routing, like EDW, needs to deal with semantic translation and this is shown as a restraining force. Over time, however, these restraining forces have become weaker as more industry standards become available.
Additional restraining forces include problems related to what data to route and the delay or latency in getting data updates distributed to various internal systems. The issues of what data to route and the delay of getting data updates distributed will likely always remain restraining forces. Data quality issues similar to EDW can occur with message routing. Obviously, it can be potentially disastrous to route poor-quality data. With message routing, however, you do not have the option of data cleansing used in conjunction with ETL software. The quality of data needs to be improved at the source for existing data and at the point of entry for new data.
Finally, the brittleness of fixed record exchanges is a maintenance issue.3 If the format of the record going to the message router is changed, it could create a problem. Because of the nature of fixed record exchanges, there is always a chance that the wrong data is routed. This brittleness problem is addressed by the tagged record structure of XML or name/value pairs. Such a structure significantly reduces the possibility of corrupting data routed by the ESB and presents an opportunity to reduce maintenance costs related to message routing programs. So, as a restraining force, the brittleness of fixed records will be reduced over time.
Web services adapters for packaged software provided by vendors also reduce costs of development. The adapters allow Web services connections with internally developed systems or packaged software. The arrow depicting the restraining force of development cost, however, is not shown as gray since there still can be other significant development costs related to an ESB.
An ESB can work with EDWs and existing middleware solutions such as an ORB. This is shown in Figure 6.7. The ESB would have adapters that, in turn, would connect to the EDW and ORB. Note that the ORB is represented as a bus much like an ESB and that it is labeled as “ORB services.” This is because an ORB provides communication for services much like an ESB.
Adopting a Service-Oriented Architecture
Web services, middleware integration (i.e. ORB services), data warehousing, and an ESB can all work together to support a service-oriented architecture. Figure 6.7 shows these technologies. This is essentially a more detailed diagram of C. R.’s organization, which was described in Chapter 2. In the bottom of Figure 2.1, you can see the internal systems in C. R.’s organization along with the repository. The three internal systems at the left in Figure 2.1 relate to the three systems at the left in Figure 6.7; this time we add the detail of middleware ORB services with an adapter for these internal systems. The two internal systems at the right in Figure 2.1 relate to the two systems at the right in Figure 6.7. The online repository at the bottom center in Figure 2.1 is shown as a data warehouse in Figure 6.7. Finally, the BI system shown connected to the repository in Figure 2.1 is also shown as connected to the data warehouse in Figure 6.7. The data warehouse, however, is not in a virtual private cloud as shown in Figure 2.1. The reasons C. R.’s organization made that change will be covered later.
The drive to use Web services is reducing the technical change issues. This makes moving to an SOA technically easier. Figure 6.9 shows how using Web services affects adoption of an SOA overall. This figure combines the force field analyses for Web services (see Figure 5.5), enterprise-wide software (see Figure 6.1), ORB middleware (see Figure 6.2), data warehousing (see Figure 6.3), and an ESB (see Figure 6.8). The combined technical restraining forces are shown at the right. The gray arrows represent the technical restraining issues that will diminish as industry adopts and expands the use of Web services.
Figure 6.9 Force field analysis of technical issues related to adopting an SOA.
Three restraining forces from enterprise-wide software (see Figure 6.1) were not added to Figure 6.9: departments have different needs, dependence on software products, and conversion to new software. These represent issues for moving to standard enterprise-wide software. Since an SOA does not require changing to enterprise-wide software, these restraining forces were dropped.
There is an addition to Figure 6.9 related to services. A restraining force has been added at the bottom right in Figure 6.9 for the identification and design of services. This is critical for an SOA and was discussed in Chapter 3 on page 30.
The analysis in Figure 6.9 is interesting because it illustrates that as the technical restraining forces shown in gray diminish, we are left with technical issues related to business and general design. The arrows at the top right represent business issues such as costs of development or concerns that a product doesn’t have all the features that might be needed. There are, of course, other design issues, but these arrows are representative of basic design issues facing any effort to create an SOA.
At the left in Figure 6.9 are the driving forces for adopting an SOA using Web services. The strength of these forces will vary by organization. Also, there very well might be additional driving forces for a particular organization. Nevertheless, by almost any measure, there are tremendous driving forces for the adoption of an SOA. You may want to try adding technical driving and restraining forces to this figure that are specific to your organization. There is space at the bottom of Figure 6.9 to add technical driving and restraining forces.
Figure 6.9 illustrates that there are some industry-wide technical issues remaining that restrain the adoption of SOAs, but those issues will diminish and, over time, the remaining restraining forces will be typical business and design issues.
This is not to diminish the business and design issues. They are not necessarily easy to solve, but they are the stuff of what developing an architecture is all about. Essentially, each organization must decide if it makes business sense to create an SOA using Web services. If it does, then there are design issues that need to be addressed.
Summary
This chapter focused on the technical change issues related to the adoption of a service-oriented architecture. It analyzed integration techniques that preceded SOAs and the ESB, which is often used in SOAs. At the end of the chapter, the analyses were assembled into a combined force field analysis of the technical change issues for adopting an SOA using Web services. The discussion showed that by combining these integration techniques:
The standardization efforts related to the use of Web services are assisting other integration techniques. This was shown in the weakening restraining forces for adopting an enterprise data warehouse and for ORB middleware.
Because the use of Web services does not require abandoning existing systems or data storage, this further reduces barriers to the adoption of an SOA as part of an integration strategy.
There are many driving forces for adopting SOAs.
Over time, many technical restraining forces will diminish and the remaining restraining forces will be typical business and design issues.
1For an explanation of the tagged record structure of XML and the brittleness of fixed record formats, see page 27.
2For some organizations, this can be a certain time of day. For others who cannot stop their operational systems, it may be necessary to provide small data extracts throughout the day.
3For an explanation of the brittleness of fixed formats, see page 27.