Managing Change with Incremental SOA Analysis
Service-oriented architecture (SOA) projects are no different from other IT projects in that larger projects tend to fail and issues regarding change can scuttle projects. This chapter introduces incremental SOA analysis. It aims to improve the project selection process in a way that also improves the chance of success for the selected project. This analysis takes into account both project size and the human change issues discussed in the previous two chapters.
The incremental SOA analysis uses three tools that address change issues. Two of the tools were discussed in earlier chapters: force field analysis and the resistance issues and suggestions worksheets. This chapter introduces a third tool: the decomposition matrix. The tools are intended to engage people in such a way that they can come to their own resolution on what might be causing human change issues.
Tools
People are much more likely to deal better with change issues if they are engaged in the change process. Chapter 8 suggested possible ways to address change issues. Of those suggestions, the three tools used in this chapter allow you to:
It is important to try using all three tools in a group setting—with the appropriate participants, of course. The tools are intended to get people talking and, hopefully, thinking differently about their design work.
Force Field Analysis
Chapters 5–7 discussed force field analysis. It engages people in the process of identifying change issues. Force field analysis can be used in a group setting if you use something like a whiteboard or flip chart.
Worksheet for Resistance Issues and Suggestions
The worksheet for resistance issues and suggestions introduced in Chapter 8 also allows a group to problem solve. As with force field analysis, the worksheets can be used with whiteboards or flip charts. The worksheets start with the resistance issues identified in the force field analysis.
Decomposition Matrix
The decomposition matrix tool generates either business process or data flow diagrams. It does this using an algebra for design decomposition that Mike Adler published in the 1980s.1
A feature of the decomposition matrix is that it does not look at all like a business process or data flow diagram. Business process diagrams, for example, are a great way to design a workflow. The problem for most of us, however, is that if we are familiar with a given workflow, it is often difficult to see how it could be significantly different. We all tend to repeat or recreate what we know. The decomposition matrix, however, requires us to only think about inputs, outputs, and how they relate to each other. The diagrams are generated for you based on the matrix of inputs, outputs, and relationships.
I have the decomposition matrix tool implemented on one of my websites.2 It is free to use. It can be used in a group setting if you have a computer with an Internet connection hooked up to a projector.
Figure 10.1 shows a decomposition matrix of inputs, outputs, and relationships. It allows you to discuss detailed issues one at a time instead of trying to juggle multiple issues all together in a design. You only need to make a series of binary decisions. Such a decision is whether a given input is related to a given output. Sometimes that can generate a great deal of discussion and bring out design issues not previously mentioned. The decomposition matrix assembles these simple decisions and generates a decomposition that might help you with your design process.
Figure 10.1 Decomposition matrix example.
The tool on my website generates either business process or data flow diagrams. Most people are familiar with business process diagrams. The data flow diagrams are a way to get at the decomposition of services in an SOA. The decomposition matrix has a specific definition of atomicity. Atomicity generally means that a business process cannot be decomposed further (see page 17 for a general definition of atomic services). The specific definition of atomicity used by the decomposition matrix is that a business process task or a data flow process is atomic if every input relates to every output in the decomposition matrix. In other words, there are check marks in every box of the matrix. Atomic tasks and processes are an important aspect of the incremental SOA analysis.
It is possible that the decomposition matrix might give you some new ideas or help you get past a sticking point in your design process. In that way, it acts much like having another designer in the room. The decomposition matrix is not a design methodology. It is meant to be a design aid. You can use it with whatever methodology you prefer since it is just another “designer” in the room.
The next section provides an example of how this tool works.
Business Process Diagram
To illustrate how the decomposition matrix works, I will use an example from a series of blog posts that start at http://www.designdecomposition.com/blog/?p=6. This example uses a set of inputs and outputs for a travel coordinator. Using those inputs and outputs, the decomposition matrix tool will generate a business process diagram.
The inputs and outputs in Figure 10.1 should be familiar to most people who have taken a business trip. They involve finding airline flights, a rental car, and hotel rooms for a set of travel dates along with making the reservations and obtaining driving instructions. Figure 10.1 shows this decomposition matrix.
You need to consider the relationship between only one row and one column at a time when using the decomposition matrix. These are the binary decisions mentioned earlier. For example, you could describe the relationship of the first row and first column as “the input of travel dates and locations that occurs before or concurrently with the output for a flight availability request.”
The portion in italics is an example of the type of phrasing you should use. You may read across the row or down a column using the italicized phrasing.
Considering just one row and one column at a time makes it easier to work with larger designs. There is no need to try to keep the entire design in your head. You just need to think about each relationship one at a time.
Arranging flights involves using the travel dates and locations to request a list of available flights. Sometimes you may need to make multiple requests with different flight times or you may make requests to multiple airlines. Figure 10.1 shows this with a check mark in the second row, flight availability response, and first column, flight availability request. The third row, flight reservation response, is not checked in the first column, because you cannot have a response before a request.
The fourth column shows the inputs that occur before or concurrently with the input to a car rental reservation request. Before making a reservation request, you need to know that cars are available for your travel dates and locations. You also need to know if flights and hotel rooms are available. You do not, however, need to reserve a room before a car. On the other hand, car rental agencies often ask for a flight number at the time of rental. So there is a check mark in the third row, flight reservation response, for the fourth column. This occurs before or concurrently with the output for a car rental reservation request.
The generated business process diagram is shown in Figure 10.2. The diagram uses a subset of the business process modeling notation (BPMN).3 The tool does not generate labels for the tasks. I have added task labels to this diagram.
Figure 10.2 Generated business process diagram.
There are a couple of ways the generated diagram can give you hints that there are problems with the check marks in your decomposition matrix:
If you have had trouble coming up with any of the labels, that could be a hint that the inputs and outputs might not have the correct check marks or perhaps an input or output was overlooked.
If the diagram is confusing, that is a hint that the check marks might not be correct. An example of something confusing is a request for something coming in after its related response.
You can “play” with the inputs and outputs to see what happens to the generated diagram. This is not a complete design tool. At some point you may want to transcribe a generated diagram into your design tool, much like you would if you used a whiteboard.
Data Flow Diagram
The next example generates a Web services API or services interface layer for legacy systems. Figure 10.3 shows the decomposition matrix. The inputs are from some type of legacy system. Some of the possible outputs are also shown in the decomposition matrix. It is obviously simpler than the real world, but it serves as an illustration of how the tool can be used.
Figure 10.3 Decomposition matrix for services.
You can phrase a relationship in Figure 10.3 as “the input of invoice is used directly or indirectly for the output of payments.” The italicized portion of the phrase is important. Note that this is different from how relationships are described for business process diagrams. In this case, we are dealing with data flow and not the sequencing that business process tasks require.
Figure 10.4 shows the decomposition of services based on the matrix.4 The processes have been labeled. Just like with the business process diagrams, the tool leaves labeling up to the user. Again, if it is difficult to label a process or if the diagram is confusing, that is a hint that the inputs and outputs may not be complete or that some check marks are missing.
Figure 10.4 Decomposition of services.
The top-level processes in Figure 10.4 represent the Web services API or service interface layer. Some of the top-level processes have multiple outputs. This indicates that the input parameters will need to specify the XML tags (in this case) to include in the output. Such input parameters are not shown in data flow diagrams, but they will be needed when you design the services. Any data flow diagram shows only the flow of data and not the control input parameters.
The services below the top level are reusable components that have been factored out. Depending on your implementation, you could implement them as services or as library code components.
Just like with the business process decomposition, this tool allows you to “play” with inputs and outputs to see the effects. At some point, you will want to transcribe the decomposition into your design tool.
Five Principles for the Incremental SOA Analysis
The incremental SOA analysis uses these three tools in a way that improves the chances of success for a project. There are five principles that provide the basis for the incremental SOA analysis:
1. Make projects as small as possible. This has already been discussed in the previous chapter, but in this technique “small” has a specific meaning. Projects involve only a single atomic task in the business process diagram generated from the business process analysis. For example, each of the tasks in Figure 10.2 would be a separate project.
2. Involve stakeholders appropriately and as much as possible. Engaging the appropriate people was discussed in Chapter 8. The incremental SOA analysis is designed for this type of engagement.
3. Make decisions as late as possible. The later you can make a decision, the more likely you will have accurate or more complete information on which to make your decision.
4. Weaken the restraining forces within the project as much as possible. Chapter 5 introduced force field analysis and described why weakening restraining forces is often better than strengthening driving forces. By weakening restraining forces, you are increasing your chances of success.
5. Realize that your SOA will never be done. For most organizations, an SOA will be ever changing because it will need to respond to the changing nature of business and technology. The primary goal of this incremental SOA analysis is to eventually position your organization so that it can respond quickly to those changes. It will provide you with a loosely coupled (see page 31) architecture that should improve your organization’s ability to change. A secondary goal is to leave you with functioning architecture whenever you stop. Budgets and other demands often derail the best-laid development plans. With this type of analysis, you should be able to restart your SOA development at some later time if work is suspended for some reason.
Incremental SOA Analysis
Figure 10.5 shows the process for incremental SOA analysis. The workflow shown in Figure 10.5 is my suggestion for how to implement the five principles for the incremental SOA analysis.
Figure 10.5 Incremental SOA analysis.
This analysis is shown as a workflow because the diagramming provides a rigor beyond a textual description. The following sections provide some notes for each of the tasks and processes in this analysis.
Business Process Analysis Lane
The business process analysis lane is where the analysis starts. (The workflow in Figure 10.5 is divided into five lanes—the business process analysis lane is at the top.) The purpose of the activity in this lane is to develop a small number of candidate projects that can move on to candidate project analysis described in the next lane. The intent is not to analyze all of the organization’s processes. The assumption is that there are some known opportunities for improvement suitable for analysis. The three tasks in this lane are described in the following subsections.
Analyze the business process with decomposition matrix or other technique
If you have a preferred analysis technique, use it. If you don’t, you might consider using the decomposition matrix tool described earlier in this chapter.
The decomposition matrix provides a way to think differently about the system you are about to analyze. This tool can be used to get information from the various stakeholders. Ideally, you should do this in a group setting to allow the stakeholders to share their views on the business process.
Modify the business process
Modify the business process until all tasks are atomic.
Analyze the business process for services
Review the atomic tasks for the candidate project. Restricting work to an atomic task is part of the principle of making projects as small as possible. A candidate project should:
Be noticeably different. The point here is to avoid just replacing something that most people don’t really see. For example, you could use Web services to replace an in-house communication protocol. There is nothing wrong with that. It just may not impress too many people with the power of SOA. What might impress people is using the connection capability of Web services to combine internal information with something from the cloud so that a business process is enhanced or made simpler.
Be the only project you do for now. What if you only had time and money to do one SOA-related project? That is consistent with the principle that your SOA is never done but that you can build it incrementally as time and money permits. So, pick something that is useful without needing a follow-up project.
Candidate Project Analysis Lane
The candidate project analysis lane analyzes the candidate project using force field analysis along with the resistance issues and suggestions worksheets. This lane adds an approach that might allow you to make projects even smaller.
Use Force Field Analysis for Each Project
As mentioned in Chapter 5, force field analysis uncovers the driving and restraining forces for the desired change related to the candidate project. You can get a group involved with the visual nature of force field analysis using flip charts or a whiteboard. Having a group inspect the completed force field analysis may allow you to discover that a project can be made smaller. For example, you may find that a restraining force is the lack of a tool to develop the service interface. You could decide that experimenting with development tools is a project unto itself. Therefore, the candidate project could be divided into two projects. One project is tool experimentation and selection. By dividing the candidate project into two projects, you eliminate a restraining force on the original candidate project and you get two smaller projects—one that is only tool selection. Presumably, the selected tool will also be used in future projects.
Use the Resistance Issues and Suggestions Worksheet for Each Project
The worksheet for resistance issues and suggestions (see page 102) lists the restraining forces found in the force field analysis and provides space for entering the suggested ways to address each restraining force. As with force field analysis, inspecting the completed worksheet may allow you to discover a way to make a project smaller. For example, one restraining force might be the lack of experience with Web services and another might be unfamiliarity with XML. The suggestions in the worksheet to address both restraining forces might be a combined XML and Web services course. That course could be a separate project. The original project could be divided into two projects. In this way, you eliminate two restraining forces on the original candidate project and you get a smaller project. In this case, the smaller project is training.
Add the Project to the Candidate Pool
If force field analysis and the resistance issues and suggestions worksheet cannot make the candidate project smaller, then that project can be added to the candidate pool. You should have at least a few projects in the pool before selecting a project for deployment.
Deployment Selection Lane
The deployment selection lane selects the project for deployment. Only one process appears in this lane, but for a given organization there could be more processes or tasks based on the organization’s project selection criteria. For example, some type of financial justification might need to be added at this point as a factor to be considered in project selection.
Select a Project with the Best Chance of Success
Following the principles of this analysis, pick the project with the shortest duration. It will most likely be the one with the greatest chance of success. The duration of the project is based on the estimation technique your organization uses, and the chance of success is determined based on inspection of the final force field analysis and worksheet for resistance issues and suggestions. Of course, these forces sometimes can be difficult to quantify. Nevertheless, the force field analysis and worksheet provide a way to inform you whether one project versus another project is more likely to succeed.
Note that there is a “+” at the base of this process in Figure 10.5. That indicates that there are subprocesses. Since these subprocesses can vary by organization, the details are not shown in the figure.
Deployment Lane
This lane has the workflow for deployment. Notice that decisions on vocabulary and interface parameters have been deferred until this time in keeping with the principle of making decisions as late as possible.
Analyze Vocabulary Needed for Interface
You might not have expected that the semantic vocabulary needs are deferred until this point. In reality, the semantic vocabulary needed is not a factor until this point. If additional vocabulary is needed, the workflow will move to the vocabulary management lane.
Analyze Parameters Needed for Interface
Here you need to determine the parameters required for the service interface. If there is a change in vocabulary or parameters, then it will be necessary to consider refactoring services.
Refactor Services Using Decomposition Matrix or other Technique
If there is a need for additional vocabulary or parameters for the interface, then there is a possibility that the services may need to be refactored. The refactoring of services is part of deployment to keep the project self-contained and complete at the end of deployment. This adheres to the principle where you need to assume your SOA may never be done. You want to be at a reasonable stopping point of completion at the end of every project. Refactoring ensures that the services are at the right level of service granularity at the end of each project.
The refactoring of services suggests using the decomposition matrix or using a technique of your choice to refactor services. If you use the decomposition matrix, the website will generate diagrams with processes that can be either services or internal functions. Each process will be factored at an atomic level. Those processes that have interfaces with the business processes will be part of a service interface. The processes that do not interface with business processes will most likely be implemented as a function. Of course, you may find yourself refactoring functions into services based on the needs of that future project. Nevertheless, you should not anticipate refactoring. The next project or projects may not require refactoring. This way, you have just as many services needed right now as opposed to creating additional services in anticipation of future needs. Besides, you might guess wrong on the factoring of future services.
Deploy Services and Business Processes
The deployment of services and business processes is shown as a process in the model because organizations will have multiple tasks related to deployment. The model ends after deployment. At this point, the system should be at a stable state with new and or updated services, a semantic vocabulary sufficient for the services, and all services at the right level of granularity.
Note that there is a “+” at the base of this process in Figure 10.5. That indicates that there are subprocesses. Since these subprocesses can vary by organization, the details are not shown in the figure.
Vocabulary Management Lane
The vocabulary management lane supports the deployment lane in cases where additional semantic vocabulary is needed.
Review Industry-Specific Vocabularies
No one should develop a new semantic vocabulary if it can be avoided. Developing a vocabulary can become a black hole from which you may not return. In many organizations, it is easy to find differing definitions of such common terms as serial number or account. It is equally easy to find differing terms that have the same definition. Arguing over who is right can be never ending.
The increasing global reach of even the smallest of organizations means that it is probably more important to use vocabulary terms and meanings consistent with the rest of the world rather than consistent within an organization. This is one reason industry groups developed standard semantic vocabularies. It is best to adopt that part of the industry vocabulary that is needed for the project’s service interface. There is a partial list of industry vocabularies on page 179. You can also use a search engine to find semantic vocabularies that apply to your industry.
Review Cross-Industry Vocabularies
If you cannot find a vocabulary designed specifically for your organization, then you should look to cross-industry vocabularies. A good place to start is the Universal Business Language (UBL), which is an OASIS standard (http://ubl.xml.org).
Develop Organization-Specific Vocabulary
As a last resort, develop an organization-specific vocabulary. If you find this necessary, develop only what is needed when it is needed. As mentioned before, this effort can easily become a black hole.
Add to the Organization’s Semantic Vocabulary
In whatever way you determine additions to the semantic vocabulary, add only what is needed when it is needed. This is in keeping with the principle of making decisions as late as possible. It also reduces the number of vocabulary decisions to only those needed to support the current project, thus keeping the project duration and complexity to a minimum.
Summary
This chapter showed how to coordinate the use of three tools to help in managing change: force field analysis, the worksheet for resistance issues and suggestions, and the decomposition matrix. Using these tools can engage people in such a way that they might come to their own resolutions on technical and human change issues. Finally, this chapter showed how to integrate these tools in an incremental SOA analysis with the aim of reducing project size and increasing the chances of project success.
1Mike Adler, “An Algebra for Data Flow Diagram Process Decomposition,” IEEE Transactions on Software Engineering, 14(2), (Feb. 1988).
2“Design Decomposition for Business Process and Data Flow Diagrams,” Barry & Associates, http://www.designdecomposition.com/.
3Business process modeling notation (BPMN), Object Management Group, http://www.bpmn.org/.
4At the time the website was implemented, the Service Oriented Architecture Modeling Language (SoaML) notation was not available. If you wish, it is not hard to manually create an SoaML diagram from the data flow diagrams. SoaML specifications are from the Object Management Group, http://www.omg.org/spec/SoaML/.