Looks like I might be getting stuck into a bit more R&D. I’ve come to see an enterprise message bus type integration topology as being where BizTalk Server is going to add the most value in the organisation I’m currently contracted to. We’re implementing a large scale SOA so ESB might just pay dividends if done correctly. So time to pull up my sleeves and get elbow deep.
What is ESB
Vendors like IBM and BEA have been out there blowing their Enterprise Service Bus trumpet the loudest of late. However as any BizTalk developer would know BizTalk Server is very much based on a publish subscribe architecture which lends itself to being used as an ESB. An ESB is a core component of SOA that offers management of services for trading partner interactions implemented on a middleware platform such as BizTalk Server. Features an ESB should offer include,
- Content Based Routing(CBR). Quite simply participants are publishing messages to a message bus, those messages need to be routed to there appropriate service or process. No rocket science here. This is core BizTalk Server stuff. Message routing is done by default by target namepace#root node name in BizTalk Server or simply root node name if no target namespace has been defined. Routing rules can be made more granular by promoting properties of a schema and setting filter expressions on that promoted property. i.e. have subscriber A consumer messages with Person@Age < 21 and another subscriber consume message with Person@Age >= 21.
- Support for a range of transport protocols. Inevitably the organisation will not be taking a green fields approach and we will have to deal with legacy systems. An ESB should be equipped with the ability to support more than just web services. BizTalk Server really leaps ahead here coming pre-packaged with a number of adapters including Oracle, SAP, SQL, File, FTP, SOAP, WCF, WSE, and many more. This provides ESB participants a range of options when interacting with an enterprise service bus. For example we may want to publish messages to the bus via web services, we could have a secondary protocol of file drop if the web services are not available. Alternatively we may be dealing with a system that simply is not compatible with web services.
- Message transformation. Again as per the above point we are going to be dealing with legacy. An ESB must be able to transform messages for interaction with services from heterogeneous systems. BizTalk Server has a powerful mapping tool. Although I probably would shy away from performing complex transformation with the BizTalk mapper and revert back to good ole handcoded XSLT.
- Security. We need to be able to securely interact with the services exposed as part of our SOA. BizTalk Server is well integrated with Microsoft Enterprise Single Sign On. A lot of the the out of the box adapters come with SSO support and coding your own adapters for use with SSO is a no brainer.
- An ESB should be based on an extensible architecture. This is what I refer to as plug an play integration. As mentioned above BizTalk Server is based on a publish subscribe architecture that is inherrently loosley coupled and extensible. i.e. there is no direct binding between a message publisher and a mesasge subsciber. The only commonality is the message contract or schema. So lets say we want to publish a person search message to the message bus. Today we might be searching for people in "System A", tomorrow we might need to also search on "System B". It would be a simple matter of plugging a new subscriber for "System B" onto the bus so that the subscribers for both the "System A" and "System B" subcriber to the person search message type.
- Introspection. Inevitably organisations are going to need some form of auditability with and ESB implementation. BizTalk Server leaps ahead again here. Business Activity Monitoring(BAM) provides relevant business data and business milestone(KPI) information to business users via the BAM Portal with alert notifications. i.e. a business user can add a watch via the sharepoint BAM portal and receive an email notification when a message with certain content makes its way onto the bus. It wouldn’t be too hard to expose a specific view for auditability purposes, alternatively BizTalk tracking can be configured to provide this level of introspection via the Health and Activity Tracking(HAT) tool of BizTalk Server. But I see HAT as very much being for non business admin types.
So where to from here. Well I’ve got to turn that theory into practice and find some time to read through the Microsoft Enterprise Service Bus Guidance documentation. Then I can setup a lab and start fleshing out the detail.
It is an odd world where an organisation attempts to go down the SOA path without first defining standards. What do Microsoft say are the phases of SOA delivery. Expose, Compose, Consume. It is vital to define service standards before that expose phase, it is vital to define approved orchestration patterns and sort out things like message subsciption and security standards before that compose phase commences, and it is vital to define UI and usability standards before that consume phase begins.
Canonical Data Model
A logical step would be to go around and map out a canonical view of all the entities in the organisation. We currently have as many definitions of entities as we have information systems. This one view of entities will form the basis for the organisations XML dialect and possibly assist with future database and business documents design. At the moment everyone pretty much does there own thing without guidance from the data manager, not cool. I would think if SOA needs anything to be successful it needs standards.
A data dictionary would include a definition of what each attribute of an entity actually is, but also a canonical view of all code table values/enumerations. i.e. why should an organisation have 1.n permutations of the eye colour "blue", n being the number of information systems present in the organisation. Blue is Blue. Agree on a definition and add it to a data dictionary. If a change to the organisation definition of "Blue" is required it should be the data manager consulting with technical teams to update various code tables, update translation sets used in B2B and legacy EAI scenarios, and release new versions of XML schema as appropriate.
Nothing revolutionary here, I’ve already done some research in this area. BizTalk Server is well integrated with Microsoft Enterprise Single Sign On. This will help us negotiate the different security models present in the organisation. Although I honestly hope most integration will be done via services with a common approach to authentication and message security. But SSO lets us map windows accounts to Oracle, SQL Server, SAP, etc accounts and stores the credentials in an encrypted store.
I was lucky enough to be heavily involved in setting up the infrastructure for my current employers BizTalk Server environment. For confidentiality reasons I of course won’t go into specifics but suffice to say we have site DR and fail over. If we loose a site we automatically fail over to another site. The important thing to note is an ESB must be delivered on a highly available infrastructure with no single point of failure. A BizTalk environment can be setup as a group with applications deployed in symmetrical fashion. Scaling out the environment is simply a matter of plugging new BizTalk Servers into the farm. There are stacks of whitepapers on this available on MSDN.
I’ve got a lot to learn.