BizTalk ESB Toolkit Claim Check

Recently I’ve been working with a client implementing an ESB on BizTalk Server 2010 naturally using the ESB Toolkit 2.1.  Along with a number of regular VETRO type interfaces that lend themselves to itinerary processing the client has a need to route a few large files around.  Why use the ESB Toolkit for this.  Well basically because all routing rules for all files are stored in UDDI and managed via an extended ESB Toolkit Portal.  So for the client it makes sense to manage all that routing config in the one place the have all routing controlled by the  service bus.

What we need to do here is avoid injecting large messages into the BizTalk message box database.  If injecting large messages into the message box the type of undesirable scenarios we could encounter include slow performance, message box bloat due to large suspended messages which may trigger throttling, and if a large message fails and makes it to the ESB Toolkit Exception database we want to avoid injecting huge messages there too.

How do we process the message with BizTalk while bypassing the mesage box.  We use what’s called a claim check pattern.  A bit of background on this integration pattern can be seen from Hohpe and Woolf here.  http://www.eaipatterns.com/StoreInLibrary.html  As the large message is received it’s checked into a repository, then a command message is sent through BizTalk.  That command message is seeded with a messageId that allows us to claim the message in the send port.  So the large message never actually goes into the message box.

In a traditional type BizTalk solution we’d write a receive pipeline component to check the message into a repository.  Then we’d write a send pipeline component to read the command message and claim the large message from the repository.  The repository can be any number of technologies.  MSMQ for example.

So how do we do this in a BizTalk solution based on the ESB Toolkit.  Its pretty easy actually!  All you need do is write a couple of itinerary services.  Your Itinerary will end up looking something like the below.  Note the “Check Message” service running under the On Ramp and the “Claim Message” service running under the Off Ramp.

For information on how to write a custom messaging itinerary service see this paint by numbers guide.  http://msdn.microsoft.com/en-us/library/ee250158(v=bts.10).aspx

The below are code fragments from a quick proof of concept, hence the hack coding.  In the below “Check” service the ResetMessageSteam method simply reads the stream which is the large binary file being received by the receive port.  ResetMessageStream then injects a command message with a message id that will later be used to claim the message.  ResetMessageStream then just returns the stream which is in this case written to the file system which is acting as my claim check repository.

The Check itinerary messaging service is registered with the ESB as an “OnRampReceive”.  See the Microsoft doco referenced above for more detail on this.  But this basically means the Check Itinerary service can be configured to run under the ESB On Ramp receive port much as a receive pipeline component would in a traditional BizTalk solution.

The below code fragment is used to claim the message on the other side of the message box.  So the command message has passed through the message box, routing details are retrieved for that command message from UDDI.  This routing information is then injected by ESBT into the message context.  The “Claim” itinerary messaging service claims the large message, in this case back from the file system.  The large file is injected into the outbound message stream and the command message is discarded.  We register the Claim itinerary messaging service as a “OffRampSend”.  See the Microsoft doco mentioned above for more detail on this.  Essentially this means the “Claim” service can be configured to run under an OffRamp send port.  Again the code below is a hack from a fast proof of concept.

So there we have it.  This is a pure messaging itinerary based ESB Toolkit solution that implements the Claim Check integration pattern and allows us to bypass the message box with our large messages.

Advertisements
This entry was posted in BizTalk Server. Bookmark the permalink.

2 Responses to BizTalk ESB Toolkit Claim Check

  1. Hi,
    We’re trying to implement this pattern wit ESB Toolkit 2.2. Everything works great except in case of delivery failure in the Off Ramp. The Claim service has already deleted the large file when we’re resuming suspended instance of the message. Any idea on how to implement this correctly ?

    • The demo code in this post isn’t really intended to be production ready for the reasons you’ve encountered. Although I’ve seen the concepts from this solution taken and applied in a number of ways. Most implementations go with a non-destructive claim. i.e. don’t delete the file or message when claiming. This is so a resume of a suspended instance will still have the file available in the claim repository, this also allows send port retires to reclaim the file from the claim repository. The problem with a non-destructive claim being that large messages build up in the claim repository so additionally logic needs to be employed to maintain the claim repository.

      One implementation processed large files seldom, 2 or 3 per week so as part of a bi-weekly environment health check they manually cleaned up the repository. Another implementation ran a scheduled job to maintain the repository which deleted all files in the repository with a file creation date > x number of days. Yet another implementation has the claim agent itself maintaining the repository. In that case each time a claim runs the last bit of logic was to destroy any files older than x days.

      I’ve also seen an implementation where the policy was to keep all exported files. So the repository was considered an archive. All files written to that archive were kept in that archive permanently as a record of what had been exported. In that case the developers simply implemented the claim with a non-destructive claim and didn’t need to implement further logic to maintain the claim repository.

      The most recent implementation I’ve seen has a nice standardized exception management framework, as is the case with ESBT. Although it is customized somewhat, however conceptually the same as ESBT. All exceptions occurring within the integration layer are handled by the one central exception management port, as is the case with ALL.Exceptions in ESBT. In this case the retries on the port executing the claim are set to 0, failed message routing is also enabled on the send port. This means no suspended instance is logged by design, the routing failure is instead immediately routed to the exception portal for action by operations support. The message body routed to the exception handler is the claim message which includes the original file name. In this case operations support would trigger the resubmit of the failed message from the publisher of the original message. No resume of suspended instances, no retires, and no requirement to maintain a repository as they have implemented a destructive claim much the same as the code in this post.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s