Yesterday we started testing on a ESBT based solution made up of a number of custom orchestration itinerary services. We had two environments built at roughly the same time with the same deployment components by the same developer. Only difference being one environment had Visual Studio and the other being a staging environment with no Visual Studio.
The solution worked perfectly time and time on the environment running Visual Studio but failed on teh staging environment. When I say fail I specifically mean the response message destined to correlate with the itinerary request response port would be published to the message box, the context properties used for correlation where set correctly and promoted which saw the message correlate back to the receive port service instance, but the response would never be routed back via the SOAP web service. In the BizTalk admin console we could see the receive ports service instance sitting there permanently in the active state with the response message correlated. Attempting to terminate would fail until we restarted the isolated host(iisreset).
The evidence didn’t add up. We picked the solution and environments back to the bones looking for vague security issues whilst we continued to test. Eventually we nailed the problem down to only occuring when including one specific custom itienrary service in the itinerary. Looking at the code there was a bit of complex message construction logic, basically the code cloned a message and copied the context properties a few times to work around a vague ESBG framework bug. Since the bug was no longer an issue the message construction logic was simplifed which saw the solution work on all environments everytime. So far I’ve been unable to nail down the exact cause of the problem, but somehow the orchestration in question was generating a message which was perfect and correlated back to the receive port but failed to be routed to the web service, a poison message. For future reference if I have issues like this look at the mesasge construction logic. In all the years I’ve been working with variants of BizTalk that was probably the strangest issue I’ve come across. If I find the exact cause I’ll post back.