Integration process monitoring - tool for everybody

Typically integration process monitoring has meant tools for IT: either for developers or for operational personel. On the other hand, reporting in many cases is developed only to match business’ needs. This kind of approach leaves users in both groups outside the true potential of well planned monitoring solution. I think that integration monitoring can be for everyone, both techical people and for business.

Monitoring product to rescue?

In my earlier post, I wrote about the qualities of that I would look for when choosing an integration monitoring solution. One of the most common pitfalls is that most of the tools in the market (at least that I know of) try to solve the problems of the technical persons and therefore are very technically oriented. You probably can drill down to the CPU usage of the servers or see details of message throughput, but if you wan’t to see whether your customer has received and then paid the bill you sent to her, you are likely out of luck.

Some tools like BizTalk360 provide limited visibility also for business users, but from the sound of the marketing material the focus is not in serving business users: “BizTalk360 UI is fully customizable, give limited access to your business Users, So they don’t bother your technical people for trivial things.” This is not to say that these kind tools are not good. In fact they do the job thay have been designed to do, and in many cases rather well. Some of them even claim to be self learning and to adapt to your environment’s behaviour. These tools also provide some reporting capabilities for IT.

One key feature of the previously mentioned products is that they sit directly on top of an integration platform product. This is naturally appeling for the customer as deployment of these products doesn’t require any customizations of the existing integration processess. The functionality is build on top of the integration platform itself, not the integration processes. There are few problems with this kind of tools though:

• You can only monitor the part of the process that is executed on the integration platform. In current business environment, integration processes are being scattered around the IT landscape: some parts might be executed in cloud and only small part goes through your ESB. The key is unified monitoring.
• You can’t adapt to the needs of the actual business process. For example if the process is executed in many phases, there most likely is no way to see one unified visualization of the process.
• Content of the data being transferred within the integration processes usually can’t be used for monitoring decision making (alerts, notifications), only the technical properties.

The other thing with these technically oriented monitoring tools on top of the integration platform products is that they can only cover small part of the most common problems that I have seen in many integration environments. In my experience, very rarely the problem at hand is the platform itself. Todays integration products are rather mature and server platforms stable. If I would have to name three most common sources for integration problems they would be:

• Bad data quality and human errors (ironic don’t you think, we are talking about automated processes after all…)
• Poorly designed integration solutions (business process is not clear, all use cases are not covered, poor quality of implementation)
• Network issues

Especially data quality is an issue that is hard to tackle. From the technical point of view your process might work just as expected, but business is not happy. One example: employee’s cost center is not up-to-date in master data and therefore all her travel expenses are booked to wrong bookkeeping account. This kind of problems should not end up to the integration consultant’s desk, but if the business user does not have any tools to solve the problem by herself, the only outcome is that the technical guys end up using their time solving non-technical problems.

My experiences of the ready-made monitoring products are unfortunately limited mainly to Microsoft world and to monitoring integration processes build on top of BizTalk Server or Azure. The experience that I have for example with Mule ESB hasn’t given much brighter view of the situation on that front. I think we are missing product that would have a bit broader approach to integration process monitoring than the current ones. I also don’t believe in free lunches: trying to hook the monitoring tool directly to an integration product can’t give the flexibility and process insights that are needed for a tool that can serve all user groups. Am I wrong?

Let’s dream

Since I havent’t found monitoring product that would meet the requirements I have (and no-one has yet pointed that I’m just crappy in Googling), I’ll try to explain what I would like to see in terms of serving different user groups.

Let’s imagine that we have a business process that is two-fold. In first part a message is processed and sent to a partner. Once the message is delivered to the partner, there might be a delay of day or two while the partner does their thing. Partner then replies with another message that is processed. Few notable points:

• From technical point of view there are two separate processes that are handled asynchronously
• From the business point of view these two parts should be considered as one process as it only matters whether the whole chain of communication has been completed or not
• As the messages go along the integration platforms execution pipeline, there are some steps that are totally irrelevant to the business user and should be invisible for her. On the other hand the technical persons need to see that also those steps have been executed successfully.

Here is visual presentation of the situation:

On the top is the process representation that the business user wants to see. Simple “traffic light” presentation whether the status of the process is ok and the main phases: started, in progress, finished. Also the two parts are shown as one process in the monitoring view. On the bottom is the technical counterpart: Two separate processes with all the detailed steps included. Each step in the technical processes present an event that is sent from the execution.

The details of the technical implementation in each step is not relevant (from the process point of view) as long as the process instance has an unique identifier that is included in each event. For example first part of the process could be executed on top of the ESB product in on-premise environment and the second part in an independed microservice deployed to a public cloud.

Naturally also the details that these separate user groups want to see about each step vary: the business user probably is not interested in the execution time of the validation step, but for example if this was an order-delivery process, the order number would be critical.

How can this kind of functionality be achieved? The answer is rather simple, but often in reality harder that you would think: You need to know the business process first. The key is being able to model the business process to the smallest detail and then to define the required steps needed to execute the process. After all the steps of the process have been identified, a subset of the steps can be picked for each user group for presenting the process. This way the “integration code” doesn’t need to know much about the business process, it only needs to be able to send event with correct identifiers. The meaning of the event is then intepreted within the monitoring tool and stored to the monitoring data store:

Visualization of the process is done according to the needs of the user groups:

Unfortunately business processes are in many cases much like moving targets: once you think you have reached the goal, the goal has moved. This is when the profession of the integration consultant is measured. Being able to help the customer to define and finalize the business processes as early as possible before the actual implementation starts is an invaluable skill.

In the next post I will take some of these ideas to reality and show how you can implement a simple event collection solution using Azure technologires with just few lines of code. Stay tuned!