How to Analyze KPI Violations [Part 1]

This post is a part of Apromore’s use-case series, where we discuss how to use Apromore for various process mining use cases. 

In business process mining, discovering KPI violations is only the first step. Business stakeholders need to further investigate causes of these violations, the impact they have on the process and make informed decisions to forestall them.  

This article shows how to analyze KPI violations in Apromore. We will create a KPI, determine the number of KPI violations and analyze these KPI violations using various techniques. We will use the Dispute Management log for our illustration. In this process, customers of a credit card provider raise claims when they detect a duplicate or an incorrect charge in their payment card. 

Let’s get started. 

Create a KPI 

In the dispute management process, a case is completed in three stages: 

• Request intake. 
• Case creation. 
• Case fulfilment. 

The business stakeholders expect that the case fulfilment stage should not take more than 2 weeks. In other words, all activities performed during the case fulfilment stage should not be greater than 2 weeks. We can therefore, create a KPI that tracks this condition.  

To begin, create a KPI and name it “Case Fulfilment Performance”. In KPI population, add an event attribute filter that retains cases that contains the “Case fulfilment” stage. This is because, some cases may not have arrived at the case fulfilment stage and such cases should not be a part of the KPI. 

In KPI condition, add two conditions. First, add an event attribute filter that retains all activity instances in the case fulfilment stage. In other words, we consider only activities performed in this stage. We can now add the second filter that retains cases where the case duration is less than 2 weeks. We can also add a basic KPI target to be greater than 80%. 

We see that the KPI was met in 55.01% of the cases, which is less than our 80%. Let’s now investigate situations where the KPI was not met. 

Perform Root Cause of KPI Violations 

When we add this KPI to the event log, we can investigate root causes for KPI violations. In the KPI window, click Investigate. Apromore provides conditions for violation causes for the KPI. 

In the first violation cause, we see that when the merchant is Etsy and the card type is credit, the KPI is violated in 97.1% of the case versus 34.8% when these conditions are not true. Let us zoom in on this cause and understand what is unique about Etsy merchant and credit card type that causes a substantial KPI violation. 

Filter Cases that Violate the KPI Condition. 

To understand the effect of Etsy merchant and credit card type on our process, we can create a filter that retains cases where the merchant is “Etsy” and the card type is “Credit”. Create this filter in Process Discoverer and save this log as “Violation cause log.” For comparison, we can create a second filter that removes cases where either condition holds. Save this log as “Violation cause negated log”. In both logs, also apply an event attribute filter that retains only activities in the case fulfilment stage. 

From the log statistics view, we can see some difference between both logs. In the violation cause log, (i.e., cases with Etsy merchant and credit card type), we see the variability ratio increase to 20.8% from 9.4% in the violation cause negated log. This higher variability ratio is indicative of an increase in non-standardized process and deviations to the process flow in cases with Etsy merchant and credit cards. 

We also see the average case duration in the violation cause log increase to 2.92 weeks from 1.58 weeks. 

Violation cause log 

Violation cause negated log 

Let’s also check the bottlenecks when our violation causes hold. 

Bottleneck Analysis 

To view the bottlenecks in the process, go to the Duration overlay and select Average. We see an average waiting time of 3.63 days before the “Generate Acknowledgement Email for Customer” activity is performed. Conversely, in the log where the violation cause was not true, the average waiting time for the same activity was 2.79 days. This 20.16-hour difference in waiting time indicates that when for disputes with Etsy merchant and credit card, there is an increased bottleneck when generating the acknowledgement email. This could mean that such disputes may be inherently complex, requiring more resource allocation. 

Furthermore, we see an increase in the average duration for the “Alert Merchant via Ethoca Alerts” activity. For cases involving Etsy and credit card type, the activity took 6.18 hours, but takes 4.96 hours otherwise. This could mean more complex communication protocol when communicating with Etsy and using credit cards. Business managers can leverage domain knowledge to address the underlying reasons for the longer delay with Etsy. 

Violation cause log 

Violation cause negated log 

Let’s now use Apromore Dashboard to understand the difference in operational performance between both logs.  

Performance Comparative Analysis 

We can compare the performance statistics of both event logs in Apromore. Select the saved event logs in Portal, right-click and click Launch Dashboard. This creates a comparative dashboard between both event logs. 

One thing is striking. It appears cases involving Etsy merchant and credit card type peaks from May to October, 2022, while other cases tanks during this period. This indicates a seasonal trend, perhaps due to shopping patterns or promotions. There is simply a higher volume of disputes involving credit card purchases from the Etsy merchant within this period. As a remedy, the business manager could add more resource allocation from May to October to reduce the bottleneck during this period. Alternatively, they could automate some tasks to reduce the workload on the available resource. 

We have established that the case duration is longer when the merchant is Etsy and a credit card is used. But we can dig deeper to investigate the activities that cause this increase in average duration. In a new view, create a chart that displays the average case duration for each activity. 

We see that activities such as “Open case” and “Close case” take about 2X more with Etsy merchant and credit card disputes while “Generate Rejection Email for Customer” and “Send Rejection Email to Customer” take more than 4X more time on average. This implies that when specific activities, such as "Open case" and "Send Rejection Email," occur within cases with Etsy merchants and credit card, the average duration for these cases is higher. This could mean these activities are resource intensive for cases involving Etsy and credit card type, or there could be some operational bottleneck when dealing with these cases. 

Let’s now see if there is any difference in rework duration. Change the Y-Axis to “Rework duration”. 

We can see an increase in the rework duration for the activity, “Calculate Provisional Credit” from 2.76 hours to 3.55 hours. This could mean there could be peculiarities when calculating the provisional credit for cases involving Etsy and credit cards. It could be a more complex transaction, integration issues with the merchant or some communication barriers between the stakeholders. The business owner can make more informed decisions based on this revelation. 

Wrapping up 

Thus far, we have discovered concrete reasons why our KPI is blatantly violated. In Part 2, we will perform predictive analysis to simulate the potential impact of changing some parameters in the process. 

Want to try it out? 

Please find the event logs here. 

By David Obembe

David is a technical writer, passionate about making complex things simple. He also loves working with data and watches YouTube documentaries in his free time.