On Measuring Abnormality

The effect of time of observation in detecting anomalies or frauds

I have been working on many anomaly and fraud detection systems for five years and what I have found in these years shows that:

• We can't use a single machine learning or statistical model to detect any abnormal behavior in a system, like an only deep neural network - which people are usually very fond of it! - or a single recurrent network or an autoencoder, etc. Instead, we have to use many classifiers or clustering processes or even statistical models and wire them together to get the result. So basically, it means we cannot do it on a shell scripting environment unless we are facing a simple problem. We need to design and build a real application.
• Regardless of what algorithm we use to detect the abnormalities, if the algorithm can't give acceptable reasoning to the customer, they are not happy.

Here we try to describe a simplified version of an algorithm that can help you to address this problem ...

Download the pdf:  On Measuring Abnormality