Benchmarking anomaly-based detection systems

Anomaly detection is a key element of intrusion-detection and other detection systems in which perturbations of normal behavior suggest the presence of intentionally or unintentionally induced attacks, faults, defects, etc. Because most anomaly detectors are based on probabilistic algorithms that exploit the intrinsic structure, or regularity, embedded in data logs, a fundamental question is whether or not such structure influences detection performance. If detector performance is indeed a function of environmental regularity, it would be critical to match detectors to environmental characteristics. In intrusion-detection settings, however, this is not done, possibly because such characteristics are not easily ascertained. This paper introduces a metric for characterizing structure in data environments, and tests the hypothesis that intrinsic structure influences probabilistic detection. In a series of experiments, an anomaly-detection algorithm was applied to benchmark suite of 165 carefully calibrated, anomaly injected datasets of varying structure. Results showed performance differences of as much as an order of magnitude, indicating that current approaches to anomaly detection may not be universally dependable.