We consider the problem of interactive iterative analysis of datasets that consist of a large number of records represented as feature vectors. The record set is known to contain a number of anomalous records that the analyst desires to locate and describe in a short and comprehensive manner. The nature of the anomaly is not known in advance (in particular, it is not known, which features or feature values identify the anomalous records, and which are irrelevant to the search), and becomes clear only in the process of analysis, as the description of the target subset is gradually refined. This situation is common in computer intrusion analysis, when a forensic analyst browses the logs to locate traces of an intrusion of unknown nature and origin, and extends to other tasks and data sets. To facilitate such ”browsing for anomalies”, we propose an unsupervised data organization technique for initial summarization and representation of data sets, and a semi-supervised learning technique for iterative modifications of the latter representation. Our approach is based on information content and Jensen-Shannon divergence and is related to information bottleneck methods. We have implemented it as a part of the Kerf log analysis toolkit.
semisupervised learning technique, data organization, interactive anomaly analysis, interactive iterative analysis, datasets, feature vectors, anomalous record sets, computer intrusion analysis, forensic analyst, information content, Jensen-Shannon divergence, Kerf log analysis toolkit
Anomaly detection (Computer security), Iterative methods (Mathematics)
Personal use of this material is permitted. Permission from IEEE must be obtained for all other uses, including reprinting/republishing this material for advertising or promotional purposes, creating new collective works for resale or redistribution to servers or lists, or reuse of any copyrighted components of this work in other works.
Aslam, Javed A.; Bratus, Sergey; and Pavlu, Virgil, "Semi-supervised data organization for interactive anomaly analysis" (2006). Computer and Information Science Faculty Publications. Paper 5. http://hdl.handle.net/2047/d20000319
Click button above to open, or right-click to save.