Mining Surprising Patterns Using Temporal Description Length

Mining Surprising Patterns Using Temporal Description Length (1998)

Cached

Download Links

[www.it.iitb.ernet.in]
[http.cs.berkeley.edu]

Other Repositories/Bibliography

DBLP

by Soumen Chakrabarti , Sunita Sarawagi , Byron Dom

Citations:

48 - 0 self

Versions

Version History

Metadata Version 1

Datum	Value	Source
TITLE	Mining surprising patterns using temporal description length	INFERENCE
AUTHOR NAME	Soumen Chakrabarti	SVM HeaderParse 0.1
AUTHOR AFFIL	IBM Almaden Research Center; 650 Harry Road	SVM HeaderParse 0.2
AUTHOR ADDR	San Jose, CA 95120	SVM HeaderParse 0.1
AUTHOR NAME	Sunita Sarawagi	SVM HeaderParse 0.1
AUTHOR AFFIL	IBM Almaden Research Center; 650 Harry Road	SVM HeaderParse 0.2
AUTHOR ADDR	San Jose, CA 95120	SVM HeaderParse 0.1
AUTHOR NAME	Byron Dom	SVM HeaderParse 0.1
AUTHOR AFFIL	IBM Almaden Research Center; 650 Harry Road	SVM HeaderParse 0.2
AUTHOR ADDR	San Jose, CA 95120	SVM HeaderParse 0.1
ABSTRACT	We propose a new notion of surprising temporal patterns in market basket data, and algorithms to find such patterns. This is distinct from finding frequent patterns as addressed in the common mining literature. We argue that once the analyst is already familiar with prevalent patterns in the data, the greatest incremental benefit is likely to be from changes in the relationship between item frequencies over time. A simple measure of surprise is the extent of departure from a model, estimated using standard multivariate time series analysis. Unfortunately, such estimation involves models, smoothing windows and parameters whose optimal choices can vary dramatically from one application to another. In contrast, we propose a precise characterization of surprise based on the number of bits in which a basket sequence can be encoded under a carefully chosen coding scheme. In this scheme it is inexpensive to encode sequences of itemsets that have steady, hence likely to be well-known, correlation between items. Conversely, a sequence with large code length hints at a possibly surprising correlation. Our notion of surprise also has the desirable property that the score of a set of items is offset by anything surprising that the user may already know from the marginal distribution of any proper subset. No parameters, such as support, confidence, or smoothing windows, need to be estimated or specified by the user. We experimented with real-life market basket data. The algorithm successfully rejected a large number of frequent sets of items that bore obvious and steady complementary relations to each other, such as cereal and milk. Instead, our algorithm found itemsets that showed statistically strong fluctuations in correlation over time. These items had no obviously complementary roles.	SVM HeaderParse 0.1
YEAR	1998	INFERENCE
VENUE TYPE	CONFERENCE	INFERENCE
PAGES	606--617	INFERENCE
CITATIONS	24 found	ParsCit 1.0