The Agglomerative Information Bottleneck (AIB) algorithm greedily compresses discrete data by iteratively merging the two elements which cause the mutual information between the data and the class labels to decreases as little as possible.

Here we test AIB on the problem of finding a discriminatively optimal quantization of a mixture of Gaussians. The data in this case is 2 dimensional:

Random data generated from a Gaussian mixture with three components (class labels are indicated by color).

We quantize this data on a fixed lattice (a 20x20 grid shown in the figures below), and construct histograms for each class.

Next we apply AIB:

This provides us with a list of parents of each column in Pcx, forming a tree of merges. We can now "cut" this tree to obtain any number of clusters.

Three "cuts" of the merge tree, showing 10, 3, and 2 clusters. The gray squares are nodes of the tree which did not have any data points which were quantized to them.

Notice that the resulting clusters do not have to be contiguous in the original space.