In order to test clustering, let's move away from the toy text examples and find a dataset that resembles the data we are expecting in the future so that we can test our approach. For our purpose, we need documents about technical topics that are already grouped together so that we can check whether our algorithm works as expected when we apply it later to the posts we hope to receive.

One standard dataset in machine learning is the 20newsgroup dataset, which contains 18,826 posts from 20 different newsgroups. Among the groups' topics are technical ones such as comp.sys.mac.hardware or sci.crypt, as well as more politics- and religion-related ones such as talk.politics.guns or soc.religion.christian. We will restrict ourselves to the technical groups. If we assume each newsgroup as one cluster, we can nicely test whether our approach of finding related posts works.

For convenience, the sklearn.datasets module also contains the fetch_20newsgroups function, which automatically downloads the data behind the scenes:

>>> import sklearn.datasets
>>> all_data = sklearn.datasets.fetch_20newsgroups(subset='all')
>>> print(len(all_data.filenames))
18846
>>> print(all_data.target_names)
['alt.atheism', 'comp.graphics', 'comp.os.ms-windows.misc', 
 'comp.sys.ibm.pc.hardware', 'comp.sys.mac.hardware', 
 'comp.windows.x', 'misc.forsale', 'rec.autos', 'rec.motorcycles', 
 'rec.sport.baseball', 'rec.sport.hockey', 'sci.crypt', 
 'sci.electronics', 'sci.med', 'sci.space', 'soc.religion.christian', 
 'talk.politics.guns', 'talk.politics.mideast', 'talk.politics.misc', 
 'talk.religion.misc']

We can choose between training and test sets:

>>> train_data = sklearn.datasets.fetch_20newsgroups(subset='train')
>>> print(len(train_data.filenames))
11314
>>> test_data = sklearn.datasets.fetch_20newsgroups(subset='test')
>>> print(len(test_data.filenames))
7532 

For simplicity's sake, we will restrict ourselves to only some newsgroups so that the overall experimentation cycle is shorter. We can achieve this with the categories parameter:

>>> groups = ['comp.graphics', 'comp.os.ms-windows.misc', 
 'comp.sys.ibm.pc.hardware', 'comp.sys.mac.hardware', 
 'comp.windows.x', 'sci.space']
>>> train_data = sklearn.datasets.fetch_20newsgroups(subset='train', 
                    categories=groups)
>>> print(len(train_data.filenames))
3529
    
>>> test_data = sklearn.datasets.fetch_20newsgroups(subset='test', 
                    categories=groups)
>>> print(len(test_data.filenames))
2349