Hyperparameters are parameters of the model that are not learned. For example, hyperparameters of our logistic regression SMS classifier include the value of the regularization term and thresholds used to remove words that appear too frequently or infrequently. In scikit-learn, hyperparameters are set through the model's constructor. In the previous examples, we did not set any arguments for LogisticRegression(); we used the default values for all of the hyperparameters. These default values are often a good start, but they may not produce the optimal model. Grid search is a common method to select the hyperparameter values that produce the best model. Grid search takes a set of possible values for each hyperparameter that should be tuned, and evaluates a model trained on each element of the Cartesian product of the sets. That is, grid search is an exhaustive search that trains and evaluates a model for each possible combination of the hyperparameter values supplied by the developer. A disadvantage of grid search is that it is computationally costly for even small sets of hyperparameter values. Fortunately, it is an embarrassingly parallel problem; many models can easily be trained and evaluated concurrently since no synchronization is required between the processes. Let's use scikit-learn's GridSearchCV() function to find better hyperparameter values:
import pandas as pd
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.linear_model.logistic import LogisticRegression
from sklearn.grid_search import GridSearchCV
from sklearn.pipeline import Pipeline
from sklearn.cross_validation import train_test_split
from sklearn.metrics import precision_score, recall_score, accuracy_score
pipeline = Pipeline([
('vect', TfidfVectorizer(stop_words='english')),
('clf', LogisticRegression())
])
parameters = {
'vect__max_df': (0.25, 0.5, 0.75),
'vect__stop_words': ('english', None),
'vect__max_features': (2500, 5000, 10000, None),
'vect__ngram_range': ((1, 1), (1, 2)),
'vect__use_idf': (True, False),
'vect__norm': ('l1', 'l2'),
'clf__penalty': ('l1', 'l2'),
'clf__C': (0.01, 0.1, 1, 10),
}
GridSearchCV() takes an estimator, a parameter space, and performance measure. The argument n_jobs specifies the maximum number of concurrent jobs; set n_jobs to -1 to use all CPU cores. Note that fit() must be called in a Python main block in order to fork additional processes; this example must be executed as a script, and not in an interactive interpreter:
if __name__ == "__main__":
grid_search = GridSearchCV(pipeline, parameters, n_jobs=-1, verbose=1, scoring='accuracy', cv=3)
df = pd.read_csv('data/sms.csv')
X, y, = df['message'], df['label']
X_train, X_test, y_train, y_test = train_test_split(X, y)
grid_search.fit(X_train, y_train)
print 'Best score: %0.3f' % grid_search.best_score_
print 'Best parameters set:'
best_parameters = grid_search.best_estimator_.get_params()
for param_name in sorted(parameters.keys()):
print ' %s: %r' % (param_name, best_parameters[param_name])
predictions = grid_search.predict(X_test)
print 'Accuracy:', accuracy_score(y_test, predictions)
print 'Precision:', precision_score(y_test, predictions)
print 'Recall:', recall_score(y_test, predictions)The following is the output of the script:
Fitting 3 folds for each of 1536 candidates, totalling 4608 fits [Parallel(n_jobs=-1)]: Done 1 jobs | elapsed: 0.2s [Parallel(n_jobs=-1)]: Done 50 jobs | elapsed: 4.0s [Parallel(n_jobs=-1)]: Done 200 jobs | elapsed: 16.9s [Parallel(n_jobs=-1)]: Done 450 jobs | elapsed: 36.7s [Parallel(n_jobs=-1)]: Done 800 jobs | elapsed: 1.1min [Parallel(n_jobs=-1)]: Done 1250 jobs | elapsed: 1.7min [Parallel(n_jobs=-1)]: Done 1800 jobs | elapsed: 2.5min [Parallel(n_jobs=-1)]: Done 2450 jobs | elapsed: 3.4min [Parallel(n_jobs=-1)]: Done 3200 jobs | elapsed: 4.4min [Parallel(n_jobs=-1)]: Done 4050 jobs | elapsed: 7.7min [Parallel(n_jobs=-1)]: Done 4608 out of 4608 | elapsed: 8.5min finished Best score: 0.983 Best parameters set: clf__C: 10 clf__penalty: 'l2' vect__max_df: 0.5 vect__max_features: None vect__ngram_range: (1, 2) vect__norm: 'l2' vect__stop_words: None vect__use_idf: True Accuracy: 0.989956958393 Precision: 0.988095238095 Recall: 0.932584269663
Optimizing the values of the hyperparameters has improved our model's recall score on the test set.