Default Implementation

The implementation of the TF-IDF model in sklearn is similar to that of the Bag of Words model. To train this model on a corpus, we use the TfidfVectorizer class utilizing the already familiar to us method .fit_transform().

Let's take a look at an example:

As you can see, aside from using a different class, the rest of the implementation is identical to that of the Bag of Words model. By default, the TF-IDF matrix is computed, as described in the previous chapter, with L2 normalization.

Customizing TF-IDF

Once again, similar to CountVectorizer, we can specify the min_df and max_df parameters to include only terms that occur in at least min_df documents and at most max_df documents. These can be specified as either absolute numbers of documents or as a proportion of the total number of documents.

Here is an example where we include only those terms that appear in exactly 2 documents by setting both min_df and max_df to 2:

To specify the n-grams to include in our matrix, we can use the ngram_range parameter. Let's include only bigrams in the resulting matrix:

These are the most commonly used parameters, however, in case you want to explore more of them, you can refer to the documentation.