Notice: This page requires JavaScript to function properly.
Please enable JavaScript in your browser settings or update your browser.
Lemmatization | Stemming and Lemmatization
Introduction to NLP

bookLemmatization

Lemmatization vs Stemming

First, let's define what lemmatization is and how it is different from stemming.

Unlike stemming, which crudely chops off word endings, lemmatization considers the context and converts the word to its meaningful base form. For example, 'am', 'are', and 'is' are all lemmatized into 'be'. This approach can significantly reduce the size of the vocabulary (the number of unique words) in large text corpora, thereby increasing efficiency when training models.

On the other hand, while lemmatization is more accurate, it is also more computationally expensive and can be time-consuming with large datasets. Furthermore, for even better accuracy, performing morphological analysis and part of speech tagging is recommended before lemmatization.

Lemmatization with NLTK

The WordNet Lemmatizer, provided by the NLTK library, leverages the WordNet corpus to perform lemmatization.

When you use the WordNet Lemmatizer, it looks up the target word in the WordNet database to find the most appropriate lemma (base form) of the word.

As mentioned above, because words can have different meanings in different contexts (e.g., "running" as a verb vs. "running" as a noun), the lemmatizer may require you to specify the part of speech (e.g., verb, noun, adjective). This helps it select the correct lemma based on the word's role in a sentence.

Let's now take a look at an example:

from nltk.stem import WordNetLemmatizer
import nltk
# Download the WordNet corpus
nltk.download('wordnet')
# Initialize the WordNet lemmatizer
lemmatizer = WordNetLemmatizer()
words = ["running", "bested"]
# Lemmatize words
lemmatized_words = [lemmatizer.lemmatize(word, 'v') for word in words] # 'v' for verb
print("Lemmatized words:", lemmatized_words)
12345678910
from nltk.stem import WordNetLemmatizer import nltk # Download the WordNet corpus nltk.download('wordnet') # Initialize the WordNet lemmatizer lemmatizer = WordNetLemmatizer() words = ["running", "bested"] # Lemmatize words lemmatized_words = [lemmatizer.lemmatize(word, 'v') for word in words] # 'v' for verb print("Lemmatized words:", lemmatized_words)
copy

As you can see, from the coding perspective, the approach is rather straightforward. Once again, in order to lemmatize words accurately across an entire corpus, it would be best to first perform part-of-speech (POS) tagging, which we will cover in the following chapter.

Завдання
test

Swipe to show code editor

Your task is to lemmatize the tokens, given the text string. Tokenization is already applied with stop words filtered out.

  1. Import the WordNet lemmatizer.

  2. Download the WordNet corpus.

  3. Initialize the WordNet lemmatizer.

  4. Lemmatize the tokens using list comprehension.

Все було зрозуміло?

Як ми можемо покращити це?

Дякуємо за ваш відгук!

Секція 2. Розділ 3
from nltk.tokenize import word_tokenize
from nltk.corpus import stopwords
# Import the WordNet lemmatizer
from ___ import ___
import nltk
nltk.download('punkt_tab')
nltk.download('stopwords')
# Download the WordNet corpus
nltk.___
stop_words = set(stopwords.words('english'))
# Initialize the WordNet lemmatizer
lemmatizer = ___
text = "The leaves on the tree were turning a bright red, indicating that fall was leaving its mark."
text = text.lower()
tokens = word_tokenize(text)
tokens = [token for token in tokens if token.lower() not in stop_words]
# Lemmatize the tokens using list comprehension
lemmatized_tokens = [___ for token in ___]
print("Lemmatized tokens:", lemmatized_tokens)
toggle bottom row
some-alt