Introduction to NLP

1. Text Preprocessing Fundamentals

We will start our journey with learning and implementing the most common text preprocessing techniques used in NLP to convert the initial raw text into a clean, standardized form.

NLP and Its Applications

Understanding Text Preprocessing

Tokenization

Tokenize a Sentence

Tokenization Using Regular Expressions

Tokenize Using Regex

Removing Stop Words

Applying Text Preprocessing in Practice

2. Stemming and Lemmatization

Without further ado, let's explore stemming and lemmatization. These techniques may enhance efficiency and effectiveness of some NLP tasks, especially when working with large text corpora and treating different words forms as the same word.

Stemming

Stem the Tokens

Lemmatization

Part of Speech Tagging

Lemmatization with POS Tagging

3. Basic Text Models

Preprocessed text should then be transformed into a numerical representation to be used in machine learning or deep learning models for various tasks such as prediction, classification, or clustering. Here, we will learn to implement the most basic yet popular text models that transform text data into numbers.

Overview of Vector Space Models

Key Types of Vector Space Models

Bag of Words

Customizing Bag of Words

Create a Bag of Words

TF-IDF

Implementing TF-IDF

4. Word Embeddings

Time to unlock the power of word embeddings and delve into advanced techniques for capturing semantic relationships between words. We will explore various embedding models such as Word2Vec, GloVe, and FastText, with a particular focus on the Word2Vec model and its implementation.

Word Embeddings Basics

CBoW and Skip-gram Models (Optional)

Implementing Word2Vec

Create Word Embeddings

Lemmatization with POS Tagging

The English language is full of words that can serve as multiple parts of speech with different meanings. For example, "running" can be a verb ("He is running.") or a noun ("Running is fun.").

As we have already seen, a lemmatizer can only accurately reduce a word to its base form if it knows the word's part of speech in the given context. POS tagging in turn provides this context, making lemmatization more precise.

Lemmatization with POS Tagging in NLTK

Since we are already familiar with both of these techniques separately, it's time to combine them. However, there is one important aspect to take into consideration, namely the difference in POS tags format between pos_tagand the format that WordNet Lemmatizer expects.

NLTK's pos_tag function utilizes the Penn Treebank tag set, which includes a wide range of tags for detailed grammatical categorization. The WordNet Lemmatizer, on the other hand, expects POS tags in a simplified form that aligns with WordNet's own categorization. Specifically, it only differentiates among nouns (n), verbs (v), adjectives (a or s for satellite adjectives), and adverbs (r).

The mapping process involves converting the detailed Penn Treebank tags to the broader categories recognized by WordNet. For example, both 'VBD' (past tense verb) and 'VBG' (gerund or present participle) from Penn Treebank would map to 'v' (verb) for use with the WordNet Lemmatizer.

Let's write a function for this purpose:

This function simply checks the first letter of the Penn Treebank tag: if it's 'J', it returns the WordNet tag for adjectives; if 'V', for verbs; if 'R', for adverbs.

For all other cases, including when the tag starts with 'N' or doesn't match any specified condition, it defaults to returning the WordNet tag for nouns. These ADJ, VERB etc. are just constants, where ADJ, ADJ_SAT, ADV, NOUN, VERB = "a", "s", "r", "n", "v".

Given this function, let's now perform lemmatization with POS tagging beforehand:

As you can see, we first performed POS tagging using the pos_tag() function, next we used list comprehension to create a list of lemmatized tokens by applying the lemmatize() method with the current token and correctly formatted tag (using our function get_wordnet_pos(tag)) as its arguments. We intentionally did not remove stop words to demonstrate that the code effectively processes all tokens.

As mentioned in one of the previous chapters, ' '.join(lemmatized_tokens) is used here to create a string, given a list of strings (tokens), so that they will be separated by a whitespace, effectively reconstructing the sentence from its individual, processed tokens.

Task

It's time to combine all the text preprocessing techniques we have learned so far to get lemmatized text without the stop words, given the initial raw text. Your task is the following:

Convert text to lowercase.
Load the list of English stop words and convert it to set.
Initialize a lemmatizer.
Tokenize the text string.
Filter out the stop words using list comprehension.
Perform POS tagging using the respective function.
Lemmatize the resulting tokens taking their POS tags into account using list comprehension.

Everything was clear?

Section 2. Chapter 5

Course Content

Introduction to NLP

Lemmatization with POS Tagging

The English language is full of words that can serve as multiple parts of speech with different meanings. For example, "running" can be a verb ("He is running.") or a noun ("Running is fun.").

Lemmatization with POS Tagging in NLTK

Let's write a function for this purpose:

This function simply checks the first letter of the Penn Treebank tag: if it's 'J', it returns the WordNet tag for adjectives; if 'V', for verbs; if 'R', for adverbs.

Given this function, let's now perform lemmatization with POS tagging beforehand:

Task

It's time to combine all the text preprocessing techniques we have learned so far to get lemmatized text without the stop words, given the initial raw text. Your task is the following:

Convert text to lowercase.
Load the list of English stop words and convert it to set.
Initialize a lemmatizer.
Tokenize the text string.
Filter out the stop words using list comprehension.
Perform POS tagging using the respective function.
Lemmatize the resulting tokens taking their POS tags into account using list comprehension.

Everything was clear?

Section 2. Chapter 5