Introduction to NLP

1. Text Preprocessing Fundamentals

We will start our journey with learning and implementing the most common text preprocessing techniques used in NLP to convert the initial raw text into a clean, standardized form.

NLP and Its Applications

Understanding Text Preprocessing

Tokenization

Tokenize a Sentence

Tokenization Using Regular Expressions

Tokenize Using Regex

Removing Stop Words

Applying Text Preprocessing in Practice

2. Stemming and Lemmatization

Without further ado, let's explore stemming and lemmatization. These techniques may enhance efficiency and effectiveness of some NLP tasks, especially when working with large text corpora and treating different words forms as the same word.

Stemming

Stem the Tokens

Lemmatization

Part of Speech Tagging

Lemmatization with POS Tagging

3. Basic Text Models

Preprocessed text should then be transformed into a numerical representation to be used in machine learning or deep learning models for various tasks such as prediction, classification, or clustering. Here, we will learn to implement the most basic yet popular text models that transform text data into numbers.

Overview of Vector Space Models

Key Types of Vector Space Models

Bag of Words

Customizing Bag of Words

Create a Bag of Words

TF-IDF

Implementing TF-IDF

4. Word Embeddings

Time to unlock the power of word embeddings and delve into advanced techniques for capturing semantic relationships between words. We will explore various embedding models such as Word2Vec, GloVe, and FastText, with a particular focus on the Word2Vec model and its implementation.

Word Embeddings Basics

CBoW and Skip-gram Models (Optional)

Implementing Word2Vec

Create Word Embeddings

Removing Stop Words

Understanding Stop Words

In NLP, the process of removing stop words is a crucial step in text preprocessing.

Stop words are common words that usually do not contribute to the meaning of a sentence, at least for the purposes of most analysis and algorithms. These include words such as "the", "is", "in", and "on".

Stop words are typically filtered out after tokenization for NLP tasks, such as sentiment analysis, topic modeling, or keyword extraction. The rationale behind removing stop words is to decrease the dataset size, thereby improving computational efficiency, and to increase the relevance of the analysis by focusing on the words that carry significant meaning.

Removing Stop Words with NLTK

To make things easier, nltk provides a comprehensive list of stop words in multiple languages, which can be easily accessed and used to filter stop words from text data.

Here’s how you can get the list of English stop words in NLTK and convert it to set:

Converting this list to a set improves the efficiency of lookups, as checking membership in a set is faster than in a list.

With this in mind, let's take a look at a complete example of how to filter out stop words from a given text:

As you can see, we should first download the stop words and perform tokenization. The next step is to use a list comprehension to create a list containing only tokens which are not stop words. The word.lower() in the if clause is essential to convert each word (token) to lower case, since nltk contains stop words exclusively in lower case.

Alternatively, we could use a usual for loop instead of a list comprehension, however, using list comprehension here is more efficient and concise.

Task

Your task is to convert the text to lowercase, load the English stop words list from nltk and convert it to a set, then tokenize the text string using the word_tokenize() function, and filter out the stop words from tokens using list comprehension.

Everything was clear?

Section 1. Chapter 7

Course Content

Introduction to NLP

Removing Stop Words

Understanding Stop Words

In NLP, the process of removing stop words is a crucial step in text preprocessing.

Removing Stop Words with NLTK

To make things easier, nltk provides a comprehensive list of stop words in multiple languages, which can be easily accessed and used to filter stop words from text data.

Here’s how you can get the list of English stop words in NLTK and convert it to set:

Converting this list to a set improves the efficiency of lookups, as checking membership in a set is faster than in a list.

With this in mind, let's take a look at a complete example of how to filter out stop words from a given text:

Alternatively, we could use a usual for loop instead of a list comprehension, however, using list comprehension here is more efficient and concise.

Task

Everything was clear?

Section 1. Chapter 7