Notice: This page requires JavaScript to function properly.
Please enable JavaScript in your browser settings or update your browser.Part of Speech Tagging | Stemming and Lemmatization
course content

Course Content

Introduction to NLP

1. Text Preprocessing Fundamentals

NLP and Its ApplicationsUnderstanding Text PreprocessingTokenizationTokenize a Sentence Tokenization Using Regular ExpressionsTokenize Using RegexRemoving Stop WordsApplying Text Preprocessing in Practice

2. Stemming and Lemmatization

StemmingStem the TokensLemmatizationPart of Speech TaggingLemmatization with POS Tagging

3. Basic Text Models

Overview of Vector Space ModelsKey Types of Vector Space ModelsBag of WordsCustomizing Bag of WordsCreate a Bag of WordsTF-IDFImplementing TF-IDF

4. Word Embeddings

Word Embeddings BasicsCBoW and Skip-gram Models (Optional)Implementing Word2VecCreate Word Embeddings

Introduction to NLP

Part of Speech TaggingPart of Speech Tagging

Understading POS Tagging

We have mentioned in the previous chapter that part of speech tagging is beneficial for lemmatization, which is its primary role in text preprocessing, so let's discuss this process in more detail here.

Part of Speech (POS) Tagging is the process of marking up a word in a text (corpus) as corresponding to a particular part of speech (e.g., noun or verb), based on both its definition and its context—i.e., its relationship with adjacent and related words in a phrase, sentence, or paragraph.

However, using the full lexical terms can become quite cumbersome, especially with large corpus, short representations known as tags are used instead. For example, "VB" instead of verb. In practice, however, different POS taggers may use a bit different tags and more detailed tags like "VBD" for verbs in past tense.

POS with NLTK

In order to perform part of speech tagging with NLTK, you should import the pos_tag() function directly from nltk and apply it on the list of strings (tokens) by passing it as the argument.

Here is an example:

As you can see, this function returns a list of tuples each containing a token and its tag. The line nltk.download('averaged_perceptron_tagger') initiates the download of the dataset and models necessary for the PerceptronTagger, which is the default POS tagger used by NLTK. This includes the trained model based on the averaged perceptron algorithm, enabling users to perform POS tagging on text data.

This tagger is based on the averaged perceptron model, a supervised learning algorithm that is efficient and effective for large-scale text processing, including POS tagging. The PerceptronTagger is chosen for its balance of speed and accuracy, making it suitable for a wide range of NLP tasks that require POS tagging. It learns weights for features based on the training data it's given, and it uses these weights to make predictions on the POS tags for unseen text.

For better visual representation, we could convert the result to a pandas DataFrame:

Alternatively, we could use pd.DataFrame(tagged_tokens, columns=['Token', 'POS tag']) without transposing the DataFrame, so that each row represents a token-tag pair.

Task

Your task is to perform part of speech tagging on the tokens of the text string and convert the resulting list of tuples to a DataFrame.

  1. Import the correct function for POS tagging.
  2. Download the model needed for NLTK's POS tagging.
  3. Perform POS tagging.
  4. Convert the result to DataFrame (do NOT transpose it).

Everything was clear?

Section 2. Chapter 4
toggle bottom row
course content

Course Content

Introduction to NLP

1. Text Preprocessing Fundamentals

NLP and Its ApplicationsUnderstanding Text PreprocessingTokenizationTokenize a Sentence Tokenization Using Regular ExpressionsTokenize Using RegexRemoving Stop WordsApplying Text Preprocessing in Practice

2. Stemming and Lemmatization

StemmingStem the TokensLemmatizationPart of Speech TaggingLemmatization with POS Tagging

3. Basic Text Models

Overview of Vector Space ModelsKey Types of Vector Space ModelsBag of WordsCustomizing Bag of WordsCreate a Bag of WordsTF-IDFImplementing TF-IDF

4. Word Embeddings

Word Embeddings BasicsCBoW and Skip-gram Models (Optional)Implementing Word2VecCreate Word Embeddings

Introduction to NLP

Part of Speech TaggingPart of Speech Tagging

Understading POS Tagging

We have mentioned in the previous chapter that part of speech tagging is beneficial for lemmatization, which is its primary role in text preprocessing, so let's discuss this process in more detail here.

Part of Speech (POS) Tagging is the process of marking up a word in a text (corpus) as corresponding to a particular part of speech (e.g., noun or verb), based on both its definition and its context—i.e., its relationship with adjacent and related words in a phrase, sentence, or paragraph.

However, using the full lexical terms can become quite cumbersome, especially with large corpus, short representations known as tags are used instead. For example, "VB" instead of verb. In practice, however, different POS taggers may use a bit different tags and more detailed tags like "VBD" for verbs in past tense.

POS with NLTK

In order to perform part of speech tagging with NLTK, you should import the pos_tag() function directly from nltk and apply it on the list of strings (tokens) by passing it as the argument.

Here is an example:

As you can see, this function returns a list of tuples each containing a token and its tag. The line nltk.download('averaged_perceptron_tagger') initiates the download of the dataset and models necessary for the PerceptronTagger, which is the default POS tagger used by NLTK. This includes the trained model based on the averaged perceptron algorithm, enabling users to perform POS tagging on text data.

This tagger is based on the averaged perceptron model, a supervised learning algorithm that is efficient and effective for large-scale text processing, including POS tagging. The PerceptronTagger is chosen for its balance of speed and accuracy, making it suitable for a wide range of NLP tasks that require POS tagging. It learns weights for features based on the training data it's given, and it uses these weights to make predictions on the POS tags for unseen text.

For better visual representation, we could convert the result to a pandas DataFrame:

Alternatively, we could use pd.DataFrame(tagged_tokens, columns=['Token', 'POS tag']) without transposing the DataFrame, so that each row represents a token-tag pair.

Task

Your task is to perform part of speech tagging on the tokens of the text string and convert the resulting list of tuples to a DataFrame.

  1. Import the correct function for POS tagging.
  2. Download the model needed for NLTK's POS tagging.
  3. Perform POS tagging.
  4. Convert the result to DataFrame (do NOT transpose it).

Everything was clear?

Section 2. Chapter 4
toggle bottom row
some-alt