Notice: This page requires JavaScript to function properly.
Please enable JavaScript in your browser settings or update your browser.Tokenize Using Regex | Text Preprocessing Fundamentals
Introduction to NLP

Tokenize Using RegexTokenize Using Regex

Task

Given a string named message, convert it lowercase, then tokenize it into words using regular expression tokenization and the corresponding nltk class. A word is a sequence of only alphanumeric characters (letters and numbers). '#Conference2023!', for example, contains one word: Conference2023.

Everything was clear?

Section 1. Chapter 6
toggle bottom row
course content

Course Content

Introduction to NLP

1. Text Preprocessing Fundamentals

NLP and Its ApplicationsUnderstanding Text PreprocessingTokenizationTokenize a Sentence Tokenization Using Regular ExpressionsTokenize Using RegexRemoving Stop WordsApplying Text Preprocessing in Practice

2. Stemming and Lemmatization

StemmingStem the TokensLemmatizationPart of Speech TaggingLemmatization with POS Tagging

3. Basic Text Models

Overview of Vector Space ModelsKey Types of Vector Space ModelsBag of WordsCustomizing Bag of WordsCreate a Bag of WordsTF-IDFImplementing TF-IDF

4. Word Embeddings

Word Embeddings BasicsCBoW and Skip-gram Models (Optional)Implementing Word2VecCreate Word Embeddings

Introduction to NLP

Tokenize Using RegexTokenize Using Regex

Task

Given a string named message, convert it lowercase, then tokenize it into words using regular expression tokenization and the corresponding nltk class. A word is a sequence of only alphanumeric characters (letters and numbers). '#Conference2023!', for example, contains one word: Conference2023.

Everything was clear?

Section 1. Chapter 6
toggle bottom row
some-alt