Clean and Convert

We have decided to create a complete chapter on the topic of text cleaning and preprocessing. As you may imagine, complete texts cannot be directly fed into an ML model. For this reason, we will apply specific preprocessing techniques.

The first step will be to remove punctuation from our column to reduce noise in our data. We will do this using regex (regular expression matching operations).

Then we will vectorize our text. Refer to the picture below for more information. Essentially, we will represent words, sentences, or even larger units of text as vectors.