Course Content
Extracting Text Meaning using TF-IDF
Words Frequency
The main goal of this chapter is to quantify the distribution of words across sentences within a given text. By determining how many sentences each unique word appears in, we aim to lay the foundation for calculating the Inverse Sentence Frequency (ISF) part of the TF-ISF score.
Setting Up a Counting Mechanism
Dictionary Initialization: We start by creating an empty dictionary named word_sentence_counts. This dictionary is designed to map each unique word to the number of sentences it appears in. The key-value pairs consist of the word as the key and its sentence occurrence count as the value.
Processing Each Sentence
Iterating Through Tokenized Sentences: The code loops through each sentence in the tokenized_sentences list, which contains sentences that have already been split into individual words (tokens).
Updating Word Counts
Word Presence Check: For every unique word in a sentence, the code checks if that word already exists in the word_sentence_counts dictionary.
- New Words: If a word is not found in the dictionary, it implies that this is the first sentence in which the word has been encountered. Consequently, the word is added to the dictionary with a count of 1;
- Existing Words: If the word is already in the dictionary, its count is incremented by 1, reflecting its appearance in an additional sentence.
Task
Iterate through each tokenized sentence, count each unique word, and update the counts in the dictionary.
Switch to desktop for real-world practiceContinue from where you are using one of the options belowThanks for your feedback!
The main goal of this chapter is to quantify the distribution of words across sentences within a given text. By determining how many sentences each unique word appears in, we aim to lay the foundation for calculating the Inverse Sentence Frequency (ISF) part of the TF-ISF score.
Setting Up a Counting Mechanism
Dictionary Initialization: We start by creating an empty dictionary named word_sentence_counts. This dictionary is designed to map each unique word to the number of sentences it appears in. The key-value pairs consist of the word as the key and its sentence occurrence count as the value.
Processing Each Sentence
Iterating Through Tokenized Sentences: The code loops through each sentence in the tokenized_sentences list, which contains sentences that have already been split into individual words (tokens).
Updating Word Counts
Word Presence Check: For every unique word in a sentence, the code checks if that word already exists in the word_sentence_counts dictionary.
- New Words: If a word is not found in the dictionary, it implies that this is the first sentence in which the word has been encountered. Consequently, the word is added to the dictionary with a count of 1;
- Existing Words: If the word is already in the dictionary, its count is incremented by 1, reflecting its appearance in an additional sentence.
Task
Iterate through each tokenized sentence, count each unique word, and update the counts in the dictionary.
Switch to desktop for real-world practiceContinue from where you are using one of the options below