Words Frequency

The main goal of this chapter is to quantify the distribution of words across sentences within a given text. By determining how many sentences each unique word appears in, we aim to lay the foundation for calculating the Inverse Sentence Frequency (ISF) part of the TF-ISF score.

Setting Up a Counting Mechanism

Dictionary Initialization: We start by creating an empty dictionary named word_sentence_counts. This dictionary is designed to map each unique word to the number of sentences it appears in. The key-value pairs consist of the word as the key and its sentence occurrence count as the value.

Processing Each Sentence

Iterating Through Tokenized Sentences: The code loops through each sentence in the tokenized_sentences list, which contains sentences that have already been split into individual words (tokens).

Updating Word Counts

Word Presence Check: For every unique word in a sentence, the code checks if that word already exists in the word_sentence_counts dictionary.

New Words: If a word is not found in the dictionary, it implies that this is the first sentence in which the word has been encountered. Consequently, the word is added to the dictionary with a count of 1;
Existing Words: If the word is already in the dictionary, its count is incremented by 1, reflecting its appearance in an additional sentence.

Завдання

Swipe to start coding

Iterate through each tokenized sentence, count each unique word, and update the counts in the dictionary.

Рішення

# Initialize an empty dictionary to store the counts of sentences containing each unique word

word_sentence_counts = {}

# Iterate through each tokenized sentence

for sentence in tokenized_sentences:

# Convert each sentence list to a set to ensure uniqueness of words

for word in set(sentence):

# Check if the word is already in the dictionary

if word not in word_sentence_counts:

# If the word is not in the dictionary, add it with a count of 1

word_sentence_counts[word] = 1

else:

# If the word is already in the dictionary, increment its count by 1

word_sentence_counts[word] += 1

# Print the counts for specific words

print('Number of sentences with the word "wedding":', word_sentence_counts['wedding'])

print('Number of sentences with the word "mr":', word_sentence_counts['mr'])

Mark tasks as Completed

Все було зрозуміло?

Дякуємо за ваш відгук!

Секція 1. Розділ 8

AVAILABLE TO ULTIMATE ONLY