Why Out-of-Vocabulary Is Inevitable
Languages like English, German, and Turkish are considered open-vocabulary languages because new words can be created at any time. This property is known as morphological productivity — the ability to generate new word forms by combining roots, prefixes, suffixes, or even inventing entirely new terms. For instance, you might encounter a made-up word like hyperconnectivity or a rarely used technical term such as transmogrification. Since it is impossible to list every possible word that could ever exist, any fixed vocabulary will eventually encounter words it does not contain. This is the core reason why out-of-vocabulary (OOV) tokens are inevitable in practical tokenization systems.
Even if you train on massive text corpora, new words, names, slang, and domain-specific terms will always appear. The infinite creativity of natural language means that your tokenizer will face words it has never seen before. As a result, handling OOV tokens becomes a fundamental challenge for any tokenization approach.
An OOV (out-of-vocabulary) token is any word or sequence not present in a tokenizer's known vocabulary. For example, unhappinesses might not be in a basic vocabulary, so it would be considered OOV.
Examples: OOV tokens can include:
- Newly coined words ("cryptojacking");
- Misspellings ("recieve");
- Domain-specific jargon ("bioinformatics");
- Foreign words or names ("Schwarzenegger").
Subword modeling—breaking words into smaller, more frequent parts—helps reduce OOVs by representing unseen words as combinations of known subwords. For example, unhappinesses might be split into un, happi, ness, es. However, this does not eliminate OOVs entirely, because truly novel character sequences, rare names, or code-like strings may still occur.
123456789101112131415161718192021222324252627# Example: Subword tokenization of an unseen word using a simple vocabulary vocab = {"un", "happi", "ness", "es", "bio", "info", "mat", "ics"} def subword_tokenize(word, vocab): tokens = [] i = 0 while i < len(word): # Try to find the longest subword in vocab that matches the current position found = False for j in range(len(word), i, -1): sub = word[i:j] if sub in vocab: tokens.append(sub) i = j found = True break if not found: # If no subword found, treat single character as OOV tokens.append(word[i]) i += 1 return tokens # Unseen word that is not in the vocabulary word = "unhappinesses" tokens = subword_tokenize(word, vocab) print("Tokens:", tokens) # Output: Tokens: ['un', 'happi', 'ness', 'es']
Grazie per i tuoi commenti!
Chieda ad AI
Chieda ad AI
Chieda pure quello che desidera o provi una delle domande suggerite per iniziare la nostra conversazione
Can you explain more about how subword tokenization helps with OOV tokens?
What are some common subword tokenization algorithms?
Are there any drawbacks to using subword modeling?
Fantastico!
Completion tasso migliorato a 11.11Why Out-of-Vocabulary Is Inevitable
Scorri per mostrare il menu
Languages like English, German, and Turkish are considered open-vocabulary languages because new words can be created at any time. This property is known as morphological productivity — the ability to generate new word forms by combining roots, prefixes, suffixes, or even inventing entirely new terms. For instance, you might encounter a made-up word like hyperconnectivity or a rarely used technical term such as transmogrification. Since it is impossible to list every possible word that could ever exist, any fixed vocabulary will eventually encounter words it does not contain. This is the core reason why out-of-vocabulary (OOV) tokens are inevitable in practical tokenization systems.
Even if you train on massive text corpora, new words, names, slang, and domain-specific terms will always appear. The infinite creativity of natural language means that your tokenizer will face words it has never seen before. As a result, handling OOV tokens becomes a fundamental challenge for any tokenization approach.
An OOV (out-of-vocabulary) token is any word or sequence not present in a tokenizer's known vocabulary. For example, unhappinesses might not be in a basic vocabulary, so it would be considered OOV.
Examples: OOV tokens can include:
- Newly coined words ("cryptojacking");
- Misspellings ("recieve");
- Domain-specific jargon ("bioinformatics");
- Foreign words or names ("Schwarzenegger").
Subword modeling—breaking words into smaller, more frequent parts—helps reduce OOVs by representing unseen words as combinations of known subwords. For example, unhappinesses might be split into un, happi, ness, es. However, this does not eliminate OOVs entirely, because truly novel character sequences, rare names, or code-like strings may still occur.
123456789101112131415161718192021222324252627# Example: Subword tokenization of an unseen word using a simple vocabulary vocab = {"un", "happi", "ness", "es", "bio", "info", "mat", "ics"} def subword_tokenize(word, vocab): tokens = [] i = 0 while i < len(word): # Try to find the longest subword in vocab that matches the current position found = False for j in range(len(word), i, -1): sub = word[i:j] if sub in vocab: tokens.append(sub) i = j found = True break if not found: # If no subword found, treat single character as OOV tokens.append(word[i]) i += 1 return tokens # Unseen word that is not in the vocabulary word = "unhappinesses" tokens = subword_tokenize(word, vocab) print("Tokens:", tokens) # Output: Tokens: ['un', 'happi', 'ness', 'es']
Grazie per i tuoi commenti!
