How to Calculate N Gram Probability

What is N-Gram Probability?

An n-gram is a contiguous sequence of n items from a sample of text. In NLP, these items are typically words, characters, or syllables. The probability of an n-gram measures how frequently that sequence appears in a given text corpus relative to all possible sequences of the same length.

N-grams are classified by their length (n):

• Unigrams (n=1): Single words (e.g., "the")
• Bigrams (n=2): Pairs of words (e.g., "the cat")
• Trigrams (n=3): Triplets of words (e.g., "the cat sat")
• N-grams (n≥4): Longer sequences of words

N-gram probabilities are essential for tasks like text generation, machine translation, speech recognition, and spam detection.

How to Calculate N-Gram Probability

Calculating n-gram probability involves counting occurrences of the n-gram in a corpus and dividing by the total number of possible n-grams of that length. Here's the step-by-step process:

1. Define the corpus: Choose a text corpus (a collection of documents) to analyze.
2. Preprocess the text: Tokenize the text into words, convert to lowercase, and remove punctuation.
3. Count n-gram occurrences: Count how many times each n-gram appears in the corpus.
4. Calculate probabilities: Divide the count of each n-gram by the total number of possible n-grams of that length.

N-Gram Probability Formula

The probability of an n-gram w₁ w₂ ... w_n is calculated as:

Where:

• Count(w₁ w₂ ... wₙ) is the number of times the n-gram appears in the corpus.
• Count(w₁ w₂ ... wₙ₋₁) is the number of times the (n-1)-gram appears in the corpus.

For unigrams (n=1), the probability is simply the count of the word divided by the total number of words in the corpus.

Worked Example

Let's calculate the probability of the bigram "the cat" in a small corpus:

Corpus: "the cat sat on the mat. the cat slept on the mat."

1. Tokenize and count bigrams:
   • "the cat" appears 2 times
   • "cat sat" appears 1 time
   • "sat on" appears 1 time
   • "on the" appears 2 times
   • "the mat" appears 2 times
   • "cat slept" appears 1 time
2. Count the trigram "the cat sat": 1 occurrence
3. Count the bigram "the cat": 2 occurrences
4. Calculate probability:
   P("the cat sat") = Count("the cat sat") / Count("the cat") = 1 / 2 = 0.5

The probability of the bigram "the cat" is 0.5, meaning it appears half as often as the unigram "the cat" in this corpus.

Applications of N-Gram Probability

N-gram probabilities are used in various NLP applications:

• Text generation: Predicting the next word in a sequence based on previous words.
• Machine translation: Estimating the likelihood of word sequences in different languages.
• Speech recognition: Converting spoken language to text by analyzing word probabilities.
• Spam detection: Identifying spammy text patterns based on n-gram frequencies.
• Autocomplete: Suggesting the next word in search queries or messaging apps.