“Bag of Words” (BoW) is a simple and commonly used method to represent text data when we are working with machine learning algorithms. It’s called “Bag of Words” because it represents each document as a bag (or set) of its words, disregarding grammar and word order but keeping track of frequency of occurrence.
In the BoW model, a text (such as a sentence or a document) is represented as a bag (multiset) of its words. The frequency of occurrence of each word is used as a feature for training a classifier.
Here’s how it typically works:
- Create a vocabulary: First, create a vocabulary of unique words from all the documents in the dataset. This vocabulary acts as a feature set.
- Create document vectors: For each document, create a vector of zeros of length equal to the size of the vocabulary.
- Count word occurrences: Go through the document word by word. For each word, increment the corresponding element in the vector. This element is found at the same position as the word is in the vocabulary.
The result is a matrix where each row represents a document and each column represents a word from the vocabulary. The value at the intersection of a row and a column is the frequency of that word in the corresponding document.
For example, given the two sentences “The cat sat on the mat” and “the cat ate the mouse”, and a vocabulary of {‘the’, ‘cat’, ‘sat’, ‘on’, ‘mat’, ‘ate’, ‘mouse’}, the BoW representation might be:
- “The cat sat on the mat”: [2, 1, 1, 1, 1, 0, 0] (because ‘the’ appears twice, ‘cat’, ‘sat’, ‘on’, and ‘mat’ each appear once, and ‘ate’ and ‘mouse’ do not appear)
- “The cat ate the mouse”: [2, 1, 0, 0, 0, 1, 1] (with a similar reasoning)
BoW is simple and effective for some tasks, but it has limitations. It discards information about the order of the words, and it treats all words as equally important. It also doesn’t capture any information about semantics or context. More complex methods like TF-IDF weighting or word embeddings can help address some of these limitations.
« Back to Glossary Index