Consider the sentence, "The quick brown fox jumps over the lazy dog." To a human, this is a simple declarative statement. To a computer, however, it is merely a sequence of characters. For machines to understand and process human language, these sequences must be converted into a numerical format they can comprehend. This is where word embeddings and vector spaces come into play, offering a way to represent meaning as geometry.
The core idea is that words can be mapped to points in a multi-dimensional space, where the distance and direction between these points capture the semantic relationships between the words. This concept is rooted in the distributional hypothesis, famously articulated by linguist J.R. Firth in 1957: "You shall know a word by the company it keeps." In essence, words that appear in similar contexts tend to have similar meanings.
Early attempts to represent words numerically involved methods like one-hot encoding, where each word was assigned a unique vector with a single "1" and the rest "0s." This approach, while simple, created extremely high-dimensional and sparse vectors, failing to capture any semantic relationships between words. For instance, the vectors for "king" and "queen" would be as distant as the vectors for "king" and "banana."
A significant leap occurred in the early 2000s with the development of neural language models. In 2003, Yoshua Bengio and colleagues introduced a neural probabilistic language model that learned "distributed representations" for words, addressing the high dimensionality issue. This laid the groundwork for what became known as word embeddings.
The 2013 paper "Efficient Estimation of Word Representations in Vector Space" by Tomas Mikolov and his team at Google introduced Word2Vec. Word2Vec employs neural network architectures, primarily Continuous Bag-of-Words (CBOW) and Skip-gram, to learn word embeddings. CBOW predicts a target word from its surrounding context, while Skip-gram predicts the context words given a target word. These models learn by analyzing massive text corpora. For example, the Word2Vec model was trained on a corpus of approximately 6 billion words.
A year later, in 2014, Jeffrey Pennington, Richard Socher, and Christopher Manning at Stanford University introduced GloVe (Global Vectors for Word Representation). Unlike Word2Vec, which focuses on local context windows, GloVe leverages global word-word co-occurrence statistics from the entire corpus. The "GloVe" model is a log-bilinear model that uses a weighted least-squares objective, aiming to learn word vectors such that their dot product relates to the logarithm of their co-occurrence probability. This global approach allows GloVe to capture both local and global relationships between words.
Another notable model is FastText, developed by Facebook AI Research and published around 2016-2017. FastText builds upon the Skip-gram model by considering sub-word information, specifically character n-grams. This allows it to generate representations for words not seen during training, which is particularly useful for morphologically rich languages.
The output of these models is a vector space where words are represented as vectors. The key insight is that the geometric relationships between these vectors encode semantic meaning. For instance, it was famously demonstrated that vector("King") - vector("Man") + vector("Woman") ≈ vector("Queen"). This type of vector arithmetic allows machines to perform analogical reasoning. Words with similar meanings, such as "doctor" and "nurse," will have vectors that are close to each other in this space, often measured by cosine similarity.
These word embeddings have found widespread application across numerous Natural Language Processing (NLP) tasks. They serve as input features for text classification (e.g., sentiment analysis, spam detection), named entity recognition, machine translation, information retrieval, and question answering systems. For example, in machine translation, word embeddings can help represent words in a language-agnostic manner, improving the model's understanding of semantic relationships between source and target languages.
While static word embeddings like Word2Vec and GloVe assign a single vector to each word, newer contextual embeddings, such as those from BERT (Bidirectional Encoder Representations from Transformers), can generate different vectors for the same word depending on its context. BERT, introduced in 2017, uses attention mechanisms to consider the full context of a word, looking at words both before and after it. Recent advancements in 2023-2024 have focused on universal text embeddings and large language model (LLM) driven approaches, aiming for models that can generalize across various tasks, domains, and languages.
The ongoing research in this area explores methods for more accurate and nuanced representation of language, pushing the boundaries of how machines understand and interact with human communication.