The New Age of Language Models: Vector Embeddings
The world is witnessing a seismic shift in language applications, and vector databases are at the core of this transformation. Understanding vectors and their importance is no longer a choice but a necessity. The article source link below sheds light on the difference in vector embeddings between models and illustrates how to utilize multiple collections of vector data in one Jupyter Notebook.
What Are Vector Embeddings and Why Are They Important?
Vector embeddings are numerical representations of data, primarily used to represent unstructured data like images, videos, audio, text, and more. They are generated by running input data through a pretrained neural network. The uniqueness of each model’s vector embedding makes working with unstructured data a complex task. The differences in neural networks necessitate the use of distinct models to process various forms of unstructured data.
How Do We Compare Vector Embeddings?
Comparing vector embeddings is an intricate process. This article explores the comparison of three different multilingual models based on MiniLM from Hugging Face. The comparison is done using the L2 distance metric and an inverted file index as the vector index. The project is inspired by Taylor Swift’s recent album release, and the data is defined right in the Jupyter Notebook.
Comparing Vector Embeddings from Different Models
The comparison involves three models: the base multilingual paraphrase model, a version fine-tuned for intent detection, and one fine-tuned by Sprylab. The vectors must be of the same length/dimension for comparison. The example uses 384-dimensional vectors, creating an abstract representation of the data in a 384-dimensional space.
Vector Embedding Comparison Data
The data used for comparison consists of sentences from four of Taylor Swift’s songs. The choice of songs is deliberate, as they form complete sentences and test a hypothesis related to love and break-up songs.
Comparing Vector Embeddings in Your Jupyter Notebook
The comparison is done using Milvus Lite, a lightweight version of Milvus, directly in a Jupyter Notebook. The code includes the necessary imports, connections, and steps to get the Sentence Transformer Models from Hugging Face, load the data sets, and query the vector stores to compare the embeddings.
Summary
This tutorial offers a comprehensive understanding of vector embeddings and their comparison. It highlights the difference between a model and its fine-tuned versions and demonstrates how one result can pop up in different embedding spaces. The article encourages further exploration with image models, different dimensionality language models, or personalized data.
