Retrieval-Augmented-Generation (RAG) on simple terms (Part-02)

Lets start with Embeddings. If I give you three animal names such as Lion ,Cat, Dog and ask you to differentiate them into two groups you can easily divide them. Cat, Dog : Domestic Pets , Lion : A carnivores animal, not domestic . Like that, machines shall be able to differentiate this. So in AI or machine learning algorithms embeddings took this job. It contains the representation of a particular object.

example lets take a simple phrase to learn about embeddings.

"The employee is eligible for dental coverage." -> [Embeddding Model] ->

[0.021, -0.184, 0.732, 0.091, ..., 0.442] <- this can take 384,768 or 1536 numbers. This is basically called dimensions. It is again a state of art deciding number of dimensions of an embedding model. 384 may be less informative and 1536 may be more informative. Based on this output becomes different. since this is a vector representation, we can perform cosine similarity to check whether , two embeddings are similar.

As an example:

A = "The employee is eligible for dental insurance." B = "The worker qualifies for dental coverage." C = "What my travel insurance offers?"

Considering the cosine similarity we can determine A and B are more relatable compare to A and C or B and C. Here how this can be performed using python.

from sentence_transformers import SentenceTransformer
from sklearn.metrics.pairwise import cosine_similarity

model = SentenceTransformer("all-MiniLM-L6-v2")

A = "The employee is eligible for dental insurance."
B = "The worker qualifies for dental coverage."
C = "What my travel insurance offers?"

embedding1 = model.encode(A)
embedding2 = model.encode(B)
embedding3 = model.encode(C)

similarity_between_A_B = cosine_similarity(
    [embedding1],
    [embedding2]
)

similarity_between_A_C = cosine_similarity(
    [embedding1],
    [embedding3]
)

similarity_between_B_C = cosine_similarity(
    [embedding2],
    [embedding3]
)

print("Embedding size :" ,len(embedding1))


print("Cosine similarity: Between A and B", similarity_between_A_B[0][0])
print("Cosine similarity: Between A and C", similarity_between_A_C[0][0])
print("Cosine similarity: Between B and C", similarity_between_B_C[0][0])

Output as follows:

Embedding size : 384
Cosine similarity: Between A and B 0.8540557
Cosine similarity: Between A and C 0.28483576
Cosine similarity: Between B and C 0.21922925

You can see A and B more relatable compared to (A and C) and (B and C). This is the underlying layer of the prediction of next token based on the similarity of the text. To have a better output of this, we need to determine better chunking strategy . We will look into that in next part.