I trained a contrastive learning model on image-text pairs, and now I want to retrieve the most similar text based on images. For this purpose i generate test images (embeddings) and test text (embeddings) using the pretrained image and text encoders.

and then I want to measure cosine similarity between each text and all image embeddings like below

“
images = (i1, i2, i3) text = (t1,t2, t3)

cosine similarity = [(t1,i1),(t1, i2), (t1, i3)], [(t2, i1), (t2, i2), (t2, i3)], [(t3, i1), (t3, i2), (t3, i3)]
“

Using this code

image_embeddings = [loaded_global_vision_encoder(image) for image in test_images]

# Generate finding embeddings finding_embeddings = [loaded_finding_encoder(tf.convert_to_tensor([finding])) for finding in test_findings]

`# Compute cosine similarity between image and finding embeddings

cosine_similarities = []
for finding_emb in finding_embeddings:
similarities = [tf.keras.losses.cosine_similarity(finding_emb, image_emb, axis=-1) for image_emb in image_embeddings]
cosine_similarities.append(similarities)

 

I then find top 1 pairs with high similarity score like this 


`image_embeddings = [
    [0.1, 0.2, 0.3],
    [0.4, 0.5, 0.6],
]

finding_embeddings = [
    [0.3, 0.4, 0.5],
    [0.6, 0.7, 0.8]
]`

`For the first finding embedding [0.3, 0.4, 0.5]:
Cosine similarities:
- With [0.1, 0.2, 0.3]: 0.9746
- With [0.4, 0.5, 0.6]: 0.9873

For the second finding embedding [0.6, 0.7, 0.8]:
Cosine similarities:
- With [0.1, 0.2, 0.3]: 0.8847
- With [0.4, 0.5, 0.6]: 0.9603

`
Step 2: Find the top-k most similar image indices
Let's assume k = 2 (you can adjust this value as needed).
For the first finding embedding [0.3, 0.4, 0.5]:

Top 2 most similar image indices: [3, 1]

For the second finding embedding [0.6, 0.7, 0.8]:
Top 2 most similar image indices: [2, 4]

results = [
    [3, 1],
    [2, 4],
]
```

using this code 

```
k = 1
results = [tf.math.top_k(similarities, k).indices.numpy() for similarities in cosine_similarities]
```


So now I don't no this retrieval part at all, and I am very confused to understand it. I do it's code like this 



```
predicted_reports = [[test_findings[int(idx)] for idx in indices] for indices in results]
```



and I think it's work like this

```
test_findings = [
    "No acute cardiopulmonary process.",
    "Heart size is normal. No focal airspace disease or effusion.",
    "Low lung volumes. No acute findings.",
    "Clear lungs. No pleural effusions or pneumothoraces.",
    "Degenerative changes in the thoracic spine."
]

results = [
    [3, 1],
    [2, 4],
    [0, 3]
]

predicted_reports = [
    [
        "Clear lungs. No pleural effusions or pneumothoraces.",
        "Heart size is normal. No focal airspace disease or effusion."
    ],
    [
        "Low lung volumes. No acute findings.",
        "Degenerative changes in the thoracic spine."
    ],
    [
        "No acute cardiopulmonary process.",
        "Clear lungs. No pleural effusions or pneumothoraces."
    ]
]

```





If it is working in that manner in which I explained it, then I would be happy because I am not satisfied with the results using this retrieval manner however i am intereted to know whether my concept about this code is right or not.