I’m using RelevancyEvaluator when looping through the source_nodes of a query_engine response, and the .evaluate() says that every source node is passing (result.passing returns True). This is of course not correct, and is preventing me from obtaining the correct source node that was used by the llm to answer my query.

The LLM I’m using is amazon/MistralLite, and the embedding model I’m using is sentence-transformers/all-MiniLM-L6-v2.

Any suggestions?

RelevancyEvaluator from llama-index is returning the wrong Pass/Fail on response source nodes.