AI Challenge Dayで学んだRAGの評価指標

from openai import AzureOpenAI
import os
from dotenv import load_dotenv
import numpy as np
from tenacity import (
    retry,
    stop_after_attempt,
    wait_random_exponential,
)  # for exponential backoff


client = AzureOpenAI(
    api_key=os.getenv("AZURE_OPENAI_API_KEY"),
    api_version=os.getenv("AZURE_OPENAI_API_VERSION"),
    azure_endpoint = os.getenv("AZURE_OPENAI_ENDPOINT")
)

ground_truth="正解テキスト"
answer="生成したテキスト"

ground_truth_embeddings = generate_embeddings(ground_truth)
answer_embeddings = generate_embeddings(answer)
ada_cosine_similarity_score = cosine_similarity_to_bin(calc_cosine_similarity(ground_truth_embeddings, answer_embeddings))
print("評価点: ", ada_cosine_similarity_score)

@retry(wait=wait_random_exponential(min=1, max=60), stop=stop_after_attempt(6))
def generate_embeddings(text, model=os.getenv("AZURE_OPENAI_EMBED_DEPLOYMENT_NAME")):
    return client.embeddings.create(input = [text], model=model).data[0].embedding

def calc_cosine_similarity(v1, v2):
    dot_product = np.dot(v1, v2)
    norm_v1 = np.linalg.norm(v1)
    norm_v2 = np.linalg.norm(v2)
    cos = dot_product / (norm_v1 * norm_v2)
    return cos

def cosine_similarity_to_bin(cosine_similarity):
    # コサイン類似度値を1~5に分類
    if cosine_similarity < 0.2:
        return 1
    elif cosine_similarity < 0.4:
        return 2
    elif cosine_similarity < 0.6:
        return 3
    elif cosine_similarity < 0.8:
        return 4
    else:  # 0.8以上1以下
        return 5

gpt_fluency_prompt_sys="""
You are an AI assistant. You will be given the definition of an evaluation metric for assessing the quality of an answer in a question-answering task. Your job is to compute an accurate evaluation score using the provided evaluation metric.
"""

gpt_relevance_prompt_user="""
Relevance measures how well the answer addresses the main aspects of the question, based on the context. Consider whether all and only the important aspects are contained in the answer when evaluating relevance. Given the context and question, score the relevance of the answer between one to five stars using the following rating scale:
One star: the answer completely lacks relevance
Two stars: the answer mostly lacks relevance
Three stars: the answer is partially relevant
Four stars: the answer is mostly relevant
Five stars: the answer has perfect relevance

This rating value should always be an integer between 1 and 5. So the rating produced should be 1 or 2 or 3 or 4 or 5.

{5個の例があったが、ページ数の都合でカット}

context: {context}
question: {question}
answer: {answer}
stars:
"""

"""
あなたはcontext(文脈)とそのcontextについてのanswer(回答)が提示されます。以下の評価から一つ選んで、answerがcontextから導き出されるかどうかを判断する必要があります:

{前の評価と類似しているため省略}

文脈：{context}
回答：{answer}
星：
"""

プロンプト「"""Relevance measures how {略} answer: {answer} stars:""" 」
GPT「3」

gpt_groundedness_prompt_user="""
You will be presented with a CONTEXT and an ANSWER about that CONTEXT. You need to decide whether the ANSWER is entailed by the CONTEXT by choosing one of the following rating:

{前の評価と類似しているため省略}

CONTEXT: {context}
ANSWER: {answer}
STARS:
"""

"""
あなたはcontext(文脈)とそのcontextについてのanswer(回答)が提示されます。以下の評価から一つ選んで、answerがcontextから導き出されるかどうかを判断する必要があります:

{前の評価と類似しているため省略}

文脈：{context}
回答：{answer}
星：
"""

gpt_similarity_prompt_user="""
Equivalence, as a metric, measures the similarity between the predicted answer and the correct answer. If the information and content in the predicted answer is similar or equivalent to the correct answer, then the value of the Equivalence metric should be high, else it should be low. Given the question, correct answer, and predicted answer, determine the value of Equivalence metric using the following rating scale:

{前の評価と類似しているため省略}

CONTEXT: {context}
ANSWER: {answer}
STARS:
"""

"""
等価性とは、メトリクとして、予測された答えと正しい答えの間の類似性を測定するものです。予測された答えの情報と内容が正しい答えと似ているか等価であれば、その等価性のメトリクの値は高くなるはずです。それ以外の場合は、その値は低くなります。以下の評価尺度を用いて、質問、正しい答え、予測した答えから等価性のメトリクの値を決定します。

{前の評価と類似しているため省略}

質問: {question}
正しい答え:{ground_truth}
回答: {answer}
星: 
"""

gpt_fluency_prompt_user="""
Fluency measures the quality of individual sentences in the answer, and whether they are well-written and grammatically correct. Consider the quality of individual sentences when evaluating fluency. Given the question and answer, score the fluency of the answer between one to five stars using the following rating scale:

{前の評価と類似しているため省略}

CONTEXT: {context}
ANSWER: {answer}
STARS:
"""

"""
流暢性は、回答の個々の文章の質や、それらがよく書かれていて文法的に正しいかどうかを測定します。流暢性を評価するときは、個々の文章の質を考慮に入れてください。質問と回答が与えられたら、以下の評価尺度を使用して、回答の流暢性を1から5つ星で評価します：

{前の評価と類似しているため省略}


質問: {question}
回答: {answer}
星: 
"""

***** Result *****
average_score: {'gpt_relevance': 3.75, 'gpt_groundedness': 1.8, 'gpt_similarity': 2.55, 'gpt_fluency': 4.975, 'ada_cosine_similarity': 4.975}
total_score: 18.050