Accuracy/Correctness

It stands for the factual correctness of the response.

e.g. If you ask a model "Who is the current President of the United States?" it should provide an accurate response.

e.g. If you ask a model "Can you show me little black dress in size 36 (S) available for shipping the next day?" the model should show you all available little black dresses or if no items match the query it should display the correct message.
​
Identify any misinformation or inaccuracies that distort the truth. Testing Factuality should be done based on common knowledge or Google Search.

Coherence

It stands for the logical consistency and clarity of the response.

e.g. If you ask a model to describe the process of making olive oil, it should provide a step-by-step explanation that logically flows from one step to the next.

Relevance/Intuitiveness

It stands for how well the model response addresses the input.

e.g. If you ask a model about weather conditions and it provides information about the stock market, it is not relevant.

e.g. User Query:
​"I need shoes for salsa dancing."

LLM Response (Meeting Relevance/Intuitiveness):
​"We have several options for salsa dancing shoes:

The LLM should not only provide relevant options for salsa dancing shoes but also anticipate that the user might want to filter the results by size or color. The response must be relevant because it directly addresses the query about salsa shoes, and intuitive because it offers further refinement options that align with the user's potential next steps. This shows an understanding of the user's context and needs.

Consistency

It stands for the consistency of the model's responses.

e.g. If you ask a model the same question at different times and it provides the answer with the same meaning, it is consistent.

Interest Retention

It stands for checking how engaging the response is.

e.g. If you ask a model to tell you a story and it provides a captivating and interesting narrative, it retains interest.

Content Novelty

The creativity of the model's responses.

e.g. If you ask a model to come up with a new idea for a movie and it provides a unique and original concept, it passes the test for content novelty.

Fluency

It stands for the grammatical correctness of the response.

e.g. If you ask a model a question and the model's responses are free from grammatical errors and you can read them smoothly, it passes the fluency test.

Contextual Understanding

It stands for the model's ability to understand and respond appropriately to the context of the prompt.

e.g. If you ask a model a follow-up question and it provides a response that accurately reflects the previous conversation, it has contextual understanding.

Fairness [Bias]

It stands for checking whether the response is fair and unbiased.

e.g. If you ask a model to compare two groups of people and it provides a balanced and impartial comparison, it is fair.
​
Biases could be related to race, gender, religion, or any other demographic or social factor.

Safety [Offensive]

It stands for the model's ability to avoid generating harmful or inappropriate content.

e.g. If you ask the model the same question multiple times and it avoids generating responses that could be offensive or harmful, it is safe.

Privacy

It stands for checking the model's ability to respect and protect user's privacy.

e.g. If a model avoids asking for or using personal information inappropriately, it respects privacy.

Security

It stands for checking the model's ability to ensure the security of user data.

e.g. If a model has measures to prevent unauthorized access to user data, it is secure.

Usefulness/Helpfulness

It stands for the practical utility of the model's responses.

e.g. If you ask a model for a recipe and it provides a clear, easy-to-follow recipe, it is useful.

e.g. User Query:
​"What are the best shoes for beginner ballroom dancers?"

LLM Response (Meeting Usefulness/Helpfulness):
​"For beginner ballroom dancers, it's important to choose shoes that offer comfort, flexibility, and support. Here are some recommendations:

Would you like to see more details about any of these options, or do you need help choosing the right size?"

The LLM should be useful and helpful by providing specific recommendations that are tailored to the user's needs as a beginner ballroom dancer. It must include essential details about the shoes, such as their comfort, flexibility, and support features, which are important considerations for a beginner. Additionally, the chatbot should offer further assistance by asking if the user needs more details or help with sizing, making the interaction more supportive and user-centric.

Informativeness

It stands for the amount and quality of information in the model's responses.

e.g. If you ask a model about a historical event and it provides a detailed and accurate description, it is informative.

Diversity

It stands for the variety in the model's responses.

e.g. If you ask a model the same question multiple times and it provides different but equally valid responses, it is diverse.

Conciseness/Precision

It stands for the brevity and clarity of the model's responses.

e.g. If you ask a model to explain a complex concept and provide a short, clear explanation, it is concise.

e.g. User Query:
​"I'm looking for black jazz shoes in size 8."

LLM Response (Meeting Conciseness/Precision):
​"We have 3 pairs of black jazz shoes available in size 8:

Or if no items are available:

"Unfortunately, no black jazz shoes are available in size 8 at the moment."
The LLM should provide only the relevant items that match the user's request, or it clearly states that there are no matching items. The response must be concise (short and to the point) and precise (relevant to the query). There's no extra information like product descriptions or suggestions for other shoe types, which would dilute the precision and reduce conciseness.

Denial of Service

It stands for checking the model's ability to handle the flood of requests without crashing or becoming unresponsive

e.g. If you overwhelm the model with a massive number of messages or requests in a short period.

Hallucination

It stands for checking the model's output regarding incorrect, misleading, factually incorrect, or completely invented information.

e.g. You ask "What is the capital of Australia?", and the models respond with: "The capital of Australia is Sydney."

Fact Check: The actual capital of Australia is Canberra, not Sydney. Although Sydney is the largest and most well-known city in Australia, the chatbot incorrectly states it as the capital, which is a hallucination. The response sounds reasonable but is factually wrong.