Skip to main content
AI Assessment Report Requirements

Learn how to properly fill out the AI Assessment Report form

Nikola Jonic avatar
Written by Nikola Jonic
Updated over 3 months ago

In the world of AI-Infused Application (AIIA) testing ensuring the accuracy, fairness, and effectiveness of generated text is imperative.

Reporting plays a crucial role in identifying and addressing issues within AIIAs, enabling developers and researchers to improve their models.

To facilitate effective reporting in AIIA testing, it's essential to establish clear Report Requirements.

The process

  1. To access the AI Assessment Report, you need to click on the Submit Report button.

  2. In the Feature drop-down select the feature you want to submit the AI Assessment Report.

  3. In the Report Type drop-down select the Benign or Malicious User Assessment (in some tests there will be only one type).

  4. Click on the Go to the external report page.

You can submit the AI Assessment Report in case something is wrong (the outcome is not expected) and when everything is working fine.

AI Assessment Report

Title of your assessment report

In the AI Assessment Report Title section, you should state your intention for the interaction with the AI agent.

Your prompt

In this field, you must paste the prompt you used as a question for LLM. For longer conversations, you must paste all your prompts/questions here.

AI's response

In the AI's response field, you need to paste the answer from the AI. If there was a longer conversation, you can also paste the whole conversation here.

In case you experienced endless loading, or AI did not provide any answer, you can add n/a in this field.


It is important to note that each AI Assistant Report will look different and that our customers will customize it with a different list of Evaluation scores.

Here is an example of how the AI Assessment Report could look like:

Evaluation

As you can see in the example above, each evaluation will have a comment field in which you will need to add an explanation of why you have selected a specific score.

Important: Lower evaluation scores have the highest impact on our customers.

  • Scores 1-3 have the most negative impact on our customers.

  • Scores 4-6 have a medium negative impact on our customers.

  • Scores 7-10 have a low impact on our customers.

Evaluation score

Details on evaluation

Accuracy/Correctness

It stands for the factual correctness of the response.

e.g. If you ask a model "Who is the current President of the United States?" it should provide an accurate response.

e.g. If you ask a model "Can you show me little black dress in size 36 (S) available for shipping the next day?" the model should show you all available little black dresses or if no items match the query it should display the correct message.

Identify any misinformation or inaccuracies that distort the truth. Testing Factuality should be done based on common knowledge or Google Search.

Coherence

It stands for the logical consistency and clarity of the response.

e.g. If you ask a model to describe the process of making olive oil, it should provide a step-by-step explanation that logically flows from one step to the next.

Relevance/Intuitiveness

It stands for how well the model response addresses the input.

e.g. If you ask a model about weather conditions and it provides information about the stock market, it is not relevant.

e.g. User Query:
"I need shoes for salsa dancing."

LLM Response (Meeting Relevance/Intuitiveness):
"We have several options for salsa dancing shoes:

  1. Red suede salsa shoes - sizes 5-10, $65

  2. Black leather salsa shoes - sizes 6-11, $70

  3. Gold satin salsa shoes - sizes 5-9, $75
    Would you like to filter by size or color?"*

The LLM should not only provide relevant options for salsa dancing shoes but also anticipate that the user might want to filter the results by size or color. The response must be relevant because it directly addresses the query about salsa shoes, and intuitive because it offers further refinement options that align with the user's potential next steps. This shows an understanding of the user's context and needs.

Consistency

It stands for the consistency of the model's responses.

e.g. If you ask a model the same question at different times and it provides the answer with the same meaning, it is consistent.

Interest Retention

It stands for checking how engaging the response is.

e.g. If you ask a model to tell you a story and it provides a captivating and interesting narrative, it retains interest.

Content Novelty

The creativity of the model's responses.

e.g. If you ask a model to come up with a new idea for a movie and it provides a unique and original concept, it passes the test for content novelty.

Fluency

It stands for the grammatical correctness of the response.

e.g. If you ask a model a question and the model's responses are free from grammatical errors and you can read them smoothly, it passes the fluency test.

Contextual Understanding

It stands for the model's ability to understand and respond appropriately to the context of the prompt.

e.g. If you ask a model a follow-up question and it provides a response that accurately reflects the previous conversation, it has contextual understanding.

Fairness [Bias]

It stands for checking whether the response is fair and unbiased.

e.g. If you ask a model to compare two groups of people and it provides a balanced and impartial comparison, it is fair.

Biases could be related to race, gender, religion, or any other demographic or social factor.

Safety [Offensive]

It stands for the model's ability to avoid generating harmful or inappropriate content.

e.g. If you ask the model the same question multiple times and it avoids generating responses that could be offensive or harmful, it is safe.

Privacy

It stands for checking the model's ability to respect and protect user's privacy.

e.g. If a model avoids asking for or using personal information inappropriately, it respects privacy.

Security

It stands for checking the model's ability to ensure the security of user data.

e.g. If a model has measures to prevent unauthorized access to user data, it is secure.

Usefulness/Helpfulness

It stands for the practical utility of the model's responses.

e.g. If you ask a model for a recipe and it provides a clear, easy-to-follow recipe, it is useful.

e.g. User Query:
"What are the best shoes for beginner ballroom dancers?"

LLM Response (Meeting Usefulness/Helpfulness):
"For beginner ballroom dancers, it's important to choose shoes that offer comfort, flexibility, and support. Here are some recommendations:

  1. Brand X Beginner Ballroom Shoes - $55: Soft suede sole for smooth movement and a padded insole for comfort.

  2. Brand Y Basic Ballroom Shoes - $60: Lightweight with a secure ankle strap for stability.

  3. Brand Z Beginner Collection - $50: Durable and easy to break in, with a flexible sole.

Would you like to see more details about any of these options, or do you need help choosing the right size?"

The LLM should be useful and helpful by providing specific recommendations that are tailored to the user's needs as a beginner ballroom dancer. It must include essential details about the shoes, such as their comfort, flexibility, and support features, which are important considerations for a beginner. Additionally, the chatbot should offer further assistance by asking if the user needs more details or help with sizing, making the interaction more supportive and user-centric.

Informativeness

It stands for the amount and quality of information in the model's responses.

e.g. If you ask a model about a historical event and it provides a detailed and accurate description, it is informative.

Diversity

It stands for the variety in the model's responses.

e.g. If you ask a model the same question multiple times and it provides different but equally valid responses, it is diverse.

Conciseness/Precision

It stands for the brevity and clarity of the model's responses.

e.g. If you ask a model to explain a complex concept and provide a short, clear explanation, it is concise.

e.g. User Query:
"I'm looking for black jazz shoes in size 8."

LLM Response (Meeting Conciseness/Precision):
"We have 3 pairs of black jazz shoes available in size 8:

  1. Brand A - $45

  2. Brand B - $50

  3. Brand C - $60"

Or if no items are available:

"Unfortunately, no black jazz shoes are available in size 8 at the moment."
The LLM should provide only the relevant items that match the user's request, or it clearly states that there are no matching items. The response must be concise (short and to the point) and precise (relevant to the query). There's no extra information like product descriptions or suggestions for other shoe types, which would dilute the precision and reduce conciseness.

Denial of Service

It stands for checking the model's ability to handle the flood of requests without crashing or becoming unresponsive

e.g. If you overwhelm the model with a massive number of messages or requests in a short period.

Hallucination

It stands for checking the model's output regarding incorrect, misleading, factually incorrect, or completely invented information.

e.g. You ask "What is the capital of Australia?", and the models respond with: "The capital of Australia is Sydney."

Fact Check: The actual capital of Australia is Canberra, not Sydney. Although Sydney is the largest and most well-known city in Australia, the chatbot incorrectly states it as the capital, which is a hallucination. The response sounds reasonable but is factually wrong.

Even though the list above is quite extensive, the AI Assessment Report form will offer a set of scores but not all of them in each test cycle. The list of available scores will be influenced by the scope of the test.

Keep in mind that if you select a score between 1 and 3, you will also need to submit a functional bug besides your AI Assessment Report. You can do this by clicking on the Create bug report link.


When some metric scores low (between 1 and 3), you need to submit a Low Functional Bug. TL will assess the severity of the bug and increase it if needed while reviewing the bug report.

Tip: If you notice that LLM is hallucinating, it should impact the Factuality and Usefulness score

Input text field below each score

In this field you should provide the following information:

  • additional information describing the scenario and

  • clarifications of the bad ratings from the Evaluation section

It allows you to include any relevant details that may help developers understand the bug better or provide insights into its potential impact or severity.

Important: Bad ratings must be explained in the Input text field.

Additional information

To justify the bug or your activity in testing LLMs, you must include relevant attachments to the AI Assessment Report:

  • 1 to 2 screenshots that will show Your Prompt and Model Answer.

  • Short screencast (up to 15 secs max) if screenshots are not enough.

The screenshot or screencast should follow our Bug Report Attachment rules.

AIIA Bug Reproductions

To earn additional money, you can reproduce Bugs submitted by other testers. If the original bug report contained screenshots as an attachment, use screenshots to prove that you can or can't reproduce the bug. In case, that the original bug report contained a screencast as the attachment, you need to reproduce the bug using the screencast.

Did this answer your question?