Skip to main content
Exploratory LLM Testing

Quickly learn about Exploratory LLM Testing specifics

Zorica Micanovic avatar
Written by Zorica Micanovic
Updated over a week ago

Exploratory LLM Testing is a powerful tool for freelance testers at Test IO. It allows you to:

  • leverage your creativity and

  • critical thinking skills to uncover potential issues and improve the overall performance of the LLM.

This method involves testing the Language Learning Model (LLM) using your prompts, rather than following a predetermined set of test cases.

Key Features of Exploratory LLM Testing

  1. Spontaneity and Flexibility: You are not bound by a script or a rigid set of instructions. You are free to interact with the system in any way you see fit, using your prompts to guide your exploration.

  2. Critical and Creative Thinking: The success of Exploratory LLM Testing largely depends on your ability to think critically and creatively. Your deep understanding of the LLM and its potential use cases, combined with your ability to think outside the box, will enable you to identify potential issues and suggest improvements.

  3. Understanding System Behavior: Exploratory LLM Testing is not just about finding bugs; it's also about understanding the system and its behavior. As you interact with the LLM using your prompts, you will gain a deeper understanding of how the system responds to different inputs and scenarios.

Types of Exploratory LLM Testing and Scenarios

There are 3 Testing Types of Exploratory LLM Testing:

  • Positive Testing is testing the LLM with valid inputs and expected use cases.

  • Negative Testing is testing the LLM with invalid inputs or unexpected use cases.

  • Edge Case Testing is testing the LLM with extreme or boundary inputs.

Below, you can find some of the examples with Pass or Fail scenarios.

Type of Exploratory LLM Testing

Pass Scenario

Fail Scenario

Positive Testing

If the LLM is expected to translate a simple sentence from English to Spanish, providing a correct English sentence should result in an accurate Spanish translation.

If you provide a correct English sentence, but the LLM translates it inaccurately or not at all.

Negative Testing

If you provide a sentence with incorrect grammar or syntax, the LLM should be able to identify the error and either correct it or provide an error message.

If you provide a sentence with incorrect grammar or syntax, and the LLM translates it without identifying the error.

Edge Case Testing

If you provide a very long sentence, the LLM should still be able to process the input and provide a reasonable output.

If you provide a very long sentence and the LLM fails to process the input or provides an incorrect output.

Exploratory LLM Testing aims not just to find bugs, but also to understand the system's behavior and suggest improvements. When the test scenario fails, it provides valuable insights that can be used to enhance the LLM's performance and usability.

Did this answer your question?