Skip to main content
LLM Testing Test Process

Quickly learn about the LLM Agent Testing test process on our platform

Zorica Micanovic avatar
Written by Zorica Micanovic
Updated over a week ago

LLM Testing at Test IO represents a specific process which encompasses several different stages:

  • Accepting the Test Invitation

  • Testing the LLM

  • Reporting

Accepting the Test Invitation

LLM Test Invitations are sent to testers who passed the LLM Testing Course and who match other requirements (location, gender, age, device combo, etc).

Remember: The LLM Test Invitation looks like any other Test Invitation from our platform. The only differences are:

  • The Test Scope

  • Included Bug Types

Testing the LLM

Unlike Manual Functional Testing where you test the application functionality (working as expected), Manual LLM Testing requires you to focus only on the quality and appropriateness of the LLM output. Depending on the LLM test type, your testing should be focused on:

  • Exploratory Testing or

  • Regression Testing

In LLM Exploratory Testing test cycles you will need to craft your messages for LLM and validate the response quality and appropriateness while in the LLM Regression Testing, you will evaluate the responses on the messages available in the Test Cases.

LLM Reporting

While in the Manual Functional Testing, you submit bugs, in the Manual LLM Testing, you need to submit Reports. Reports in LLM Testing could be:

  1. Bug Reports or

  2. Validation Reports

The difference between LLM Bug Reports and LLM Validation Reports is in the response outcome. If the LLM is showing behavior that should not be present, you should report a bug. In cases when LLM performs as expected, you should submit a Validation Report.

Remember: Some tests will have only Bug Reports in the scope of the test while other tests will have both enabled.

Did this answer your question?