"If you spot any confusing or contradicting information, please ask the Team Leader via the chat rather than making assumptions."
Motivation
The scope of a test defines what areas of a customer's product are supposed to get tested, what functionalities to focus on, what bug types the customer is interested in, and what areas or features should not be tested by any means.
If something is in scope, please test it; if something is out of scope, it should not be tested. Understanding the scope of a test is crucial to be a successful tester on our platform.
What information makes up the scope of a test?
Terms of this test
First things first – The Terms of this test section will summarize the scope of the test and determine the language that should be used on your reports. This section is the first one you'll notice in a cycle because failing to follow these instructions will result in consequences for you and your Test IO account.
Test environment
The URL in the Access section at the top of the test overview page defines what website or app should be tested. Any other websites or apps are not under test (unless stated differently in the test description), including other builds of the same app. For more information, see Testing Environment.
You can only see access information once the test starts (mind the countdown at the top), confirm you want to participate in the cycle, and then agree with the cycle terms.
Features
Each test contains at least one of what we call features. A feature can describe an area of a product, e.g. a landing page, product overview page or product detail page of a webshop, it can describe a process like a checkout process of a website or an app, or it can even be a particular functionality that should be tested. For example:
When a customer sets up a new test, so-called standard features will be added by default. Customers can use these standard features and modify them if they want or add completely new features. You can learn more about some of our standard features here.
You can only submit bugs if a corresponding feature exists in the test. For example, if none of the given features covers the checkout process, you cannot submit bugs related to it.
Sometimes the customer may choose a standard set of features, and it may not match the names of the features on the site or application being tested. In such cases, we recommend asking for support from the test cycle team leader in the test cycle chat.
Bug types and severities
Customers are usually only interested in certain bug types and/or severities, not in all of them at once (Functional, Content, Visual, Usability). You can see which bug types and severities are in scope by checking out the payout table in the right sidebar on the test overview page:
If a bug type/severity is not listed there, you won't be able to select it in the bug form, and it is thus out of scope. E.g. if the payout table does not list the bug type Visual, you may not report visual bugs.
Test instructions
Test instructions are displayed below the Access section and they consist of three sections:
The goal of this test: This section usually covers the general purpose of the test. It often names what to focus on, e.g. new feature that is about to be released.
Out of scope: What is not supposed to get tested is specifically stated here. On live websites, you may oftentimes not complete orders at the end of a checkout process, or some areas of a website or an app might not be ready yet. Make sure to stick to this information – disregarding them can have big consequences.
Additional requirements: If there is anything else to point out, information will be given in this section. This might be information concerning the whole test, credentials for user accounts, dummy payment information, etc.
Attachments
In some cases, one or more attachments might be provided. They will be found in the right sidebar on the test overview page. Those are usually Spreadsheets, Documents, Images, Videos or PDF files given by the customer directly. The attachments might contain important information or requirements from the customer so make sure you checked them carefully before starting to test the app or website.
Requested devices
You will usually be invited to the cycle to use a specific device of yours. This device is displayed in the right sidebar on the test overview page. For ideal device coverage, our distribution algorithm invites testers for different devices. You may only use the requested devices that were assigned to you. When testing websites, you must not only use the correct device but also one of the available browsers on the list.
If the seats for the requested device are taken, you may be able to join the test with a different device if there are other slots opened. On the 'Submit a Bug' page you can check with which device you are in the test.
Chat information
The team leader or Customer Success Manager might have provided additional information about the scope or posted important reminders. Please always check the chat if you see that there are new messages. Whenever you have any questions about the environment or the scope of the cycle, please use the chat to ask for the Team Leader or even other testers' help.
Out of scope
Out of scope means that something is not supposed to be tested. Note that the following points are always out of scope (unless stated differently in the test description):
Legal problems. We are not legal advisors, and the severity of a bug is not determined by legal provisions, frameworks, or standards.
Problems related to browser extensions, ad-blockers, or virus scanners, e.g. blocking certain contents or even the execution of apps.
Website content should not be changed, e.g. by using the Google Translator service to translate the website into a different language, as it may lead to unexpected behaviour.
Set up problems in tests.
Issues associated with geolocation. e.g. If you experience a 403 error while trying to access the link from the testing environment section of the test cycle, it most likely means that the customer deliberately blocked your location and that you should not participate in the test. You can read more about location issues in our article.
Setup problems
Setup problems are not legitimate bugs – it only means there is a problem with the test setup or the test environment. Please report them to Team Leader via the test chat who will in turn contact the CSM. Here are a few examples:
The URL given in the test environment section of a test is wrong (404 error).
The test environment can only be accessed with credentials, but none are given or the given ones don't work.
The test environment requires a proxy or VPN to be accessed but this information is missing in the instructions (403, 1020 errors).
An app cannot be downloaded or installed.
The environment is not available for some time while testing. Remember the customer might update the environment during a test run, so make sure your issues are not temporary ones.
The customer provided login or payment credentials for user accounts but they don't work.
A link on a staging environment redirects to their live website. The link was not updated to point to the staging environment during the test environment setup.
The device requirements for a test e.g. list iOS 9 but the app under test can only be installed on iOS 10 and above.
The scope is unclear or contains contradicting information
If you don't understand something about the scope or if you find contradicting information, please contact the test's Team Leader via the test chat as described in the article Confusing Instructions. Sharing such information helps us improve the setups of future tests.
Why is information sometimes not consistent?
When customers intend to run a new test but the test setup of a previous test can be reused somehow, they will often duplicate that old test and adjust severities as well as additional requirements. Features and instructions are usually not adjusted, so sometimes irrelevant information might be shown to you or feature descriptions and additional requirements may contradict each other. Since features were duplicated but additional requirements were specifically written for the new test, additional requirements will often override feature descriptions.
If you spot any confusing or contradicting information, please ask the Team Leader via the chat rather than making assumptions.