All posts by anjotestio

Invitation System Update

Here’s what’s new —

Device-specific Invitations:

We will now send out invitations for a specific device or OS you have listed in your profile.
This means if we have a test where some requested devices are uncommon, you will be invited for one of those devices. For example, we might invite you to test on a Samsung Galaxy S6. When you accept this invitation, you must use this device to test, even if there are other devices in the test you also might own.

This also applies to operating systems. If we invite you to test using a certain OS, you can use any device you have operating on this OS, but you must test on the OS for which you are invited. Reproductions can be done with any device in scope.

Tester Segmentation:

The system will be classifying testers based on performance. If you are really good at rapid tests, you may be marked as a “rapid test tester”. What impact does this have? We reserve extra invitations for testers who are parts of these performance-based segments

If you perform better in certain test types, you will have a better chance at being invited to these test types.
If you are not part of one of these specialist segments, you will still receive invitations to those tests.
Greenhorn testers (less than 50 bugs reported) have their own segment and will not be rated on their performance initially. As soon as you have reported 51 or more bugs our system will classify you based on all bugs you reported. So make sure you’re always doing your best work, even as a Greenhorn tester.

Test Distribution:

We consider test distribution updates the most important change.
Tests will be distributed equally among testers within one segment.
If you test as well as another tester, and your device coverage is the same, both of you should get about the same number of invites.
Many of you have told us lately that you are not getting as many tests as other testers. This issue should now be fixed. If you think this is not the case, please let us know!

The Leaderboards:

Points you score for reporting bugs and other activities do not impact the invitations you receive anymore.
We will keep the leaderboards active so you can compare yourself with other testers, if you want to.
But remember, rejected bugs will influence your performance rating and you are less likely to be in a top segment if you get a lot of bugs rejected.

Coming up —

Tester Profile:

Currently, you cannot see your own performance indicators. This is something we would like to change in the future.
To do so we are in the planning phase of developing a “profile page” where you can see things like “number of bugs posted,” “bug acceptance rate,” and other performance indicators.

New vs. experienced testers

Here at test IO we call our new testers “Greenhorns” and you will be a Greenhorn for your first 50 bugs.
After you posted 50 bugs we see you as experienced enough to know the basics of bug reporting here at test IO and you will lose your newbie status and all the advantages which come with it.
For a Greenhorn, team leaders will more often request further information if you forgot important steps or the quality of the bug is not good enough.

For testers which are not Greenhorns any more, the rules change a lot. Your bugs can be straight rejected for the following reasons:

  • Multiple grammar mistakes (not just a typo)
  • Not understandable description or results
  • Missing important steps in the description
  • Wrongly selected feature
  • Unclear bug title
  • Clearly not reading/understanding the test description
  • Incomplete evidence
    example: Attachments show a lot of stuff, but not what is important
    example: missing crash logs

So as a tester you should make sure every bug you submit fits the standards of quality and is as good as it can be.