Invitation System Update

Here’s what’s new —

Device-specific Invitations:

We will now send out invitations for a specific device or OS you have listed in your profile.
This means if we have a test where some requested devices are uncommon, you will be invited for one of those devices. For example, we might invite you to test on a Samsung Galaxy S6. When you accept this invitation, you must use this device to test, even if there are other devices in the test you also might own.

This also applies to operating systems. If we invite you to test using a certain OS, you can use any device you have operating on this OS, but you must test on the OS for which you are invited. Reproductions can be done with any device in scope.

Tester Segmentation:

The system will be classifying testers based on performance. If you are really good at rapid tests, you may be marked as a “rapid test tester”. What impact does this have? We reserve extra invitations for testers who are parts of these performance-based segments

If you perform better in certain test types, you will have a better chance at being invited to these test types.
If you are not part of one of these specialist segments, you will still receive invitations to those tests.
Greenhorn testers (less than 50 bugs reported) have their own segment and will not be rated on their performance initially. As soon as you have reported 51 or more bugs our system will classify you based on all bugs you reported. So make sure you’re always doing your best work, even as a Greenhorn tester.

Test Distribution:

We consider test distribution updates the most important change.
Tests will be distributed equally among testers within one segment.
If you test as well as another tester, and your device coverage is the same, both of you should get about the same number of invites.
Many of you have told us lately that you are not getting as many tests as other testers. This issue should now be fixed. If you think this is not the case, please let us know!

The Leaderboards:

Points you score for reporting bugs and other activities do not impact the invitations you receive anymore.
We will keep the leaderboards active so you can compare yourself with other testers, if you want to.
But remember, rejected bugs will influence your performance rating and you are less likely to be in a top segment if you get a lot of bugs rejected.

Coming up —

Tester Profile:

Currently, you cannot see your own performance indicators. This is something we would like to change in the future.
To do so we are in the planning phase of developing a “profile page” where you can see things like “number of bugs posted,” “bug acceptance rate,” and other performance indicators.

New vs. experienced testers

Here at test IO we call our new testers “Greenhorns” and you will be a Greenhorn for your first 50 bugs.
After you posted 50 bugs we see you as experienced enough to know the basics of bug reporting here at test IO and you will lose your newbie status and all the advantages which come with it.
For a Greenhorn, team leaders will more often request further information if you forgot important steps or the quality of the bug is not good enough.

For testers which are not Greenhorns any more, the rules change a lot. Your bugs can be straight rejected for the following reasons:

  • Multiple grammar mistakes (not just a typo)
  • Not understandable description or results
  • Missing important steps in the description
  • Wrongly selected feature
  • Unclear bug title
  • Clearly not reading/understanding the test description
  • Incomplete evidence
    example: Attachments show a lot of stuff, but not what is important
    example: missing crash logs

So as a tester you should make sure every bug you submit fits the standards of quality and is as good as it can be.

test IO tester ms_frodo shares his 5 top tips to crowd testing success!

1st tip: Thoroughly read test instructions before starting testing
Often some testers start testing, without reading all the test instructions. As a result many bugs raised by them are rejected with one of the following reasons: the most popular “bug is out of scope” and two less popular “this is not a bug” or “this bug is known”. This tip will allow you to escape rejected bugs, as result, keeping your tester ranking and not wasting time for nothing.

2nd tip: Increase quality of your bug report
When you start filling out the bug report, make sure that you include all important information needed for creating your bug. Improve your own English level for writing (if it is not already very good) bug reports grammatically correct and understandable for reviewer and customer. Use rule of three ‘W’ (What, Where, When) in your bug title. This tip will allow you to write quality bugs, as result again escaping rejected bugs. And one more plus, the reviewer and customer will enjoy working with you and your bug reports.

3rd tip: Ensure that your bug isn’t a duplicate
Before raising a bug, check the bug list and ensure that your bug isn’t a duplicate. Sometimes I firstly raise the bug and immediately after I check the bug list to ensure that my bug isn’t a duplicate. Most rejected bugs are rejected for the reason “this bug is a duplicate..”. This tip is very simple, but many testers forget about it and as a result, they have many rejected bugs.

4th tip: Check the full functionality of the app during testing
When you start to execute testing of the app, ensure that you checked all features inside this app. Sometimes, high or critical bugs are located in the most prominent places, I mean features like “upload button doesn’t work” or “unable to sign up via facebook network”. It’s not necessary to go to the deep into the app, in order to find some serious bug. You need to test thoroughly all app features to ensure they work correctly and find the most hidden bugs.

5th tip: Go hand in hand with the test platform
What do I mean? I mean that you should always read and review latest news and updates of the testing platform. Usually testing platforms update their rules of testing and your job can suffer, if you are not aware of the latest updates of your testing platform. Also, if you are a novice it’ll be useful for you to read articles, which are present on the test platform, because there you can learn a lot of useful things, which will help you in your job.

Keep in mind these rules and they will certainly help to improve your work

Relevant bugs are valuable bugs!

Put yourself in the shoes of a developer or product owner, you have run a crowd test and there are lots of bugs for you to work on and fix. Then you start to work through the bugs and realise that some of them are just not relevant to you, this may be due to the environment, the behaviour to trigger or multiple other reasons. The result, the irrelevant bugs are rejected.

So what can you do to ensure you are reporting relevant bugs?

  1.  Step back into the shoes of the developer and try to understand what they are looking for. The test description and features should give you a very good idea of what is relevant.
  2. Consider the target market of the app or website. If the app is designed for the Russian market then chances are nobody will be interested in a bug relating to a content issue when you change your phone settings to Italian.
  3. If you have to follow steps a real user would never follow in typical usage, then there is no real value provided in your report and it would be an edge case. Edge case bugs are very likely to be rejected. If the bug is never going to impact a customer’s users, they most likely don’t care about it.



Insider insights: 5 Golden rules to get your bugs accepted.

It’s the one question we are asked more than any other, “How can I get more of my bugs accepted?” Luckily we have experience of nearly half a million bugs reports and hundreds of customers, so we have a pretty good idea of exactly what is required! We believe there are 5 golden rules to follow to maximise your bug acceptance and we thought it was about time we shared them with you.


It might sound obvious but without reading the feature descriptions, instructions and test chat fully, you will not know exactly what the customer is looking for. Even if you find bugs, they may not be the ones they are currently interested in, so check what they’re asking for in the test.


If you can’t reproduce your bug there is very little chance the customer will be able to and nobody is interested in bugs which only happen once!


Something we see often are reports where something clearly goes wrong but the behaviour to trigger this was pretty strange and does not reflect the target user behaviour. Put yourself in the customer’s shoes, you are only going to be interested in bugs which your actual customers are likely to encounter, not ones that happen when you do some random sequence of events. On this point it is always worth considering who the target audience for the product will be.


For most tests there will be a known bug list of existing bugs so firstly make sure your bug is not on the list, also don’t forget to check the bugs already reported by other testers in the current test. When you are typing your bug title you will be shown possible similar bugs, so make sure you keep a close eye on this as well as being aware of what is on the list in general.


It’s another fairly obvious one but if your report and evidence is not clear and well written, the customer may not understand your bug immediately and as a result, reject it. You can find guidance on quality here and specifically on your title here.

Follow these 5 golden rules to high acceptance rates of your bugs.

Writing the perfect bug title

Often the bug title is easily overlooked as simply a necessary way of separating your bug from another bug, but hold on as you are missing a great opportunity to make your bug great. If you write a good bug title another tester or QA engineer should have a very good idea exactly what the issue is, how severe of an issue it may be and how likely they are to truly value your bug.

So let’s get down to the task of defining a good bug title. The title should be as concise as possible but contain all of the necessary information. Ask yourself what is the actual bug (tip: this is often not the result of the bug!) and put this information first. You also want to include some location information so another user can locate your issue. Finally, add any additional information to support your bug.

So if we use this approach with a real bug we get something like this:

[Issue] Location / additional information

[Broken button] Buy now button on summer savings page / Error #123 displayed.

[Crash] Sort menu / App crashes while opening the menu

[Infinite loading] Video on “know our product” page / When trying to play the video

Now go forth and make your bug titles as good as they can be!